Signature Calculations for the Authorization Header:Transferring Payload in Multiple Chunks (Chunked Upload) (AWS Signature Version4)
As described in theOverview, when authenticating requests using theAuthorization header, you have an option of uploading the payload inchunks. You can send data in fixed size or variable size chunks. This section describes thesignature calculation process in chunked upload, how you create the chunk body, and how thedelayed signing works where you first upload the chunk, and send its signature in thesubsequent chunk. The example section (seeExample: PUT Object) shows signature calculationsand resultingAuthorization headers that you can use as a test suite to verifyyour code.
When transferring data in a series of chunks, you must do one of the following:
Explicitly specify the total content length (object length in bytes plusmetadata in each chunk) using the
Content-LengthHTTP header. To dothis, you must pre-compute the total length of the payload, including themetadata that you send in each chunk, before starting your request.Specify the
Transfer-EncodingHTTP header. If you include theTransfer-Encodingheader and specify any value other thanidentity, you must omit theContent-Lengthheader.
For all requests, you must include thex-amz-decoded-content-lengthheader, specifying the size of the object in bytes.
Each chunk signature calculation includes the signature of the previous chunk. To begin,you create aseed signature using only the headers. You use the seedsignature in the signature calculation of the first chunk. For each subsequent chunk, youcreate a chunk signature that includes the signature of the previous chunk. Thus, the chunksignatures are chained together; that is, the signature of chunkn is afunctionF(chunk n, signature(chunk n-1)). The chaining ensures thatyou send the chunks in the correct order.
To perform a chunked upload, do the following:
Decide the payload chunk size. You need this when you write the code.
The chunk size must be at least 8 KB. We recommend a chunk size of a least 64 KB for betterperformance. This chunk size applies to all chunks except the last one. The lastchunk you send can be smaller than 8 KB. If your payload is small and can fit intoone chunk, then it can be smaller than the 8 KB.
Create the seed signature for inclusion in the first chunk. For more information,seeCalculating the SeedSignature.
Create the first chunk and stream it. For more information, seeDefining the Chunk Body.
For each subsequent chunk, calculate the chunk signature that includes theprevious signature in the string you sign, construct the chunk, and send it. Formore information, seeDefining the Chunk Body.
Send the final additional chunk, which is the same as the other chunks in theconstruction, but it has zero data bytes. For more information, seeDefining the Chunk Body.
Calculating the SeedSignature
The following diagram illustrates the process of calculating the seedsignature.
The following table describes the functions that are shown in the diagram. You need to implement code for these functions.
| Function | Description |
|---|---|
Lowercase() | Convert the string to lowercase. |
Hex() | Lowercase base 16 encoding. |
SHA256Hash() | Secure Hash Algorithm (SHA) cryptographic hash function. |
HMAC-SHA256() | Computes HMAC by using the SHA256 algorithm with the signing key provided. This is the final signature. |
Trim() | Remove any leading or trailing whitespace. |
UriEncode() | URI encode every byte. UriEncode() must enforce the following rules:
The standard UriEncode functions provided by your development platform may not work because of differences in implementation and related ambiguity in the underlying RFCs. We recommend that you write your own custom UriEncode function to ensure that your encoding will work. To see an example of a UriEncode function in Java, seeJava Utilities on the GitHub website. |
For information about the signing process, seeSignature Calculations for the Authorization Header:Transferring Payload in a Single Chunk (AWS Signature Version 4). Theprocess is the same, except that the creation ofCanonicalRequest differsas follows:
In addition to the request headers you plan to add, you must include thefollowing headers:
Header Description x-amz-content-sha256This header is required for all AWS Signature Version 4requests. Set the value to
STREAMING-AWS4-HMAC-SHA256-PAYLOADtoindicate that the signature covers only headers and thatthere is no payload.Content-EncodingSet the value to
aws-chunked.Amazon S3 supports multiple content encoding values. You can specify your custom content-encoding when using the Signature Version 4 streaming API.
For example:
Content-Encoding : aws-chunked,gzipAmazon S3 stores the resulting object without the
aws-chunkedvalue in thecontent-encodingheader. Ifaws-chunkedis the only value that you pass in thecontent-encodingheader, S3 considers thecontent-encodingheader empty and does not return this header when your retrieve the object.x-amz-decoded-content-lengthSet the value to the length, in bytes, of the data to bechunked, without counting any metadata. For example, if you areuploading a 4 GB file, set the value to 4294967296. This is theraw size of the object to be uploaded (data you want to store inAmazon S3). Content-LengthSet the value to the actual size of the transmitted HTTP body, which includes thelength of your data (value set for
x-amz-decoded-content-length), plus chunkmetadata. Each chunk has metadata, such as the signature ofthe previous chunk. Chunk calculations are discussed in thefollowing section. If you include theTransfer-Encodingheader and specify anyvalue other thanidentity, you must not includetheContent-Lengthheader.
You send the first chunk with the seed signature. You must construct the chunk asdescribed in the following section.
Defining the Chunk Body
All chunks include some metadata. Each chunk must conform to the followingstructure:
string(IntHexBase(chunk-size)) + ";chunk-signature=" +signature + \r\n +chunk-data + \r\nWhere:
IntHexBase()is a function that you write to convert an integerchunk-size to hexadecimal. For example, if chunk-size is 65536, hexadecimalstring is "10000".chunk-sizeis the size, in bytes, of thechunk-data, without metadata. For example, if you are uploading a 65 KB objectand using a chunk size of 64 KB, you upload the data in three chunks: the firstwould be 64 KB, the second 1 KB, and the final chunk with 0 bytes.signatureFor each chunk, you calculate the signature using the following string to sign.For the first chunk, you use the seed-signature as the previous signature.
The size of the final chunk data that you send is 0, although the chunk body stillcontains metadata, including the signature of the previous chunk.
Example: PUT Object
You can use the examples in this section as a reference to check signaturecalculations in your code. Before you review the examples, note the following:
The signature calculations in these examples use the following examplesecurity credentials.
Parameter Value AWSAccessKeyIdAKIAIOSFODNN7EXAMPLEAWSSecretAccessKeywJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEYAll examples use the request timestamp 20130524T000000Z (
Fri, 24 May 2013 00:00:00GMT).All examples use
examplebucketas the bucket name.The bucket is assumed to be in theUS East (N. Virginia) Region, and the credential
Scopeand theSigning Keycalculations useus-east-1as theRegion specifier. For more information, seeRegions and Endpoints in theAmazon Web Services General Reference.You can use either path style or virtual-hosted style requests. The followingexamples use virtual-hosted style requests, for example:
https://examplebucket.s3.amazonaws.com/photos/photo1.jpgFor more information, seeVirtualHosting of Buckets in theAmazon Simple Storage Service User Guide.
The following example sends aPUT request to upload an object. Thesignature calculations assume the following:
You are uploading a 65 KB text file, and the file content is aone-character string made up of the letter 'a'.
The chunk size is 64 KB. As a result, the payload is uploaded in threechunks, 64 KB, 1 KB, and the final chunk with 0 bytes of chunk data.
The resulting object has the key name
chunkObject.txt.You are requesting
REDUCED_REDUNDANCYas the storage class byadding thex-amz-storage-classrequest header.
For information about the API action, seePutObject. The general request syntax is as follows:
PUT /examplebucket/chunkObject.txt HTTP/1.1Host: s3.amazonaws.comx-amz-date: 20130524T000000Z x-amz-storage-class: REDUCED_REDUNDANCYAuthorization:SignatureToBeCalculatedx-amz-content-sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOADContent-Encoding: aws-chunkedx-amz-decoded-content-length: 66560Content-Length: 66824<Payload>The following steps show signature calculations.
Seed signature — Create String to Sign
CanonicalRequest
PUT/examplebucket/chunkObject.txtcontent-encoding:aws-chunkedcontent-length:66824host:s3.amazonaws.comx-amz-content-sha256:STREAMING-AWS4-HMAC-SHA256-PAYLOADx-amz-date:20130524T000000Zx-amz-decoded-content-length:66560x-amz-storage-class:REDUCED_REDUNDANCYcontent-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-classSTREAMING-AWS4-HMAC-SHA256-PAYLOADIn the canonical request, the third line is empty because thereare no query parameters in the request. The last line is theconstant string provided as the value of the hashed Payload, whichshould be same as the value of
x-amz-content-sha256header.StringToSign
AWS4-HMAC-SHA25620130524T000000Z20130524/us-east-1/s3/aws4_requestcee3fed04b70f867d036f722359b0b1f2f0e5dc0efadbc082b76c4c60e316455For information about each of line in the string to sign, seethe diagram that explains seed signature calculation.
SigningKey
signing key = HMAC-SHA256(HMAC-SHA256(HMAC-SHA256(HMAC-SHA256("AWS4" + "<YourSecretAccessKey>","20130524"),"us-east-1"),"s3"),"aws4_request")Seed Signature
4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9Authorization header
The resulting Authorization header is as follows:
AWS4-HMAC-SHA256 Credential=AKIAIOSFODNN7EXAMPLE/20130524/us-east-1/s3/aws4_request,SignedHeaders=content-encoding;content-length;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-storage-class,Signature=4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9Chunk 1: (65536 bytes, with value 97 for letter 'a')
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD20130524T000000Z20130524/us-east-1/s3/aws4_request4f232c4386841ef735655705268965c44a0e4690baa4adea153f7db9fa80a0a9e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855bf718b6f653bebc184e1479f1935b8da974d701b893afcf49e701f3e2f9f9c5aFor information about each line in the string to sign, see thepreceding diagram that shows various components of the string tosign (for example, the last three lines are,
previous-signature,hash(""), andhash(current-chunk-data)).Chunk signature:
ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648Chunk data sent:
10000;chunk-signature=ad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648<65536-bytes>
Chunk 2: (1024 bytes, with value 97 for letter 'a')
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD20130524T000000Z20130524/us-east-1/s3/aws4_requestad80c730a21e5b8d04586a2213dd63b9a0e99e0e2307b0ade35a65485a288648e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b8552edc986847e209b4016e141a6dc8716d3207350f416969382d431539bf292e4aChunk signature:
0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497Chunk data sent:
400;chunk-signature=0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497<1024 bytes>
Chunk 3: (0 byte data)
Chunk string to sign:
AWS4-HMAC-SHA256-PAYLOAD20130524T000000Z20130524/us-east-1/s3/aws4_request0055627c9e194cb4542bae2aa5492e3c1575bbb81b612b7d234b86a503ef5497e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855Chunk signature:
b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9Chunk data sent:
0;chunk-signature=b6c6ea8a5354eaf15b3cb7646744f4275b71ea724fed81ceb9323e279d449df9
