This page describes how to complete a simple migration from Amazon Simple Storage Service (Amazon S3) toCloud Storage. In a simple migration, you use your existing tools andlibraries for generating authenticated REST requests to Amazon S3 tosend authenticated requests to Cloud Storage instead.

If you are new to Cloud Storage and won't be using the API directly,considerusing the Google Cloud console to set up and manage transfers. TheGoogle Cloud console provides a graphical interface to Cloud Storage thatlets you accomplish many of your storage tasks using just a browser,including migration of your data from Amazon S3 to Cloud Storage.

If you want Cloud Storage to store a backup of your Amazon S3 data,consider usingevent-driven transfers, which use Amazon S3 EventNotifications to automatically keep a Cloud Storage bucket in syncwith your Amazon S3 source.

Migrate from Amazon S3 to Cloud Storage in a simple migration scenario

In order to make requests to Cloud Storage, you need to complete thefollowing steps:

• Set adefault Google Cloud project.
• Get anHMAC (hash-based message authentication code) key.

• In your existing tools or libraries, make the following changes:

  • Change the request endpoint to use the Cloud StorageXML API request endpoint.
  • Replace the Amazon Web Services (AWS) access and secret key with thecorresponding Cloud Storage access ID and secret(collectively called your Cloud Storage HMAC key).

  • Make sure yourx-amz- headers use supported Cloud Storagevalues. For example,x-amz-storage-class should use one of theavailableCloud Storage storage classes.

    When you use the Cloud Storage XML API in a simple migrationscenario, specifying theAWS signature identifier in theAuthorization header lets Cloud Storage know to expectx-amz-* headers and Amazon S3 ACL XML syntax in your request.Cloud Storage processesx-amz-* headers that have anx-goog-* equivalent, such as those listed in theheaders table,and it processes thex-amz-decoded-content-length header.

After you make these changes, you can start using your existing tools andlibraries to send HMAC requests to Cloud Storage.

For example, the following samples demonstrate how to listCloud Storage buckets using the Amazon S3 SDK:

import("context""fmt""io""time""github.com/aws/aws-sdk-go/aws""github.com/aws/aws-sdk-go/aws/credentials""github.com/aws/aws-sdk-go/aws/session""github.com/aws/aws-sdk-go/service/s3")funclistGCSBuckets(wio.Writer,googleAccessKeyIDstring,googleAccessKeySecretstring)error{// googleAccessKeyID := "Your Google Access Key ID"// googleAccessKeySecret := "Your Google Access Key Secret"// Create a new client and do the following:// 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.// 2. Use Cloud Storage HMAC Credentials.sess:=session.Must(session.NewSession(&aws.Config{Region:aws.String("auto"),Endpoint:aws.String("https://storage.googleapis.com"),Credentials:credentials.NewStaticCredentials(googleAccessKeyID,googleAccessKeySecret,""),}))client:=s3.New(sess)ctx:=context.Background()ctx,cancel:=context.WithTimeout(ctx,time.Second*10)defercancel()result,err:=client.ListBucketsWithContext(ctx,&s3.ListBucketsInput{})iferr!=nil{returnfmt.Errorf("ListBucketsWithContext: %w",err)}fmt.Fprintf(w,"Buckets:")for_,b:=rangeresult.Buckets{fmt.Fprintf(w,"%s\n",aws.StringValue(b.Name))}returnnil}

importcom.amazonaws.auth.AWSStaticCredentialsProvider;importcom.amazonaws.auth.BasicAWSCredentials;importcom.amazonaws.client.builder.AwsClientBuilder;importcom.amazonaws.services.s3.AmazonS3;importcom.amazonaws.services.s3.AmazonS3ClientBuilder;importcom.amazonaws.services.s3.model.Bucket;importjava.util.List;publicclassListGcsBuckets{publicstaticvoidlistGcsBuckets(StringgoogleAccessKeyId,StringgoogleAccessKeySecret){// String googleAccessKeyId = "your-google-access-key-id";// String googleAccessKeySecret = "your-google-access-key-secret";// Create a BasicAWSCredentials using Cloud Storage HMAC credentials.BasicAWSCredentialsgoogleCreds=newBasicAWSCredentials(googleAccessKeyId,googleAccessKeySecret);// Create a new client and do the following:// 1. Change the endpoint URL to use the Google Cloud Storage XML API endpoint.// 2. Use Cloud Storage HMAC Credentials.AmazonS3interopClient=AmazonS3ClientBuilder.standard().withEndpointConfiguration(newAwsClientBuilder.EndpointConfiguration("https://storage.googleapis.com","auto")).withCredentials(newAWSStaticCredentialsProvider(googleCreds)).build();// Call GCS to list current bucketsList<Bucket>buckets=interopClient.listBuckets();// Print bucket namesSystem.out.println("Buckets:");for(Bucketbucket:buckets){System.out.println(bucket.getName());}// Explicitly clean up client resources.interopClient.shutdown();}

importboto3# type: ignoredeflist_gcs_buckets(google_access_key_id:str,google_access_key_secret:str)->List[str]:"""Lists all Cloud Storage buckets using AWS SDK for Python (boto3)    Positional arguments:        google_access_key_id: hash-based message authentication code (HMAC) access ID        google_access_key_secret: HMAC access secret    Returned value is a list of strings, one for each bucket name.    To use this sample:    1. Create a Cloud Storage HMAC key: https://cloud.google.com/storage/docs/authentication/managing-hmackeys#create    2. Change endpoint_url to a Google Cloud Storage XML API endpoint.    To learn more about HMAC: https://cloud.google.com/storage/docs/authentication/hmackeys#overview    """client=boto3.client("s3",region_name="auto",endpoint_url="https://storage.googleapis.com",aws_access_key_id=google_access_key_id,aws_secret_access_key=google_access_key_secret,)# Call GCS to list current bucketsresponse=client.list_buckets()# Return list of bucket namesresults=[]forbucketinresponse["Buckets"]:results.append(bucket["Name"])print(bucket["Name"])# Can remove if not needed after developmentreturnresults

Set a default project

To use Cloud Storage in a simple migration scenario, it's recommendedthat you set a defaultproject, which Cloud Storage uses toperformcertain operations, such asGET service orPUT bucket. If youdon't set a default project, you mustspecify a project header.in certain requests.

To set a default project:

1. Open theCloud Storage Settings page in theGoogle Cloud console.
2. Select theInteroperability tab.

3. ClickSetPROJECT-ID as default project, located in thesectionDefault project for interoperable access.

   If the project is already the default project, you seePROJECT-ID is your default project for interoperable access.

This project is now your default project. You can change your default project atany time by choosing a different project and following these steps.

Alternatively specify a project header

Instead of, or in addition to, setting a default project, you can use thex-amz-project-id header in individual requests that require you to specify aproject.

• A request that usesx-amz-project-id uses the project specified in theheader, even if there is an existing default project.

Thex-amz-project-id header is useful when:

• You're working with multiple projects.
• Your requests are made by a service account associated with a differentproject, because service accounts use their parent project as the defaultproject.

Note that Amazon S3 does not haveprojects, so depending on the toolsor client libraries you use, specifying ax-amz-project-id header may not bean option. In this case, you shouldset a default project.

Use HMAC keys

To use the Cloud Storage XML API in a simple migration scenario, useCloud Storagehash-based message authentication code (HMAC) keysfor the credentials. Typically, you shouldcreate an HMAC key that isassociated with a service account; however, you can alternatively use oneassociated with a user account.

Authenticate in a simple migration scenario

Use the Authorization header

For operations in a simple migration scenario that require authentication, youinclude anAuthorization request header just like you do for requests toAmazon S3. TheAuthorization header syntax for an Amazon S3 request is:

Authorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

In a simple migration scenario, you only change the header to use yourCloud Storage HMAC access ID and make sure theSignature you attachis calculated with your Cloud Storage HMAC secret key:

Authorization:ALGORITHM Credential=GOOG-ACCESS-ID/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

The parts of theAuthorization header are:

• ALGORITHM: The signature algorithm and version that you are using.UsingAWS4-HMAC-SHA256 indicates that you are using an HMAC V4 signatureand you intend to sendx-amz-* headers. You can also useGOOG4-HMAC-SHA256, which indicates that you are using an HMAC V4 signatureand you intend to sendx-goog-* headers, orGOOG4-RSA-SHA256, whichindicates that you are using an RSA V4 signature and you intend to sendx-goog-* headers.

• GOOG-ACCESS-ID: The access ID identifies the entity that ismaking and signing the request. In a simple migration, replace the AmazonWeb Service (AWS) access key ID you use to access Amazon S3 with yourCloud Storage HMAC access ID. Your Cloud Storage HMACaccess ID starts withGOOG.

• CREDENTIAL_SCOPE: The credential scope, asdefined in the signature. In a simple migration, you don't need tochange credential scope if you are usingAWS4-HMAC-SHA256 for yourALGORITHM value.

• SIGNED_HEADERS: A semicolon-separated list of names of headersthat must be included to sign this request. All headers should be lowercaseand sorted by character code.

  An example of an Amazon S3-style signed header string looks like:

  content-type;host;x-amz-date

  In a simple migration, you don't need to make any changes to the signedheader string.

• SIGNATURE: Thesignature that allows the request to beauthenticated. In a simple migration, replace the AWS access keyinformation with the equivalent Cloud Storage HMAC key information.

Sample authentication request

The following examples upload an object named/europe/france/paris.jpg to abucket namedmy-travel-maps, apply the predefined ACLpublic-read, anddefine a custom metadata header for reviewers. Here is the request to a bucketin Amazon S3:

PUT europe/france/paris.jpg HTTP/1.1Host: my-travel-maps.s3.amazonaws.comDate: Mon, 11 Mar 2019 23:46:19 GMTContent-Length: 888814Content-Type: image/jpgx-amz-acl: public-readx-amz-date:20190311T192918Zx-amz-meta-reviewer: joe,janeAuthorization: AWS4-HMAC-SHA256 Credential=AWS-ACCESS-KEY/20190311/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-acl;x-amz-date;x-amz-meta-reviewer, Signature=SIGNATURE

Here is the request for a bucket in Cloud Storage:

PUT europe/france/paris.jpg HTTP/1.1Host: my-travel-maps.storage.googleapis.comDate: Mon, 11 Mar 2019 23:46:19 GMTContent-Length: 888814Content-Type: image/jpgx-amz-acl: public-readx-amz-date:20190311T192918Zx-amz-meta-reviewer: joe,janeAuthorization: AWS4-HMAC-SHA256 Credential=GOOG-ACCESS-ID/20190311/us-east-1/s3/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-acl;x-amz-date;x-amz-meta-reviewer, Signature=SIGNATURE

Here is the corresponding canonical request that was created for this request:

PUT/europe/france/paris.jpgcontent-length:888814content-type:image/jpghost:my-travel-maps.storage.googleapis.comx-amz-acl:public-readx-amz-date:20190311T192918Zx-amz-meta-reviewer:joe,janecontent-length,content-type,host,x-amz-acl,x-amz-date,x-amz-meta-reviewer82e3da8b3f35989512e8d428add7eca73ab0e5f36586e66fbad8e1051343cbd2

Here is the corresponding string-to-sign that was created for this request:

AWS4-HMAC-SHA25620190311T192918Z20190311/us-east-1/s3/aws4_request73918a5ff373d7a03e406fbf9ea35675396b06fca2af76c27a5c451fa783ef65

This request did not provide a Content-MD5 header, so an empty string is shownin the second line of the message.

Access control in a simple migration scenario

To support simple migrations, Cloud Storage accepts ACLs produced byAmazon S3. In a simple migration scenario, you useAWS as your signatureidentifier, which tells Cloud Storage to expect ACL syntax usingAmazon S3 ACL XML syntax. You should ensure that the Amazon S3 ACLs youuse map to the Cloud Storage ACL model. For example, if your tools andlibraries use Amazon S3's ACL syntax to grant bucketWRITE permission, thenthey must also grant bucketREAD permission because Cloud Storagepermissions areconcentric. You do not need to specify bothWRITE andREAD permission when you grantWRITE permission using theCloud Storage syntax.

Cloud Storage supports Amazon S3 ACL syntax in the followingscenarios:

• In a request to Cloud Storage to retrieve ACLs (for example, aGETObject orGET Bucket request), Cloud Storage returns Amazon S3ACL syntax.
• In a request to Cloud Storage to apply ACLs (for example, aPUTObject orPUT Bucket request), Cloud Storage expects to receiveAmazon S3 ACL syntax.

TheAuthorization header in a simple migration scenario usesAWS for thesignature identifier, but with your Cloud Storage HMAC access ID.

Authorization: AWS4-HMAC-SHA256 Credential=GOOG-ACCESS-ID/CREDENTIAL_SCOPE, SignedHeaders=SIGNED_HEADERS, Signature=SIGNATURE

The following example shows aGET request to Cloud Storage to returnthe ACLs for an object.

GET europe/france/paris.jpg?acl HTTP/1.1Host: my-travel-maps.storage.googleapis.comDate: Thu, 21 Feb 2019 23:50:10 GMTContent-Type: application/xmlX-Amz-Date: 20190221T235010ZAuthorization: AWS4-HMAC-SHA256 Credential=GOOGMC5PDPA5JLZYQMHQHRAX/20190221/region/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=29088b1d6dfeb2549f6ff67bc3744abb7e45475f0ad60400485805415bbfc534

The response to the request includes the ACL using Amazon S3 ACL syntax.

<?xml version='1.0' encoding='UTF-8'?><AccessControlPolicy>    <Owner>        <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490        </ID>        <DisplayName>OwnerName</DisplayName>    </Owner>    <AccessControlList>        <Grant>            <Grantee xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'                xsi:type='CanonicalUser'>                <ID>00b4903a972faa8bcce9382686e9129676f1cd6e5def1f5663affc2ba4652490</ID>                <DisplayName>UserName</DisplayName>            </Grantee>            <Permission>FULL_CONTROL</Permission>        </Grant>    </AccessControlList></AccessControlPolicy>

The following example shows aPUT request to Cloud Storage to setthe ACLs for an object. The example shows a request body with Amazon S3ACL syntax.

PUT europe/france/paris.jpg?acl HTTP/1.1Host: my-travel-maps.storage.googleapis.comDate: Thu, 21 Feb 2019 23:50:10 GMTContent-Type: application/xmlContent-Length: 337X-Amz-Date: 20190221T235010ZAuthorization: AWS4-HMAC-SHA256 Credential=GOOGMC5PDPA5JLZYQMHQHRAX/20190221/region/s3/aws4_request, SignedHeaders=host;x-amz-date, Signature=29088b1d6dfeb2549f6ff67bc3744abb7e45475f0ad60400485805415bbfc534<?xml version='1.0' encoding='utf-8'?><AccessControlPolicy>  <AccessControlList>    <Grant>      <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="AmazonCustomerByEmail">        <EmailAddress>jeffersonloveshiking@gmail.com</EmailAddress>      </Grantee>      <Permission>FULL_CONTROL</Permission>    </Grant>  </AccessControlList></AccessControlPolicy>

Finally, in a simple migration scenario, you can also use theGOOG1 signatureidentifier in theAuthorization header. In this case, you must use theCloud Storage ACL syntax and ensure that all of yourx-amz-* headersare changed tox-goog-*. While this is possible, we recommend that you chooseafull migration in order to use all the benefits ofCloud Storage.

Support for XML API compatibility with Amazon S3

For discussions about XML API interoperability, see Stack Overflow using the taggoogle-cloud-storage.

What's next

• Plan a migration from Amazon S3.
• Transfer your data to Cloud Storage from external sources, suchas Amazon S3 and Microsoft Azure Blob Storage, using Storage Transfer Service.
• Create event-driven transfers that use Amazon S3 Event Notificationsto keep a Cloud Storage bucket in sync with Amazon S3.