Google Cloud provides tools, products, guidance, and professionalservices to help you migrate data fromAmazon Simple Storage Service (Amazon S3) toCloud Storage.This document discusses how to design, implement, and validate a plan to migratefrom Amazon S3 to Cloud Storage. The document describes a portion ofthe overall migration process in which you create an inventory of Amazon S3artifacts and create a plan for how to handle the migration process.

The discussion in this document is intended for cloud administrators who wantdetails about how to plan and implement a migration process. It's also intendedfor decision-makers who are evaluating the opportunity to migrate and who wantto explore what migration might look like.

This document is part of a multi-part series about migrating from AWS toGoogle Cloud that includes the following documents:

• Get started
• Migrate from Amazon EC2 to Compute Engine
• Migrate from Amazon S3 to Cloud Storage (this document)
• Migrate from Amazon EKS to Google Kubernetes Engine
• Migrate from Amazon RDS and Amazon Aurora for MySQL to Cloud SQL for MySQL
• Migrate from Amazon RDS and Amazon Aurora for PostgreSQL to Cloud SQL for PostgreSQL and AlloyDB for PostgreSQL
• Migrate from Amazon RDS for SQL Server to Cloud SQL for SQL Server
• Migrate from AWS Lambda to Cloud Run

For this migration to Google Cloud, we recommend that you followthe migration framework described inMigrate to Google Cloud: Get started.

The following diagram illustrates the path of your migration journey.

You might migrate from your source environment to Google Cloud in a seriesof iterations—for example, you might migrate some workloads first and otherslater. For each separate migration iteration, you follow the phases of thegeneral migration framework:

1. Assess and discover your workloads and data.
2. Plan and build a foundation on Google Cloud.
3. Migrate your workloads and data to Google Cloud.
4. Optimize your Google Cloud environment.

For more information about the phases of this framework, seeMigrate to Google Cloud: Get started.

To design an effective migration plan, we recommend that you validate each stepof the plan, and ensure that you have a rollback strategy. To help you validateyour migration plan, seeMigrate to Google Cloud: Best practices for validating a migration plan.

Assess the source environment

In the assessment phase, you determine the requirements and dependencies tomigrate your source environment to Google Cloud.

The assessment phase is crucial for the success of your migration. You need togain deep knowledge about the workloads you want to migrate, their requirements,their dependencies, and about your current environment. You need to understandyour starting point to successfully plan and execute a Google Cloudmigration.

The assessment phase consists of the following tasks:

1. Build a comprehensive inventory of your workloads.
2. Catalog your workloads according to their properties and dependencies.
3. Train and educate your teams on Google Cloud.
4. Build experiments and proofs of concept on Google Cloud.
5. Calculate the total cost of ownership (TCO) of the target environment.
6. Choose the migration strategy for your workloads.
7. Choose your migration tools.
8. Define the migration plan and timeline.
9. Validate your migration plan.

For more information about the assessment phase and these tasks, seeMigrate to Google Cloud: Assess and discover your workloads.The following sections are based on information in that document.

Build an inventory of your Amazon S3 buckets

To scope your migration, you create two inventories: an inventory of yourAmazon S3 buckets, and an inventory of the objects that are stored in thebuckets.

After you build the inventory of your Amazon S3 buckets, refine the inventoryby considering the following data points about each Amazon S3 bucket:

• How you've configured Amazon S3 bucket server-side encryption.
• Your settings for Amazon S3 bucket identity and access management.
• The configuration for S3 Block Public Access.
• Any cost allocation tags for Amazon S3 buckets.
• The configuration for S3 Object Lock.
• How you're accessing the Amazon S3 bucket.
• How you've configured Requester Pays.
• The settings for Amazon S3 object versioning.
• The configuration for AWS Backup policies for Amazon S3.
• Whether you're using Amazon S3 Intelligent-Tiering.
• How you've configured for Amazon S3 object replication.
• The Amazon S3 object lifecycle.

We also recommend that you gather data about your Amazon S3 buckets that letsyou compute aggregate statistics about the objects that each bucketcontains. For example, if you gather the total object size, average object size,and object count, it can help youestimate the time and cost that's needed to migrate from an Amazon S3 bucket to a Cloud Storage bucket.

To build the inventory of your Amazon S3 buckets and to gather data points aboutyour Amazon S3 buckets, you can implement data-collection mechanisms andprocesses that rely on AWS tools, such as the following:

• Amazon S3 monitoring tools
• S3 Analytics
• AWS Multi-Account Multi-Region Data Aggregation
• AWS APIs
• AWS developer tools
• The AWS command-line interface

To help you avoid issues during the migration, and to help estimate the effortneeded for the migration, we recommend that you evaluate how Amazon S3 bucketfeatures map to similar Cloud Storage bucket features. The followingtable summarizes this mapping.

As noted earlier, the features listed in the preceding table might look similarwhen you compare them. However, differences in the design and implementation ofthe features in the two cloud providers can have significant effects on yourmigration from Amazon S3 to Cloud Storage.

Build an inventory of the objects stored in your Amazon S3 objects

After you build the inventory of your Amazon S3 buckets, we recommend that youbuild an inventory of the objects stored in these buckets by using theAmazon S3 inventory tool.

To build the inventory of your Amazon S3 objects, consider the following foreach object:

• Amazon S3 object name
• Amazon S3 object size
• Amazon S3 object metadata
• Amazon S3 object subresources
• Amazon S3 object versions, and if you need to migrate these versions
• Amazon S3 object presigned URLs
• Amazon S3 object transformations
• Amazon S3 object tags
• Amazon S3 object storage classes
• Amazon S3 object archiving

We also recommend that you gather data about your Amazon S3 objects tounderstand how often you and your workloads create, update, and delete Amazon S3objects.

To help you avoid issues during the migration, and to help estimate the effortneeded for the migration, we recommend that you evaluate how Amazon S3 objectfeatures map to similar Cloud Storage object features. The followingtable summarizes this mapping.

As noted earlier, the features listed in the preceding table might look similarwhen you compare them. However, differences in the design and implementation ofthe features in the two cloud providers can have significant effects on yourmigration from Amazon S3 to Cloud Storage.

Complete the assessment

After you build the inventories from your Amazon S3 environment, complete therest of the activities of the assessment phase as described inMigrate to Google Cloud: Assess and discover your workloads.

Plan and build your foundation

In the plan and build phase, you provision and configure the infrastructure todo the following:

• Support your workloads in your Google Cloud environment.
• Connect your source environment and your Google Cloud environment tocomplete the migration.

The plan and build phase is composed of the following tasks:

1. Build a resource hierarchy.
2. Configure Google Cloud's Identity and Access Management (IAM).
3. Set up billing.
4. Set up network connectivity.
5. Harden your security.
6. Set up logging, monitoring, and alerting.

For more information about each of these tasks, see theMigrate to Google Cloud: Plan and build your foundation.

Migrate data and workloads from Amazon S3 to Cloud Storage

To migrate data from Amazon S3 to Cloud Storage, we recommend that youdesign a data migration plan by following the guidance inMigrate to Google Cloud: Transfer your large datasets.That document recommends usingStorage Transfer Service,a Google Cloud product that lets you migrate data from several sources toCloud Storage, such as from on-premises environments or from othercloud storage providers. Storage Transfer Service supports several types of datatransfer jobs, such as the following:

• Run-once transfer jobs,which transfer data from Amazon S3 or other supported sources toCloud Storage on demand.
• Scheduled transfer jobs,which transfer data from Amazon S3 or other supported sources toCloud Storage on a schedule.
• Event-driven transfer jobs,which automatically transfer data when Amazon S3 sends Amazon S3 EventNotifications to Amazon Simple Queue Service (SQS).

To implement a data migration plan, you can configure one or more data transferjobs. For example, to reduce the length of cut-over windows during themigration, you can implement acontinuous replication data migration strategy as follows:

1. Configure a run-once transfer job to copy the data from an Amazon S3bucket to the Cloud Storage bucket.
2. Perform data validation and consistency checks to compare data in theAmazon S3 bucket against the copied data in the Cloud Storage bucket.
3. Set up event-driven transfer jobs to automatically transfer data fromthe Amazon S3 bucket to the Cloud Storage bucket when the contentof the Amazon S3 bucket changes.
4. Stop the workloads and services that have access to the data that'sbeing migrated (that is, to the data that's involved in the previous step).

5. Refactor workloads to use Cloud Storage instead of Amazon S3.You can refactor your workloads by using one of the following approaches,or by using the approaches in sequence:

   • Simple migration from Amazon S3 to Cloud Storage.In a simple migration, you use your existing tools and libraries thatgenerate authenticated REST requests to Amazon S3 toinstead generate authenticated requests to Cloud Storage.
   • Fully migrate from Amazon S3 to Cloud Storage.In a full migration, you can use all of the features ofCloud Storage, including multiple projects and OAuth 2.0 forauthentication.

6. Wait for the replication to fully synchronize Cloud Storagewith Amazon S3.

7. Start your workloads.

8. When you no longer need your Amazon S3 environment as a fallback option,retire it.

Storage Transfer Service can preservecertain metadata when you migrate objects from a supported source to Cloud Storage.We recommend that you assess whether Storage Transfer Service can migratethe Amazon S3 metadata that you're interested in.

When you design your data migration plan, we recommend that you also assess AWSnetwork egress costs and your Amazon S3 costs. For example, consider thefollowing options to transfer data:

• Across the public internet.
• By using an interconnect link.
• By usingAmazon CloudFront.

The option that you choose can have an impact on your AWS network egress costsand your Amazon S3 costs. The option can also affect the amount of effort andresources that you need in order to provision and configure the infrastructure.For more information about costs, see the following:

• Transfer from Amazon S3 to Cloud Storage: Egress options
• Understanding data transfer charges in the AWS documentation
• Amazon S3 pricing

When you migrate data from Amazon S3 to Cloud Storage, we recommendthat youuse VPC Service Controls to build a perimeter that explicitly denies communication between Google Cloud services unlessthe services are authorized.

Optimize your Google Cloud environment

Optimization is the last phase of your migration. In this phase, you iterate onoptimization tasks until your target environment meets your optimizationrequirements. The steps of each iteration are as follows:

1. Assess your current environment, teams, and optimization loop.
2. Establish your optimization requirements and goals.
3. Optimize your environment and your teams.
4. Tune the optimization loop.

You repeat this sequence until you've achieved your optimization goals.

For more information about optimizing your Google Cloud environment, seeMigrate to Google Cloud: Optimize your environment andGoogle Cloud Well-Architected Framework: Performance optimization.

What's next

• Read aboutother AWS to Google Cloud migration journeys.
• Learn how tocompare AWS and Azure services to Google Cloud.
• Learn when tofind help for your migrations.
• For more reference architectures, diagrams, and best practices, explore theCloud Architecture Center.

Contributors

Author:Marco Ferrari | Cloud Solutions Architect