In the normal world, synchronizing two buckets is as simple as doingaws s3 sync <SYNC_OPTIONS> <SOURCE_BUCKET> <DESTINATION_BUCKET>. However, due to the information security needs of some of my customers, it's occasionally necessary to perform data-synchronizations between two S3 buckets, but using methods that amount to "offline" transfers.

To illustrate what is meant by "offline":

1. Create a transfer-archive from a data source
2. Copy the transfer-archive across a security boundary
3. Unpack the transfer-archive to its final destination

Note that things are a bit more involved than the summary of the process – but this gives you the gist of the major effort-points.

The first time you do an offline bucket sync, transferring the entirety of a bucket is typically the goal. However, for a refresh-sync – particularly for a bucket of greater than a trivial content-size, this can be sub-ideal. For example, it might be necessary to do monthly syncs of a bucket that grows by a few Gigabytes per month. After a year, a full sync can mean having to move tens to hundreds of gigabytes. A better way is to only sync the deltas – copying only what's changed between the current and immediately-prior sync-tasks (a few GiB rather than tens to hundreds).

The AWS CLI tools don't really have a "sync only the files that have been added/modified since<DATE>". That said, it's not super difficult to work around that gap. A simple shell script like the following works a trick:

for FILE in $( aws s3 ls --recursive s3://<SOURCE_BUCKET>/ | \   awk '$1 > "2019-03-01 00:00:00" {print $4}' )do   echo "Downloading ${FILE}"   install -bDm 000644 <( aws s3 cp "s3://<SOURCE_BUCKET>/${FILE}" - ) \     "<STAGING_DIR>/${FILE}"done

To explain the above:

1. Create a list of files to iterate:
   1. Invoke a subprocess using the$() notation. Within that subprocess...
   2. Invoke the AWS CLI's S3 module to recursively list the source-bucket's contents (aws s3 ls --recursive)
   3. Pipe the output toawk – looking for any date-string that's newer than the value ins3 ls's first output-column (the file-modification date column) and print out only the fourth column (the S3 object-path)The output from the subprocess is captured as an iterable list-structure
2. Use afor loop-method to iterate the previously-assembled list, assigning each S3 object-path to the${FILE} variable
3. Since I hate sending programs off to do things in silence (I don't trust them to not hang), my first looped-command is to say what's happening via theecho "Downloading ${FILE}" directive.

4. Theinstall line makes use of some niftiness within both BASH and the AWS CLI's S3 command:

   1. By specifying "-" as the "destination" for the file-copy operation, you tell the S3 command to write the fetched object-contents toSTDOUT.
   2. BASH allows you take a stream of output and assign a file-handle to it by surrounding the output-producing command with<( ).
   3. Invoking theinstall command with the-D flag tells the command to "create all necessary path-elements to place the source 'file' in the desired location within the filesystem, even if none of the intervening directory structure exists, yet."

   Putting it all together, theinstall operation takes the streameds3 cp output, and installs it as a file (with mode 000644) at the location derived from theSTAGING_DIR plus the S3 object-path ...thus preserving theSOURCE_BUCKET's content-structure within theSTAGING_DIR

Obviously, this method really only works for additive/substitutive deltas. If you need to account for deletions and/or moves, this approach will be insufficient.