Repository files navigation

WAL-E is a program designed to perform continuous archiving ofPostgreSQL WAL files and base backups.

WAL-E is obsolete. Though it has been used recently, nobody routinelyreviews patches or fixes regressions that are occasionally introducedby changing libraries and Python versions. It is also not fast asmore modern archivers.

Some alternatives for you to consider:

WAL-G is one alternative that usesa similar model as WAL-E, but is also a much more expansive piece ofsoftware, supporting many databases and compression formats. WAL-Ghas the capability to read, but not write, WAL-E archives. LikeWAL-E, it is piece of software that tends to be developed from thecloud service provider set of priorities.

pgBackRest has a more traditional model(by standards of Postgres archives predating cloud blob storage) and anarrower focus on Postgres. In general, it makes more concessions toand implements more polished features for what independent systemoperators might encounter.

To correspond on using WAL-E or to collaborate on its development, donot hesitate to send mail to the mailing list atwal-e@googlegroups.com (archives and subscription settings).Github issues are also currently being used to track known problems,so please feel free to submit those.

If no up-to-date packages are available to you via a package manager,this command can work on most operating systems:

sudo python3 -m pip install wal-e[aws,azure,google,swift]

You can omit storage services you do not wish to use from the abovelist.

WAL-E has these key commands:

• backup-fetch
• backup-push
• wal-fetch
• wal-push
• delete

All of these operators work in a context of several environmentvariables that WAL-E reads. The variables set depend on the storageprovider being used, and are detailed below.

WAL-E's organizing concept is the PREFIX. Prefixes must be setuniquely for eachwriting database, and prefix all objects storedfor a given database. For example:s3://bucket/databasename.

Of these, the "push" operators send backup data to storage and "fetch"operators get backup data from storage.

wal commands are called by Postgres'sarchive_command andrestore_command to fetch or pull write ahead log, andbackupcommands are used to fetch or push a hot backup of the base databasethat WAL segments can be applied to. Finally, thedelete commandis used to prune the archives as to retain a finite number of backups.

• WALE_S3_PREFIX (e.g.s3://bucket/path/optionallymorepath)
• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• AWS_REGION (e.g.us-east-1)

Optional:

• WALE_S3_ENDPOINT: SeeManually specifying the S3 Endpoint
• WALE_S3_STORAGE_CLASS: One of: STANDARD (default), REDUCED_REDUNDANCY,GLACIER, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, DEEP_ARCHIVE
• AWS_SECURITY_TOKEN: When using AWS STS
• Pass--aws-instance-profile to gather credentials from theInstance Profile. See Using AWS IAM Instance Profiles.

Example below is based on the following blob storage in Azure in theresource groupresgroup :https://store1.blob.core.windows.net/container1/nextpath

• WALE_WABS_PREFIX (e.g.wabs://container1/nextpath)
• WABS_ACCOUNT_NAME (e.g.store1)
• WABS_ACCESS_KEY (Use key1 from runningazure storage account keyslist store1 --resource-group resgroup You will need to have theAzure CLI installed for this to work.)
• WABS_SAS_TOKEN (You only need this if you have not provided anWABS_ACCESS_KEY)

• WALE_GS_PREFIX (e.g.gs://bucket/path/optionallymorepath)
• GOOGLE_APPLICATION_CREDENTIALS

• WALE_SWIFT_PREFIX (e.g.swift://container/path/optionallymorepath)
• SWIFT_AUTHURL
• SWIFT_TENANT
• SWIFT_USER
• SWIFT_PASSWORD

Optional Variables:

• SWIFT_AUTH_VERSION which defaults to2. Some object stores such asSoftlayer require version1.
• SWIFT_ENDPOINT_TYPE defaults topublicURL, this may be set tointernalURL on object stores like Rackspace Cloud Files in orderto use the internal network.

• WALE_FILE_PREFIX (e.g.file://localhost/backups/pg)

• python (>= 3.4)
• lzop
• psql (>= 8.4)
• pv

This software also has Python dependencies: installing withpipwill attempt to resolve them:

• gevent>=1.1.1
• boto>=2.40.0
• azure==3.0.0
• google-cloud-storage>=1.4.0
• python-swiftclient>=3.0.0
• python-keystoneclient>=3.0.0

It is possible to use WAL-E without the dependencies of back-endstorage one does not use installed: the imports for those are onlyperformed if the storage configuration demands their use.

Pushing a base backup to S3:

$ AWS_SECRET_ACCESS_KEY=... wal-e                     \  -k AWS_ACCESS_KEY_ID                                \  --s3-prefix=s3://some-bucket/directory/or/whatever  \  backup-push /var/lib/my/database

Sending a WAL segment to WABS:

$ WABS_ACCESS_KEY=... wal-e                                   \  -a WABS_ACCOUNT_NAME                                        \  --wabs-prefix=wabs://some-bucket/directory/or/whatever      \  wal-push /var/lib/my/database/pg_xlog/WAL_SEGMENT_LONG_HEX

Push a base backup to Swift:

$ WALE_SWIFT_PREFIX="swift://my_container_name"              \  SWIFT_AUTHURL="http://my_keystone_url/v2.0/"               \  SWIFT_TENANT="my_tennant"                                  \  SWIFT_USER="my_user"                                       \  SWIFT_PASSWORD="my_password" wal-e                         \  backup-push /var/lib/my/database

Push a base backup to Google Cloud Storage:

$ WALE_GS_PREFIX="gs://some-bucket/directory-or-whatever"     \  GOOGLE_APPLICATION_CREDENTIALS=...                          \  wal-e backup-push /var/lib/my/database

It is generally recommended that one use some sort of environmentvariable management with WAL-E: working with it this way is less verbose,less prone to error, and less likely to expose secret information inlogs.

envdir, part of thedaemontools package is one recommended approachto setting environment variables. One can prepare anenvdir-compatible directory like so:

# Assumption: the group is trusted to read secret information# S3 Setup$ umask u=rwx,g=rx,o=$ mkdir -p /etc/wal-e.d/env$ echo "secret-key-content" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY$ echo "access-key" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID$ echo 's3://some-bucket/directory/or/whatever' > \  /etc/wal-e.d/env/WALE_S3_PREFIX$ chown -R root:postgres /etc/wal-e.d# Assumption: the group is trusted to read secret information# WABS Setup$ umask u=rwx,g=rx,o=$ mkdir -p /etc/wal-e.d/env$ echo "secret-key-content" > /etc/wal-e.d/env/WABS_ACCESS_KEY$ echo "access-key" > /etc/wal-e.d/env/WABS_ACCOUNT_NAME$ echo 'wabs://some-container/directory/or/whatever' > \  /etc/wal-e.d/env/WALE_WABS_PREFIX$ chown -R root:postgres /etc/wal-e.d

After having done this preparation, it is possible to run WAL-Ecommands much more simply, with less risk of accidentally usingincorrect values:

$ envdir /etc/wal-e.d/env wal-e backup-push ...$ envdir /etc/wal-e.d/env wal-e wal-push ...

envdir is conveniently combined with the archive_command functionalityused by PostgreSQL to enable continuous archiving. To enablecontinuous archiving, one needs to editpostgresql.conf andrestart the server. The important settings to enable continuousarchiving are related here:

wal_level = archive # hot_standby and logical in 9.x is also acceptablearchive_mode = onarchive_command = 'envdir /etc/wal-e.d/env wal-e wal-push %p'archive_timeout = 60

Every segment archived will be noted in the PostgreSQL log.

A base backup (viabackup-push) can be uploaded at any time, butthis must be done at least once in order to perform a restoration. Itmust be done again if you decided to skip archiving any WAL segments:replication will not be able to continue if there are any gaps in thestored WAL segments.

backup-push,backup-fetch,wal-push,wal-fetch representthe primary functionality of WAL-E and must reside on the database machine.Unlikewal-push andwal-fetch commands, which function as describedabove, thebackup-push andbackup-fetch require a little additionalexplanation.

By defaultbackup-push will include all user defined tablespaces inthe database backup. please see thebackup-fetch section below forWAL-E's tablespace restoration behavior.

Usebackup-fetch to restore a base backup from storage.

This command makes use of theLATEST pseudo-backup-name to find abackup to download:

$ envdir /etc/wal-e.d/fetch-env wal-e               \--s3-prefix=s3://some-bucket/directory/or/whatever  \backup-fetch /var/lib/my/database LATEST

Also allowed is naming a backup specifically as seen inbackup-list, which can be useful for restoring older backups forthe purposes of point in time recovery:

$ envdir /etc/wal-e.d/fetch-env wal-e               \--s3-prefix=s3://some-bucket/directory/or/whatever  \backup-fetch                                        \/var/lib/my/database base_LONGWALNUMBER_POSITION_NUMBER

One will need to provide arecovery.conf file to recover WALsegments associated with the backup. In short,recovery.conf needsto be created in the Postgres's data directory with content like:

restore_command = 'envdir /etc/wal-e.d/env wal-e wal-fetch %f %p'standby_mode = on

A database with such a recovery.conf set will poll WAL-E storage forWAL indefinitely. You can exit recovery by runningpg_ctl promote.

If you wish to perform Point In Time Recovery (PITR) can addrecoverytargets torecovery.conf, looking like this:

recovery_target_time = '2017-02-01 19:58:55'

There are several other ways to specify recovery target,e.g. transaction id.

Regardless of recovery target, the result by default is Postgres willpause recovery at this time, allowing inspection before promotion.Seerecovery targets for details on how to customize what happenswhen the target criterion is reached.

If and only if you are using Tablespaces, you will need to consideradditional issues on how runbackup-fetch. The options are:

• User-directed Restore

  WAL-E expects that tablespace symlinks will be in place prior to abackup-fetch run. This means prepare your target path byinsuring${PG_CLUSTER_DIRECTORY}/pg_tblspc contains all requiredsymlinks before restoration time. If any expected symlink does notexistbackup-fetch will fail.

• Blind Restore

  If you are unable to reproduce tablespace storage structures priorto runningbackup-fetch you can set the option flag--blind-restore. This will direct WAL-E to skip the symlinkverification process and place all data directly in the${PG_CLUSTER_DIRECTORY}/pg_tblspc path.

• Restoration Specification

  You can provide a restoration specification file to WAL-E using thebackup-fetch--restore-spec RESTORE_SPEC option. This specmust be valid JSON and contain all contained tablespaces as well asthe target storage path they require, and the symlink postgresexpects for the tablespace. Here is an example for a cluster with asingle tablespace:

  {    "12345": {        "loc": "/data/postgres/tablespaces/tblspc001/",        "link": "pg_tblspc/12345"    },    "tablespaces": [        "12345"    ],}

  Given this information WAL-E will create the data storage directoryand symlink it appropriately in${PG_CLUSTER_DIRECTORY}/pg_tblspc.

These are commands that are not used expressly for backup or WALpushing and fetching, but are important to the monitoring ormaintenance of WAL-E archived databases. Unlike the critical fouroperators for taking and restoring backups (backup-push,backup-fetch,wal-push,wal-fetch) that must reside on thedatabase machine, these commands can be productively run from anycomputer with the appropriate _PREFIX set and the necessary credentials tomanipulate or read data there.

backup-list is useful for listing base backups that are complete for agiven WAL-E context. Some fields are only filled in when the--detail option is passed tobackup-list[1].

Firstly, the fields that are filled in regardless of if--detailis passed or not:

Secondly, the fields that are filled in only when--detail ispassed:

delete contains additional subcommands that are used for deletingdata from storage for various reasons. These commands are organizedseparately because thedelete subcommand itself takes options thatapply to any subcommand that does deletion, such as--confirm.

All deletions are designed to be reentrant and idempotent: there areno negative consequences if one runs several deletions at once or ifone resubmits the same deletion command several times, with or withoutcanceling other deletions that may be concurrent.

These commands have adry-run mode that is the default. Thecommand is basically optimized for not deleting data except in a veryspecific circumstance to avoid operator error. Should a dry-run beperformed,wal-e will instead simply report every key it wouldotherwise delete if it was not running in dry-run mode, along withprominent HINT-lines for every key noting that nothing was actuallydeleted from the blob store.

Toactually delete any data, one must pass--confirm towal-edelete. If one passes both--dry-run and--confirm, a dryrun will be performed, regardless of the order of options passed.

Currently, these kinds of deletions are supported. Examples omitenvironment variable configuration for clarity:

• before: Delete all backups and wal segment files before thegiven base-backup name. This does not include the base backuppassed: it will remain a viable backup.

  Example:

  $ wal-e delete [--confirm] before base_00000004000002DF000000A6_03626144

• retain: Leave the given number of backups in place, and deleteall base backups and wal segment files older than them.

  Example:

  $ wal-e delete [--confirm] retain 5

• old-versions: Delete all backups and wal file segments with anolder format. This is only intended to be run after a major WAL-Eversion upgrade and the subsequent base-backup. If no base backupis successfully performed first, one is more exposed to data lossuntil one does perform a base backup.

  Example:

  $ wal-e delete [--confirm] old-versions

• everything: Delete all backups and wal file segments in thecontext. This is appropriate if one is decommissioning a databaseand has no need for its archives.

  Example:

  $ wal-e delete [--confirm] everything

All assets pushed to storage are run through the program "lzop" whichcompresses the object using the very fast lzo compression algorithm.It takes roughly 2 CPU seconds to compress a gigabyte, which whensending things to storage at about 25MB/s occupies about 5% CPU time.Compression ratios are expected to make file sizes 50% or less of theoriginal file size in most cases, making backups and restorationsconsiderably faster.

Because storage services generally require the Content-Length headerof a stored object to be set up-front, it is necessary to completelyfinish compressing an entire input file and storing the compressedoutput in a temporary file. Thus, the temporary file directory needsto be big enough and fast enough to support this, although this toolis designed to avoid calling fsync(), so some memory can be leveraged.

Base backups first have their files consolidated into disjoint tarfiles of limited length to avoid the relatively large per-file transferoverhead. This has the effect of making base backups and restoresmuch faster when many small relations and ancillary files areinvolved.

To encrypt backups as well as compress them, first generate a key pairusinggpg --gen-key. You don't need the private key on the machineto back up, but you will need it to restore. The private key may havea password, but to restore, the password should be present in GPGagent. WAL-E does not support entering GPG passwords via a tty device.

Once this is done, set theWALE_GPG_KEY_ID environment variable orthe--gpg-key-id command line option to the ID of the secret keyfor backup and restore commands.

Here's an example of how you can restore with a private key that has apassword, by forcing decryption of an arbitrary file with the correctkey to unlock the GPG keychain:

# This assumes you have "keychain" gpg-agent installed.eval $( keychain --eval --agents gpg )# If you want default gpg-agent, use this instead# eval $( gpg-agent --daemon )# Force storing the private key password in the agent.  Here you# will need to enter the key password.export TEMPFILE=`tempfile`gpg --recipient "$WALE_GPG_KEY_ID" --encrypt "$TEMPFILE"gpg --decrypt "$TEMPFILE".gpg || exit 1rm "$TEMPFILE" "$TEMPFILE".gpgunset TEMPFILE# Now use wal-e to fetch the backup.wal-e backup-fetch [...]# If you have WAL segments encrypted, don't forget to add# restore_command to recovery.conf, e.g.## restore_command = 'wal-e wal-fetch "%f" "%p"'# Start the restoration postgres server in a context where you have# gpg-agent's environment variables initialized, such as the current# shell.pg_ctl -D [...] start

To reduce the read load on base backups, they are sent through thetoolpv first. To use this rate-limited-read mode, use the option--cluster-read-rate-limit as seen inwal-e backup-push.

WAL-E supports logging configuration with following environmentvariables:

• WALE_LOG_DESTINATION comma separated values,syslog andstderr are supported. The default is equivalent to:syslog,stderr.
• WALE_SYSLOG_FACILITY fromLOCAL0 toLOCAL7 andUSER.

To restrict log statements to warnings and errors, use the--terseoption.

In certain situations, thewal-push process can take long enoughthat it can't keep up with WAL segments being produced by Postgres,which can lead to unbounded disk usage and an eventual crash of thedatabase.

One can instruct WAL-E to pool WAL segments together and send them ingroups by passing the--pool-size parameter towal-push. Thiscan increase throughput significantly.

As of version 1.x,--pool-size defaults to 32.

Note: You can also use this parameter when callingbackup-fetchandbackup-push (it defaults to 4).

Storing credentials on AWS EC2 instances has usability and securitydrawbacks. When using WAL-E with AWS S3 and AWS EC2, most uses ofWAL-E would benefit from use with theAWS Instance Profile feature,which automatically generates and rotates credentials on behalf of aninstance.

To instruct WAL-E to use these credentials for access to S3, pass the--aws-instance-profile flag.

Instance profiles maynot be preferred in more complex scenarioswhen one has multiple AWS IAM policies written for multiple programsrun on an instance, or an existing key management infrastructure.

If one wishes to target WAL-E against an alternate S3 endpoint(e.g. Ceph RADOS), one can set theWALE_S3_ENDPOINT environmentvariable. This can also be used take fine-grained control overendpoints and calling conventions with AWS.

The format is that of:

protocol+convention://hostname:port

Where valid protocols arehttp andhttps, and conventions arepath,virtualhost, andsubdomain.

Example:

# Turns off encryption and specifies us-west-1 endpoint.WALE_S3_ENDPOINT=http+path://s3-us-west-1.amazonaws.com:80# For radosgw.WALE_S3_ENDPOINT=http+path://hostname# As seen when using Deis, which uses radosgw.WALE_S3_ENDPOINT=http+path://deis-store-gateway:8888

Development is heavily reliant on the tooltox being existent withinthe development environment. All additional dependencies of WAL-E aremanaged bytox. In addition, the coding conventions are checked bythetox configuration included with WAL-E.

To run the tests, run:

$ tox -e py35

To run a somewhat more lengthy suite of integration tests thatcommunicate with a real blob store account, one might runtox likethis:

$ WALE_S3_INTEGRATION_TESTS=TRUE      \  AWS_ACCESS_KEY_ID=[AKIA...]         \  AWS_SECRET_ACCESS_KEY=[...]         \  WALE_WABS_INTEGRATION_TESTS=TRUE    \  WABS_ACCOUNT_NAME=[...]             \  WABS_ACCESS_KEY=[...]               \  WALE_GS_INTEGRATION_TESTS=TRUE      \  GOOGLE_APPLICATION_CREDENTIALS=[~/my-credentials.json] \  tox -e py35 -- -n 8

Looking carefully at the above, notice the-n 8 added thetoxinvocation. This-n 8 is after a-- that indicates totoxthat the subsequent arguments are for the underlying test programpytest.

This is to enable parallel test execution, which makes the integrationtests complete a small fraction of the time it would take otherwise.It is a design requirement of new tests that parallel execution not besacrificed.

Coverage testing can be used by combining any of these usingpytest-cov, e.g.:tox -- --cov wal_e andtox -- --cov wal_e --cov-report html; see htmlcov/index.html.