Repository files navigation

This repository contains fork of pg_arman by Postgres Professional withblock level incremental backup support.

pg_arman is a backup and recovery manager for PostgreSQL servers able to dodifferential and full backup as well as restore a cluster to astate defined by a given recovery target. It is designed to performperiodic backups of an existing PostgreSQL server, combined with WALarchives to provide a way to recover a server in case of failure ofserver because of a reason or another. Its differential backupfacility reduces the amount of data necessary to be taken betweentwo consecutive backups.

The latest version of this software can be found on the project website athttps://github.com/postgrespro/pg_arman. Original fork of pg_arman can befound athttps://github.com/michaelpq/pg_arman.

Compiling pg_arman requires a PostgreSQL installation to be in placeas well as a raw source tree. Pass the path to the PostgreSQL source treeto make, in the top_srcdir variable:

make USE_PGXS=1 top_srcdir=<path to PostgreSQL source tree>

In addition, you must have pg_config in $PATH.

The current version of pg_arman is compatible with PostgreSQL 9.5 andupper versions.

pg_arman has been tested on Linux and Unix-based platforms.

All the documentation you can findhere.

The test suite of pg_arman is available in the code tree and can belaunched in a way similar to common PostgreSQL extensions and modules:

make installcheck

Idea of block level incremental backup is that you may backup only blockschanged since last full backup. It gives two major benefits: taking backupsfaster and making backups smaller.

The major question here is how to get the list of changed blocks. Sinceeach block contains LSN number, changed blocks could be retrieved by full scanof all the blocks. But this approach consumes as much server IO as fullbackup.

This is why we implemented alternative approaches to retrievelist of changed blocks.

1. Scan WAL archive and extract changed blocks from it. However, shortcomingof these approach is requirement to have WAL archive.

2. Track bitmap of changes blocks inside PostgreSQL (ptrack). It introducessome overhead to PostgreSQL performance. On our experiments it appears to beless than 3%.

These two approaches were implemented in this fork of pg_arman. The secondapproach requirespatch for PostgreSQL.

You need build and installPGPRO9_5_ptrack branch of PostgreSQL orapply this patch to PostgreSQL 9.5.

You need to enable WAL archive by adding following lines to postgresql.conf:

wal_level = archivearchive_command = 'test ! -f /home/postgres/backup/arman/wal/%f && cp %p /home/postgres/backup/arman/wal/%f'wal_log_hints = on

Example backup (assuming PostgreSQL is running):

# Init pg_aramn backup folderpg_arman init -B /home/postgres/backup/pgarmancat<<__EOF__ >> /home/postgres/backup/pgarman/pg_arman.iniARCLOG_PATH = '/home/postgres/backup/arman/wal'__EOF__# Make full backup with 2 thread and verbose mode.pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2# Validate backuppg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman# Show backups informationpg_arman show -B /home/postgres/backup/pgarman# Now you can insert or update some data in your database# Then start the incremental backup.pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b page -v -j 2pg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman# You should see that increment is really smallpg_arman show -B /home/postgres/backup/pgarman

For restore after remove your pgdata you can use:

pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose

The advantage of this approach is that you don't have to save WAL archive. You will need to enable ptrack in postgresql.conf (restart required).

ptrack_enable = on

Also, some WALs still need to be fetched in order to get consistent backup. pg_arman can fetch them trough the streaming replication protocol. Thus, you also need toenable streaming replication connection.

Example backup (assuming PostgreSQL is running):

# Init pg_aramn backup folderpg_arman init -B /home/postgres/backup/pgarmancat<<__EOF__ >> /home/postgres/backup/pgarman/pg_arman.iniARCLOG_PATH = '/home/postgres/backup/arman/wal'__EOF__# Make full backup with 2 thread and verbose mode.pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b full -v -j 2 --stream# Validate backuppg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman# Show backups informationpg_arman show -B /home/postgres/backup/pgarman# Now you can insert or update some data in your database# Then start the incremental backup.pg_arman backup -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -b ptrack -v -j 2 --streampg_arman validate -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman# You should see that increment is really smallpg_arman show -B /home/postgres/backup/pgarman

For restore after remove your pgdata you can use:

pg_arman restore -B /home/postgres/backup/pgarman -D /home/postgres/pgdata/arman -j 4 --verbose --stream

pg_arman can be distributed under the PostgreSQL license. See COPYRIGHTfile for more information. pg_arman is a fork of the existing projectpg_rman, initially created and maintained by NTT and Itagaki Takahiro.