Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit0d8c9c1

Browse files
committed
Generate backup manifests for base backups, and validate them.
A manifest is a JSON document which includes (1) the file name, size,last modification time, and an optional checksum for each file backedup, (2) timelines and LSNs for whatever WAL will need to be replayedto make the backup consistent, and (3) a checksum for the manifestitself. By default, we use CRC-32C when checksumming data files,because we are trying to detect corruption and user error, not foil anadversary. However, pg_basebackup and the server-side BASE_BACKUPcommand now have options to select a different algorithm, so userswanting a cryptographic hash function can select SHA-224, SHA-256,SHA-384, or SHA-512. Users not wanting file checksums at all candisable them, or disable generating of the backup manifest altogether.Using a cryptographic hash function in place of CRC-32C consumessignificantly more CPU cycles, which may slow down backups in somecases.A new tool called pg_validatebackup can validate a backup against themanifest. If no checksums are present, it can still check that theright files exist and that they have the expected sizes. If checksumsare present, it can also verify that each file has the expectedchecksum. Additionally, it calls pg_waldump to verify that theexpected WAL files are present and parseable. Only plain formatbackups can be validated directly, but tar format backups can bevalidated after extracting them.Robert Haas, with help, ideas, review, and testing from David Steele,Stephen Frost, Andrew Dunstan, Rushabh Lathia, Suraj Kharage, TusharAhuja, Rajkumar Raghuwanshi, Mark Dilger, Davinder Singh, JeevanChalke, Amit Kapila, Andres Freund, and Noah Misch.Discussion:http://postgr.es/m/CA+TgmoZV8dw1H2bzZ9xkKwdrk8+XYa+DC9H=F7heO2zna5T6qg@mail.gmail.com
1 parentce77abe commit0d8c9c1

27 files changed

+3614
-32
lines changed

‎doc/src/sgml/protocol.sgml

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2466,7 +2466,7 @@ The commands accepted in replication mode are:
24662466
</varlistentry>
24672467

24682468
<varlistentry id="protocol-replication-base-backup" xreflabel="BASE_BACKUP">
2469-
<term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ]
2469+
<term><literal>BASE_BACKUP</literal> [ <literal>LABEL</literal> <replaceable>'label'</replaceable> ] [ <literal>PROGRESS</literal> ] [ <literal>FAST</literal> ] [ <literal>WAL</literal> ] [ <literal>NOWAIT</literal> ] [ <literal>MAX_RATE</literal> <replaceable>rate</replaceable> ] [ <literal>TABLESPACE_MAP</literal> ] [ <literal>NOVERIFY_CHECKSUMS</literal> ] [ <literal>MANIFEST</literal> <replaceable>manifest_option</replaceable> ] [ <literal>MANIFEST_CHECKSUMS</literal> <replaceable>checksum_algorithm</replaceable> ]
24702470
<indexterm><primary>BASE_BACKUP</primary></indexterm>
24712471
</term>
24722472
<listitem>
@@ -2576,6 +2576,41 @@ The commands accepted in replication mode are:
25762576
</para>
25772577
</listitem>
25782578
</varlistentry>
2579+
2580+
<varlistentry>
2581+
<term><literal>MANIFEST</literal></term>
2582+
<listitem>
2583+
<para>
2584+
When this option is specified with a value of <literal>yes</literal>
2585+
or <literal>force-escape</literal>, a backup manifest is created
2586+
and sent along with the backup. The manifest is a list of every
2587+
file present in the backup with the exception of any WAL files that
2588+
may be included. It also stores the size, last modification time, and
2589+
an optional checksum for each file.
2590+
A value of <literal>force-escape</literal> forces all filenames
2591+
to be hex-encoded; otherwise, this type of encoding is performed only
2592+
for files whose names are non-UTF8 octet sequences.
2593+
<literal>force-escape</literal> is intended primarily for testing
2594+
purposes, to be sure that clients which read the backup manifest
2595+
can handle this case. For compatibility with previous releases,
2596+
the default is <literal>MANIFEST 'no'</literal>.
2597+
</para>
2598+
</listitem>
2599+
</varlistentry>
2600+
2601+
<varlistentry>
2602+
<term><literal>MANIFEST_CHECKSUMS</literal></term>
2603+
<listitem>
2604+
<para>
2605+
Specifies the algorithm that should be applied to each file included
2606+
in the backup manifest. Currently, the available
2607+
algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
2608+
<literal>SHA224</literal>, <literal>SHA256</literal>,
2609+
<literal>SHA384</literal>, and <literal>SHA512</literal>.
2610+
The default is <literal>CRC32C</literal>.
2611+
</para>
2612+
</listitem>
2613+
</varlistentry>
25792614
</variablelist>
25802615
</para>
25812616
<para>

‎doc/src/sgml/ref/allfiles.sgml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,7 @@ Complete list of usable sgml source files in this directory.
211211
<!ENTITY pgResetwal SYSTEM "pg_resetwal.sgml">
212212
<!ENTITY pgRestore SYSTEM "pg_restore.sgml">
213213
<!ENTITY pgRewind SYSTEM "pg_rewind.sgml">
214+
<!ENTITY pgValidateBackup SYSTEM "pg_validatebackup.sgml">
214215
<!ENTITY pgtestfsync SYSTEM "pgtestfsync.sgml">
215216
<!ENTITY pgtesttiming SYSTEM "pgtesttiming.sgml">
216217
<!ENTITY pgupgrade SYSTEM "pgupgrade.sgml">

‎doc/src/sgml/ref/pg_basebackup.sgml

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -561,6 +561,70 @@ PostgreSQL documentation
561561
</para>
562562
</listitem>
563563
</varlistentry>
564+
565+
<varlistentry>
566+
<term><option>--no-manifest</option></term>
567+
<listitem>
568+
<para>
569+
Disables generation of a backup manifest. If this option is not
570+
specified, the server will generate and send a backup manifest
571+
which can be verified using <xref linkend="app-pgvalidatebackup" />.
572+
The manifest is a list of every file present in the backup with the
573+
exception of any WAL files that may be included. It also stores the
574+
size, last modification time, and an optional checksum for each file.
575+
</para>
576+
</listitem>
577+
</varlistentry>
578+
579+
<varlistentry>
580+
<term><option>--manifest-force-encode</option></term>
581+
<listitem>
582+
<para>
583+
Forces all filenames in the backup manifest to be hex-encoded.
584+
If this option is not specified, only non-UTF8 filenames are
585+
hex-encoded. This option is mostly intended to test that tools which
586+
read a backup manifest file properly handle this case.
587+
</para>
588+
</listitem>
589+
</varlistentry>
590+
591+
<varlistentry>
592+
<term><option>--manifest-checksums=<replaceable class="parameter">algorithm</replaceable></option></term>
593+
<listitem>
594+
<para>
595+
Specifies the checksum algorithm that should be applied to each file
596+
included in the backup manifest. Currently, the available
597+
algorithms are <literal>NONE</literal>, <literal>CRC32C</literal>,
598+
<literal>SHA224</literal>, <literal>SHA256</literal>,
599+
<literal>SHA384</literal>, and <literal>SHA512</literal>.
600+
The default is <literal>CRC32C</literal>.
601+
</para>
602+
<para>
603+
If <literal>NONE</literal> is selected, the backup manifest will
604+
not contain any checksums. Otherwise, it will contain a checksum
605+
of each file in the backup using the specified algorithm. In addition,
606+
the manifest will always contain a <literal>SHA256</literal>
607+
checksum of its own contents. The <literal>SHA</literal> algorithms
608+
are significantly more CPU-intensive than <literal>CRC32C</literal>,
609+
so selecting one of them may increase the time required to complete
610+
the backup.
611+
</para>
612+
<para>
613+
Using a SHA hash function provides a cryptographically secure digest
614+
of each file for users who wish to verify that the backup has not been
615+
tampered with, while the CRC32C algorithm provides a checksum which is
616+
much faster to calculate and good at catching errors due to accidental
617+
changes but is not resistant to targeted modifications. Note that, to
618+
be useful against an adversary who has access to the backup, the backup
619+
manifest would need to be stored securely elsewhere or otherwise
620+
verified not to have been modified since the backup was taken.
621+
</para>
622+
<para>
623+
<xref linkend="app-pgvalidatebackup" /> can be used to check the
624+
integrity of a backup against the backup manifest.
625+
</para>
626+
</listitem>
627+
</varlistentry>
564628
</variablelist>
565629
</para>
566630

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp