Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitffd5365

Browse files
committed
Replace BASE_BACKUP COMPRESSION_LEVEL option with COMPRESSION_DETAIL.
There are more compression parameters that can be specified than justan integer compression level, so rename the new COMPRESSION_LEVELoption to COMPRESSION_DETAIL before it gets released. Introduce aflexible syntax for that option to allow arbitrary options to bespecified without needing to adjust the main replication grammar,and common code to parse it that is shared between the client andthe server.This commit doesn't actually add any new compression parameters,so the only user-visible change is that you can now type somethinglike pg_basebackup --compress gzip:level=5 instead of writing justpg_basebackup --compress gzip:5. However, it should make it easy toadd new options. If for example gzip starts offering fries, we cansupport pg_basebackup --compress gzip:level=5,fries=true for thebenefit of users who want fries with that.Along the way, this fixes a few things in pg_basebackup so that thepg_basebackup can be used with a server-side compression algorithmthat pg_basebackup itself does not understand. For example,pg_basebackup --compress server-lz4 could still succeed even ifonly the server and not the client has LZ4 support, provided thatthe other options to pg_basebackup don't require the client todecompress the archive.Patch by me. Reviewed by Justin Pryzby and Dagfinn Ilmari Mannsåker.Discussion:http://postgr.es/m/CA+TgmoYvpetyRAbbg1M8b3-iHsaN4nsgmWPjOENu5-doHuJ7fA@mail.gmail.com
1 parent4a39f87 commitffd5365

File tree

17 files changed

+662
-331
lines changed

17 files changed

+662
-331
lines changed

‎doc/src/sgml/protocol.sgml‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2731,14 +2731,24 @@ The commands accepted in replication mode are:
27312731
</varlistentry>
27322732

27332733
<varlistentry>
2734-
<term><literal>COMPRESSION_LEVEL</literal> <replaceable>level</replaceable></term>
2734+
<term><literal>COMPRESSION_DETAIL</literal> <replaceable>detail</replaceable></term>
27352735
<listitem>
27362736
<para>
2737-
Specifies the compression level to be used. This should only be
2738-
used in conjunction with the <literal>COMPRESSION</literal> option.
2739-
For <literal>gzip</literal> the value should be an integer between 1
2740-
and 9, for <literal>lz4</literal> between 1 and 12, and for
2741-
<literal>zstd</literal> it should be between 1 and 22.
2737+
Specifies details for the chosen compression method. This should only
2738+
be used in conjunction with the <literal>COMPRESSION</literal>
2739+
option. If the value is an integer, it specifies the compression
2740+
level. Otherwise, it should be a comma-separated list of items,
2741+
each of the form <literal>keyword</literal> or
2742+
<literal>keyword=value</literal>. Currently, the only supported
2743+
keyword is <literal>level</literal>, which sets the compression
2744+
level.
2745+
</para>
2746+
2747+
<para>
2748+
For <literal>gzip</literal> the compression level should be an
2749+
integer between 1 and 9, for <literal>lz4</literal> an integer
2750+
between 1 and 12, and for <literal>zstd</literal> an integer
2751+
between 1 and 22.
27422752
</para>
27432753
</listitem>
27442754
</varlistentry>

‎doc/src/sgml/ref/pg_basebackup.sgml‎

Lines changed: 16 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -399,9 +399,9 @@ PostgreSQL documentation
399399

400400
<varlistentry>
401401
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
402-
<term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
402+
<term><option>-Z [{client|server}-]<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
403403
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
404-
<term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable></option>[:<replaceable>level</replaceable>]</term>
404+
<term><option>--compress=[{client|server}-]<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
405405
<listitem>
406406
<para>
407407
Requests compression of the backup. If <literal>client</literal> or
@@ -419,13 +419,20 @@ PostgreSQL documentation
419419
<para>
420420
The compression method can be set to <literal>gzip</literal>,
421421
<literal>lz4</literal>, <literal>zstd</literal>, or
422-
<literal>none</literal> for no compression. A compression level can
423-
optionally be specified, by appending the level number after a colon
424-
(<literal>:</literal>). If no level is specified, the default
425-
compression level will be used. If only a level is specified without
426-
mentioning an algorithm, <literal>gzip</literal> compression will be
427-
used if the level is greater than 0, and no compression will be used if
428-
the level is 0.
422+
<literal>none</literal> for no compression. A compression detail
423+
string can optionally be specified. If the detail string is an
424+
integer, it specifies the compression level. Otherwise, it should be
425+
a comma-separated list of items, each of the form
426+
<literal>keyword</literal> or <literal>keyword=value</literal>.
427+
Currently, the only supported keyword is <literal>level</literal>,
428+
which sets the compression level.
429+
</para>
430+
<para>
431+
If no compression level is specified, the default compression level
432+
will be used. If only a level is specified without mentioning an
433+
algorithm, <literal>gzip</literal> compression will be used if the
434+
level is greater than 0, and no compression will be used if the level
435+
is 0.
429436
</para>
430437
<para>
431438
When the tar format is used with <literal>gzip</literal>,

‎src/backend/replication/basebackup.c‎

Lines changed: 32 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
#include<time.h>
1818

1919
#include"access/xlog_internal.h"/* for pg_start/stop_backup */
20+
#include"common/backup_compression.h"
2021
#include"common/file_perm.h"
2122
#include"commands/defrem.h"
2223
#include"lib/stringinfo.h"
@@ -54,14 +55,6 @@
5455
*/
5556
#defineSINK_BUFFER_LENGTHMax(32768, BLCKSZ)
5657

57-
typedefenum
58-
{
59-
BACKUP_COMPRESSION_NONE,
60-
BACKUP_COMPRESSION_GZIP,
61-
BACKUP_COMPRESSION_LZ4,
62-
BACKUP_COMPRESSION_ZSTD
63-
}basebackup_compression_type;
64-
6558
typedefstruct
6659
{
6760
constchar*label;
@@ -75,8 +68,8 @@ typedef struct
7568
booluse_copytblspc;
7669
BaseBackupTargetHandle*target_handle;
7770
backup_manifest_optionmanifest;
78-
basebackup_compression_typecompression;
79-
intcompression_level;
71+
bc_algorithmcompression;
72+
bc_specificationcompression_specification;
8073
pg_checksum_typemanifest_checksum_type;
8174
}basebackup_options;
8275

@@ -713,12 +706,14 @@ parse_basebackup_options(List *options, basebackup_options *opt)
713706
char*target_str=NULL;
714707
char*target_detail_str=NULL;
715708
boolo_compression= false;
716-
boolo_compression_level= false;
709+
boolo_compression_detail= false;
710+
char*compression_detail_str=NULL;
717711

718712
MemSet(opt,0,sizeof(*opt));
719713
opt->manifest=MANIFEST_OPTION_NO;
720714
opt->manifest_checksum_type=CHECKSUM_TYPE_CRC32C;
721715
opt->compression=BACKUP_COMPRESSION_NONE;
716+
opt->compression_specification.algorithm=BACKUP_COMPRESSION_NONE;
722717

723718
foreach(lopt,options)
724719
{
@@ -885,29 +880,21 @@ parse_basebackup_options(List *options, basebackup_options *opt)
885880
ereport(ERROR,
886881
(errcode(ERRCODE_SYNTAX_ERROR),
887882
errmsg("duplicate option \"%s\"",defel->defname)));
888-
if (strcmp(optval,"none")==0)
889-
opt->compression=BACKUP_COMPRESSION_NONE;
890-
elseif (strcmp(optval,"gzip")==0)
891-
opt->compression=BACKUP_COMPRESSION_GZIP;
892-
elseif (strcmp(optval,"lz4")==0)
893-
opt->compression=BACKUP_COMPRESSION_LZ4;
894-
elseif (strcmp(optval,"zstd")==0)
895-
opt->compression=BACKUP_COMPRESSION_ZSTD;
896-
else
883+
if (!parse_bc_algorithm(optval,&opt->compression))
897884
ereport(ERROR,
898885
(errcode(ERRCODE_SYNTAX_ERROR),
899-
errmsg("unrecognized compression algorithm: \"%s\"",
886+
errmsg("unrecognized compression algorithm \"%s\"",
900887
optval)));
901888
o_compression= true;
902889
}
903-
elseif (strcmp(defel->defname,"compression_level")==0)
890+
elseif (strcmp(defel->defname,"compression_detail")==0)
904891
{
905-
if (o_compression_level)
892+
if (o_compression_detail)
906893
ereport(ERROR,
907894
(errcode(ERRCODE_SYNTAX_ERROR),
908895
errmsg("duplicate option \"%s\"",defel->defname)));
909-
opt->compression_level=defGetInt32(defel);
910-
o_compression_level= true;
896+
compression_detail_str=defGetString(defel);
897+
o_compression_detail= true;
911898
}
912899
else
913900
ereport(ERROR,
@@ -949,10 +936,25 @@ parse_basebackup_options(List *options, basebackup_options *opt)
949936
opt->target_handle=
950937
BaseBackupGetTargetHandle(target_str,target_detail_str);
951938

952-
if (o_compression_level&& !o_compression)
939+
if (o_compression_detail&& !o_compression)
953940
ereport(ERROR,
954941
(errcode(ERRCODE_SYNTAX_ERROR),
955-
errmsg("compression level requires compression")));
942+
errmsg("compression detail requires compression")));
943+
944+
if (o_compression)
945+
{
946+
char*error_detail;
947+
948+
parse_bc_specification(opt->compression,compression_detail_str,
949+
&opt->compression_specification);
950+
error_detail=
951+
validate_bc_specification(&opt->compression_specification);
952+
if (error_detail!=NULL)
953+
ereport(ERROR,
954+
errcode(ERRCODE_SYNTAX_ERROR),
955+
errmsg("invalid compression specification: %s",
956+
error_detail));
957+
}
956958
}
957959

958960

@@ -998,11 +1000,11 @@ SendBaseBackup(BaseBackupCmd *cmd)
9981000

9991001
/* Set up server-side compression, if client requested it */
10001002
if (opt.compression==BACKUP_COMPRESSION_GZIP)
1001-
sink=bbsink_gzip_new(sink,opt.compression_level);
1003+
sink=bbsink_gzip_new(sink,&opt.compression_specification);
10021004
elseif (opt.compression==BACKUP_COMPRESSION_LZ4)
1003-
sink=bbsink_lz4_new(sink,opt.compression_level);
1005+
sink=bbsink_lz4_new(sink,&opt.compression_specification);
10041006
elseif (opt.compression==BACKUP_COMPRESSION_ZSTD)
1005-
sink=bbsink_zstd_new(sink,opt.compression_level);
1007+
sink=bbsink_zstd_new(sink,&opt.compression_specification);
10061008

10071009
/* Set up progress reporting. */
10081010
sink=bbsink_progress_new(sink,opt.progress);

‎src/backend/replication/basebackup_gzip.c‎

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_gzip_ops = {
5656
#endif
5757

5858
/*
59-
* Create a new basebackup sink that performs gzip compression using the
60-
* designated compression level.
59+
* Create a new basebackup sink that performs gzip compression.
6160
*/
6261
bbsink*
63-
bbsink_gzip_new(bbsink*next,intcompresslevel)
62+
bbsink_gzip_new(bbsink*next,bc_specification*compress)
6463
{
64+
intcompresslevel;
65+
6566
#ifndefHAVE_LIBZ
6667
ereport(ERROR,
6768
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,15 +72,14 @@ bbsink_gzip_new(bbsink *next, int compresslevel)
7172
bbsink_gzip*sink;
7273

7374
Assert(next!=NULL);
74-
Assert(compresslevel >=0&&compresslevel <=9);
7575

76-
if (compresslevel==0)
76+
if ((compress->options&BACKUP_COMPRESSION_OPTION_LEVEL)==0)
7777
compresslevel=Z_DEFAULT_COMPRESSION;
78-
elseif (compresslevel<0||compresslevel>9)
79-
ereport(ERROR,
80-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
81-
errmsg("gzip compression level %d is out of range",
82-
compresslevel)));
78+
else
79+
{
80+
compresslevel=compress->level;
81+
Assert(compresslevel >=1&&compresslevel <=9);
82+
}
8383

8484
sink=palloc0(sizeof(bbsink_gzip));
8585
*((constbbsink_ops**)&sink->base.bbs_ops)=&bbsink_gzip_ops;

‎src/backend/replication/basebackup_lz4.c‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -56,12 +56,13 @@ const bbsink_ops bbsink_lz4_ops = {
5656
#endif
5757

5858
/*
59-
* Create a new basebackup sink that performs lz4 compression using the
60-
* designated compression level.
59+
* Create a new basebackup sink that performs lz4 compression.
6160
*/
6261
bbsink*
63-
bbsink_lz4_new(bbsink*next,intcompresslevel)
62+
bbsink_lz4_new(bbsink*next,bc_specification*compress)
6463
{
64+
intcompresslevel;
65+
6566
#ifndefUSE_LZ4
6667
ereport(ERROR,
6768
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -72,11 +73,13 @@ bbsink_lz4_new(bbsink *next, int compresslevel)
7273

7374
Assert(next!=NULL);
7475

75-
if (compresslevel<0||compresslevel>12)
76-
ereport(ERROR,
77-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
78-
errmsg("lz4 compression level %d is out of range",
79-
compresslevel)));
76+
if ((compress->options&BACKUP_COMPRESSION_OPTION_LEVEL)==0)
77+
compresslevel=0;
78+
else
79+
{
80+
compresslevel=compress->level;
81+
Assert(compresslevel >=1&&compresslevel <=12);
82+
}
8083

8184
sink=palloc0(sizeof(bbsink_lz4));
8285
*((constbbsink_ops**)&sink->base.bbs_ops)=&bbsink_lz4_ops;

‎src/backend/replication/basebackup_zstd.c‎

Lines changed: 11 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -55,12 +55,13 @@ const bbsink_ops bbsink_zstd_ops = {
5555
#endif
5656

5757
/*
58-
* Create a new basebackup sink that performs zstd compression using the
59-
* designated compression level.
58+
* Create a new basebackup sink that performs zstd compression.
6059
*/
6160
bbsink*
62-
bbsink_zstd_new(bbsink*next,intcompresslevel)
61+
bbsink_zstd_new(bbsink*next,bc_specification*compress)
6362
{
63+
intcompresslevel;
64+
6465
#ifndefUSE_ZSTD
6566
ereport(ERROR,
6667
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
@@ -71,11 +72,13 @@ bbsink_zstd_new(bbsink *next, int compresslevel)
7172

7273
Assert(next!=NULL);
7374

74-
if (compresslevel<0||compresslevel>22)
75-
ereport(ERROR,
76-
(errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
77-
errmsg("zstd compression level %d is out of range",
78-
compresslevel)));
75+
if ((compress->options&BACKUP_COMPRESSION_OPTION_LEVEL)==0)
76+
compresslevel=0;
77+
else
78+
{
79+
compresslevel=compress->level;
80+
Assert(compresslevel >=1&&compresslevel <=22);
81+
}
7982

8083
sink=palloc0(sizeof(bbsink_zstd));
8184
*((constbbsink_ops**)&sink->base.bbs_ops)=&bbsink_zstd_ops;

‎src/bin/pg_basebackup/bbstreamer.h‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222
#ifndefBBSTREAMER_H
2323
#defineBBSTREAMER_H
2424

25+
#include"common/backup_compression.h"
2526
#include"lib/stringinfo.h"
2627
#include"pqexpbuffer.h"
2728

@@ -200,17 +201,17 @@ bbstreamer_buffer_until(bbstreamer *streamer, const char **data, int *len,
200201
*/
201202
externbbstreamer*bbstreamer_plain_writer_new(char*pathname,FILE*file);
202203
externbbstreamer*bbstreamer_gzip_writer_new(char*pathname,FILE*file,
203-
intcompresslevel);
204+
bc_specification*compress);
204205
externbbstreamer*bbstreamer_extractor_new(constchar*basepath,
205206
constchar*(*link_map) (constchar*),
206207
void (*report_output_file) (constchar*));
207208

208209
externbbstreamer*bbstreamer_gzip_decompressor_new(bbstreamer*next);
209210
externbbstreamer*bbstreamer_lz4_compressor_new(bbstreamer*next,
210-
intcompresslevel);
211+
bc_specification*compress);
211212
externbbstreamer*bbstreamer_lz4_decompressor_new(bbstreamer*next);
212213
externbbstreamer*bbstreamer_zstd_compressor_new(bbstreamer*next,
213-
intcompresslevel);
214+
bc_specification*compress);
214215
externbbstreamer*bbstreamer_zstd_decompressor_new(bbstreamer*next);
215216
externbbstreamer*bbstreamer_tar_parser_new(bbstreamer*next);
216217
externbbstreamer*bbstreamer_tar_terminator_new(bbstreamer*next);

‎src/bin/pg_basebackup/bbstreamer_gzip.c‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,8 @@ const bbstreamer_ops bbstreamer_gzip_decompressor_ops = {
7676
* closed so that the data may be written there.
7777
*/
7878
bbstreamer*
79-
bbstreamer_gzip_writer_new(char*pathname,FILE*file,intcompresslevel)
79+
bbstreamer_gzip_writer_new(char*pathname,FILE*file,
80+
bc_specification*compress)
8081
{
8182
#ifdefHAVE_LIBZ
8283
bbstreamer_gzip_writer*streamer;
@@ -115,11 +116,11 @@ bbstreamer_gzip_writer_new(char *pathname, FILE *file, int compresslevel)
115116
}
116117
}
117118

118-
if (gzsetparams(streamer->gzfile,compresslevel,
119+
if (gzsetparams(streamer->gzfile,compress->level,
119120
Z_DEFAULT_STRATEGY)!=Z_OK)
120121
{
121122
pg_log_error("could not set compression level %d: %s",
122-
compresslevel,get_gz_error(streamer->gzfile));
123+
compress->level,get_gz_error(streamer->gzfile));
123124
exit(1);
124125
}
125126

‎src/bin/pg_basebackup/bbstreamer_lz4.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ const bbstreamer_ops bbstreamer_lz4_decompressor_ops = {
6767
* blocks.
6868
*/
6969
bbstreamer*
70-
bbstreamer_lz4_compressor_new(bbstreamer*next,intcompresslevel)
70+
bbstreamer_lz4_compressor_new(bbstreamer*next,bc_specification*compress)
7171
{
7272
#ifdefUSE_LZ4
7373
bbstreamer_lz4_frame*streamer;
@@ -89,7 +89,7 @@ bbstreamer_lz4_compressor_new(bbstreamer *next, int compresslevel)
8989
prefs=&streamer->prefs;
9090
memset(prefs,0,sizeof(LZ4F_preferences_t));
9191
prefs->frameInfo.blockSizeID=LZ4F_max256KB;
92-
prefs->compressionLevel=compresslevel;
92+
prefs->compressionLevel=compress->level;
9393

9494
/*
9595
* Find out the compression bound, it specifies the minimum destination

‎src/bin/pg_basebackup/bbstreamer_zstd.c‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ const bbstreamer_ops bbstreamer_zstd_decompressor_ops = {
6363
* blocks.
6464
*/
6565
bbstreamer*
66-
bbstreamer_zstd_compressor_new(bbstreamer*next,intcompresslevel)
66+
bbstreamer_zstd_compressor_new(bbstreamer*next,bc_specification*compress)
6767
{
6868
#ifdefUSE_ZSTD
6969
bbstreamer_zstd_frame*streamer;
@@ -85,7 +85,7 @@ bbstreamer_zstd_compressor_new(bbstreamer *next, int compresslevel)
8585

8686
/* Initialize stream compression preferences */
8787
ZSTD_CCtx_setParameter(streamer->cctx,ZSTD_c_compressionLevel,
88-
compresslevel);
88+
compress->level);
8989

9090
/* Initialize the ZSTD output buffer. */
9191
streamer->zstd_outBuf.dst=streamer->base.bbs_buffer.data;

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp