Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit042a923

Browse files
committed
Rework compression options of pg_receivewal
Sincebabbbb5 and the introduction of LZ4 in pg_receivewal, thecompression of the WAL archived is controlled by two options:- --compression-method with "gzip", "none" or "lz4" as possible value.- --compress=N to specify a compression level. This includes abackward-incompatible change where a value of 0 leads to a failureinstead of no compression enforced.This commit takes advantage ofa4b5754 and3603f7c to rework thecompression options of pg_receivewal, as of:- The removal of --compression-method.- The extenction of --compress to use the same grammar as pg_basebackup,with a METHOD:DETAIL format, where a METHOD is "gzip", "none" or "lz4"and a DETAIL is a comma-separated list of options, the only keywordsupported is now "level" to control the compression level. If only aninteger is specified as value of this option, "none" is implied on 0and "gzip" is implied otherwise. This brings back --compress to bebackward-compatible with ~14, while still supporting LZ4.This has also the advantage of centralizing the set of checks used bypg_receivewal to validate its compression options.Author: Michael PaquierReviewed-by: Robert Haas, Georgios KokolatosDiscussion:https://postgr.es/m/YlPQGNAAa04raObK@paquier.xyz
1 parentd27323d commit042a923

File tree

3 files changed

+121
-79
lines changed

3 files changed

+121
-79
lines changed

‎doc/src/sgml/ref/pg_receivewal.sgml

Lines changed: 22 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -263,37 +263,36 @@ PostgreSQL documentation
263263
</varlistentry>
264264

265265
<varlistentry>
266-
<term><option>--compression-method=<replaceable class="parameter">method</replaceable></option></term>
266+
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
267+
<term><option>-Z <replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
268+
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
269+
<term><option>--compress=<replaceable class="parameter">method</replaceable>[:<replaceable>detail</replaceable>]</option></term>
267270
<listitem>
268271
<para>
269-
Enables compression of write-ahead logs using the specified method.
270-
Supported values are <literal>gzip</literal>, <literal>lz4</literal>
271-
(if <productname>PostgreSQL</productname> was compiled with
272-
<option>--with-lz4</option>), and <literal>none</literal>.
272+
Enables compression of write-ahead logs.
273273
</para>
274-
275274
<para>
276-
The suffix <filename>.gz</filename> will automatically be added to
277-
all filenames when using <literal>gzip</literal>, and the suffix
278-
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
275+
The compression method can be set to <literal>gzip</literal>,
276+
<literal>lz4</literal> (if <productname>PostgreSQL</productname>
277+
was compiled with <option>--with-lz4</option>) or
278+
<literal>none</literal> for no compression.
279+
A compression detail string can optionally be specified. If the
280+
detail string is an integer, it specifies the compression level.
281+
Otherwise, it should be a comma-separated list of items, each of the
282+
form <literal>keyword</literal> or <literal>keyword=value</literal>.
283+
Currently, the only supported keyword is <literal>level</literal>.
279284
</para>
280-
</listitem>
281-
</varlistentry>
282-
283-
<varlistentry>
284-
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
285-
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
286-
<listitem>
287285
<para>
288-
Specifies the compression level (<literal>1</literal> through
289-
<literal>9</literal>, <literal>1</literal> being worst compression
290-
and <literal>9</literal> being best compression) for WAL segments
291-
compressed with <application>gzip</application>.
286+
If no compression level is specified, the default compression level
287+
will be used. If only a level is specified without mentioning an
288+
algorithm, <literal>gzip</literal> compression will be used if the
289+
level is greater than 0, and no compression will be used if the level
290+
is 0.
292291
</para>
293-
294292
<para>
295-
This option requires <option>--compression-method</option> to be
296-
specified with <literal>gzip</literal>.
293+
The suffix <filename>.gz</filename> will automatically be added to
294+
all filenames when using <literal>gzip</literal>, and the suffix
295+
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
297296
</para>
298297
</listitem>
299298
</varlistentry>

‎src/bin/pg_basebackup/pg_receivewal.c

Lines changed: 90 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,8 @@ static XLogRecPtr endpos = InvalidXLogRecPtr;
5757

5858

5959
staticvoidusage(void);
60+
staticvoidparse_compress_options(char*option,char**algorithm,
61+
char**detail);
6062
staticDIR*get_destination_dir(char*dest_folder);
6163
staticvoidclose_destination_dir(DIR*dest_dir,char*dest_folder);
6264
staticXLogRecPtrFindStreamingStart(uint32*tli);
@@ -90,9 +92,8 @@ usage(void)
9092
printf(_(" --synchronous flush write-ahead log immediately after writing\n"));
9193
printf(_(" -v, --verbose output verbose messages\n"));
9294
printf(_(" -V, --version output version information, then exit\n"));
93-
printf(_(" --compression-method=METHOD\n"
94-
" method to compress logs\n"));
95-
printf(_(" -Z, --compress=1-9 compress logs with given compression level\n"));
95+
printf(_(" -Z, --compress=METHOD[:DETAIL]\n"
96+
" compress as specified\n"));
9697
printf(_(" -?, --help show this help, then exit\n"));
9798
printf(_("\nConnection options:\n"));
9899
printf(_(" -d, --dbname=CONNSTR connection string\n"));
@@ -108,6 +109,66 @@ usage(void)
108109
printf(_("%s home page: <%s>\n"),PACKAGE_NAME,PACKAGE_URL);
109110
}
110111

112+
/*
113+
* Basic parsing of a value specified for -Z/--compress
114+
*
115+
* The parsing consists of a METHOD:DETAIL string fed later on to a more
116+
* advanced routine in charge of proper validation checks. This only extracts
117+
* METHOD and DETAIL. If only an integer is found, the method is implied by
118+
* the value specified.
119+
*/
120+
staticvoid
121+
parse_compress_options(char*option,char**algorithm,char**detail)
122+
{
123+
char*sep;
124+
char*endp;
125+
longresult;
126+
127+
/*
128+
* Check whether the compression specification consists of a bare integer.
129+
*
130+
* For backward-compatibility, assume "none" if the integer found is zero
131+
* and "gzip" otherwise.
132+
*/
133+
result=strtol(option,&endp,10);
134+
if (*endp=='\0')
135+
{
136+
if (result==0)
137+
{
138+
*algorithm=pstrdup("none");
139+
*detail=NULL;
140+
}
141+
else
142+
{
143+
*algorithm=pstrdup("gzip");
144+
*detail=pstrdup(option);
145+
}
146+
return;
147+
}
148+
149+
/*
150+
* Check whether there is a compression detail following the algorithm
151+
* name.
152+
*/
153+
sep=strchr(option,':');
154+
if (sep==NULL)
155+
{
156+
*algorithm=pstrdup(option);
157+
*detail=NULL;
158+
}
159+
else
160+
{
161+
char*alg;
162+
163+
alg=palloc((sep-option)+1);
164+
memcpy(alg,option,sep-option);
165+
alg[sep-option]='\0';
166+
167+
*algorithm=alg;
168+
*detail=pstrdup(sep+1);
169+
}
170+
}
171+
111172
/*
112173
* Check if the filename looks like a WAL file, letting caller know if this
113174
* WAL segment is partial and/or compressed.
@@ -651,7 +712,6 @@ main(int argc, char **argv)
651712
{"if-not-exists",no_argument,NULL,3},
652713
{"synchronous",no_argument,NULL,4},
653714
{"no-sync",no_argument,NULL,5},
654-
{"compression-method",required_argument,NULL,6},
655715
{NULL,0,NULL,0}
656716
};
657717

@@ -660,6 +720,10 @@ main(int argc, char **argv)
660720
char*db_name;
661721
uint32hi,
662722
lo;
723+
pg_compress_specificationcompression_spec;
724+
char*compression_detail=NULL;
725+
char*compression_algorithm_str="none";
726+
char*error_detail=NULL;
663727

664728
pg_logging_init(argv[0]);
665729
progname=get_progname(argv[0]);
@@ -728,9 +792,8 @@ main(int argc, char **argv)
728792
verbose++;
729793
break;
730794
case'Z':
731-
if (!option_parse_int(optarg,"-Z/--compress",1,9,
732-
&compresslevel))
733-
exit(1);
795+
parse_compress_options(optarg,&compression_algorithm_str,
796+
&compression_detail);
734797
break;
735798
/* action */
736799
case1:
@@ -748,17 +811,6 @@ main(int argc, char **argv)
748811
case5:
749812
do_sync= false;
750813
break;
751-
case6:
752-
if (pg_strcasecmp(optarg,"gzip")==0)
753-
compression_algorithm=PG_COMPRESSION_GZIP;
754-
elseif (pg_strcasecmp(optarg,"lz4")==0)
755-
compression_algorithm=PG_COMPRESSION_LZ4;
756-
elseif (pg_strcasecmp(optarg,"none")==0)
757-
compression_algorithm=PG_COMPRESSION_NONE;
758-
else
759-
pg_fatal("invalid value \"%s\" for option %s",
760-
optarg,"--compression-method");
761-
break;
762814
default:
763815
/* getopt_long already emitted a complaint */
764816
pg_log_error_hint("Try \"%s --help\" for more information.",progname);
@@ -810,24 +862,33 @@ main(int argc, char **argv)
810862
exit(1);
811863
}
812864

813-
814865
/*
815-
* Compression-related options.
866+
* Compression options
816867
*/
868+
if (!parse_compress_algorithm(compression_algorithm_str,
869+
&compression_algorithm))
870+
pg_fatal("unrecognized compression algorithm \"%s\"",
871+
compression_algorithm_str);
872+
873+
parse_compress_specification(compression_algorithm,compression_detail,
874+
&compression_spec);
875+
error_detail=validate_compress_specification(&compression_spec);
876+
if (error_detail!=NULL)
877+
pg_fatal("invalid compression specification: %s",
878+
error_detail);
879+
880+
/* Extract the compression level, if found in the specification */
881+
if ((compression_spec.options&PG_COMPRESSION_OPTION_LEVEL)!=0)
882+
compresslevel=compression_spec.level;
883+
817884
switch (compression_algorithm)
818885
{
819886
casePG_COMPRESSION_NONE:
820-
if (compresslevel!=0)
821-
{
822-
pg_log_error("cannot use --compress with --compression-method=%s",
823-
"none");
824-
pg_log_error_hint("Try \"%s --help\" for more information.",progname);
825-
exit(1);
826-
}
887+
/* nothing to do */
827888
break;
828889
casePG_COMPRESSION_GZIP:
829890
#ifdefHAVE_LIBZ
830-
if (compresslevel==0)
891+
if ((compression_spec.options&PG_COMPRESSION_OPTION_LEVEL)==0)
831892
{
832893
pg_log_info("no value specified for --compress, switching to default");
833894
compresslevel=Z_DEFAULT_COMPRESSION;
@@ -838,15 +899,7 @@ main(int argc, char **argv)
838899
#endif
839900
break;
840901
casePG_COMPRESSION_LZ4:
841-
#ifdefUSE_LZ4
842-
if (compresslevel!=0)
843-
{
844-
pg_log_error("cannot use --compress with --compression-method=%s",
845-
"lz4");
846-
pg_log_error_hint("Try \"%s --help\" for more information.",progname);
847-
exit(1);
848-
}
849-
#else
902+
#ifndefUSE_LZ4
850903
pg_fatal("this build does not support compression with %s",
851904
"LZ4");
852905
#endif

‎src/bin/pg_basebackup/t/020_pg_receivewal.pl

Lines changed: 9 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,9 @@
3434
['pg_receivewal','-D',$stream_dir,'--synchronous','--no-sync' ],
3535
'failure if --synchronous specified with --no-sync');
3636
$primary->command_fails_like(
37-
[
38-
'pg_receivewal','-D',$stream_dir,'--compression-method','none',
39-
'--compress','1'
40-
],
41-
qr/\Qpg_receivewal: error: cannot use --compress with --compression-method=none/,
42-
'failure if --compress specified with --compression-method=none');
37+
['pg_receivewal','-D',$stream_dir,'--compress','none:1', ],
38+
qr/\Qpg_receivewal: error: invalid compression specification: compression algorithm "none" does not accept a compression level/,
39+
'failure if --compress none:N (where N > 0)');
4340

4441
# Slot creation and drop
4542
my$slot_name ='test';
@@ -48,7 +45,7 @@
4845
'creating a replication slot');
4946
my$slot =$primary->slot($slot_name);
5047
is($slot->{'slot_type'},'physical','physical replication slot was created');
51-
is($slot->{'restart_lsn'},'','restart LSN of new slot is null');
48+
is($slot->{'restart_lsn'},'','restart LSN of new slot is null');
5249
$primary->command_ok(['pg_receivewal','--slot',$slot_name,'--drop-slot' ],
5350
'dropping a replication slot');
5451
is($primary->slot($slot_name)->{'slot_type'},
@@ -93,15 +90,10 @@
9390
chomp($nextlsn);
9491
$primary->psql('postgres','INSERT INTO test_table VALUES (2);');
9592

96-
# Note the trailing whitespace after the value of --compress, that is
97-
# a valid value.
9893
$primary->command_ok(
9994
[
100-
'pg_receivewal','-D',
101-
$stream_dir,'--verbose',
102-
'--endpos',$nextlsn,
103-
'--compression-method','gzip',
104-
'--compress','1',
95+
'pg_receivewal','-D',$stream_dir,'--verbose',
96+
'--endpos',$nextlsn,'--compress','gzip:1',
10597
'--no-loop'
10698
],
10799
"streaming some WAL using ZLIB compression");
@@ -153,13 +145,11 @@
153145
# Stream up to the given position.
154146
$primary->command_ok(
155147
[
156-
'pg_receivewal','-D',
157-
$stream_dir,'--verbose',
158-
'--endpos',$nextlsn,
159-
'--no-loop','--compression-method',
148+
'pg_receivewal','-D',$stream_dir,'--verbose',
149+
'--endpos',$nextlsn,'--no-loop','--compress',
160150
'lz4'
161151
],
162-
'streaming some WAL using --compression-method=lz4');
152+
'streaming some WAL using --compress=lz4');
163153

164154
# Verify that the stored files are generated with their expected
165155
# names.

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp