Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitbabbbb5

Browse files
committed
Add support for LZ4 compression in pg_receivewal
pg_receivewal gains a new option, --compression-method=lz4, availablewhen the code is compiled with --with-lz4. Similarly to gzip, thisgives the possibility to compress archived WAL segments with LZ4. Thisoption is not compatible with --compress.The implementation uses LZ4 frames, and is compatible with simple lz4commands. Like gzip, using --synchronous ensures that any data will beflushed to disk within the current .partial segment, so as it ispossible to retrieve as much WAL data as possible even from anon-completed segment (this requires completing the partial file withzeros up to the WAL segment size supported by the backend afterdecompression, but this is the same as gzip).The calculation of the streaming start LSN is able to transparently findand check LZ4-compressed segments. Contrary to gzip where theuncompressed size is directly stored in the object read, the LZ4 chunkprotocol does not store the uncompressed data by default. There iscontentSize that can be used with LZ4 frames by that would not help ifusing an archive that includes segments compressed with the defaults ofa "lz4" command, where this is not stored. So, this commit has takenthe most extensible approach by decompressing the already-archivedsegment to check its uncompressed size, through a blank output buffer inchunks of 64kB (no actual performance difference noticed with 8kB, 16kBor 32kB, and the operation in itself is actually fast).Tests have been added to verify the creation and correctness of thegenerated LZ4 files. The latter is achieved by the use of command"lz4", if found in the environment.The tar-based WAL method in walmethods.c, used now only bypg_basebackup, does not know yet about LZ4. Its code could be extendedfor this purpose.Author: Georgios KokolatosReviewed-by: Michael Paquier, Jian Guo, Magnus Hagander, Dilip KumarDiscussion:https://postgr.es/m/ZCm1J5vfyQ2E6dYvXz8si39HQ2gwxSZ3IpYaVgYa3lUwY88SLapx9EEnOf5uEwrddhx2twG7zYKjVeuP5MwZXCNPybtsGouDsAD1o2L_I5E=@pm.me
1 parent5cd7eb1 commitbabbbb5

File tree

7 files changed

+388
-13
lines changed

7 files changed

+388
-13
lines changed

‎doc/src/sgml/ref/pg_receivewal.sgml

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -268,13 +268,15 @@ PostgreSQL documentation
268268
<listitem>
269269
<para>
270270
Enables compression of write-ahead logs using the specified method.
271-
Supported values <literal>gzip</literal>, and
272-
<literal>none</literal>.
271+
Supported values <literal>gzip</literal>, <literal>lz4</literal>
272+
(if <productname>PostgreSQL</productname> was compiled with
273+
<option>--with-lz4</option>), and <literal>none</literal>.
273274
</para>
274275

275276
<para>
276277
The suffix <filename>.gz</filename> will automatically be added to
277-
all filenames when using <literal>gzip</literal>
278+
all filenames when using <literal>gzip</literal>, and the suffix
279+
<filename>.lz4</filename> is added when using <literal>lz4</literal>.
278280
</para>
279281
</listitem>
280282
</varlistentry>

‎src/Makefile.global.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -350,6 +350,7 @@ XGETTEXT = @XGETTEXT@
350350

351351
GZIP= gzip
352352
BZIP2= bzip2
353+
LZ4= lz4
353354

354355
DOWNLOAD = wget -O$@ --no-use-server-timestamps
355356
#DOWNLOAD = curl -o $@

‎src/bin/pg_basebackup/Makefile

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ top_builddir = ../../..
1919
include$(top_builddir)/src/Makefile.global
2020

2121
# make these available to TAP test scripts
22+
exportLZ4
2223
exportTAR
2324
# Note that GZIP cannot be used directly as this environment variable is
2425
# used by the command "gzip" to pass down options, so stick with a different

‎src/bin/pg_basebackup/pg_receivewal.c

Lines changed: 157 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,10 @@
3232
#include"receivelog.h"
3333
#include"streamutil.h"
3434

35+
#ifdefHAVE_LIBLZ4
36+
#include"lz4frame.h"
37+
#endif
38+
3539
/* Time to sleep between reconnection attempts */
3640
#defineRECONNECT_SLEEP_TIME 5
3741

@@ -136,6 +140,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
136140
return true;
137141
}
138142

143+
/* File looks like a completed LZ4-compressed WAL file */
144+
if (fname_len==XLOG_FNAME_LEN+strlen(".lz4")&&
145+
strcmp(filename+XLOG_FNAME_LEN,".lz4")==0)
146+
{
147+
*ispartial= false;
148+
*wal_compression_method=COMPRESSION_LZ4;
149+
return true;
150+
}
151+
139152
/* File looks like a partial uncompressed WAL file */
140153
if (fname_len==XLOG_FNAME_LEN+strlen(".partial")&&
141154
strcmp(filename+XLOG_FNAME_LEN,".partial")==0)
@@ -154,6 +167,15 @@ is_xlogfilename(const char *filename, bool *ispartial,
154167
return true;
155168
}
156169

170+
/* File looks like a partial LZ4-compressed WAL file */
171+
if (fname_len==XLOG_FNAME_LEN+strlen(".lz4.partial")&&
172+
strcmp(filename+XLOG_FNAME_LEN,".lz4.partial")==0)
173+
{
174+
*ispartial= true;
175+
*wal_compression_method=COMPRESSION_LZ4;
176+
return true;
177+
}
178+
157179
/* File does not look like something we know */
158180
return false;
159181
}
@@ -278,12 +300,20 @@ FindStreamingStart(uint32 *tli)
278300
/*
279301
* Check that the segment has the right size, if it's supposed to be
280302
* completed. For non-compressed segments just check the on-disk size
281-
* and see if it matches a completed segment. For gzip-compressed
303+
* and see if it matches a completed segment.For gzip-compressed
282304
* segments, look at the last 4 bytes of the compressed file, which is
283305
* where the uncompressed size is located for files with a size lower
284306
* than 4GB, and then compare it to the size of a completed segment.
285307
* The 4 last bytes correspond to the ISIZE member according to
286308
* http://www.zlib.org/rfc-gzip.html.
309+
*
310+
* For LZ4-compressed segments, uncompress the file in a throw-away
311+
* buffer keeping track of the uncompressed size, then compare it to
312+
* the size of a completed segment. Per its protocol, LZ4 does not
313+
* store the uncompressed size of an object by default. contentSize
314+
* is one possible way to do that, but we need to rely on a method
315+
* where WAL segments could have been compressed by a different source
316+
* than pg_receivewal, like an archive_command with lz4.
287317
*/
288318
if (!ispartial&&wal_compression_method==COMPRESSION_NONE)
289319
{
@@ -350,6 +380,114 @@ FindStreamingStart(uint32 *tli)
350380
continue;
351381
}
352382
}
383+
elseif (!ispartial&&wal_compression_method==COMPRESSION_LZ4)
384+
{
385+
#ifdefHAVE_LIBLZ4
386+
#defineLZ4_CHUNK_SZ64 * 1024/* 64kB as maximum chunk size read */
387+
intfd;
388+
ssize_tr;
389+
size_tuncompressed_size=0;
390+
charfullpath[MAXPGPATH*2];
391+
char*outbuf;
392+
char*readbuf;
393+
LZ4F_decompressionContext_tctx=NULL;
394+
LZ4F_decompressOptions_tdec_opt;
395+
LZ4F_errorCode_tstatus;
396+
397+
memset(&dec_opt,0,sizeof(dec_opt));
398+
snprintf(fullpath,sizeof(fullpath),"%s/%s",basedir,dirent->d_name);
399+
400+
fd=open(fullpath,O_RDONLY |PG_BINARY,0);
401+
if (fd<0)
402+
{
403+
pg_log_error("could not open file \"%s\": %m",fullpath);
404+
exit(1);
405+
}
406+
407+
status=LZ4F_createDecompressionContext(&ctx,LZ4F_VERSION);
408+
if (LZ4F_isError(status))
409+
{
410+
pg_log_error("could not create LZ4 decompression context: %s",
411+
LZ4F_getErrorName(status));
412+
exit(1);
413+
}
414+
415+
outbuf=pg_malloc0(LZ4_CHUNK_SZ);
416+
readbuf=pg_malloc0(LZ4_CHUNK_SZ);
417+
do
418+
{
419+
char*readp;
420+
char*readend;
421+
422+
r=read(fd,readbuf,LZ4_CHUNK_SZ);
423+
if (r<0)
424+
{
425+
pg_log_error("could not read file \"%s\": %m",fullpath);
426+
exit(1);
427+
}
428+
429+
/* Done reading the file */
430+
if (r==0)
431+
break;
432+
433+
/* Process one chunk */
434+
readp=readbuf;
435+
readend=readbuf+r;
436+
while (readp<readend)
437+
{
438+
size_tout_size=LZ4_CHUNK_SZ;
439+
size_tread_size=readend-readp;
440+
441+
memset(outbuf,0,LZ4_CHUNK_SZ);
442+
status=LZ4F_decompress(ctx,outbuf,&out_size,
443+
readp,&read_size,&dec_opt);
444+
if (LZ4F_isError(status))
445+
{
446+
pg_log_error("could not decompress file \"%s\": %s",
447+
fullpath,
448+
LZ4F_getErrorName(status));
449+
exit(1);
450+
}
451+
452+
readp+=read_size;
453+
uncompressed_size+=out_size;
454+
}
455+
456+
/*
457+
* No need to continue reading the file when the
458+
* uncompressed_size exceeds WalSegSz, even if there are still
459+
* data left to read. However, if uncompressed_size is equal
460+
* to WalSegSz, it should verify that there is no more data to
461+
* read.
462+
*/
463+
}while (uncompressed_size <=WalSegSz&&r>0);
464+
465+
close(fd);
466+
pg_free(outbuf);
467+
pg_free(readbuf);
468+
469+
status=LZ4F_freeDecompressionContext(ctx);
470+
if (LZ4F_isError(status))
471+
{
472+
pg_log_error("could not free LZ4 decompression context: %s",
473+
LZ4F_getErrorName(status));
474+
exit(1);
475+
}
476+
477+
if (uncompressed_size!=WalSegSz)
478+
{
479+
pg_log_warning("compressed segment file \"%s\" has incorrect uncompressed size %ld, skipping",
480+
dirent->d_name,uncompressed_size);
481+
continue;
482+
}
483+
#else
484+
pg_log_error("could not check file \"%s\"",
485+
dirent->d_name);
486+
pg_log_error("this build does not support compression with %s",
487+
"LZ4");
488+
exit(1);
489+
#endif
490+
}
353491

354492
/* Looks like a valid segment. Remember that we saw it. */
355493
if ((segno>high_segno)||
@@ -650,6 +788,8 @@ main(int argc, char **argv)
650788
case6:
651789
if (pg_strcasecmp(optarg,"gzip")==0)
652790
compression_method=COMPRESSION_GZIP;
791+
elseif (pg_strcasecmp(optarg,"lz4")==0)
792+
compression_method=COMPRESSION_LZ4;
653793
elseif (pg_strcasecmp(optarg,"none")==0)
654794
compression_method=COMPRESSION_NONE;
655795
else
@@ -746,6 +886,22 @@ main(int argc, char **argv)
746886
pg_log_error("this build does not support compression with %s",
747887
"gzip");
748888
exit(1);
889+
#endif
890+
break;
891+
caseCOMPRESSION_LZ4:
892+
#ifdefHAVE_LIBLZ4
893+
if (compresslevel!=0)
894+
{
895+
pg_log_error("cannot use --compress with --compression-method=%s",
896+
"lz4");
897+
fprintf(stderr,_("Try \"%s --help\" for more information.\n"),
898+
progname);
899+
exit(1);
900+
}
901+
#else
902+
pg_log_error("this build does not support compression with %s",
903+
"LZ4");
904+
exit(1);
749905
#endif
750906
break;
751907
}

‎src/bin/pg_basebackup/t/020_pg_receivewal.pl

Lines changed: 64 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@
55
use warnings;
66
use PostgreSQL::Test::Utils;
77
use PostgreSQL::Test::Cluster;
8-
use Test::Moretests=>37;
8+
use Test::Moretests=>42;
99

1010
program_help_ok('pg_receivewal');
1111
program_version_ok('pg_receivewal');
@@ -138,21 +138,77 @@
138138
"gzip verified the integrity of compressed WAL segments");
139139
}
140140

141+
# Check LZ4 compression if available
142+
SKIP:
143+
{
144+
skip"postgres was not built with LZ4 support", 5
145+
if (!check_pg_config("#define HAVE_LIBLZ4 1"));
146+
147+
# Generate more WAL including one completed, compressed segment.
148+
$primary->psql('postgres','SELECT pg_switch_wal();');
149+
$nextlsn =
150+
$primary->safe_psql('postgres','SELECT pg_current_wal_insert_lsn();');
151+
chomp($nextlsn);
152+
$primary->psql('postgres','INSERT INTO test_table VALUES (3);');
153+
154+
# Stream up to the given position.
155+
$primary->command_ok(
156+
[
157+
'pg_receivewal','-D',
158+
$stream_dir,'--verbose',
159+
'--endpos',$nextlsn,
160+
'--no-loop','--compression-method',
161+
'lz4'
162+
],
163+
'streaming some WAL using --compression-method=lz4');
164+
165+
# Verify that the stored files are generated with their expected
166+
# names.
167+
my@lz4_wals =glob"$stream_dir/*.lz4";
168+
is(scalar(@lz4_wals), 1,
169+
"one WAL segment compressed with LZ4 was created");
170+
my@lz4_partial_wals =glob"$stream_dir/*.lz4.partial";
171+
is(scalar(@lz4_partial_wals),
172+
1,"one partial WAL segment compressed with LZ4 was created");
173+
174+
# Verify that the start streaming position is computed correctly by
175+
# comparing it with the partial file generated previously. The name
176+
# of the previous partial, now-completed WAL segment is updated, keeping
177+
# its base number.
178+
$partial_wals[0] =~s/(\.gz)?\.partial$/.lz4/;
179+
is($lz4_wals[0]eq$partial_wals[0],
180+
1,"one partial WAL segment is now completed");
181+
# Update the list of partial wals with the current one.
182+
@partial_wals =@lz4_partial_wals;
183+
184+
# Check the integrity of the completed segment, if LZ4 is an available
185+
# command.
186+
my$lz4 =$ENV{LZ4};
187+
skip"program lz4 is not found in your system", 1
188+
if ( !defined$lz4
189+
||$lz4eq''
190+
|| system_log($lz4,'--version') != 0);
191+
192+
my$lz4_is_valid = system_log($lz4,'-t',@lz4_wals);
193+
is($lz4_is_valid, 0,
194+
"lz4 verified the integrity of compressed WAL segments");
195+
}
196+
141197
# Verify that the start streaming position is computed and that the value is
142-
# correct regardless of whetherZLIB is available.
198+
# correct regardless of whetherany compression is available.
143199
$primary->psql('postgres','SELECT pg_switch_wal();');
144200
$nextlsn =
145201
$primary->safe_psql('postgres','SELECT pg_current_wal_insert_lsn();');
146202
chomp($nextlsn);
147-
$primary->psql('postgres','INSERT INTO test_table VALUES (3);');
203+
$primary->psql('postgres','INSERT INTO test_table VALUES (4);');
148204
$primary->command_ok(
149205
[
150206
'pg_receivewal','-D',$stream_dir,'--verbose',
151207
'--endpos',$nextlsn,'--no-loop'
152208
],
153209
"streaming some WAL");
154210

155-
$partial_wals[0] =~s/(\.gz)?.partial//;
211+
$partial_wals[0] =~s/(\.gz|\.lz4)?.partial//;
156212
ok(-e$partial_wals[0],"check that previously partial WAL is now complete");
157213

158214
# Permissions on WAL files should be default
@@ -190,15 +246,15 @@
190246

191247
# Switch to a new segment, to make sure that the segment retained by the
192248
# slot is still streamed. This may not be necessary, but play it safe.
193-
$primary->psql('postgres','INSERT INTO test_table VALUES (4);');
249+
$primary->psql('postgres','INSERT INTO test_table VALUES (5);');
194250
$primary->psql('postgres','SELECT pg_switch_wal();');
195251
$nextlsn =
196252
$primary->safe_psql('postgres','SELECT pg_current_wal_insert_lsn();');
197253
chomp($nextlsn);
198254

199255
# Add a bit more data to accelerate the end of the next pg_receivewal
200256
# commands.
201-
$primary->psql('postgres','INSERT INTO test_table VALUES (5);');
257+
$primary->psql('postgres','INSERT INTO test_table VALUES (6);');
202258

203259
# Check case where the slot does not exist.
204260
$primary->command_fails_like(
@@ -253,13 +309,13 @@
253309
# on the new timeline.
254310
my$walfile_after_promotion =$standby->safe_psql('postgres',
255311
"SELECT pg_walfile_name(pg_current_wal_insert_lsn());");
256-
$standby->psql('postgres','INSERT INTO test_table VALUES (6);');
312+
$standby->psql('postgres','INSERT INTO test_table VALUES (7);');
257313
$standby->psql('postgres','SELECT pg_switch_wal();');
258314
$nextlsn =
259315
$standby->safe_psql('postgres','SELECT pg_current_wal_insert_lsn();');
260316
chomp($nextlsn);
261317
# This speeds up the operation.
262-
$standby->psql('postgres','INSERT INTO test_table VALUES (7);');
318+
$standby->psql('postgres','INSERT INTO test_table VALUES (8);');
263319

264320
# Now try to resume from the slot after the promotion.
265321
my$timeline_dir =$primary->basedir .'/timeline_wal';

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp