Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit4035cd5

Browse files
committed
Add support for LZ4 with compression of full-page writes in WAL
The logic is implemented so as there can be a choice in the compressionused when building a WAL record, and an extra per-record bit is used totrack down if a block is compressed with PGLZ, LZ4 or nothing.wal_compression, the existing parameter, is changed to an enum withsupport for the following backward-compatible values:- "off", the default, to not use compression.- "pglz" or "on", to compress FPWs with PGLZ.- "lz4", the new mode, to compress FPWs with LZ4.Benchmarking has showed that LZ4 outclasses easily PGLZ. ZSTD would bealso an interesting choice, but going just with LZ4 for now makes thepatch minimalistic as toast compression is already able to use LZ4, sothere is no need to worry about any build-related needs for thisimplementation.Author: Andrey Borodin, Justin PryzbyReviewed-by: Dilip Kumar, Michael PaquierDiscussion:https://postgr.es/m/3037310D-ECB7-4BF1-AF20-01C10BB33A33@yandex-team.ru
1 parentcc2c7d6 commit4035cd5

File tree

13 files changed

+191
-52
lines changed

13 files changed

+191
-52
lines changed

‎doc/src/sgml/config.sgml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3128,23 +3128,27 @@ include_dir 'conf.d'
31283128
</varlistentry>
31293129

31303130
<varlistentry id="guc-wal-compression" xreflabel="wal_compression">
3131-
<term><varname>wal_compression</varname> (<type>boolean</type>)
3131+
<term><varname>wal_compression</varname> (<type>enum</type>)
31323132
<indexterm>
31333133
<primary><varname>wal_compression</varname> configuration parameter</primary>
31343134
</indexterm>
31353135
</term>
31363136
<listitem>
31373137
<para>
3138-
When this parameter is <literal>on</literal>, the <productname>PostgreSQL</productname>
3138+
This parameter enables compression of WAL using the specified
3139+
compression method.
3140+
When enabled, the <productname>PostgreSQL</productname>
31393141
server compresses full page images written to WAL when
31403142
<xref linkend="guc-full-page-writes"/> is on or during a base backup.
31413143
A compressed page image will be decompressed during WAL replay.
3142-
The default value is <literal>off</literal>.
3143-
Only superusers can change this setting.
3144+
The supported methods are <literal>pglz</literal> and
3145+
<literal>lz4</literal> (if <productname>PostgreSQL</productname> was
3146+
compiled with <option>--with-lz4</option>). The default value is
3147+
<literal>off</literal>. Only superusers can change this setting.
31443148
</para>
31453149

31463150
<para>
3147-
Turning this parameter on can reduce the WAL volume without
3151+
Enabling compression can reduce the WAL volume without
31483152
increasing the risk of unrecoverable data corruption,
31493153
but at the cost of some extra CPU spent on the compression during
31503154
WAL logging and on the decompression during WAL replay.

‎doc/src/sgml/install-windows.sgml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -299,7 +299,7 @@ $ENV{MSBFLAGS}="/m";
299299
<term><productname>LZ4</productname></term>
300300
<listitem><para>
301301
Required for supporting <productname>LZ4</productname> compression
302-
method for compressingthetable data. Binaries and source can be
302+
method for compressing table or WAL data. Binaries and source can be
303303
downloaded from
304304
<ulink url="https://github.com/lz4/lz4/releases"></ulink>.
305305
</para></listitem>

‎doc/src/sgml/installation.sgml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -270,7 +270,8 @@ su - postgres
270270
<para>
271271
You need <productname>LZ4</productname>, if you want to support
272272
compression of data with this method; see
273-
<xref linkend="guc-default-toast-compression"/>.
273+
<xref linkend="guc-default-toast-compression"/> and
274+
<xref linkend="guc-wal-compression"/>.
274275
</para>
275276
</listitem>
276277

@@ -980,7 +981,7 @@ build-postgresql:
980981
<para>
981982
Build with <productname>LZ4</productname> compression support.
982983
This allows the use of <productname>LZ4</productname> for
983-
compression of table data.
984+
compression of tableand WALdata.
984985
</para>
985986
</listitem>
986987
</varlistentry>

‎doc/src/sgml/standalone-profile.xsl

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,10 @@ variant without links and references to the main documentation.
5252
<xsl:text>the configuration parameter default_toast_compression</xsl:text>
5353
</xsl:template>
5454

55+
<xsl:templatematch="xref[@linkend='guc-wal-compression']">
56+
<xsl:text>the configuration parameter wal_compression</xsl:text>
57+
</xsl:template>
58+
5559
<xsl:templatematch="xref[@linkend='install-windows']">
5660
<xsl:text>the documentation</xsl:text>
5761
</xsl:template>

‎src/backend/access/transam/xlog.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,7 +98,7 @@ char *XLogArchiveCommand = NULL;
9898
boolEnableHotStandby= false;
9999
boolfullPageWrites= true;
100100
boolwal_log_hints= false;
101-
boolwal_compression=false;
101+
intwal_compression=WAL_COMPRESSION_NONE;
102102
char*wal_consistency_checking_string=NULL;
103103
bool*wal_consistency_checking=NULL;
104104
boolwal_init_zero= true;

‎src/backend/access/transam/xloginsert.c

Lines changed: 64 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,8 +33,20 @@
3333
#include"storage/proc.h"
3434
#include"utils/memutils.h"
3535

36-
/* Buffer size required to store a compressed version of backup block image */
37-
#definePGLZ_MAX_BLCKSZ PGLZ_MAX_OUTPUT(BLCKSZ)
36+
/*
37+
* Guess the maximum buffer size required to store a compressed version of
38+
* backup block image.
39+
*/
40+
#ifdefUSE_LZ4
41+
#include<lz4.h>
42+
#defineLZ4_MAX_BLCKSZLZ4_COMPRESSBOUND(BLCKSZ)
43+
#else
44+
#defineLZ4_MAX_BLCKSZ0
45+
#endif
46+
47+
#definePGLZ_MAX_BLCKSZPGLZ_MAX_OUTPUT(BLCKSZ)
48+
49+
#defineCOMPRESS_BUFSIZEMax(PGLZ_MAX_BLCKSZ, LZ4_MAX_BLCKSZ)
3850

3951
/*
4052
* For each block reference registered with XLogRegisterBuffer, we fill in
@@ -58,7 +70,7 @@ typedef struct
5870
* backup block data in XLogRecordAssemble() */
5971

6072
/* buffer to store a compressed version of backup block image */
61-
charcompressed_page[PGLZ_MAX_BLCKSZ];
73+
charcompressed_page[COMPRESS_BUFSIZE];
6274
}registered_buffer;
6375

6476
staticregistered_buffer*registered_buffers;
@@ -628,7 +640,7 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
628640
/*
629641
* Try to compress a block image if wal_compression is enabled
630642
*/
631-
if (wal_compression)
643+
if (wal_compression!=WAL_COMPRESSION_NONE)
632644
{
633645
is_compressed=
634646
XLogCompressBackupBlock(page,bimg.hole_offset,
@@ -665,8 +677,29 @@ XLogRecordAssemble(RmgrId rmid, uint8 info,
665677

666678
if (is_compressed)
667679
{
680+
/* The current compression is stored in the WAL record */
668681
bimg.length=compressed_len;
669-
bimg.bimg_info |=BKPIMAGE_IS_COMPRESSED;
682+
683+
/* Set the compression method used for this block */
684+
switch ((WalCompression)wal_compression)
685+
{
686+
caseWAL_COMPRESSION_PGLZ:
687+
bimg.bimg_info |=BKPIMAGE_COMPRESS_PGLZ;
688+
break;
689+
690+
caseWAL_COMPRESSION_LZ4:
691+
#ifdefUSE_LZ4
692+
bimg.bimg_info |=BKPIMAGE_COMPRESS_LZ4;
693+
#else
694+
elog(ERROR,"LZ4 is not supported by this build");
695+
#endif
696+
break;
697+
698+
caseWAL_COMPRESSION_NONE:
699+
Assert(false);/* cannot happen */
700+
break;
701+
/* no default case, so that compiler will warn */
702+
}
670703

671704
rdt_datas_last->data=regbuf->compressed_page;
672705
rdt_datas_last->len=compressed_len;
@@ -853,12 +886,34 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
853886
else
854887
source=page;
855888

889+
switch ((WalCompression)wal_compression)
890+
{
891+
caseWAL_COMPRESSION_PGLZ:
892+
len=pglz_compress(source,orig_len,dest,PGLZ_strategy_default);
893+
break;
894+
895+
caseWAL_COMPRESSION_LZ4:
896+
#ifdefUSE_LZ4
897+
len=LZ4_compress_default(source,dest,orig_len,
898+
COMPRESS_BUFSIZE);
899+
if (len <=0)
900+
len=-1;/* failure */
901+
#else
902+
elog(ERROR,"LZ4 is not supported by this build");
903+
#endif
904+
break;
905+
906+
caseWAL_COMPRESSION_NONE:
907+
Assert(false);/* cannot happen */
908+
break;
909+
/* no default case, so that compiler will warn */
910+
}
911+
856912
/*
857-
* We recheck the actual size even ifpglz_compress() reports success and
858-
*seeif the number of bytes saved by compression is larger than the
859-
*lengthof extra data needed for the compressed version of block image.
913+
* We recheck the actual size even ifcompression reports success and see
914+
* if the number of bytes saved by compression is larger than the length
915+
* of extra data needed for the compressed version of block image.
860916
*/
861-
len=pglz_compress(source,orig_len,dest,PGLZ_strategy_default);
862917
if (len >=0&&
863918
len+extra_bytes<orig_len)
864919
{

‎src/backend/access/transam/xlogreader.c

Lines changed: 46 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,9 @@
1818
#include"postgres.h"
1919

2020
#include<unistd.h>
21+
#ifdefUSE_LZ4
22+
#include<lz4.h>
23+
#endif
2124

2225
#include"access/transam.h"
2326
#include"access/xlog_internal.h"
@@ -1290,7 +1293,7 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
12901293

12911294
blk->apply_image= ((blk->bimg_info&BKPIMAGE_APPLY)!=0);
12921295

1293-
if (blk->bimg_info&BKPIMAGE_IS_COMPRESSED)
1296+
if (BKPIMAGE_COMPRESSED(blk->bimg_info))
12941297
{
12951298
if (blk->bimg_info&BKPIMAGE_HAS_HOLE)
12961299
COPY_HEADER_FIELD(&blk->hole_length,sizeof(uint16));
@@ -1335,29 +1338,28 @@ DecodeXLogRecord(XLogReaderState *state, XLogRecord *record, char **errormsg)
13351338
}
13361339

13371340
/*
1338-
* cross-check that bimg_len < BLCKSZ if the IS_COMPRESSED
1339-
* flag is set.
1341+
* Cross-check that bimg_len < BLCKSZ if it is compressed.
13401342
*/
1341-
if ((blk->bimg_info&BKPIMAGE_IS_COMPRESSED)&&
1343+
if (BKPIMAGE_COMPRESSED(blk->bimg_info)&&
13421344
blk->bimg_len==BLCKSZ)
13431345
{
13441346
report_invalid_record(state,
1345-
"BKPIMAGE_IS_COMPRESSED set, but block image length %u at %X/%X",
1347+
"BKPIMAGE_COMPRESSED set, but block image length %u at %X/%X",
13461348
(unsignedint)blk->bimg_len,
13471349
LSN_FORMAT_ARGS(state->ReadRecPtr));
13481350
gotoerr;
13491351
}
13501352

13511353
/*
1352-
* cross-check that bimg_len = BLCKSZ if neither HAS_HOLEnor
1353-
*IS_COMPRESSED flag is set.
1354+
* cross-check that bimg_len = BLCKSZ if neither HAS_HOLEis
1355+
*set nor COMPRESSED().
13541356
*/
13551357
if (!(blk->bimg_info&BKPIMAGE_HAS_HOLE)&&
1356-
!(blk->bimg_info&BKPIMAGE_IS_COMPRESSED)&&
1358+
!BKPIMAGE_COMPRESSED(blk->bimg_info)&&
13571359
blk->bimg_len!=BLCKSZ)
13581360
{
13591361
report_invalid_record(state,
1360-
"neither BKPIMAGE_HAS_HOLE norBKPIMAGE_IS_COMPRESSED set, but block image length is %u at %X/%X",
1362+
"neither BKPIMAGE_HAS_HOLE norBKPIMAGE_COMPRESSED set, but block image length is %u at %X/%X",
13611363
(unsignedint)blk->data_len,
13621364
LSN_FORMAT_ARGS(state->ReadRecPtr));
13631365
gotoerr;
@@ -1555,17 +1557,49 @@ RestoreBlockImage(XLogReaderState *record, uint8 block_id, char *page)
15551557
bkpb=&record->blocks[block_id];
15561558
ptr=bkpb->bkp_image;
15571559

1558-
if (bkpb->bimg_info&BKPIMAGE_IS_COMPRESSED)
1560+
if (BKPIMAGE_COMPRESSED(bkpb->bimg_info))
15591561
{
15601562
/* If a backup block image is compressed, decompress it */
1561-
if (pglz_decompress(ptr,bkpb->bimg_len,tmp.data,
1562-
BLCKSZ-bkpb->hole_length, true)<0)
1563+
booldecomp_success= true;
1564+
1565+
if ((bkpb->bimg_info&BKPIMAGE_COMPRESS_PGLZ)!=0)
1566+
{
1567+
if (pglz_decompress(ptr,bkpb->bimg_len,tmp.data,
1568+
BLCKSZ-bkpb->hole_length, true)<0)
1569+
decomp_success= false;
1570+
}
1571+
elseif ((bkpb->bimg_info&BKPIMAGE_COMPRESS_LZ4)!=0)
1572+
{
1573+
#ifdefUSE_LZ4
1574+
if (LZ4_decompress_safe(ptr,tmp.data,
1575+
bkpb->bimg_len,BLCKSZ-bkpb->hole_length) <=0)
1576+
decomp_success= false;
1577+
#else
1578+
report_invalid_record(record,"image at %X/%X compressed with %s not supported by build, block %d",
1579+
(uint32) (record->ReadRecPtr >>32),
1580+
(uint32)record->ReadRecPtr,
1581+
"LZ4",
1582+
block_id);
1583+
return false;
1584+
#endif
1585+
}
1586+
else
1587+
{
1588+
report_invalid_record(record,"image at %X/%X compressed with unknown method, block %d",
1589+
(uint32) (record->ReadRecPtr >>32),
1590+
(uint32)record->ReadRecPtr,
1591+
block_id);
1592+
return false;
1593+
}
1594+
1595+
if (!decomp_success)
15631596
{
15641597
report_invalid_record(record,"invalid compressed image at %X/%X, block %d",
15651598
LSN_FORMAT_ARGS(record->ReadRecPtr),
15661599
block_id);
15671600
return false;
15681601
}
1602+
15691603
ptr=tmp.data;
15701604
}
15711605

‎src/backend/utils/misc/guc.c

Lines changed: 26 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -540,6 +540,22 @@ static struct config_enum_entry default_toast_compression_options[] = {
540540
{NULL,0, false}
541541
};
542542

543+
staticconststructconfig_enum_entrywal_compression_options[]= {
544+
{"pglz",WAL_COMPRESSION_PGLZ, false},
545+
#ifdefUSE_LZ4
546+
{"lz4",WAL_COMPRESSION_LZ4, false},
547+
#endif
548+
{"on",WAL_COMPRESSION_PGLZ, false},
549+
{"off",WAL_COMPRESSION_NONE, false},
550+
{"true",WAL_COMPRESSION_PGLZ, true},
551+
{"false",WAL_COMPRESSION_NONE, true},
552+
{"yes",WAL_COMPRESSION_PGLZ, true},
553+
{"no",WAL_COMPRESSION_NONE, true},
554+
{"1",WAL_COMPRESSION_PGLZ, true},
555+
{"0",WAL_COMPRESSION_NONE, true},
556+
{NULL,0, false}
557+
};
558+
543559
/*
544560
* Options for enum values stored in other modules
545561
*/
@@ -1304,16 +1320,6 @@ static struct config_bool ConfigureNamesBool[] =
13041320
NULL,NULL,NULL
13051321
},
13061322

1307-
{
1308-
{"wal_compression",PGC_SUSET,WAL_SETTINGS,
1309-
gettext_noop("Compresses full-page writes written in WAL file."),
1310-
NULL
1311-
},
1312-
&wal_compression,
1313-
false,
1314-
NULL,NULL,NULL
1315-
},
1316-
13171323
{
13181324
{"wal_init_zero",PGC_SUSET,WAL_SETTINGS,
13191325
gettext_noop("Writes zeroes to new WAL files before first use."),
@@ -4816,6 +4822,16 @@ static struct config_enum ConfigureNamesEnum[] =
48164822
NULL,NULL,NULL
48174823
},
48184824

4825+
{
4826+
{"wal_compression",PGC_SUSET,WAL_SETTINGS,
4827+
gettext_noop("Compresses full-page writes written in WAL file with specified method."),
4828+
NULL
4829+
},
4830+
&wal_compression,
4831+
WAL_COMPRESSION_NONE,wal_compression_options,
4832+
NULL,NULL,NULL
4833+
},
4834+
48194835
{
48204836
{"wal_level",PGC_POSTMASTER,WAL_SETTINGS,
48214837
gettext_noop("Sets the level of information written to the WAL."),

‎src/backend/utils/misc/postgresql.conf.sample

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -218,7 +218,8 @@
218218
#full_page_writes = on# recover from partial page writes
219219
#wal_log_hints = off# also do full page writes of non-critical updates
220220
# (change requires restart)
221-
#wal_compression = off# enable compression of full-page writes
221+
#wal_compression = off# enables compression of full-page writes;
222+
# off, pglz, lz4, or on
222223
#wal_init_zero = on# zero-fill new WAL files
223224
#wal_recycle = on# recycle WAL files
224225
#wal_buffers = -1# min 32kB, -1 sets based on shared_buffers

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp