Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit5e73a60

Browse files
committed
Switch pg_dump to use compression specifications
Compression specifications are currently used by pg_basebackup andpg_receivewal, and are able to let the user control in an extended waythe method and level of compression used. As an effect of this commit,pg_dump's -Z/--compress is now able to use more than just an integer, asof the grammar "method[:detail]".The method can be either "none" or "gzip", and can optionally take adetail string. If the detail string is only an integer, it defines thecompression level. A comma-separated list of keywords can also be usedmethod allows for more options, the only keyword supported now is"level".The change is backward-compatible, hence specifying only an integerleads to no compression for a level of 0 and gzip compression when thelevel is greater than 0.Most of the code changes are straight-forward, as pg_dump was relying onan integer tracking the compression level to check for gzip or nocompression. These are changed to use a compression specification andthe algorithm stored in it.As of this change, note that the dump format is not bumped because thereis no need yet to track the compression algorithm in the TOC entries.Hence, we still rely on the compression level to make the differencewhen reading them. This will be mandatory once a new compression methodis added, though.In order to keep the code simpler when parsing the compressionspecification, the code is changed so as pg_dump now fails hard whenusing gzip on -Z/--compress without its support compiled, rather thanenforcing no compression without the user knowing about it exceptthrough a warning. Like before this commit, archive and custom formatsare compressed by default when the code is compiled with gzip, and leftuncompressed without gzip.Author: Georgios KokolatosReviewed-by: Michael PaquierDiscussion:https://postgr.es/m/O4mutIrCES8ZhlXJiMvzsivT7ztAMja2lkdL1LJx6O5f22I2W8PBIeLKz7mDLwxHoibcnRAYJXm1pH4tyUNC4a8eDzLn22a6Pb1S74Niexg=@pm.me
1 parentedf12e7 commit5e73a60

File tree

14 files changed

+260
-160
lines changed

14 files changed

+260
-160
lines changed

‎doc/src/sgml/ref/pg_dump.sgml

Lines changed: 30 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -644,17 +644,39 @@ PostgreSQL documentation
644644
</varlistentry>
645645

646646
<varlistentry>
647-
<term><option>-Z <replaceable class="parameter">0..9</replaceable></option></term>
648-
<term><option>--compress=<replaceable class="parameter">0..9</replaceable></option></term>
647+
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
648+
<term><option>-Z <replaceable class="parameter">method</replaceable></option>[:<replaceable>detail</replaceable>]</term>
649+
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
650+
<term><option>--compress=<replaceable class="parameter">method</replaceable></option>[:<replaceable>detail</replaceable>]</term>
649651
<listitem>
650652
<para>
651-
Specify the compression level to use. Zero means no compression.
653+
Specify the compression method and/or the compression level to use.
654+
The compression method can be set to <literal>gzip</literal> or
655+
<literal>none</literal> for no compression.
656+
A compression detail string can optionally be specified. If the
657+
detail string is an integer, it specifies the compression level.
658+
Otherwise, it should be a comma-separated list of items, each of the
659+
form <literal>keyword</literal> or <literal>keyword=value</literal>.
660+
Currently, the only supported keyword is <literal>level</literal>.
661+
</para>
662+
663+
<para>
664+
If no compression level is specified, the default compression
665+
level will be used. If only a level is specified without mentioning
666+
an algorithm, <literal>gzip</literal> compression will be used if
667+
the level is greater than <literal>0</literal>, and no compression
668+
will be used if the level is <literal>0</literal>.
669+
</para>
670+
671+
<para>
652672
For the custom and directory archive formats, this specifies compression of
653-
individual table-data segments, and the default is to compress
654-
at a moderate level.
655-
For plain text output, setting a nonzero compression level causes
656-
the entire output file to be compressed, as though it had been
657-
fed through <application>gzip</application>; but the default is not to compress.
673+
individual table-data segments, and the default is to compress using
674+
<literal>gzip</literal> at a moderate level. For plain text output,
675+
setting a nonzero compression level causes the entire output file to be compressed,
676+
as though it had been fed through <application>gzip</application>; but the default
677+
is not to compress.
678+
</para>
679+
<para>
658680
The tar archive format currently does not support compression at all.
659681
</para>
660682
</listitem>

‎src/bin/pg_dump/compress_io.c

Lines changed: 44 additions & 63 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@
6464
/* typedef appears in compress_io.h */
6565
structCompressorState
6666
{
67-
CompressionAlgorithmcomprAlg;
67+
pg_compress_specificationcompression_spec;
6868
WriteFuncwriteF;
6969

7070
#ifdefHAVE_LIBZ
@@ -74,9 +74,6 @@ struct CompressorState
7474
#endif
7575
};
7676

77-
staticvoidParseCompressionOption(intcompression,CompressionAlgorithm*alg,
78-
int*level);
79-
8077
/* Routines that support zlib compressed data I/O */
8178
#ifdefHAVE_LIBZ
8279
staticvoidInitCompressorZlib(CompressorState*cs,intlevel);
@@ -93,57 +90,30 @@ static void ReadDataFromArchiveNone(ArchiveHandle *AH, ReadFunc readF);
9390
staticvoidWriteDataToArchiveNone(ArchiveHandle*AH,CompressorState*cs,
9491
constchar*data,size_tdLen);
9592

96-
/*
97-
* Interprets a numeric 'compression' value. The algorithm implied by the
98-
* value (zlib or none at the moment), is returned in *alg, and the
99-
* zlib compression level in *level.
100-
*/
101-
staticvoid
102-
ParseCompressionOption(intcompression,CompressionAlgorithm*alg,int*level)
103-
{
104-
if (compression==Z_DEFAULT_COMPRESSION||
105-
(compression>0&&compression <=9))
106-
*alg=COMPR_ALG_LIBZ;
107-
elseif (compression==0)
108-
*alg=COMPR_ALG_NONE;
109-
else
110-
{
111-
pg_fatal("invalid compression code: %d",compression);
112-
*alg=COMPR_ALG_NONE;/* keep compiler quiet */
113-
}
114-
115-
/* The level is just the passed-in value. */
116-
if (level)
117-
*level=compression;
118-
}
119-
12093
/* Public interface routines */
12194

12295
/* Allocate a new compressor */
12396
CompressorState*
124-
AllocateCompressor(intcompression,WriteFuncwriteF)
97+
AllocateCompressor(constpg_compress_specificationcompression_spec,
98+
WriteFuncwriteF)
12599
{
126100
CompressorState*cs;
127-
CompressionAlgorithmalg;
128-
intlevel;
129-
130-
ParseCompressionOption(compression,&alg,&level);
131101

132102
#ifndefHAVE_LIBZ
133-
if (alg==COMPR_ALG_LIBZ)
103+
if (compression_spec.algorithm==PG_COMPRESSION_GZIP)
134104
pg_fatal("not built with zlib support");
135105
#endif
136106

137107
cs= (CompressorState*)pg_malloc0(sizeof(CompressorState));
138108
cs->writeF=writeF;
139-
cs->comprAlg=alg;
109+
cs->compression_spec=compression_spec;
140110

141111
/*
142112
* Perform compression algorithm specific initialization.
143113
*/
144114
#ifdefHAVE_LIBZ
145-
if (alg==COMPR_ALG_LIBZ)
146-
InitCompressorZlib(cs,level);
115+
if (cs->compression_spec.algorithm==PG_COMPRESSION_GZIP)
116+
InitCompressorZlib(cs,cs->compression_spec.level);
147117
#endif
148118

149119
returncs;
@@ -154,15 +124,12 @@ AllocateCompressor(int compression, WriteFunc writeF)
154124
* out with ahwrite().
155125
*/
156126
void
157-
ReadDataFromArchive(ArchiveHandle*AH,intcompression,ReadFuncreadF)
127+
ReadDataFromArchive(ArchiveHandle*AH,pg_compress_specificationcompression_spec,
128+
ReadFuncreadF)
158129
{
159-
CompressionAlgorithmalg;
160-
161-
ParseCompressionOption(compression,&alg,NULL);
162-
163-
if (alg==COMPR_ALG_NONE)
130+
if (compression_spec.algorithm==PG_COMPRESSION_NONE)
164131
ReadDataFromArchiveNone(AH,readF);
165-
if (alg==COMPR_ALG_LIBZ)
132+
if (compression_spec.algorithm==PG_COMPRESSION_GZIP)
166133
{
167134
#ifdefHAVE_LIBZ
168135
ReadDataFromArchiveZlib(AH,readF);
@@ -179,18 +146,23 @@ void
179146
WriteDataToArchive(ArchiveHandle*AH,CompressorState*cs,
180147
constvoid*data,size_tdLen)
181148
{
182-
switch (cs->comprAlg)
149+
switch (cs->compression_spec.algorithm)
183150
{
184-
caseCOMPR_ALG_LIBZ:
151+
casePG_COMPRESSION_GZIP:
185152
#ifdefHAVE_LIBZ
186153
WriteDataToArchiveZlib(AH,cs,data,dLen);
187154
#else
188155
pg_fatal("not built with zlib support");
189156
#endif
190157
break;
191-
caseCOMPR_ALG_NONE:
158+
casePG_COMPRESSION_NONE:
192159
WriteDataToArchiveNone(AH,cs,data,dLen);
193160
break;
161+
casePG_COMPRESSION_LZ4:
162+
/* fallthrough */
163+
casePG_COMPRESSION_ZSTD:
164+
pg_fatal("invalid compression method");
165+
break;
194166
}
195167
}
196168

@@ -201,7 +173,7 @@ void
201173
EndCompressor(ArchiveHandle*AH,CompressorState*cs)
202174
{
203175
#ifdefHAVE_LIBZ
204-
if (cs->comprAlg==COMPR_ALG_LIBZ)
176+
if (cs->compression_spec.algorithm==PG_COMPRESSION_GZIP)
205177
EndCompressorZlib(AH,cs);
206178
#endif
207179
free(cs);
@@ -453,20 +425,27 @@ cfopen_read(const char *path, const char *mode)
453425
{
454426
cfp*fp;
455427

428+
pg_compress_specificationcompression_spec= {0};
429+
456430
#ifdefHAVE_LIBZ
457431
if (hasSuffix(path,".gz"))
458-
fp=cfopen(path,mode,1);
432+
{
433+
compression_spec.algorithm=PG_COMPRESSION_GZIP;
434+
fp=cfopen(path,mode,compression_spec);
435+
}
459436
else
460437
#endif
461438
{
462-
fp=cfopen(path,mode,0);
439+
compression_spec.algorithm=PG_COMPRESSION_NONE;
440+
fp=cfopen(path,mode,compression_spec);
463441
#ifdefHAVE_LIBZ
464442
if (fp==NULL)
465443
{
466444
char*fname;
467445

468446
fname=psprintf("%s.gz",path);
469-
fp=cfopen(fname,mode,1);
447+
compression_spec.algorithm=PG_COMPRESSION_GZIP;
448+
fp=cfopen(fname,mode,compression_spec);
470449
free_keep_errno(fname);
471450
}
472451
#endif
@@ -479,26 +458,27 @@ cfopen_read(const char *path, const char *mode)
479458
* be a filemode as accepted by fopen() and gzopen() that indicates writing
480459
* ("w", "wb", "a", or "ab").
481460
*
482-
* If 'compression' isnon-zero, a gzip compressed stream is opened, and
483-
*'compression' indicates the compression levelused. The ".gz" suffix
484-
*is automatically added to'path' in that case.
461+
* If 'compression_spec.algorithm' isGZIP, a gzip compressed stream is opened,
462+
*and 'compression_spec.level'used. The ".gz" suffix is automatically added to
463+
* 'path' in that case.
485464
*
486465
* On failure, return NULL with an error code in errno.
487466
*/
488467
cfp*
489-
cfopen_write(constchar*path,constchar*mode,intcompression)
468+
cfopen_write(constchar*path,constchar*mode,
469+
constpg_compress_specificationcompression_spec)
490470
{
491471
cfp*fp;
492472

493-
if (compression==0)
494-
fp=cfopen(path,mode,0);
473+
if (compression_spec.algorithm==PG_COMPRESSION_NONE)
474+
fp=cfopen(path,mode,compression_spec);
495475
else
496476
{
497477
#ifdefHAVE_LIBZ
498478
char*fname;
499479

500480
fname=psprintf("%s.gz",path);
501-
fp=cfopen(fname,mode,compression);
481+
fp=cfopen(fname,mode,compression_spec);
502482
free_keep_errno(fname);
503483
#else
504484
pg_fatal("not built with zlib support");
@@ -509,26 +489,27 @@ cfopen_write(const char *path, const char *mode, int compression)
509489
}
510490

511491
/*
512-
* Opens file 'path' in 'mode'. If'compression' isnon-zero, the file
492+
* Opens file 'path' in 'mode'. If compression isGZIP, the file
513493
* is opened with libz gzopen(), otherwise with plain fopen().
514494
*
515495
* On failure, return NULL with an error code in errno.
516496
*/
517497
cfp*
518-
cfopen(constchar*path,constchar*mode,intcompression)
498+
cfopen(constchar*path,constchar*mode,
499+
constpg_compress_specificationcompression_spec)
519500
{
520501
cfp*fp=pg_malloc(sizeof(cfp));
521502

522-
if (compression!=0)
503+
if (compression_spec.algorithm==PG_COMPRESSION_GZIP)
523504
{
524505
#ifdefHAVE_LIBZ
525-
if (compression!=Z_DEFAULT_COMPRESSION)
506+
if (compression_spec.level!=Z_DEFAULT_COMPRESSION)
526507
{
527508
/* user has specified a compression level, so tell zlib to use it */
528509
charmode_compression[32];
529510

530511
snprintf(mode_compression,sizeof(mode_compression),"%s%d",
531-
mode,compression);
512+
mode,compression_spec.level);
532513
fp->compressedfp=gzopen(path,mode_compression);
533514
}
534515
else

‎src/bin/pg_dump/compress_io.h

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,12 +21,6 @@
2121
#defineZLIB_OUT_SIZE4096
2222
#defineZLIB_IN_SIZE4096
2323

24-
typedefenum
25-
{
26-
COMPR_ALG_NONE,
27-
COMPR_ALG_LIBZ
28-
}CompressionAlgorithm;
29-
3024
/* Prototype for callback function to WriteDataToArchive() */
3125
typedefvoid (*WriteFunc) (ArchiveHandle*AH,constchar*buf,size_tlen);
3226

@@ -46,8 +40,10 @@ typedef size_t (*ReadFunc) (ArchiveHandle *AH, char **buf, size_t *buflen);
4640
/* struct definition appears in compress_io.c */
4741
typedefstructCompressorStateCompressorState;
4842

49-
externCompressorState*AllocateCompressor(intcompression,WriteFuncwriteF);
50-
externvoidReadDataFromArchive(ArchiveHandle*AH,intcompression,
43+
externCompressorState*AllocateCompressor(constpg_compress_specificationcompression_spec,
44+
WriteFuncwriteF);
45+
externvoidReadDataFromArchive(ArchiveHandle*AH,
46+
constpg_compress_specificationcompression_spec,
5147
ReadFuncreadF);
5248
externvoidWriteDataToArchive(ArchiveHandle*AH,CompressorState*cs,
5349
constvoid*data,size_tdLen);
@@ -56,9 +52,13 @@ extern void EndCompressor(ArchiveHandle *AH, CompressorState *cs);
5652

5753
typedefstructcfpcfp;
5854

59-
externcfp*cfopen(constchar*path,constchar*mode,intcompression);
55+
externcfp*cfopen(constchar*path,constchar*mode,
56+
constpg_compress_specificationcompression_spec);
57+
externcfp*cfdopen(intfd,constchar*mode,
58+
pg_compress_specificationcompression_spec);
6059
externcfp*cfopen_read(constchar*path,constchar*mode);
61-
externcfp*cfopen_write(constchar*path,constchar*mode,intcompression);
60+
externcfp*cfopen_write(constchar*path,constchar*mode,
61+
constpg_compress_specificationcompression_spec);
6262
externintcfread(void*ptr,intsize,cfp*fp);
6363
externintcfwrite(constvoid*ptr,intsize,cfp*fp);
6464
externintcfgetc(cfp*fp);

‎src/bin/pg_dump/pg_backup.h

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@
2323
#ifndefPG_BACKUP_H
2424
#definePG_BACKUP_H
2525

26+
#include"common/compression.h"
2627
#include"fe_utils/simple_list.h"
2728
#include"libpq-fe.h"
2829

@@ -143,7 +144,8 @@ typedef struct _restoreOptions
143144

144145
intnoDataForFailedTables;
145146
intexit_on_error;
146-
intcompression;
147+
pg_compress_specificationcompression_spec;/* Specification for
148+
* compression */
147149
intsuppressDumpWarnings;/* Suppress output of WARNING entries
148150
* to stderr */
149151
boolsingle_txn;
@@ -303,7 +305,8 @@ extern Archive *OpenArchive(const char *FileSpec, const ArchiveFormat fmt);
303305

304306
/* Create a new archive */
305307
externArchive*CreateArchive(constchar*FileSpec,constArchiveFormatfmt,
306-
constintcompression,booldosync,ArchiveModemode,
308+
constpg_compress_specificationcompression_spec,
309+
booldosync,ArchiveModemode,
307310
SetupWorkerPtrTypesetupDumpWorker);
308311

309312
/* The --list option */

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp