Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit00cdd83

Browse files
committed
Adopt the GNU convention for handling tar-archive members exceeding 8GB.
The POSIX standard for tar headers requires archive member sizes to beprinted in octal with at most 11 digits, limiting the representable filesize to 8GB. However, GNU tar and apparently most other modern tarssupport a convention in which oversized values can be stored in base-256,allowing any practical file to be a tar member. Adopt this conventionto remove two limitations:* pg_dump with -Ft output format failed if the contents of any one tableexceeded 8GB.* pg_basebackup failed if the data directory contained any file exceeding8GB. (This would be a fatal problem for installations configured with atable segment size of 8GB or more, and it has also been seen to fail whenlarge core dump files exist in the data directory.)File sizes under 8GB are still printed in octal, so that no compatibilityissues are created except in cases that would have failed entirely before.In addition, this patch fixes several bugs in the same area:* In 9.3 and later, we'd defined tarCreateHeader's file-size argument assize_t, which meant that on 32-bit machines it would write a corrupt tarheader for file sizes between 4GB and 8GB, even though no error was raised.This broke both "pg_dump -Ft" and pg_basebackup for such cases.* pg_restore from a tar archive would fail on tables of size between 4GBand 8GB, on machines where either "size_t" or "unsigned long" is 32 bits.This happened even with an archive file not affected by the previous bug.* pg_basebackup would fail if there were files of size between 4GB and 8GB,even on 64-bit machines.* In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size,on 64-bit big-endian machines.In view of these potential data-loss bugs, back-patch to all supportedbranches, even though removal of the documented 8GB limit might otherwisebe considered a new feature rather than a bug fix.
1 parent074c5cf commit00cdd83

File tree

6 files changed

+125
-116
lines changed

6 files changed

+125
-116
lines changed

‎doc/src/sgml/ref/pg_dump.sgml

Lines changed: 5 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -272,12 +272,12 @@ PostgreSQL documentation
272272
<listitem>
273273
<para>
274274
Output a <command>tar</command>-format archive suitable for input
275-
into <application>pg_restore</application>. The tar-format is
276-
compatible with the directory-format; extracting a tar-format
275+
into <application>pg_restore</application>. The tarformat is
276+
compatible with the directoryformat: extracting a tar-format
277277
archive produces a valid directory-format archive.
278-
However, the tar-format does not support compression and has a
279-
limit of 8 GB onthesizeofindividual tables. Also, the relative
280-
order of table data items cannot bechanged during restore.
278+
However, the tarformat does not support compression. Also, when
279+
using tar formattherelative orderoftable data items cannot be
280+
changed during restore.
281281
</para>
282282
</listitem>
283283
</varlistentry>
@@ -1140,15 +1140,6 @@ CREATE DATABASE foo WITH TEMPLATE template0;
11401140
catalogs might be left in the wrong state.
11411141
</para>
11421142

1143-
<para>
1144-
Members of tar archives are limited to a size less than 8 GB.
1145-
(This is an inherent limitation of the tar file format.) Therefore
1146-
this format cannot be used if the textual representation of any one table
1147-
exceeds that size. The total size of a tar archive and any of the
1148-
other output formats is not limited, except possibly by the
1149-
operating system.
1150-
</para>
1151-
11521143
<para>
11531144
The dump file produced by <application>pg_dump</application>
11541145
does not contain the statistics used by the optimizer to make

‎src/backend/replication/basebackup.c

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -698,7 +698,7 @@ SendBackupHeader(List *tablespaces)
698698
}
699699
else
700700
{
701-
Sizelen;
701+
Sizelen;
702702

703703
len=strlen(ti->oid);
704704
pq_sendint(&buf,len,4);
@@ -1131,13 +1131,6 @@ sendDir(char *path, int basepathlen, bool sizeonly, List *tablespaces,
11311131
*/
11321132

11331133

1134-
/*
1135-
* Maximum file size for a tar member: The limit inherent in the
1136-
* format is 2^33-1 bytes (nearly 8 GB). But we don't want to exceed
1137-
* what we can represent in pgoff_t.
1138-
*/
1139-
#defineMAX_TAR_MEMBER_FILELEN (((int64) 1 << Min(33, sizeof(pgoff_t)*8 - 1)) - 1)
1140-
11411134
/*
11421135
* Given the member, write the TAR header & send the file.
11431136
*
@@ -1166,15 +1159,6 @@ sendFile(char *readfilename, char *tarfilename, struct stat * statbuf,
11661159
errmsg("could not open file \"%s\": %m",readfilename)));
11671160
}
11681161

1169-
/*
1170-
* Some compilers will throw a warning knowing this test can never be true
1171-
* because pgoff_t can't exceed the compared maximum on their platform.
1172-
*/
1173-
if (statbuf->st_size>MAX_TAR_MEMBER_FILELEN)
1174-
ereport(ERROR,
1175-
(errmsg("archive member \"%s\" too large for tar format",
1176-
tarfilename)));
1177-
11781162
_tarWriteHeader(tarfilename,NULL,statbuf);
11791163

11801164
while ((cnt=fread(buf,1,Min(sizeof(buf),statbuf->st_size-len),fp))>0)

‎src/bin/pg_basebackup/pg_basebackup.c

Lines changed: 6 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -781,7 +781,7 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
781781
boolin_tarhdr= true;
782782
boolskip_file= false;
783783
size_ttarhdrsz=0;
784-
size_tfilesz=0;
784+
pgoff_tfilesz=0;
785785

786786
#ifdefHAVE_LIBZ
787787
gzFileztarfile=NULL;
@@ -1046,7 +1046,7 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
10461046

10471047
skip_file= (strcmp(&tarhdr[0],"recovery.conf")==0);
10481048

1049-
sscanf(&tarhdr[124],"%11o", (unsignedint*)&filesz);
1049+
filesz=read_tar_number(&tarhdr[124],12);
10501050

10511051
padding= ((filesz+511)& ~511)-filesz;
10521052
filesz+=padding;
@@ -1139,7 +1139,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
11391139
charcurrent_path[MAXPGPATH];
11401140
charfilename[MAXPGPATH];
11411141
constchar*mapped_tblspc_path;
1142-
intcurrent_len_left;
1142+
pgoff_tcurrent_len_left=0;
11431143
intcurrent_padding=0;
11441144
boolbasetablespace;
11451145
char*copybuf=NULL;
@@ -1208,20 +1208,10 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
12081208
}
12091209
totaldone+=512;
12101210

1211-
if (sscanf(copybuf+124,"%11o",&current_len_left)!=1)
1212-
{
1213-
fprintf(stderr,_("%s: could not parse file size\n"),
1214-
progname);
1215-
disconnect_and_exit(1);
1216-
}
1211+
current_len_left=read_tar_number(&copybuf[124],12);
12171212

12181213
/* Set permissions on the file */
1219-
if (sscanf(&copybuf[100],"%07o ",&filemode)!=1)
1220-
{
1221-
fprintf(stderr,_("%s: could not parse file mode\n"),
1222-
progname);
1223-
disconnect_and_exit(1);
1224-
}
1214+
filemode=read_tar_number(&copybuf[100],8);
12251215

12261216
/*
12271217
* All files are padded up to 512 bytes
@@ -2180,7 +2170,7 @@ main(int argc, char **argv)
21802170
if (replication_slot&& !streamwal)
21812171
{
21822172
fprintf(stderr,
2183-
_("%s: replication slots can only be used with WAL streaming\n"),
2173+
_("%s: replication slots can only be used with WAL streaming\n"),
21842174
progname);
21852175
fprintf(stderr,_("Try \"%s --help\" for more information.\n"),
21862176
progname);

‎src/bin/pg_dump/pg_backup_tar.c

Lines changed: 20 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -78,13 +78,6 @@ typedef struct
7878
ArchiveHandle*AH;
7979
}TAR_MEMBER;
8080

81-
/*
82-
* Maximum file size for a tar member: The limit inherent in the
83-
* format is 2^33-1 bytes (nearly 8 GB). But we don't want to exceed
84-
* what we can represent in pgoff_t.
85-
*/
86-
#defineMAX_TAR_MEMBER_FILELEN (((int64) 1 << Min(33, sizeof(pgoff_t)*8 - 1)) - 1)
87-
8881
typedefstruct
8982
{
9083
inthasSeek;
@@ -1049,7 +1042,7 @@ isValidTarHeader(char *header)
10491042
intsum;
10501043
intchk=tarChecksum(header);
10511044

1052-
sscanf(&header[148],"%8o",&sum);
1045+
sum=read_tar_number(&header[148],8);
10531046

10541047
if (sum!=chk)
10551048
return false;
@@ -1091,13 +1084,6 @@ _tarAddFile(ArchiveHandle *AH, TAR_MEMBER *th)
10911084
strerror(errno));
10921085
fseeko(tmp,0,SEEK_SET);
10931086

1094-
/*
1095-
* Some compilers will throw a warning knowing this test can never be true
1096-
* because pgoff_t can't exceed the compared maximum on their platform.
1097-
*/
1098-
if (th->fileLen>MAX_TAR_MEMBER_FILELEN)
1099-
exit_horribly(modulename,"archive member too large for tar format\n");
1100-
11011087
_tarWriteHeader(th);
11021088

11031089
while ((cnt=fread(buf,1,sizeof(buf),tmp))>0)
@@ -1222,11 +1208,10 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12221208
{
12231209
lclContext*ctx= (lclContext*)AH->formatData;
12241210
charh[512];
1225-
chartag[100];
1211+
chartag[100+1];
12261212
intsum,
12271213
chk;
1228-
size_tlen;
1229-
unsigned longullen;
1214+
pgoff_tlen;
12301215
pgoff_thPos;
12311216
boolgotBlock= false;
12321217

@@ -1249,7 +1234,7 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12491234

12501235
/* Calc checksum */
12511236
chk=tarChecksum(h);
1252-
sscanf(&h[148],"%8o",&sum);
1237+
sum=read_tar_number(&h[148],8);
12531238

12541239
/*
12551240
* If the checksum failed, see if it is a null block. If so, silently
@@ -1272,27 +1257,31 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12721257
}
12731258
}
12741259

1275-
sscanf(&h[0],"%99s",tag);
1276-
sscanf(&h[124],"%12lo",&ullen);
1277-
len= (size_t)ullen;
1260+
/* Name field is 100 bytes, might not be null-terminated */
1261+
strlcpy(tag,&h[0],100+1);
1262+
1263+
len=read_tar_number(&h[124],12);
12781264

12791265
{
1280-
charbuf[100];
1266+
charposbuf[32];
1267+
charlenbuf[32];
12811268

1282-
snprintf(buf,sizeof(buf),INT64_FORMAT, (int64)hPos);
1283-
ahlog(AH,3,"TOC Entry %s at %s (length %lu, checksum %d)\n",
1284-
tag,buf, (unsigned long)len,sum);
1269+
snprintf(posbuf,sizeof(posbuf),UINT64_FORMAT, (uint64)hPos);
1270+
snprintf(lenbuf,sizeof(lenbuf),UINT64_FORMAT, (uint64)len);
1271+
ahlog(AH,3,"TOC Entry %s at %s (length %s, checksum %d)\n",
1272+
tag,posbuf,lenbuf,sum);
12851273
}
12861274

12871275
if (chk!=sum)
12881276
{
1289-
charbuf[100];
1277+
charposbuf[32];
12901278

1291-
snprintf(buf,sizeof(buf),INT64_FORMAT, (int64)ftello(ctx->tarFH));
1279+
snprintf(posbuf,sizeof(posbuf),UINT64_FORMAT,
1280+
(uint64)ftello(ctx->tarFH));
12921281
exit_horribly(modulename,
12931282
"corrupt tar header found in %s "
12941283
"(expected %d, computed %d) file position %s\n",
1295-
tag,sum,chk,buf);
1284+
tag,sum,chk,posbuf);
12961285
}
12971286

12981287
th->targetFile=pg_strdup(tag);
@@ -1307,7 +1296,8 @@ _tarWriteHeader(TAR_MEMBER *th)
13071296
{
13081297
charh[512];
13091298

1310-
tarCreateHeader(h,th->targetFile,NULL,th->fileLen,0600,04000,02000,time(NULL));
1299+
tarCreateHeader(h,th->targetFile,NULL,th->fileLen,
1300+
0600,04000,02000,time(NULL));
13111301

13121302
/* Now write the completed header. */
13131303
if (fwrite(h,1,512,th->tarFH)!=512)

‎src/include/pgtar.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,7 @@ enum tarError
1919
TAR_SYMLINK_TOO_LONG
2020
};
2121

22-
externenumtarErrortarCreateHeader(char*h,constchar*filename,constchar*linktarget,size_tsize,mode_tmode,uid_tuid,gid_tgid,time_tmtime);
22+
externenumtarErrortarCreateHeader(char*h,constchar*filename,constchar*linktarget,
23+
pgoff_tsize,mode_tmode,uid_tuid,gid_tgid,time_tmtime);
24+
externuint64read_tar_number(constchar*s,intlen);
2325
externinttarChecksum(char*header);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp