Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit5f5e68b

Browse files
committed
Adopt the GNU convention for handling tar-archive members exceeding 8GB.
The POSIX standard for tar headers requires archive member sizes to beprinted in octal with at most 11 digits, limiting the representable filesize to 8GB. However, GNU tar and apparently most other modern tarssupport a convention in which oversized values can be stored in base-256,allowing any practical file to be a tar member. Adopt this conventionto remove two limitations:* pg_dump with -Ft output format failed if the contents of any one tableexceeded 8GB.* pg_basebackup failed if the data directory contained any file exceeding8GB. (This would be a fatal problem for installations configured with atable segment size of 8GB or more, and it has also been seen to fail whenlarge core dump files exist in the data directory.)File sizes under 8GB are still printed in octal, so that no compatibilityissues are created except in cases that would have failed entirely before.In addition, this patch fixes several bugs in the same area:* In 9.3 and later, we'd defined tarCreateHeader's file-size argument assize_t, which meant that on 32-bit machines it would write a corrupt tarheader for file sizes between 4GB and 8GB, even though no error was raised.This broke both "pg_dump -Ft" and pg_basebackup for such cases.* pg_restore from a tar archive would fail on tables of size between 4GBand 8GB, on machines where either "size_t" or "unsigned long" is 32 bits.This happened even with an archive file not affected by the previous bug.* pg_basebackup would fail if there were files of size between 4GB and 8GB,even on 64-bit machines.* In 9.3 and later, "pg_basebackup -Ft" failed entirely, for any file size,on 64-bit big-endian machines.In view of these potential data-loss bugs, back-patch to all supportedbranches, even though removal of the documented 8GB limit might otherwisebe considered a new feature rather than a bug fix.
1 parenta35c5b7 commit5f5e68b

File tree

6 files changed

+124
-115
lines changed

6 files changed

+124
-115
lines changed

‎doc/src/sgml/ref/pg_dump.sgml

Lines changed: 5 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -272,12 +272,12 @@ PostgreSQL documentation
272272
<listitem>
273273
<para>
274274
Output a <command>tar</command>-format archive suitable for input
275-
into <application>pg_restore</application>. The tar-format is
276-
compatible with the directory-format; extracting a tar-format
275+
into <application>pg_restore</application>. The tarformat is
276+
compatible with the directoryformat: extracting a tar-format
277277
archive produces a valid directory-format archive.
278-
However, the tar-format does not support compression and has a
279-
limit of 8 GB onthesizeofindividual tables. Also, the relative
280-
order of table data items cannot bechanged during restore.
278+
However, the tarformat does not support compression. Also, when
279+
using tar formattherelative orderoftable data items cannot be
280+
changed during restore.
281281
</para>
282282
</listitem>
283283
</varlistentry>
@@ -1121,15 +1121,6 @@ CREATE DATABASE foo WITH TEMPLATE template0;
11211121
catalogs might be left in the wrong state.
11221122
</para>
11231123

1124-
<para>
1125-
Members of tar archives are limited to a size less than 8 GB.
1126-
(This is an inherent limitation of the tar file format.) Therefore
1127-
this format cannot be used if the textual representation of any one table
1128-
exceeds that size. The total size of a tar archive and any of the
1129-
other output formats is not limited, except possibly by the
1130-
operating system.
1131-
</para>
1132-
11331124
<para>
11341125
The dump file produced by <application>pg_dump</application>
11351126
does not contain the statistics used by the optimizer to make

‎src/backend/replication/basebackup.c

Lines changed: 1 addition & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -698,7 +698,7 @@ SendBackupHeader(List *tablespaces)
698698
}
699699
else
700700
{
701-
Sizelen;
701+
Sizelen;
702702

703703
len=strlen(ti->oid);
704704
pq_sendint(&buf,len,4);
@@ -1131,13 +1131,6 @@ sendDir(char *path, int basepathlen, bool sizeonly, List *tablespaces,
11311131
*/
11321132

11331133

1134-
/*
1135-
* Maximum file size for a tar member: The limit inherent in the
1136-
* format is 2^33-1 bytes (nearly 8 GB). But we don't want to exceed
1137-
* what we can represent in pgoff_t.
1138-
*/
1139-
#defineMAX_TAR_MEMBER_FILELEN (((int64) 1 << Min(33, sizeof(pgoff_t)*8 - 1)) - 1)
1140-
11411134
/*
11421135
* Given the member, write the TAR header & send the file.
11431136
*
@@ -1166,15 +1159,6 @@ sendFile(char *readfilename, char *tarfilename, struct stat * statbuf,
11661159
errmsg("could not open file \"%s\": %m",readfilename)));
11671160
}
11681161

1169-
/*
1170-
* Some compilers will throw a warning knowing this test can never be true
1171-
* because pgoff_t can't exceed the compared maximum on their platform.
1172-
*/
1173-
if (statbuf->st_size>MAX_TAR_MEMBER_FILELEN)
1174-
ereport(ERROR,
1175-
(errmsg("archive member \"%s\" too large for tar format",
1176-
tarfilename)));
1177-
11781162
_tarWriteHeader(tarfilename,NULL,statbuf);
11791163

11801164
while ((cnt=fread(buf,1,Min(sizeof(buf),statbuf->st_size-len),fp))>0)

‎src/bin/pg_basebackup/pg_basebackup.c

Lines changed: 5 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -780,7 +780,7 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
780780
boolin_tarhdr= true;
781781
boolskip_file= false;
782782
size_ttarhdrsz=0;
783-
size_tfilesz=0;
783+
pgoff_tfilesz=0;
784784

785785
#ifdefHAVE_LIBZ
786786
gzFileztarfile=NULL;
@@ -1045,7 +1045,7 @@ ReceiveTarFile(PGconn *conn, PGresult *res, int rownum)
10451045

10461046
skip_file= (strcmp(&tarhdr[0],"recovery.conf")==0);
10471047

1048-
sscanf(&tarhdr[124],"%11o", (unsignedint*)&filesz);
1048+
filesz=read_tar_number(&tarhdr[124],12);
10491049

10501050
padding= ((filesz+511)& ~511)-filesz;
10511051
filesz+=padding;
@@ -1138,7 +1138,7 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
11381138
charcurrent_path[MAXPGPATH];
11391139
charfilename[MAXPGPATH];
11401140
constchar*mapped_tblspc_path;
1141-
intcurrent_len_left;
1141+
pgoff_tcurrent_len_left=0;
11421142
intcurrent_padding=0;
11431143
boolbasetablespace;
11441144
char*copybuf=NULL;
@@ -1207,20 +1207,10 @@ ReceiveAndUnpackTarFile(PGconn *conn, PGresult *res, int rownum)
12071207
}
12081208
totaldone+=512;
12091209

1210-
if (sscanf(copybuf+124,"%11o",&current_len_left)!=1)
1211-
{
1212-
fprintf(stderr,_("%s: could not parse file size\n"),
1213-
progname);
1214-
disconnect_and_exit(1);
1215-
}
1210+
current_len_left=read_tar_number(&copybuf[124],12);
12161211

12171212
/* Set permissions on the file */
1218-
if (sscanf(&copybuf[100],"%07o ",&filemode)!=1)
1219-
{
1220-
fprintf(stderr,_("%s: could not parse file mode\n"),
1221-
progname);
1222-
disconnect_and_exit(1);
1223-
}
1213+
filemode=read_tar_number(&copybuf[100],8);
12241214

12251215
/*
12261216
* All files are padded up to 512 bytes

‎src/bin/pg_dump/pg_backup_tar.c

Lines changed: 20 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -78,13 +78,6 @@ typedef struct
7878
ArchiveHandle*AH;
7979
}TAR_MEMBER;
8080

81-
/*
82-
* Maximum file size for a tar member: The limit inherent in the
83-
* format is 2^33-1 bytes (nearly 8 GB). But we don't want to exceed
84-
* what we can represent in pgoff_t.
85-
*/
86-
#defineMAX_TAR_MEMBER_FILELEN (((int64) 1 << Min(33, sizeof(pgoff_t)*8 - 1)) - 1)
87-
8881
typedefstruct
8982
{
9083
inthasSeek;
@@ -1049,7 +1042,7 @@ isValidTarHeader(char *header)
10491042
intsum;
10501043
intchk=tarChecksum(header);
10511044

1052-
sscanf(&header[148],"%8o",&sum);
1045+
sum=read_tar_number(&header[148],8);
10531046

10541047
if (sum!=chk)
10551048
return false;
@@ -1091,13 +1084,6 @@ _tarAddFile(ArchiveHandle *AH, TAR_MEMBER *th)
10911084
strerror(errno));
10921085
fseeko(tmp,0,SEEK_SET);
10931086

1094-
/*
1095-
* Some compilers will throw a warning knowing this test can never be true
1096-
* because pgoff_t can't exceed the compared maximum on their platform.
1097-
*/
1098-
if (th->fileLen>MAX_TAR_MEMBER_FILELEN)
1099-
exit_horribly(modulename,"archive member too large for tar format\n");
1100-
11011087
_tarWriteHeader(th);
11021088

11031089
while ((cnt=fread(buf,1,sizeof(buf),tmp))>0)
@@ -1222,11 +1208,10 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12221208
{
12231209
lclContext*ctx= (lclContext*)AH->formatData;
12241210
charh[512];
1225-
chartag[100];
1211+
chartag[100+1];
12261212
intsum,
12271213
chk;
1228-
size_tlen;
1229-
unsigned longullen;
1214+
pgoff_tlen;
12301215
pgoff_thPos;
12311216
boolgotBlock= false;
12321217

@@ -1249,7 +1234,7 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12491234

12501235
/* Calc checksum */
12511236
chk=tarChecksum(h);
1252-
sscanf(&h[148],"%8o",&sum);
1237+
sum=read_tar_number(&h[148],8);
12531238

12541239
/*
12551240
* If the checksum failed, see if it is a null block. If so, silently
@@ -1272,27 +1257,31 @@ _tarGetHeader(ArchiveHandle *AH, TAR_MEMBER *th)
12721257
}
12731258
}
12741259

1275-
sscanf(&h[0],"%99s",tag);
1276-
sscanf(&h[124],"%12lo",&ullen);
1277-
len= (size_t)ullen;
1260+
/* Name field is 100 bytes, might not be null-terminated */
1261+
strlcpy(tag,&h[0],100+1);
1262+
1263+
len=read_tar_number(&h[124],12);
12781264

12791265
{
1280-
charbuf[100];
1266+
charposbuf[32];
1267+
charlenbuf[32];
12811268

1282-
snprintf(buf,sizeof(buf),INT64_FORMAT, (int64)hPos);
1283-
ahlog(AH,3,"TOC Entry %s at %s (length %lu, checksum %d)\n",
1284-
tag,buf, (unsigned long)len,sum);
1269+
snprintf(posbuf,sizeof(posbuf),UINT64_FORMAT, (uint64)hPos);
1270+
snprintf(lenbuf,sizeof(lenbuf),UINT64_FORMAT, (uint64)len);
1271+
ahlog(AH,3,"TOC Entry %s at %s (length %s, checksum %d)\n",
1272+
tag,posbuf,lenbuf,sum);
12851273
}
12861274

12871275
if (chk!=sum)
12881276
{
1289-
charbuf[100];
1277+
charposbuf[32];
12901278

1291-
snprintf(buf,sizeof(buf),INT64_FORMAT, (int64)ftello(ctx->tarFH));
1279+
snprintf(posbuf,sizeof(posbuf),UINT64_FORMAT,
1280+
(uint64)ftello(ctx->tarFH));
12921281
exit_horribly(modulename,
12931282
"corrupt tar header found in %s "
12941283
"(expected %d, computed %d) file position %s\n",
1295-
tag,sum,chk,buf);
1284+
tag,sum,chk,posbuf);
12961285
}
12971286

12981287
th->targetFile=pg_strdup(tag);
@@ -1307,7 +1296,8 @@ _tarWriteHeader(TAR_MEMBER *th)
13071296
{
13081297
charh[512];
13091298

1310-
tarCreateHeader(h,th->targetFile,NULL,th->fileLen,0600,04000,02000,time(NULL));
1299+
tarCreateHeader(h,th->targetFile,NULL,th->fileLen,
1300+
0600,04000,02000,time(NULL));
13111301

13121302
/* Now write the completed header. */
13131303
if (fwrite(h,1,512,th->tarFH)!=512)

‎src/include/pgtar.h

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,5 +19,7 @@ enum tarError
1919
TAR_SYMLINK_TOO_LONG
2020
};
2121

22-
externenumtarErrortarCreateHeader(char*h,constchar*filename,constchar*linktarget,size_tsize,mode_tmode,uid_tuid,gid_tgid,time_tmtime);
22+
externenumtarErrortarCreateHeader(char*h,constchar*filename,constchar*linktarget,
23+
pgoff_tsize,mode_tmode,uid_tuid,gid_tgid,time_tmtime);
24+
externuint64read_tar_number(constchar*s,intlen);
2325
externinttarChecksum(char*header);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp