Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitf009591

Browse files
committed
Cope with data-offset-less archive files during out-of-order restores.
pg_dump produces custom-format archive files that lack data offsetswhen it is unable to seek its output. Up to now that's been a hazardfor pg_restore. But if pg_restore is able to seek in the archivefile, there is no reason to throw up our hands when asked to restoredata blocks out of order. Instead, whenever we are searching for adata block, record the locations of the blocks we passed over (thatis, fill in the missing data-offset fields in our in-memory copy ofthe TOC data). Then, when we hit a case that requires goingbackwards, we can just seek back.Also track the furthest point that we've searched to, and seek backto there when beginning a search for a new data block. This avoidspossible O(N^2) time consumption, by ensuring that each data blockis examined at most twice. (On Unix systems, that's at most twiceper parallel-restore job; but since Windows uses threads here, thethreads can share block location knowledge, reducing the amount ofduplicated work.)We can also improve the code a bit by using fseeko() to skip overdata blocks during the search.This is all of some use even in simple restores, but it's reallysignificant for parallel pg_restore. In that case, we requireseekability of the input already, and we will very probably needto do out-of-order restores.Back-patch to v12, as this fixes a regression introduced by commit548e509. Before that, parallel restore avoided requestingout-of-order restores, so it would work on a data-offset-lessarchive. Now it will again.Ideally this patch would include some test coverage, but there areother open bugs that need to be fixed before we can extend ourcoverage of parallel restore very much. Plan to revisit that later.David Gilman and Tom Lane; reviewed by Justin PryzbyDiscussion:https://postgr.es/m/CALBH9DDuJ+scZc4MEvw5uO-=vRyR2=QF9+Yh=3hPEnKHWfS81A@mail.gmail.com
1 parenta8d0732 commitf009591

File tree

2 files changed

+117
-34
lines changed

2 files changed

+117
-34
lines changed

‎doc/src/sgml/ref/pg_restore.sgml

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -246,12 +246,14 @@ PostgreSQL documentation
246246
<term><option>--jobs=<replaceable class="parameter">number-of-jobs</replaceable></option></term>
247247
<listitem>
248248
<para>
249-
Run the most time-consuming parts
250-
of <application>pg_restore</application> &mdash; those which load data,
251-
create indexes, or create constraints &mdash; using multiple
252-
concurrent jobs. This option can dramatically reduce the time
249+
Run the most time-consuming steps
250+
of <application>pg_restore</application> &mdash; those that load data,
251+
create indexes, or create constraints &mdash; concurrently, using up
252+
to <replaceable class="parameter">number-of-jobs</replaceable>
253+
concurrent sessions. This option can dramatically reduce the time
253254
to restore a large database to a server running on a
254-
multiprocessor machine.
255+
multiprocessor machine. This option is ignored when emitting a script
256+
rather than connecting directly to a database server.
255257
</para>
256258

257259
<para>
@@ -274,8 +276,7 @@ PostgreSQL documentation
274276
Only the custom and directory archive formats are supported
275277
with this option.
276278
The input must be a regular file or directory (not, for example, a
277-
pipe). This option is ignored when emitting a script rather
278-
than connecting directly to a database server. Also, multiple
279+
pipe or standard input). Also, multiple
279280
jobs cannot be used together with the
280281
option <option>--single-transaction</option>.
281282
</para>

‎src/bin/pg_dump/pg_backup_custom.c

Lines changed: 109 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,8 @@ typedef struct
7070
{
7171
CompressorState*cs;
7272
inthasSeek;
73+
/* lastFilePos is used only when reading, and may be invalid if !hasSeek */
74+
pgoff_tlastFilePos;/* position after last data block we've read */
7375
}lclContext;
7476

7577
typedefstruct
@@ -181,8 +183,13 @@ InitArchiveFmt_Custom(ArchiveHandle *AH)
181183

182184
ReadHead(AH);
183185
ReadToc(AH);
184-
}
185186

187+
/*
188+
* Remember location of first data block (i.e., the point after TOC)
189+
* in case we have to search for desired data blocks.
190+
*/
191+
ctx->lastFilePos=_getFilePos(AH,ctx);
192+
}
186193
}
187194

188195
/*
@@ -418,13 +425,62 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
418425
{
419426
/*
420427
* We cannot seek directly to the desired block. Instead, skip over
421-
* block headers until we find the one we want. This could fail if we
422-
* are asked to restore items out-of-order.
428+
* block headers until we find the one we want. Remember the
429+
* positions of skipped-over blocks, so that if we later decide we
430+
* need to read one, we'll be able to seek to it.
431+
*
432+
* When our input file is seekable, we can do the search starting from
433+
* the point after the last data block we scanned in previous
434+
* iterations of this function.
423435
*/
424-
_readBlockHeader(AH,&blkType,&id);
436+
if (ctx->hasSeek)
437+
{
438+
if (fseeko(AH->FH,ctx->lastFilePos,SEEK_SET)!=0)
439+
fatal("error during file seek: %m");
440+
}
425441

426-
while (blkType!=EOF&&id!=te->dumpId)
442+
for (;;)
427443
{
444+
pgoff_tthisBlkPos=_getFilePos(AH,ctx);
445+
446+
_readBlockHeader(AH,&blkType,&id);
447+
448+
if (blkType==EOF||id==te->dumpId)
449+
break;
450+
451+
/* Remember the block position, if we got one */
452+
if (thisBlkPos >=0)
453+
{
454+
TocEntry*otherte=getTocEntryByDumpId(AH,id);
455+
456+
if (otherte&&otherte->formatData)
457+
{
458+
lclTocEntry*othertctx= (lclTocEntry*)otherte->formatData;
459+
460+
/*
461+
* Note: on Windows, multiple threads might access/update
462+
* the same lclTocEntry concurrently, but that should be
463+
* safe as long as we update dataPos before dataState.
464+
* Ideally, we'd use pg_write_barrier() to enforce that,
465+
* but the needed infrastructure doesn't exist in frontend
466+
* code. But Windows only runs on machines with strong
467+
* store ordering, so it should be okay for now.
468+
*/
469+
if (othertctx->dataState==K_OFFSET_POS_NOT_SET)
470+
{
471+
othertctx->dataPos=thisBlkPos;
472+
othertctx->dataState=K_OFFSET_POS_SET;
473+
}
474+
elseif (othertctx->dataPos!=thisBlkPos||
475+
othertctx->dataState!=K_OFFSET_POS_SET)
476+
{
477+
/* sanity check */
478+
pg_log_warning("data block %d has wrong seek position",
479+
id);
480+
}
481+
}
482+
}
483+
428484
switch (blkType)
429485
{
430486
caseBLK_DATA:
@@ -440,7 +496,6 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
440496
blkType);
441497
break;
442498
}
443-
_readBlockHeader(AH,&blkType,&id);
444499
}
445500
}
446501
else
@@ -452,20 +507,18 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
452507
_readBlockHeader(AH,&blkType,&id);
453508
}
454509

455-
/* Produce suitable failure message if we fell off end of file */
510+
/*
511+
* If we reached EOF without finding the block we want, then either it
512+
* doesn't exist, or it does but we lack the ability to seek back to it.
513+
*/
456514
if (blkType==EOF)
457515
{
458-
if (tctx->dataState==K_OFFSET_POS_NOT_SET)
459-
fatal("could not find block ID %d in archive -- "
460-
"possibly due to out-of-order restore request, "
461-
"which cannot be handled due to lack of data offsets in archive",
462-
te->dumpId);
463-
elseif (!ctx->hasSeek)
516+
if (!ctx->hasSeek)
464517
fatal("could not find block ID %d in archive -- "
465518
"possibly due to out-of-order restore request, "
466519
"which cannot be handled due to non-seekable input file",
467520
te->dumpId);
468-
else/* huh, the dataPos led us to EOF? */
521+
else
469522
fatal("could not find block ID %d in archive -- "
470523
"possibly corrupt archive",
471524
te->dumpId);
@@ -491,6 +544,20 @@ _PrintTocData(ArchiveHandle *AH, TocEntry *te)
491544
blkType);
492545
break;
493546
}
547+
548+
/*
549+
* If our input file is seekable but lacks data offsets, update our
550+
* knowledge of where to start future searches from. (Note that we did
551+
* not update the current TE's dataState/dataPos. We could have, but
552+
* there is no point since it will not be visited again.)
553+
*/
554+
if (ctx->hasSeek&&tctx->dataState==K_OFFSET_POS_NOT_SET)
555+
{
556+
pgoff_tcurPos=_getFilePos(AH,ctx);
557+
558+
if (curPos>ctx->lastFilePos)
559+
ctx->lastFilePos=curPos;
560+
}
494561
}
495562

496563
/*
@@ -548,6 +615,7 @@ _skipBlobs(ArchiveHandle *AH)
548615
staticvoid
549616
_skipData(ArchiveHandle*AH)
550617
{
618+
lclContext*ctx= (lclContext*)AH->formatData;
551619
size_tblkLen;
552620
char*buf=NULL;
553621
intbuflen=0;
@@ -556,19 +624,27 @@ _skipData(ArchiveHandle *AH)
556624
blkLen=ReadInt(AH);
557625
while (blkLen!=0)
558626
{
559-
if (blkLen>buflen)
627+
if (ctx->hasSeek)
560628
{
561-
if (buf)
562-
free(buf);
563-
buf= (char*)pg_malloc(blkLen);
564-
buflen=blkLen;
629+
if (fseeko(AH->FH,blkLen,SEEK_CUR)!=0)
630+
fatal("error during file seek: %m");
565631
}
566-
if ((cnt=fread(buf,1,blkLen,AH->FH))!=blkLen)
632+
else
567633
{
568-
if (feof(AH->FH))
569-
fatal("could not read from input file: end of file");
570-
else
571-
fatal("could not read from input file: %m");
634+
if (blkLen>buflen)
635+
{
636+
if (buf)
637+
free(buf);
638+
buf= (char*)pg_malloc(blkLen);
639+
buflen=blkLen;
640+
}
641+
if ((cnt=fread(buf,1,blkLen,AH->FH))!=blkLen)
642+
{
643+
if (feof(AH->FH))
644+
fatal("could not read from input file: end of file");
645+
else
646+
fatal("could not read from input file: %m");
647+
}
572648
}
573649

574650
blkLen=ReadInt(AH);
@@ -804,6 +880,9 @@ _Clone(ArchiveHandle *AH)
804880
{
805881
lclContext*ctx= (lclContext*)AH->formatData;
806882

883+
/*
884+
* Each thread must have private lclContext working state.
885+
*/
807886
AH->formatData= (lclContext*)pg_malloc(sizeof(lclContext));
808887
memcpy(AH->formatData,ctx,sizeof(lclContext));
809888
ctx= (lclContext*)AH->formatData;
@@ -813,10 +892,13 @@ _Clone(ArchiveHandle *AH)
813892
fatal("compressor active");
814893

815894
/*
895+
* We intentionally do not clone TOC-entry-local state: it's useful to
896+
* share knowledge about where the data blocks are across threads.
897+
* _PrintTocData has to be careful about the order of operations on that
898+
* state, though.
899+
*
816900
* Note: we do not make a local lo_buf because we expect at most one BLOBS
817-
* entry per archive, so no parallelism is possible. Likewise,
818-
* TOC-entry-local state isn't an issue because any one TOC entry is
819-
* touched by just one worker child.
901+
* entry per archive, so no parallelism is possible.
820902
*/
821903
}
822904

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp