Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit7d5c83b

Browse files
pg_dump: Reduce memory usage of dumps with statistics.
Right now, pg_dump stores all generated commands for statistics inmemory. These commands can be quite large and therefore cansignificantly increase pg_dump's memory footprint. To fix, waituntil we are about to write out the commands before generatingthem, and be sure to free the commands after writing. This isimplemented via a new defnDumper callback that works much like thedataDumper one but is specifically designed for TOC entries.Custom dumps that include data might write the TOC twice (to updatedata offset information), which would ordinarily cause pg_dump torun the attribute statistics queries twice. However, as a hack, wesave the length of the written-out entry in the first pass and skipover it in the second. While there is no known technical issuewith executing the queries multiple times and rewriting theresults, it's expensive and feels risky, so let's avoid it.As an exception, we _do_ execute the queries twice for the tarformat. This format does a second pass through the TOC to generatethe restore.sql file. pg_restore doesn't use this file, so even ifthe second round of queries returns different results than thefirst, it won't corrupt the output; the archive and restore.sqlfile will just have different content. A follow-up commit willteach pg_dump to gather attribute statistics in batches, which ourtesting indicates more than makes up for the added expense ofrunning the queries twice.Author: Corey Huinker <corey.huinker@gmail.com>Co-authored-by: Nathan Bossart <nathandbossart@gmail.com>Reviewed-by: Jeff Davis <pgsql@j-davis.com>Discussion:https://postgr.es/m/CADkLM%3Dc%2Br05srPy9w%2B-%2BnbmLEo15dKXYQ03Q_xyK%2BriJerigLQ%40mail.gmail.com
1 parente3cc039 commit7d5c83b

File tree

4 files changed

+120
-16
lines changed

4 files changed

+120
-16
lines changed

‎src/bin/pg_dump/pg_backup.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -285,6 +285,7 @@ typedef int DumpId;
285285
* Function pointer prototypes for assorted callback methods.
286286
*/
287287

288+
typedefchar*(*DefnDumperPtr) (Archive*AH,constvoid*userArg);
288289
typedefint (*DataDumperPtr) (Archive*AH,constvoid*userArg);
289290

290291
typedefvoid (*SetupWorkerPtrType) (Archive*AH);

‎src/bin/pg_dump/pg_backup_archiver.c

Lines changed: 81 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1266,6 +1266,9 @@ ArchiveEntry(Archive *AHX, CatalogId catalogId, DumpId dumpId,
12661266
newToc->dataDumperArg=opts->dumpArg;
12671267
newToc->hadDumper=opts->dumpFn ? true : false;
12681268

1269+
newToc->defnDumper=opts->defnFn;
1270+
newToc->defnDumperArg=opts->defnArg;
1271+
12691272
newToc->formatData=NULL;
12701273
newToc->dataLength=0;
12711274

@@ -2621,7 +2624,45 @@ WriteToc(ArchiveHandle *AH)
26212624
WriteStr(AH,te->tag);
26222625
WriteStr(AH,te->desc);
26232626
WriteInt(AH,te->section);
2624-
WriteStr(AH,te->defn);
2627+
2628+
if (te->defnLen)
2629+
{
2630+
/*
2631+
* defnLen should only be set for custom format's second call to
2632+
* WriteToc(), which rewrites the TOC in place to update data
2633+
* offsets. Instead of calling the defnDumper a second time
2634+
* (which could involve re-executing queries), just skip writing
2635+
* the entry. While regenerating the definition should
2636+
* theoretically produce the same result as before, it's expensive
2637+
* and feels risky.
2638+
*
2639+
* The custom format only calls WriteToc() a second time if
2640+
* fseeko() is usable (see _CloseArchive() in pg_backup_custom.c),
2641+
* so we can safely use it without checking. For other formats,
2642+
* we fail because one of our assumptions must no longer hold
2643+
* true.
2644+
*
2645+
* XXX This is a layering violation, but the alternative is an
2646+
* awkward and complicated callback infrastructure for this
2647+
* special case. This might be worth revisiting in the future.
2648+
*/
2649+
if (AH->format!=archCustom)
2650+
pg_fatal("unexpected TOC entry in WriteToc(): %d %s %s",
2651+
te->dumpId,te->desc,te->tag);
2652+
2653+
if (fseeko(AH->FH,te->defnLen,SEEK_CUR!=0))
2654+
pg_fatal("error during file seek: %m");
2655+
}
2656+
elseif (te->defnDumper)
2657+
{
2658+
char*defn=te->defnDumper((Archive*)AH,te->defnDumperArg);
2659+
2660+
te->defnLen=WriteStr(AH,defn);
2661+
pg_free(defn);
2662+
}
2663+
else
2664+
WriteStr(AH,te->defn);
2665+
26252666
WriteStr(AH,te->dropStmt);
26262667
WriteStr(AH,te->copyStmt);
26272668
WriteStr(AH,te->namespace);
@@ -3849,7 +3890,7 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx)
38493890

38503891
/*
38513892
* Actually print the definition. Normally we can just print the defn
3852-
* string if any, but we havethree special cases:
3893+
* string if any, but we havefour special cases:
38533894
*
38543895
* 1. A crude hack for suppressing AUTHORIZATION clause that old pg_dump
38553896
* versions put into CREATE SCHEMA. Don't mutate the variant for schema
@@ -3862,6 +3903,11 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx)
38623903
* 3. ACL LARGE OBJECTS entries need special processing because they
38633904
* contain only one copy of the ACL GRANT/REVOKE commands, which we must
38643905
* apply to each large object listed in the associated BLOB METADATA.
3906+
*
3907+
* 4. Entries with a defnDumper need to call it to generate the
3908+
* definition. This is primarily intended to provide a way to save memory
3909+
* for objects that would otherwise need a lot of it (e.g., statistics
3910+
* data).
38653911
*/
38663912
if (ropt->noOwner&&
38673913
strcmp(te->desc,"SCHEMA")==0&&strncmp(te->defn,"--",2)!=0)
@@ -3877,6 +3923,39 @@ _printTocEntry(ArchiveHandle *AH, TocEntry *te, const char *pfx)
38773923
{
38783924
IssueACLPerBlob(AH,te);
38793925
}
3926+
elseif (te->defnLen&&AH->format!=archTar)
3927+
{
3928+
/*
3929+
* If defnLen is set, the defnDumper has already been called for this
3930+
* TOC entry. We don't normally expect a defnDumper to be called for
3931+
* a TOC entry a second time in _printTocEntry(), but there's an
3932+
* exception. The tar format first calls WriteToc(), which scans the
3933+
* entire TOC, and then it later calls RestoreArchive() to generate
3934+
* restore.sql, which scans the TOC again. There doesn't appear to be
3935+
* a good way to prevent a second defnDumper call in this case without
3936+
* storing the definition in memory, which defeats the purpose. This
3937+
* second defnDumper invocation should generate the same output as the
3938+
* first, but even if it doesn't, the worst-case scenario is that
3939+
* restore.sql might have different statistics data than the archive.
3940+
*
3941+
* In all other cases, encountering a TOC entry a second time in
3942+
* _printTocEntry() is unexpected, so we fail because one of our
3943+
* assumptions must no longer hold true.
3944+
*
3945+
* XXX This is a layering violation, but the alternative is an awkward
3946+
* and complicated callback infrastructure for this special case. This
3947+
* might be worth revisiting in the future.
3948+
*/
3949+
pg_fatal("unexpected TOC entry in _printTocEntry(): %d %s %s",
3950+
te->dumpId,te->desc,te->tag);
3951+
}
3952+
elseif (te->defnDumper)
3953+
{
3954+
char*defn=te->defnDumper((Archive*)AH,te->defnDumperArg);
3955+
3956+
te->defnLen=ahprintf(AH,"%s\n\n",defn);
3957+
pg_free(defn);
3958+
}
38803959
elseif (te->defn&&strlen(te->defn)>0)
38813960
{
38823961
ahprintf(AH,"%s\n\n",te->defn);

‎src/bin/pg_dump/pg_backup_archiver.h

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -368,6 +368,10 @@ struct _tocEntry
368368
constvoid*dataDumperArg;/* Arg for above routine */
369369
void*formatData;/* TOC Entry data specific to file format */
370370

371+
DefnDumperPtrdefnDumper;/* routine to dump definition statement */
372+
constvoid*defnDumperArg;/* arg for above routine */
373+
size_tdefnLen;/* length of dumped definition */
374+
371375
/* working state while dumping/restoring */
372376
pgoff_tdataLength;/* item's data size; 0 if none or unknown */
373377
intreqs;/* do we need schema and/or data of object
@@ -407,6 +411,8 @@ typedef struct _archiveOpts
407411
intnDeps;
408412
DataDumperPtrdumpFn;
409413
constvoid*dumpArg;
414+
DefnDumperPtrdefnFn;
415+
constvoid*defnArg;
410416
}ArchiveOpts;
411417
#defineARCHIVE_OPTS(...) &(ArchiveOpts){__VA_ARGS__}
412418
/* Called to add a TOC entry */

‎src/bin/pg_dump/pg_dump.c

Lines changed: 32 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -10554,17 +10554,21 @@ appendNamedArgument(PQExpBuffer out, Archive *fout, const char *argname,
1055410554
}
1055510555

1055610556
/*
10557-
*dumpRelationStats --
10557+
*dumpRelationStats_dumper --
1055810558
*
10559-
* Dump command to import stats into the relation on the new database.
10559+
* Generate command to import stats into the relation on the new database.
10560+
* This routine is called by the Archiver when it wants the statistics to be
10561+
* dumped.
1056010562
*/
10561-
staticvoid
10562-
dumpRelationStats(Archive *fout, constRelStatsInfo *rsinfo)
10563+
staticchar *
10564+
dumpRelationStats_dumper(Archive *fout, constvoid *userArg)
1056310565
{
10566+
const RelStatsInfo *rsinfo = (RelStatsInfo *) userArg;
1056410567
const DumpableObject *dobj = &rsinfo->dobj;
1056510568
PGresult *res;
1056610569
PQExpBuffer query;
10567-
PQExpBuffer out;
10570+
PQExpBufferData out_data;
10571+
PQExpBuffer out = &out_data;
1056810572
inti_attname;
1056910573
inti_inherited;
1057010574
inti_null_frac;
@@ -10581,10 +10585,6 @@ dumpRelationStats(Archive *fout, const RelStatsInfo *rsinfo)
1058110585
inti_range_empty_frac;
1058210586
inti_range_bounds_histogram;
1058310587

10584-
/* nothing to do if we are not dumping statistics */
10585-
if (!fout->dopt->dumpStatistics)
10586-
return;
10587-
1058810588
query = createPQExpBuffer();
1058910589
if (!fout->is_prepared[PREPQUERY_GETATTRIBUTESTATS])
1059010590
{
@@ -10620,7 +10620,7 @@ dumpRelationStats(Archive *fout, const RelStatsInfo *rsinfo)
1062010620
resetPQExpBuffer(query);
1062110621
}
1062210622

10623-
out = createPQExpBuffer();
10623+
initPQExpBuffer(out);
1062410624

1062510625
/* restore relation stats */
1062610626
appendPQExpBufferStr(out, "SELECT * FROM pg_catalog.pg_restore_relation_stats(\n");
@@ -10764,17 +10764,35 @@ dumpRelationStats(Archive *fout, const RelStatsInfo *rsinfo)
1076410764

1076510765
PQclear(res);
1076610766

10767+
destroyPQExpBuffer(query);
10768+
return out->data;
10769+
}
10770+
10771+
/*
10772+
* dumpRelationStats --
10773+
*
10774+
* Make an ArchiveEntry for the relation statistics. The Archiver will take
10775+
* care of gathering the statistics and generating the restore commands when
10776+
* they are needed.
10777+
*/
10778+
static void
10779+
dumpRelationStats(Archive *fout, const RelStatsInfo *rsinfo)
10780+
{
10781+
const DumpableObject *dobj = &rsinfo->dobj;
10782+
10783+
/* nothing to do if we are not dumping statistics */
10784+
if (!fout->dopt->dumpStatistics)
10785+
return;
10786+
1076710787
ArchiveEntry(fout, nilCatalogId, createDumpId(),
1076810788
ARCHIVE_OPTS(.tag = dobj->name,
1076910789
.namespace = dobj->namespace->dobj.name,
1077010790
.description = "STATISTICS DATA",
1077110791
.section = rsinfo->section,
10772-
.createStmt = out->data,
10792+
.defnFn = dumpRelationStats_dumper,
10793+
.defnArg = rsinfo,
1077310794
.deps = dobj->dependencies,
1077410795
.nDeps = dobj->nDeps));
10775-
10776-
destroyPQExpBuffer(out);
10777-
destroyPQExpBuffer(query);
1077810796
}
1077910797

1078010798
/*

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp