Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitb69bf30

Browse files
committed
Protect against multixact members wraparound
Multixact member files are subject to early wraparound overflow andremoval: if the average multixact size is above a certain threshold (seenote below) the protections against offset overflow are not enough:during multixact truncation at checkpoint time, somepg_multixact/members files would be removed because the server considersthem to be old and not needed anymore. This leads to loss of files thatare critical to interpret existing tuples's Xmax values.To protect against this, since we don't have enough info in pg_controland we can't modify it in old branches, we maintain shared memory stateabout the oldest value that we need to keep; we use this during newmultixact creation to abort if an old still-needed file would getoverwritten. This value is kept up to date by checkpoints, which makesit not completely accurate but should be good enough. We start emittingwarnings sometime earlier, so that the eventual multixact-shutdowndoesn't take DBAs completely by surprise (more precisely: once 20members SLRU segments are remaining before shutdown.)On troublesome average multixact size: The threshold size depends on themultixact freeze parameters. The oldest age is related to the greater ofmultixact_freeze_table_age and multixact_freeze_min_age: anythingolder than that should be removed promptly by autovacuum. If autovacuumis keeping up with multixact freezing, the troublesome multixact averagesize is(2^32-1) / Max(freeze table age, freeze min age)or around 28 members per multixact. Having an average multixact sizelarger than that will eventually cause new multixact data to overwritethe data area for older multixacts. (If autovacuum is not able to keepup, or there are errors in vacuuming, the actual maximum ismultixact_freeeze_max_age instead, at which point multixact generationis stopped completely. The default value for this limit is 400 million,which means that the multixact size that would cause trouble is about 10members).Initial bug report by Timothy Garnett, bug #12990Backpatch to 9.3, where the problem was introduced.Authors: Álvaro Herrera, Thomas MunroReviews: Thomas Munro, Amit Kapila, Robert Haas, Kevin Grittner
1 parentdfbaed4 commitb69bf30

File tree

2 files changed

+187
-25
lines changed

2 files changed

+187
-25
lines changed

‎src/backend/access/transam/multixact.c

Lines changed: 180 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,9 @@ typedef struct MultiXactStateData
213213
MultiXactIdmultiStopLimit;
214214
MultiXactIdmultiWrapLimit;
215215

216+
/* support for members anti-wraparound measures */
217+
MultiXactOffsetoffsetStopLimit;
218+
216219
/*
217220
* Per-backend data starts here. We have two arrays stored in the area
218221
* immediately following the MultiXactStateData struct. Each is indexed by
@@ -341,6 +344,10 @@ static bool MultiXactOffsetPrecedes(MultiXactOffset offset1,
341344
MultiXactOffsetoffset2);
342345
staticvoidExtendMultiXactOffset(MultiXactIdmulti);
343346
staticvoidExtendMultiXactMember(MultiXactOffsetoffset,intnmembers);
347+
staticvoidDetermineSafeOldestOffset(MultiXactIdoldestMXact);
348+
staticboolMultiXactOffsetWouldWrap(MultiXactOffsetboundary,
349+
MultiXactOffsetstart,uint32distance);
350+
staticMultiXactOffsetread_offset_for_multi(MultiXactIdmulti);
344351
staticvoidWriteMZeroPageXlogRec(intpageno,uint8info);
345352

346353

@@ -967,7 +974,7 @@ GetNewMultiXactId(int nmembers, MultiXactOffset *offset)
967974

968975
/*
969976
* To avoid swamping the postmaster with signals, we issue the autovac
970-
* request only once per 64Ktransaction starts. This still gives
977+
* request only once per 64Kmultis generated. This still gives
971978
* plenty of chances before we get into real trouble.
972979
*/
973980
if (IsUnderPostmaster&& (result %65536)==0)
@@ -1043,6 +1050,47 @@ GetNewMultiXactId(int nmembers, MultiXactOffset *offset)
10431050
else
10441051
*offset=nextOffset;
10451052

1053+
/*----------
1054+
* Protect against overrun of the members space as well, with the
1055+
* following rules:
1056+
*
1057+
* If we're past offsetStopLimit, refuse to generate more multis.
1058+
* If we're close to offsetStopLimit, emit a warning.
1059+
*
1060+
* Arbitrarily, we start emitting warnings when we're 20 segments or less
1061+
* from offsetStopLimit.
1062+
*
1063+
* Note we haven't updated the shared state yet, so if we fail at this
1064+
* point, the multixact ID we grabbed can still be used by the next guy.
1065+
*
1066+
* Note that there is no point in forcing autovacuum runs here: the
1067+
* multixact freeze settings would have to be reduced for that to have any
1068+
* effect.
1069+
*----------
1070+
*/
1071+
#defineOFFSET_WARN_SEGMENTS20
1072+
if (MultiXactOffsetWouldWrap(MultiXactState->offsetStopLimit,nextOffset,
1073+
nmembers))
1074+
ereport(ERROR,
1075+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1076+
errmsg("multixact \"members\" limit exceeded"),
1077+
errdetail_plural("This command would create a multixact with %u members, which exceeds remaining space (%u member.)",
1078+
"This command would create a multixact with %u members, which exceeds remaining space (%u members.)",
1079+
MultiXactState->offsetStopLimit-nextOffset-1,
1080+
nmembers,
1081+
MultiXactState->offsetStopLimit-nextOffset-1),
1082+
errhint("Execute a database-wide VACUUM in database with OID %u, with reduced vacuum_multixact_freeze_min_age and vacuum_multixact_freeze_table_age settings.",
1083+
MultiXactState->oldestMultiXactDB)));
1084+
elseif (MultiXactOffsetWouldWrap(MultiXactState->offsetStopLimit,
1085+
nextOffset,
1086+
nmembers+MULTIXACT_MEMBERS_PER_PAGE*SLRU_PAGES_PER_SEGMENT*OFFSET_WARN_SEGMENTS))
1087+
ereport(WARNING,
1088+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
1089+
errmsg("database with OID %u must be vacuumed before %d more multixact members are used",
1090+
MultiXactState->oldestMultiXactDB,
1091+
MultiXactState->offsetStopLimit-nextOffset+nmembers),
1092+
errhint("Execute a database-wide VACUUM in that database, with reduced vacuum_multixact_freeze_min_age and vacuum_multixact_freeze_table_age settings.")));
1093+
10461094
ExtendMultiXactMember(nextOffset,nmembers);
10471095

10481096
/*
@@ -1899,6 +1947,12 @@ StartupMultiXact(void)
18991947
*/
19001948
pageno=MXOffsetToMemberPage(offset);
19011949
MultiXactMemberCtl->shared->latest_page_number=pageno;
1950+
1951+
/*
1952+
* compute the oldest member we need to keep around to avoid old member
1953+
* data overrun.
1954+
*/
1955+
DetermineSafeOldestOffset(MultiXactState->oldestMultiXactId);
19021956
}
19031957

19041958
/*
@@ -1992,6 +2046,8 @@ TrimMultiXact(void)
19922046
}
19932047

19942048
LWLockRelease(MultiXactMemberControlLock);
2049+
2050+
DetermineSafeOldestOffset(MultiXactState->oldestMultiXactId);
19952051
}
19962052

19972053
/*
@@ -2099,7 +2155,7 @@ SetMultiXactIdLimit(MultiXactId oldest_datminmxid, Oid oldest_datoid)
20992155
*
21002156
* Note: This differs from the magic number used in
21012157
* SetTransactionIdLimit() since vacuum itself will never generate new
2102-
* multis.
2158+
* multis. XXX actually it does, if it needs to freeze old multis.
21032159
*/
21042160
multiStopLimit=multiWrapLimit-100;
21052161
if (multiStopLimit<FirstMultiXactId)
@@ -2142,6 +2198,8 @@ SetMultiXactIdLimit(MultiXactId oldest_datminmxid, Oid oldest_datoid)
21422198
curMulti=MultiXactState->nextMXact;
21432199
LWLockRelease(MultiXactGenLock);
21442200

2201+
DetermineSafeOldestOffset(oldest_datminmxid);
2202+
21452203
/* Log the info */
21462204
ereport(DEBUG1,
21472205
(errmsg("MultiXactId wrap limit is %u, limited by database with OID %u",
@@ -2228,13 +2286,16 @@ MultiXactAdvanceNextMXact(MultiXactId minMulti,
22282286

22292287
/*
22302288
* Update our oldestMultiXactId value, but only if it's more recent than
2231-
* what we had.
2289+
* what we had. However, even if not, always update the oldest multixact
2290+
* offset limit.
22322291
*/
22332292
void
22342293
MultiXactAdvanceOldest(MultiXactIdoldestMulti,OidoldestMultiDB)
22352294
{
22362295
if (MultiXactIdPrecedes(MultiXactState->oldestMultiXactId,oldestMulti))
22372296
SetMultiXactIdLimit(oldestMulti,oldestMultiDB);
2297+
else
2298+
DetermineSafeOldestOffset(oldestMulti);
22382299
}
22392300

22402301
/*
@@ -2401,6 +2462,121 @@ GetOldestMultiXactId(void)
24012462
returnoldestMXact;
24022463
}
24032464

2465+
/*
2466+
* Based on the given oldest MultiXactId, determine what's the oldest member
2467+
* offset and install the limit info in MultiXactState, where it can be used to
2468+
* prevent overrun of old data in the members SLRU area.
2469+
*/
2470+
staticvoid
2471+
DetermineSafeOldestOffset(MultiXactIdoldestMXact)
2472+
{
2473+
MultiXactOffsetoldestOffset;
2474+
2475+
/*
2476+
* Can't do this while initdb'ing or in the startup process while
2477+
* replaying WAL: the segment file to read might have not yet been
2478+
* created, or already been removed.
2479+
*/
2480+
if (IsBootstrapProcessingMode()||InRecovery)
2481+
return;
2482+
2483+
/*
2484+
* We determine the safe upper bound for offsets of new xacts by reading
2485+
* the offset of the oldest multixact, and going back one segment. This
2486+
* way, the sequence of multixact member segments will always have a
2487+
* one-segment hole at a minimum. We start spewing warnings a few
2488+
* complete segments before that.
2489+
*/
2490+
oldestOffset=read_offset_for_multi(oldestMXact);
2491+
/* move back to start of the corresponding segment */
2492+
oldestOffset-=oldestOffset /MULTIXACT_MEMBERS_PER_PAGE*SLRU_PAGES_PER_SEGMENT;
2493+
2494+
LWLockAcquire(MultiXactGenLock,LW_EXCLUSIVE);
2495+
/* always leave one segment before the wraparound point */
2496+
MultiXactState->offsetStopLimit=oldestOffset-
2497+
(MULTIXACT_MEMBERS_PER_PAGE*SLRU_PAGES_PER_SEGMENT);
2498+
LWLockRelease(MultiXactGenLock);
2499+
}
2500+
2501+
/*
2502+
* Return whether adding "distance" to "start" would move past "boundary".
2503+
*
2504+
* We use this to determine whether the addition is "wrapping around" the
2505+
* boundary point, hence the name. The reason we don't want to use the regular
2506+
* 2^31-modulo arithmetic here is that we want to be able to use the whole of
2507+
* the 2^32-1 space here, allowing for more multixacts that would fit
2508+
* otherwise. See also SlruScanDirCbRemoveMembers.
2509+
*/
2510+
staticbool
2511+
MultiXactOffsetWouldWrap(MultiXactOffsetboundary,MultiXactOffsetstart,
2512+
uint32distance)
2513+
{
2514+
MultiXactOffsetfinish;
2515+
2516+
Assert(distance >=0);
2517+
2518+
/*
2519+
* Note that offset number 0 is not used (see GetMultiXactIdMembers), so
2520+
* if the addition wraps around the UINT_MAX boundary, skip that value.
2521+
*/
2522+
finish=start+distance;
2523+
if (finish<start)
2524+
finish++;
2525+
2526+
/*-----------------------------------------------------------------------
2527+
* When the boundary is numerically greater than the starting point, any
2528+
* value numerically between the two is not wrapped:
2529+
*
2530+
*<----S----B---->
2531+
*[---) = F wrapped past B (and UINT_MAX)
2532+
* [---) = F not wrapped
2533+
* [----] = F wrapped past B
2534+
*
2535+
* When the boundary is numerically less than the starting point (i.e. the
2536+
* UINT_MAX wraparound occurs somewhere in between) then all values in
2537+
* between are wrapped:
2538+
*
2539+
*<----B----S---->
2540+
*[---) = F not wrapped past B (but wrapped past UINT_MAX)
2541+
* [---) = F wrapped past B (and UINT_MAX)
2542+
* [----] = F not wrapped
2543+
*-----------------------------------------------------------------------
2544+
*/
2545+
if (start<boundary)
2546+
{
2547+
returnfinish >=boundary||finish<start;
2548+
}
2549+
else
2550+
{
2551+
returnfinish >=boundary&&finish<start;
2552+
}
2553+
}
2554+
2555+
/*
2556+
* Read the offset of the first member of the given multixact.
2557+
*/
2558+
staticMultiXactOffset
2559+
read_offset_for_multi(MultiXactIdmulti)
2560+
{
2561+
MultiXactOffsetoffset;
2562+
intpageno;
2563+
intentryno;
2564+
intslotno;
2565+
MultiXactOffset*offptr;
2566+
2567+
pageno=MultiXactIdToOffsetPage(multi);
2568+
entryno=MultiXactIdToOffsetEntry(multi);
2569+
2570+
/* lock is acquired by SimpleLruReadPage_ReadOnly */
2571+
slotno=SimpleLruReadPage_ReadOnly(MultiXactOffsetCtl,pageno,multi);
2572+
offptr= (MultiXactOffset*)MultiXactOffsetCtl->shared->page_buffer[slotno];
2573+
offptr+=entryno;
2574+
offset=*offptr;
2575+
LWLockRelease(MultiXactOffsetControlLock);
2576+
2577+
returnoffset;
2578+
}
2579+
24042580
/*
24052581
* SlruScanDirectory callback.
24062582
*This callback deletes segments that are outside the range determined by
@@ -2533,26 +2709,7 @@ TruncateMultiXact(void)
25332709
* First, compute the safe truncation point for MultiXactMember. This is
25342710
* the starting offset of the oldest multixact.
25352711
*/
2536-
{
2537-
intpageno;
2538-
intslotno;
2539-
intentryno;
2540-
MultiXactOffset*offptr;
2541-
2542-
/* lock is acquired by SimpleLruReadPage_ReadOnly */
2543-
2544-
pageno=MultiXactIdToOffsetPage(oldestMXact);
2545-
entryno=MultiXactIdToOffsetEntry(oldestMXact);
2546-
2547-
slotno=SimpleLruReadPage_ReadOnly(MultiXactOffsetCtl,pageno,
2548-
oldestMXact);
2549-
offptr= (MultiXactOffset*)
2550-
MultiXactOffsetCtl->shared->page_buffer[slotno];
2551-
offptr+=entryno;
2552-
oldestOffset=*offptr;
2553-
2554-
LWLockRelease(MultiXactOffsetControlLock);
2555-
}
2712+
oldestOffset=read_offset_for_multi(oldestMXact);
25562713

25572714
/*
25582715
* To truncate MultiXactMembers, we need to figure out the active page

‎src/backend/bootstrap/bootstrap.c

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -397,6 +397,12 @@ AuxiliaryProcessMain(int argc, char *argv[])
397397
proc_exit(1);/* should never return */
398398

399399
caseBootstrapProcess:
400+
/*
401+
* There was a brief instant during which mode was Normal; this is
402+
* okay. We need to be in bootstrap mode during BootStrapXLOG for
403+
* the sake of multixact initialization.
404+
*/
405+
SetProcessingMode(BootstrapProcessing);
400406
bootstrap_signals();
401407
BootStrapXLOG();
402408
BootstrapModeMain();
@@ -459,8 +465,7 @@ BootstrapModeMain(void)
459465
inti;
460466

461467
Assert(!IsUnderPostmaster);
462-
463-
SetProcessingMode(BootstrapProcessing);
468+
Assert(IsBootstrapProcessingMode());
464469

465470
/*
466471
* Do backend-like initialization for bootstrap mode

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp