Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit0f45ff7

Browse files
tglsfdcpull[bot]
authored andcommitted
Improve heuristics for compressing the KnownAssignedXids array.
Previously, we'd compress only when the active range of array entriesreached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids).If max_connections is large, the first term could result in notcompressing for a long time, resulting in much wastage of cycles inhot-standby backends scanning the array to take snapshots. Get ridof that term, and just bound it to 2 * pArray->numKnownAssignedXids.That however creates the opposite risk, that we might spend too mucheffort compressing. Hence, consider compressing only once every 128commit records. (This frequency was chosen by benchmarking. Whilewe only tried one benchmark scenario, the results seem stable overa fairly wide range of frequencies.)Also, force compression when processing RecoveryInfo WAL records(which should be infrequent); the old code could perform compressionthen, but would do so only after the same array-range check as forthe transaction-commit path.Also, opportunistically run compression if the startup process is aboutto wait for WAL, though not oftener than once a second. This shouldprevent cases where we waste lots of time by leaving the arraynot-compressed for long intervals due to low WAL traffic.Lastly, add a simple check to keep us from uselessly compressingwhen the array storage is already compact.Back-patch, as the performance problem is worse in pre-v14 branchesthan in HEAD.Simon Riggs and Michail Nikolaev, with help from Tom Lane andAndres Freund.Discussion:https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com
1 parent6406881 commit0f45ff7

File tree

3 files changed

+110
-30
lines changed

3 files changed

+110
-30
lines changed

‎src/backend/access/transam/xlogrecovery.c

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3566,6 +3566,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
35663566
elog(LOG,"waiting for WAL to become available at %X/%X",
35673567
LSN_FORMAT_ARGS(RecPtr));
35683568

3569+
/* Do background tasks that might benefit us later. */
3570+
KnownAssignedTransactionIdsIdleMaintenance();
3571+
35693572
(void)WaitLatch(&XLogRecoveryCtl->recoveryWakeupLatch,
35703573
WL_LATCH_SET |WL_TIMEOUT |
35713574
WL_EXIT_ON_PM_DEATH,
@@ -3832,6 +3835,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
38323835
streaming_reply_sent= true;
38333836
}
38343837

3838+
/* Do any background tasks that might benefit us later. */
3839+
KnownAssignedTransactionIdsIdleMaintenance();
3840+
38353841
/* Update pg_stat_recovery_prefetch before sleeping. */
38363842
XLogPrefetcherComputeStats(xlogprefetcher);
38373843

‎src/backend/storage/ipc/procarray.c

Lines changed: 103 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,17 @@ typedef enum GlobalVisHorizonKind
257257
VISHORIZON_TEMP
258258
}GlobalVisHorizonKind;
259259

260+
/*
261+
* Reason codes for KnownAssignedXidsCompress().
262+
*/
263+
typedefenumKAXCompressReason
264+
{
265+
KAX_NO_SPACE,/* need to free up space at array end */
266+
KAX_PRUNE,/* we just pruned old entries */
267+
KAX_TRANSACTION_END,/* we just committed/removed some XIDs */
268+
KAX_STARTUP_PROCESS_IDLE/* startup process is about to sleep */
269+
}KAXCompressReason;
270+
260271

261272
staticProcArrayStruct*procArray;
262273

@@ -336,7 +347,7 @@ static void DisplayXidCache(void);
336347
#endif/* XIDCACHE_DEBUG */
337348

338349
/* Primitives for KnownAssignedXids array handling for standby */
339-
staticvoidKnownAssignedXidsCompress(boolforce);
350+
staticvoidKnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock);
340351
staticvoidKnownAssignedXidsAdd(TransactionIdfrom_xid,TransactionIdto_xid,
341352
boolexclusive_lock);
342353
staticboolKnownAssignedXidsSearch(TransactionIdxid,boolremove);
@@ -4509,6 +4520,17 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
45094520
LWLockRelease(ProcArrayLock);
45104521
}
45114522

4523+
/*
4524+
* KnownAssignedTransactionIdsIdleMaintenance
4525+
*Opportunistically do maintenance work when the startup process
4526+
*is about to go idle.
4527+
*/
4528+
void
4529+
KnownAssignedTransactionIdsIdleMaintenance(void)
4530+
{
4531+
KnownAssignedXidsCompress(KAX_STARTUP_PROCESS_IDLE, false);
4532+
}
4533+
45124534

45134535
/*
45144536
* Private module functions to manipulate KnownAssignedXids
@@ -4591,50 +4613,101 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
45914613
* so there is an optimal point for any workload mix. We use a heuristic to
45924614
* decide when to compress the array, though trimming also helps reduce
45934615
* frequency of compressing. The heuristic requires us to track the number of
4594-
* currently valid XIDs in the array.
4616+
* currently valid XIDs in the array (N). Except in special cases, we'll
4617+
* compress when S >= 2N. Bounding S at 2N in turn bounds the time for
4618+
* taking a snapshot to be O(N), which it would have to be anyway.
45954619
*/
45964620

45974621

45984622
/*
45994623
* Compress KnownAssignedXids by shifting valid data down to the start of the
46004624
* array, removing any gaps.
46014625
*
4602-
* A compression step is forced if "force" istrue, otherwise we do it
4603-
* only if a heuristic indicates it's a good time to do it.
4626+
* A compression step is forced if "reason" isKAX_NO_SPACE, otherwise
4627+
*we do itonly if a heuristic indicates it's a good time to do it.
46044628
*
4605-
* Caller must hold ProcArrayLock in exclusive mode.
4629+
* Compression requires holding ProcArrayLock in exclusive mode.
4630+
* Caller must pass haveLock = true if it already holds the lock.
46064631
*/
46074632
staticvoid
4608-
KnownAssignedXidsCompress(boolforce)
4633+
KnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock)
46094634
{
46104635
ProcArrayStruct*pArray=procArray;
46114636
inthead,
4612-
tail;
4637+
tail,
4638+
nelements;
46134639
intcompress_index;
46144640
inti;
46154641

4616-
/* no spinlock required since we hold ProcArrayLock exclusively */
4642+
/* Counters for compression heuristics */
4643+
staticunsignedinttransactionEndsCounter;
4644+
staticTimestampTzlastCompressTs;
4645+
4646+
/* Tuning constants */
4647+
#defineKAX_COMPRESS_FREQUENCY 128/* in transactions */
4648+
#defineKAX_COMPRESS_IDLE_INTERVAL 1000/* in ms */
4649+
4650+
/*
4651+
* Since only the startup process modifies the head/tail pointers, we
4652+
* don't need a lock to read them here.
4653+
*/
46174654
head=pArray->headKnownAssignedXids;
46184655
tail=pArray->tailKnownAssignedXids;
4656+
nelements=head-tail;
46194657

4620-
if (!force)
4658+
/*
4659+
* If we can choose whether to compress, use a heuristic to avoid
4660+
* compressing too often or not often enough. "Compress" here simply
4661+
* means moving the values to the beginning of the array, so it is not as
4662+
* complex or costly as typical data compression algorithms.
4663+
*/
4664+
if (nelements==pArray->numKnownAssignedXids)
46214665
{
46224666
/*
4623-
* If we can choose how much to compress, use a heuristic to avoid
4624-
* compressing too often or not often enough.
4625-
*
4626-
* Heuristic is if we have a large enough current spread and less than
4627-
* 50% of the elements are currently in use, then compress. This
4628-
* should ensure we compress fairly infrequently. We could compress
4629-
* less often though the virtual array would spread out more and
4630-
* snapshots would become more expensive.
4667+
* When there are no gaps between head and tail, don't bother to
4668+
* compress, except in the KAX_NO_SPACE case where we must compress to
4669+
* create some space after the head.
4670+
*/
4671+
if (reason!=KAX_NO_SPACE)
4672+
return;
4673+
}
4674+
elseif (reason==KAX_TRANSACTION_END)
4675+
{
4676+
/*
4677+
* Consider compressing only once every so many commits. Frequency
4678+
* determined by benchmarks.
46314679
*/
4632-
intnelements=head-tail;
4680+
if ((transactionEndsCounter++) %KAX_COMPRESS_FREQUENCY!=0)
4681+
return;
46334682

4634-
if (nelements<4*PROCARRAY_MAXPROCS||
4635-
nelements<2*pArray->numKnownAssignedXids)
4683+
/*
4684+
* Furthermore, compress only if the used part of the array is less
4685+
* than 50% full (see comments above).
4686+
*/
4687+
if (nelements<2*pArray->numKnownAssignedXids)
46364688
return;
46374689
}
4690+
elseif (reason==KAX_STARTUP_PROCESS_IDLE)
4691+
{
4692+
/*
4693+
* We're about to go idle for lack of new WAL, so we might as well
4694+
* compress. But not too often, to avoid ProcArray lock contention
4695+
* with readers.
4696+
*/
4697+
if (lastCompressTs!=0)
4698+
{
4699+
TimestampTzcompress_after;
4700+
4701+
compress_after=TimestampTzPlusMilliseconds(lastCompressTs,
4702+
KAX_COMPRESS_IDLE_INTERVAL);
4703+
if (GetCurrentTimestamp()<compress_after)
4704+
return;
4705+
}
4706+
}
4707+
4708+
/* Need to compress, so get the lock if we don't have it. */
4709+
if (!haveLock)
4710+
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
46384711

46394712
/*
46404713
* We compress the array by reading the valid values from tail to head,
@@ -4650,9 +4723,16 @@ KnownAssignedXidsCompress(bool force)
46504723
compress_index++;
46514724
}
46524725
}
4726+
Assert(compress_index==pArray->numKnownAssignedXids);
46534727

46544728
pArray->tailKnownAssignedXids=0;
46554729
pArray->headKnownAssignedXids=compress_index;
4730+
4731+
if (!haveLock)
4732+
LWLockRelease(ProcArrayLock);
4733+
4734+
/* Update timestamp for maintenance. No need to hold lock for this. */
4735+
lastCompressTs=GetCurrentTimestamp();
46564736
}
46574737

46584738
/*
@@ -4724,18 +4804,11 @@ KnownAssignedXidsAdd(TransactionId from_xid, TransactionId to_xid,
47244804
*/
47254805
if (head+nxids>pArray->maxKnownAssignedXids)
47264806
{
4727-
/* must hold lock to compress */
4728-
if (!exclusive_lock)
4729-
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
4730-
4731-
KnownAssignedXidsCompress(true);
4807+
KnownAssignedXidsCompress(KAX_NO_SPACE,exclusive_lock);
47324808

47334809
head=pArray->headKnownAssignedXids;
47344810
/* note: we no longer care about the tail pointer */
47354811

4736-
if (!exclusive_lock)
4737-
LWLockRelease(ProcArrayLock);
4738-
47394812
/*
47404813
* If it still won't fit then we're out of memory
47414814
*/
@@ -4929,7 +5002,7 @@ KnownAssignedXidsRemoveTree(TransactionId xid, int nsubxids,
49295002
KnownAssignedXidsRemove(subxids[i]);
49305003

49315004
/* Opportunistically compress the array */
4932-
KnownAssignedXidsCompress(false);
5005+
KnownAssignedXidsCompress(KAX_TRANSACTION_END, true);
49335006
}
49345007

49355008
/*
@@ -5004,7 +5077,7 @@ KnownAssignedXidsRemovePreceding(TransactionId removeXid)
50045077
}
50055078

50065079
/* Opportunistically compress the array */
5007-
KnownAssignedXidsCompress(false);
5080+
KnownAssignedXidsCompress(KAX_PRUNE, true);
50085081
}
50095082

50105083
/*

‎src/include/storage/procarray.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
3939
TransactionIdmax_xid);
4040
externvoidExpireAllKnownAssignedTransactionIds(void);
4141
externvoidExpireOldKnownAssignedTransactionIds(TransactionIdxid);
42+
externvoidKnownAssignedTransactionIdsIdleMaintenance(void);
4243

4344
externintGetMaxSnapshotXidCount(void);
4445
externintGetMaxSnapshotSubxidCount(void);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp