Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit6e8ad11

Browse files
committed
Improve heuristics for compressing the KnownAssignedXids array.
Previously, we'd compress only when the active range of array entriesreached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids).If max_connections is large, the first term could result in notcompressing for a long time, resulting in much wastage of cycles inhot-standby backends scanning the array to take snapshots. Get ridof that term, and just bound it to 2 * pArray->numKnownAssignedXids.That however creates the opposite risk, that we might spend too mucheffort compressing. Hence, consider compressing only once every 128commit records. (This frequency was chosen by benchmarking. Whilewe only tried one benchmark scenario, the results seem stable overa fairly wide range of frequencies.)Also, force compression when processing RecoveryInfo WAL records(which should be infrequent); the old code could perform compressionthen, but would do so only after the same array-range check as forthe transaction-commit path.Also, opportunistically run compression if the startup process is aboutto wait for WAL, though not oftener than once a second. This shouldprevent cases where we waste lots of time by leaving the arraynot-compressed for long intervals due to low WAL traffic.Lastly, add a simple check to keep us from uselessly compressingwhen the array storage is already compact.Back-patch, as the performance problem is worse in pre-v14 branchesthan in HEAD.Simon Riggs and Michail Nikolaev, with help from Tom Lane andAndres Freund.Discussion:https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com
1 parentaca695f commit6e8ad11

File tree

3 files changed

+110
-30
lines changed

3 files changed

+110
-30
lines changed

‎src/backend/access/transam/xlog.c

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12514,6 +12514,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
1251412514
wait_time=wal_retrieve_retry_interval-
1251512515
TimestampDifferenceMilliseconds(last_fail_time,now);
1251612516

12517+
/* Do background tasks that might benefit us later. */
12518+
KnownAssignedTransactionIdsIdleMaintenance();
12519+
1251712520
(void)WaitLatch(&XLogCtl->recoveryWakeupLatch,
1251812521
WL_LATCH_SET |WL_TIMEOUT |
1251912522
WL_EXIT_ON_PM_DEATH,
@@ -12776,6 +12779,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
1277612779
streaming_reply_sent= true;
1277712780
}
1277812781

12782+
/* Do any background tasks that might benefit us later. */
12783+
KnownAssignedTransactionIdsIdleMaintenance();
12784+
1277912785
/*
1278012786
* Wait for more WAL to arrive. Time out after 5 seconds
1278112787
* to react to a trigger file promptly and to check if the

‎src/backend/storage/ipc/procarray.c

Lines changed: 103 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,17 @@ static ProcArrayStruct *procArray;
104104
staticPGPROC*allProcs;
105105
staticPGXACT*allPgXact;
106106

107+
/*
108+
* Reason codes for KnownAssignedXidsCompress().
109+
*/
110+
typedefenumKAXCompressReason
111+
{
112+
KAX_NO_SPACE,/* need to free up space at array end */
113+
KAX_PRUNE,/* we just pruned old entries */
114+
KAX_TRANSACTION_END,/* we just committed/removed some XIDs */
115+
KAX_STARTUP_PROCESS_IDLE/* startup process is about to sleep */
116+
}KAXCompressReason;
117+
107118
/*
108119
* Cache to reduce overhead of repeated calls to TransactionIdIsInProgress()
109120
*/
@@ -166,7 +177,7 @@ static bool HaveVirtualXIDsDelayingChkptGuts(VirtualTransactionId *vxids,
166177
intnvxids,inttype);
167178

168179
/* Primitives for KnownAssignedXids array handling for standby */
169-
staticvoidKnownAssignedXidsCompress(boolforce);
180+
staticvoidKnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock);
170181
staticvoidKnownAssignedXidsAdd(TransactionIdfrom_xid,TransactionIdto_xid,
171182
boolexclusive_lock);
172183
staticboolKnownAssignedXidsSearch(TransactionIdxid,boolremove);
@@ -3540,6 +3551,17 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
35403551
LWLockRelease(ProcArrayLock);
35413552
}
35423553

3554+
/*
3555+
* KnownAssignedTransactionIdsIdleMaintenance
3556+
*Opportunistically do maintenance work when the startup process
3557+
*is about to go idle.
3558+
*/
3559+
void
3560+
KnownAssignedTransactionIdsIdleMaintenance(void)
3561+
{
3562+
KnownAssignedXidsCompress(KAX_STARTUP_PROCESS_IDLE, false);
3563+
}
3564+
35433565

35443566
/*
35453567
* Private module functions to manipulate KnownAssignedXids
@@ -3622,50 +3644,101 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
36223644
* so there is an optimal point for any workload mix. We use a heuristic to
36233645
* decide when to compress the array, though trimming also helps reduce
36243646
* frequency of compressing. The heuristic requires us to track the number of
3625-
* currently valid XIDs in the array.
3647+
* currently valid XIDs in the array (N). Except in special cases, we'll
3648+
* compress when S >= 2N. Bounding S at 2N in turn bounds the time for
3649+
* taking a snapshot to be O(N), which it would have to be anyway.
36263650
*/
36273651

36283652

36293653
/*
36303654
* Compress KnownAssignedXids by shifting valid data down to the start of the
36313655
* array, removing any gaps.
36323656
*
3633-
* A compression step is forced if "force" istrue, otherwise we do it
3634-
* only if a heuristic indicates it's a good time to do it.
3657+
* A compression step is forced if "reason" isKAX_NO_SPACE, otherwise
3658+
*we do itonly if a heuristic indicates it's a good time to do it.
36353659
*
3636-
* Caller must hold ProcArrayLock in exclusive mode.
3660+
* Compression requires holding ProcArrayLock in exclusive mode.
3661+
* Caller must pass haveLock = true if it already holds the lock.
36373662
*/
36383663
staticvoid
3639-
KnownAssignedXidsCompress(boolforce)
3664+
KnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock)
36403665
{
36413666
ProcArrayStruct*pArray=procArray;
36423667
inthead,
3643-
tail;
3668+
tail,
3669+
nelements;
36443670
intcompress_index;
36453671
inti;
36463672

3647-
/* no spinlock required since we hold ProcArrayLock exclusively */
3673+
/* Counters for compression heuristics */
3674+
staticunsignedinttransactionEndsCounter;
3675+
staticTimestampTzlastCompressTs;
3676+
3677+
/* Tuning constants */
3678+
#defineKAX_COMPRESS_FREQUENCY 128/* in transactions */
3679+
#defineKAX_COMPRESS_IDLE_INTERVAL 1000/* in ms */
3680+
3681+
/*
3682+
* Since only the startup process modifies the head/tail pointers, we
3683+
* don't need a lock to read them here.
3684+
*/
36483685
head=pArray->headKnownAssignedXids;
36493686
tail=pArray->tailKnownAssignedXids;
3687+
nelements=head-tail;
36503688

3651-
if (!force)
3689+
/*
3690+
* If we can choose whether to compress, use a heuristic to avoid
3691+
* compressing too often or not often enough. "Compress" here simply
3692+
* means moving the values to the beginning of the array, so it is not as
3693+
* complex or costly as typical data compression algorithms.
3694+
*/
3695+
if (nelements==pArray->numKnownAssignedXids)
36523696
{
36533697
/*
3654-
* If we can choose how much to compress, use a heuristic to avoid
3655-
* compressing too often or not often enough.
3656-
*
3657-
* Heuristic is if we have a large enough current spread and less than
3658-
* 50% of the elements are currently in use, then compress. This
3659-
* should ensure we compress fairly infrequently. We could compress
3660-
* less often though the virtual array would spread out more and
3661-
* snapshots would become more expensive.
3698+
* When there are no gaps between head and tail, don't bother to
3699+
* compress, except in the KAX_NO_SPACE case where we must compress to
3700+
* create some space after the head.
36623701
*/
3663-
intnelements=head-tail;
3702+
if (reason!=KAX_NO_SPACE)
3703+
return;
3704+
}
3705+
elseif (reason==KAX_TRANSACTION_END)
3706+
{
3707+
/*
3708+
* Consider compressing only once every so many commits. Frequency
3709+
* determined by benchmarks.
3710+
*/
3711+
if ((transactionEndsCounter++) %KAX_COMPRESS_FREQUENCY!=0)
3712+
return;
36643713

3665-
if (nelements<4*PROCARRAY_MAXPROCS||
3666-
nelements<2*pArray->numKnownAssignedXids)
3714+
/*
3715+
* Furthermore, compress only if the used part of the array is less
3716+
* than 50% full (see comments above).
3717+
*/
3718+
if (nelements<2*pArray->numKnownAssignedXids)
36673719
return;
36683720
}
3721+
elseif (reason==KAX_STARTUP_PROCESS_IDLE)
3722+
{
3723+
/*
3724+
* We're about to go idle for lack of new WAL, so we might as well
3725+
* compress. But not too often, to avoid ProcArray lock contention
3726+
* with readers.
3727+
*/
3728+
if (lastCompressTs!=0)
3729+
{
3730+
TimestampTzcompress_after;
3731+
3732+
compress_after=TimestampTzPlusMilliseconds(lastCompressTs,
3733+
KAX_COMPRESS_IDLE_INTERVAL);
3734+
if (GetCurrentTimestamp()<compress_after)
3735+
return;
3736+
}
3737+
}
3738+
3739+
/* Need to compress, so get the lock if we don't have it. */
3740+
if (!haveLock)
3741+
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
36693742

36703743
/*
36713744
* We compress the array by reading the valid values from tail to head,
@@ -3681,9 +3754,16 @@ KnownAssignedXidsCompress(bool force)
36813754
compress_index++;
36823755
}
36833756
}
3757+
Assert(compress_index==pArray->numKnownAssignedXids);
36843758

36853759
pArray->tailKnownAssignedXids=0;
36863760
pArray->headKnownAssignedXids=compress_index;
3761+
3762+
if (!haveLock)
3763+
LWLockRelease(ProcArrayLock);
3764+
3765+
/* Update timestamp for maintenance. No need to hold lock for this. */
3766+
lastCompressTs=GetCurrentTimestamp();
36873767
}
36883768

36893769
/*
@@ -3755,18 +3835,11 @@ KnownAssignedXidsAdd(TransactionId from_xid, TransactionId to_xid,
37553835
*/
37563836
if (head+nxids>pArray->maxKnownAssignedXids)
37573837
{
3758-
/* must hold lock to compress */
3759-
if (!exclusive_lock)
3760-
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
3761-
3762-
KnownAssignedXidsCompress(true);
3838+
KnownAssignedXidsCompress(KAX_NO_SPACE,exclusive_lock);
37633839

37643840
head=pArray->headKnownAssignedXids;
37653841
/* note: we no longer care about the tail pointer */
37663842

3767-
if (!exclusive_lock)
3768-
LWLockRelease(ProcArrayLock);
3769-
37703843
/*
37713844
* If it still won't fit then we're out of memory
37723845
*/
@@ -3960,7 +4033,7 @@ KnownAssignedXidsRemoveTree(TransactionId xid, int nsubxids,
39604033
KnownAssignedXidsRemove(subxids[i]);
39614034

39624035
/* Opportunistically compress the array */
3963-
KnownAssignedXidsCompress(false);
4036+
KnownAssignedXidsCompress(KAX_TRANSACTION_END, true);
39644037
}
39654038

39664039
/*
@@ -4035,7 +4108,7 @@ KnownAssignedXidsRemovePreceding(TransactionId removeXid)
40354108
}
40364109

40374110
/* Opportunistically compress the array */
4038-
KnownAssignedXidsCompress(false);
4111+
KnownAssignedXidsCompress(KAX_PRUNE, true);
40394112
}
40404113

40414114
/*

‎src/include/storage/procarray.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
7474
TransactionIdmax_xid);
7575
externvoidExpireAllKnownAssignedTransactionIds(void);
7676
externvoidExpireOldKnownAssignedTransactionIds(TransactionIdxid);
77+
externvoidKnownAssignedTransactionIdsIdleMaintenance(void);
7778

7879
externintGetMaxSnapshotXidCount(void);
7980
externintGetMaxSnapshotSubxidCount(void);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp