Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit06998ea

Browse files
committed
Improve heuristics for compressing the KnownAssignedXids array.
Previously, we'd compress only when the active range of array entriesreached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids).If max_connections is large, the first term could result in notcompressing for a long time, resulting in much wastage of cycles inhot-standby backends scanning the array to take snapshots. Get ridof that term, and just bound it to 2 * pArray->numKnownAssignedXids.That however creates the opposite risk, that we might spend too mucheffort compressing. Hence, consider compressing only once every 128commit records. (This frequency was chosen by benchmarking. Whilewe only tried one benchmark scenario, the results seem stable overa fairly wide range of frequencies.)Also, force compression when processing RecoveryInfo WAL records(which should be infrequent); the old code could perform compressionthen, but would do so only after the same array-range check as forthe transaction-commit path.Also, opportunistically run compression if the startup process is aboutto wait for WAL, though not oftener than once a second. This shouldprevent cases where we waste lots of time by leaving the arraynot-compressed for long intervals due to low WAL traffic.Lastly, add a simple check to keep us from uselessly compressingwhen the array storage is already compact.Back-patch, as the performance problem is worse in pre-v14 branchesthan in HEAD.Simon Riggs and Michail Nikolaev, with help from Tom Lane andAndres Freund.Discussion:https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com
1 parent7715a3c commit06998ea

File tree

3 files changed

+110
-30
lines changed

3 files changed

+110
-30
lines changed

‎src/backend/access/transam/xlog.c

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12741,6 +12741,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
1274112741
wait_time=wal_retrieve_retry_interval-
1274212742
TimestampDifferenceMilliseconds(last_fail_time,now);
1274312743

12744+
/* Do background tasks that might benefit us later. */
12745+
KnownAssignedTransactionIdsIdleMaintenance();
12746+
1274412747
(void)WaitLatch(&XLogCtl->recoveryWakeupLatch,
1274512748
WL_LATCH_SET |WL_TIMEOUT |
1274612749
WL_EXIT_ON_PM_DEATH,
@@ -13002,6 +13005,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
1300213005
streaming_reply_sent= true;
1300313006
}
1300413007

13008+
/* Do any background tasks that might benefit us later. */
13009+
KnownAssignedTransactionIdsIdleMaintenance();
13010+
1300513011
/*
1300613012
* Wait for more WAL to arrive. Time out after 5 seconds
1300713013
* to react to a trigger file promptly and to check if the

‎src/backend/storage/ipc/procarray.c

Lines changed: 103 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,17 @@ typedef enum GlobalVisHorizonKind
257257
VISHORIZON_TEMP
258258
}GlobalVisHorizonKind;
259259

260+
/*
261+
* Reason codes for KnownAssignedXidsCompress().
262+
*/
263+
typedefenumKAXCompressReason
264+
{
265+
KAX_NO_SPACE,/* need to free up space at array end */
266+
KAX_PRUNE,/* we just pruned old entries */
267+
KAX_TRANSACTION_END,/* we just committed/removed some XIDs */
268+
KAX_STARTUP_PROCESS_IDLE/* startup process is about to sleep */
269+
}KAXCompressReason;
270+
260271

261272
staticProcArrayStruct*procArray;
262273

@@ -341,7 +352,7 @@ static bool HaveVirtualXIDsDelayingChkptGuts(VirtualTransactionId *vxids,
341352
intnvxids,inttype);
342353

343354
/* Primitives for KnownAssignedXids array handling for standby */
344-
staticvoidKnownAssignedXidsCompress(boolforce);
355+
staticvoidKnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock);
345356
staticvoidKnownAssignedXidsAdd(TransactionIdfrom_xid,TransactionIdto_xid,
346357
boolexclusive_lock);
347358
staticboolKnownAssignedXidsSearch(TransactionIdxid,boolremove);
@@ -4556,6 +4567,17 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
45564567
LWLockRelease(ProcArrayLock);
45574568
}
45584569

4570+
/*
4571+
* KnownAssignedTransactionIdsIdleMaintenance
4572+
*Opportunistically do maintenance work when the startup process
4573+
*is about to go idle.
4574+
*/
4575+
void
4576+
KnownAssignedTransactionIdsIdleMaintenance(void)
4577+
{
4578+
KnownAssignedXidsCompress(KAX_STARTUP_PROCESS_IDLE, false);
4579+
}
4580+
45594581

45604582
/*
45614583
* Private module functions to manipulate KnownAssignedXids
@@ -4638,50 +4660,101 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
46384660
* so there is an optimal point for any workload mix. We use a heuristic to
46394661
* decide when to compress the array, though trimming also helps reduce
46404662
* frequency of compressing. The heuristic requires us to track the number of
4641-
* currently valid XIDs in the array.
4663+
* currently valid XIDs in the array (N). Except in special cases, we'll
4664+
* compress when S >= 2N. Bounding S at 2N in turn bounds the time for
4665+
* taking a snapshot to be O(N), which it would have to be anyway.
46424666
*/
46434667

46444668

46454669
/*
46464670
* Compress KnownAssignedXids by shifting valid data down to the start of the
46474671
* array, removing any gaps.
46484672
*
4649-
* A compression step is forced if "force" istrue, otherwise we do it
4650-
* only if a heuristic indicates it's a good time to do it.
4673+
* A compression step is forced if "reason" isKAX_NO_SPACE, otherwise
4674+
*we do itonly if a heuristic indicates it's a good time to do it.
46514675
*
4652-
* Caller must hold ProcArrayLock in exclusive mode.
4676+
* Compression requires holding ProcArrayLock in exclusive mode.
4677+
* Caller must pass haveLock = true if it already holds the lock.
46534678
*/
46544679
staticvoid
4655-
KnownAssignedXidsCompress(boolforce)
4680+
KnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock)
46564681
{
46574682
ProcArrayStruct*pArray=procArray;
46584683
inthead,
4659-
tail;
4684+
tail,
4685+
nelements;
46604686
intcompress_index;
46614687
inti;
46624688

4663-
/* no spinlock required since we hold ProcArrayLock exclusively */
4689+
/* Counters for compression heuristics */
4690+
staticunsignedinttransactionEndsCounter;
4691+
staticTimestampTzlastCompressTs;
4692+
4693+
/* Tuning constants */
4694+
#defineKAX_COMPRESS_FREQUENCY 128/* in transactions */
4695+
#defineKAX_COMPRESS_IDLE_INTERVAL 1000/* in ms */
4696+
4697+
/*
4698+
* Since only the startup process modifies the head/tail pointers, we
4699+
* don't need a lock to read them here.
4700+
*/
46644701
head=pArray->headKnownAssignedXids;
46654702
tail=pArray->tailKnownAssignedXids;
4703+
nelements=head-tail;
46664704

4667-
if (!force)
4705+
/*
4706+
* If we can choose whether to compress, use a heuristic to avoid
4707+
* compressing too often or not often enough. "Compress" here simply
4708+
* means moving the values to the beginning of the array, so it is not as
4709+
* complex or costly as typical data compression algorithms.
4710+
*/
4711+
if (nelements==pArray->numKnownAssignedXids)
46684712
{
46694713
/*
4670-
* If we can choose how much to compress, use a heuristic to avoid
4671-
* compressing too often or not often enough.
4672-
*
4673-
* Heuristic is if we have a large enough current spread and less than
4674-
* 50% of the elements are currently in use, then compress. This
4675-
* should ensure we compress fairly infrequently. We could compress
4676-
* less often though the virtual array would spread out more and
4677-
* snapshots would become more expensive.
4714+
* When there are no gaps between head and tail, don't bother to
4715+
* compress, except in the KAX_NO_SPACE case where we must compress to
4716+
* create some space after the head.
4717+
*/
4718+
if (reason!=KAX_NO_SPACE)
4719+
return;
4720+
}
4721+
elseif (reason==KAX_TRANSACTION_END)
4722+
{
4723+
/*
4724+
* Consider compressing only once every so many commits. Frequency
4725+
* determined by benchmarks.
46784726
*/
4679-
intnelements=head-tail;
4727+
if ((transactionEndsCounter++) %KAX_COMPRESS_FREQUENCY!=0)
4728+
return;
46804729

4681-
if (nelements<4*PROCARRAY_MAXPROCS||
4682-
nelements<2*pArray->numKnownAssignedXids)
4730+
/*
4731+
* Furthermore, compress only if the used part of the array is less
4732+
* than 50% full (see comments above).
4733+
*/
4734+
if (nelements<2*pArray->numKnownAssignedXids)
46834735
return;
46844736
}
4737+
elseif (reason==KAX_STARTUP_PROCESS_IDLE)
4738+
{
4739+
/*
4740+
* We're about to go idle for lack of new WAL, so we might as well
4741+
* compress. But not too often, to avoid ProcArray lock contention
4742+
* with readers.
4743+
*/
4744+
if (lastCompressTs!=0)
4745+
{
4746+
TimestampTzcompress_after;
4747+
4748+
compress_after=TimestampTzPlusMilliseconds(lastCompressTs,
4749+
KAX_COMPRESS_IDLE_INTERVAL);
4750+
if (GetCurrentTimestamp()<compress_after)
4751+
return;
4752+
}
4753+
}
4754+
4755+
/* Need to compress, so get the lock if we don't have it. */
4756+
if (!haveLock)
4757+
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
46854758

46864759
/*
46874760
* We compress the array by reading the valid values from tail to head,
@@ -4697,9 +4770,16 @@ KnownAssignedXidsCompress(bool force)
46974770
compress_index++;
46984771
}
46994772
}
4773+
Assert(compress_index==pArray->numKnownAssignedXids);
47004774

47014775
pArray->tailKnownAssignedXids=0;
47024776
pArray->headKnownAssignedXids=compress_index;
4777+
4778+
if (!haveLock)
4779+
LWLockRelease(ProcArrayLock);
4780+
4781+
/* Update timestamp for maintenance. No need to hold lock for this. */
4782+
lastCompressTs=GetCurrentTimestamp();
47034783
}
47044784

47054785
/*
@@ -4771,18 +4851,11 @@ KnownAssignedXidsAdd(TransactionId from_xid, TransactionId to_xid,
47714851
*/
47724852
if (head+nxids>pArray->maxKnownAssignedXids)
47734853
{
4774-
/* must hold lock to compress */
4775-
if (!exclusive_lock)
4776-
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
4777-
4778-
KnownAssignedXidsCompress(true);
4854+
KnownAssignedXidsCompress(KAX_NO_SPACE,exclusive_lock);
47794855

47804856
head=pArray->headKnownAssignedXids;
47814857
/* note: we no longer care about the tail pointer */
47824858

4783-
if (!exclusive_lock)
4784-
LWLockRelease(ProcArrayLock);
4785-
47864859
/*
47874860
* If it still won't fit then we're out of memory
47884861
*/
@@ -4976,7 +5049,7 @@ KnownAssignedXidsRemoveTree(TransactionId xid, int nsubxids,
49765049
KnownAssignedXidsRemove(subxids[i]);
49775050

49785051
/* Opportunistically compress the array */
4979-
KnownAssignedXidsCompress(false);
5052+
KnownAssignedXidsCompress(KAX_TRANSACTION_END, true);
49805053
}
49815054

49825055
/*
@@ -5051,7 +5124,7 @@ KnownAssignedXidsRemovePreceding(TransactionId removeXid)
50515124
}
50525125

50535126
/* Opportunistically compress the array */
5054-
KnownAssignedXidsCompress(false);
5127+
KnownAssignedXidsCompress(KAX_PRUNE, true);
50555128
}
50565129

50575130
/*

‎src/include/storage/procarray.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
3939
TransactionIdmax_xid);
4040
externvoidExpireAllKnownAssignedTransactionIds(void);
4141
externvoidExpireOldKnownAssignedTransactionIds(TransactionIdxid);
42+
externvoidKnownAssignedTransactionIdsIdleMaintenance(void);
4243

4344
externintGetMaxSnapshotXidCount(void);
4445
externintGetMaxSnapshotSubxidCount(void);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp