Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit55fa993

Browse files
committed
Improve heuristics for compressing the KnownAssignedXids array.
Previously, we'd compress only when the active range of array entriesreached Max(4 * PROCARRAY_MAXPROCS, 2 * pArray->numKnownAssignedXids).If max_connections is large, the first term could result in notcompressing for a long time, resulting in much wastage of cycles inhot-standby backends scanning the array to take snapshots. Get ridof that term, and just bound it to 2 * pArray->numKnownAssignedXids.That however creates the opposite risk, that we might spend too mucheffort compressing. Hence, consider compressing only once every 128commit records. (This frequency was chosen by benchmarking. Whilewe only tried one benchmark scenario, the results seem stable overa fairly wide range of frequencies.)Also, force compression when processing RecoveryInfo WAL records(which should be infrequent); the old code could perform compressionthen, but would do so only after the same array-range check as forthe transaction-commit path.Also, opportunistically run compression if the startup process is aboutto wait for WAL, though not oftener than once a second. This shouldprevent cases where we waste lots of time by leaving the arraynot-compressed for long intervals due to low WAL traffic.Lastly, add a simple check to keep us from uselessly compressingwhen the array storage is already compact.Back-patch, as the performance problem is worse in pre-v14 branchesthan in HEAD.Simon Riggs and Michail Nikolaev, with help from Tom Lane andAndres Freund.Discussion:https://postgr.es/m/CALdSSPgahNUD_=pB_j=1zSnDBaiOtqVfzo8Ejt5J_k7qZiU1Tw@mail.gmail.com
1 parent5dfc2b7 commit55fa993

File tree

3 files changed

+110
-30
lines changed

3 files changed

+110
-30
lines changed

‎src/backend/access/transam/xlogrecovery.c

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3565,6 +3565,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
35653565
elog(LOG,"waiting for WAL to become available at %X/%X",
35663566
LSN_FORMAT_ARGS(RecPtr));
35673567

3568+
/* Do background tasks that might benefit us later. */
3569+
KnownAssignedTransactionIdsIdleMaintenance();
3570+
35683571
(void)WaitLatch(&XLogRecoveryCtl->recoveryWakeupLatch,
35693572
WL_LATCH_SET |WL_TIMEOUT |
35703573
WL_EXIT_ON_PM_DEATH,
@@ -3831,6 +3834,9 @@ WaitForWALToBecomeAvailable(XLogRecPtr RecPtr, bool randAccess,
38313834
streaming_reply_sent= true;
38323835
}
38333836

3837+
/* Do any background tasks that might benefit us later. */
3838+
KnownAssignedTransactionIdsIdleMaintenance();
3839+
38343840
/* Update pg_stat_recovery_prefetch before sleeping. */
38353841
XLogPrefetcherComputeStats(xlogprefetcher);
38363842

‎src/backend/storage/ipc/procarray.c

Lines changed: 103 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -256,6 +256,17 @@ typedef enum GlobalVisHorizonKind
256256
VISHORIZON_TEMP
257257
}GlobalVisHorizonKind;
258258

259+
/*
260+
* Reason codes for KnownAssignedXidsCompress().
261+
*/
262+
typedefenumKAXCompressReason
263+
{
264+
KAX_NO_SPACE,/* need to free up space at array end */
265+
KAX_PRUNE,/* we just pruned old entries */
266+
KAX_TRANSACTION_END,/* we just committed/removed some XIDs */
267+
KAX_STARTUP_PROCESS_IDLE/* startup process is about to sleep */
268+
}KAXCompressReason;
269+
259270

260271
staticProcArrayStruct*procArray;
261272

@@ -335,7 +346,7 @@ static void DisplayXidCache(void);
335346
#endif/* XIDCACHE_DEBUG */
336347

337348
/* Primitives for KnownAssignedXids array handling for standby */
338-
staticvoidKnownAssignedXidsCompress(boolforce);
349+
staticvoidKnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock);
339350
staticvoidKnownAssignedXidsAdd(TransactionIdfrom_xid,TransactionIdto_xid,
340351
boolexclusive_lock);
341352
staticboolKnownAssignedXidsSearch(TransactionIdxid,boolremove);
@@ -4508,6 +4519,17 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
45084519
LWLockRelease(ProcArrayLock);
45094520
}
45104521

4522+
/*
4523+
* KnownAssignedTransactionIdsIdleMaintenance
4524+
*Opportunistically do maintenance work when the startup process
4525+
*is about to go idle.
4526+
*/
4527+
void
4528+
KnownAssignedTransactionIdsIdleMaintenance(void)
4529+
{
4530+
KnownAssignedXidsCompress(KAX_STARTUP_PROCESS_IDLE, false);
4531+
}
4532+
45114533

45124534
/*
45134535
* Private module functions to manipulate KnownAssignedXids
@@ -4590,50 +4612,101 @@ ExpireOldKnownAssignedTransactionIds(TransactionId xid)
45904612
* so there is an optimal point for any workload mix. We use a heuristic to
45914613
* decide when to compress the array, though trimming also helps reduce
45924614
* frequency of compressing. The heuristic requires us to track the number of
4593-
* currently valid XIDs in the array.
4615+
* currently valid XIDs in the array (N). Except in special cases, we'll
4616+
* compress when S >= 2N. Bounding S at 2N in turn bounds the time for
4617+
* taking a snapshot to be O(N), which it would have to be anyway.
45944618
*/
45954619

45964620

45974621
/*
45984622
* Compress KnownAssignedXids by shifting valid data down to the start of the
45994623
* array, removing any gaps.
46004624
*
4601-
* A compression step is forced if "force" istrue, otherwise we do it
4602-
* only if a heuristic indicates it's a good time to do it.
4625+
* A compression step is forced if "reason" isKAX_NO_SPACE, otherwise
4626+
*we do itonly if a heuristic indicates it's a good time to do it.
46034627
*
4604-
* Caller must hold ProcArrayLock in exclusive mode.
4628+
* Compression requires holding ProcArrayLock in exclusive mode.
4629+
* Caller must pass haveLock = true if it already holds the lock.
46054630
*/
46064631
staticvoid
4607-
KnownAssignedXidsCompress(boolforce)
4632+
KnownAssignedXidsCompress(KAXCompressReasonreason,boolhaveLock)
46084633
{
46094634
ProcArrayStruct*pArray=procArray;
46104635
inthead,
4611-
tail;
4636+
tail,
4637+
nelements;
46124638
intcompress_index;
46134639
inti;
46144640

4615-
/* no spinlock required since we hold ProcArrayLock exclusively */
4641+
/* Counters for compression heuristics */
4642+
staticunsignedinttransactionEndsCounter;
4643+
staticTimestampTzlastCompressTs;
4644+
4645+
/* Tuning constants */
4646+
#defineKAX_COMPRESS_FREQUENCY 128/* in transactions */
4647+
#defineKAX_COMPRESS_IDLE_INTERVAL 1000/* in ms */
4648+
4649+
/*
4650+
* Since only the startup process modifies the head/tail pointers, we
4651+
* don't need a lock to read them here.
4652+
*/
46164653
head=pArray->headKnownAssignedXids;
46174654
tail=pArray->tailKnownAssignedXids;
4655+
nelements=head-tail;
46184656

4619-
if (!force)
4657+
/*
4658+
* If we can choose whether to compress, use a heuristic to avoid
4659+
* compressing too often or not often enough. "Compress" here simply
4660+
* means moving the values to the beginning of the array, so it is not as
4661+
* complex or costly as typical data compression algorithms.
4662+
*/
4663+
if (nelements==pArray->numKnownAssignedXids)
46204664
{
46214665
/*
4622-
* If we can choose how much to compress, use a heuristic to avoid
4623-
* compressing too often or not often enough.
4624-
*
4625-
* Heuristic is if we have a large enough current spread and less than
4626-
* 50% of the elements are currently in use, then compress. This
4627-
* should ensure we compress fairly infrequently. We could compress
4628-
* less often though the virtual array would spread out more and
4629-
* snapshots would become more expensive.
4666+
* When there are no gaps between head and tail, don't bother to
4667+
* compress, except in the KAX_NO_SPACE case where we must compress to
4668+
* create some space after the head.
4669+
*/
4670+
if (reason!=KAX_NO_SPACE)
4671+
return;
4672+
}
4673+
elseif (reason==KAX_TRANSACTION_END)
4674+
{
4675+
/*
4676+
* Consider compressing only once every so many commits. Frequency
4677+
* determined by benchmarks.
46304678
*/
4631-
intnelements=head-tail;
4679+
if ((transactionEndsCounter++) %KAX_COMPRESS_FREQUENCY!=0)
4680+
return;
46324681

4633-
if (nelements<4*PROCARRAY_MAXPROCS||
4634-
nelements<2*pArray->numKnownAssignedXids)
4682+
/*
4683+
* Furthermore, compress only if the used part of the array is less
4684+
* than 50% full (see comments above).
4685+
*/
4686+
if (nelements<2*pArray->numKnownAssignedXids)
46354687
return;
46364688
}
4689+
elseif (reason==KAX_STARTUP_PROCESS_IDLE)
4690+
{
4691+
/*
4692+
* We're about to go idle for lack of new WAL, so we might as well
4693+
* compress. But not too often, to avoid ProcArray lock contention
4694+
* with readers.
4695+
*/
4696+
if (lastCompressTs!=0)
4697+
{
4698+
TimestampTzcompress_after;
4699+
4700+
compress_after=TimestampTzPlusMilliseconds(lastCompressTs,
4701+
KAX_COMPRESS_IDLE_INTERVAL);
4702+
if (GetCurrentTimestamp()<compress_after)
4703+
return;
4704+
}
4705+
}
4706+
4707+
/* Need to compress, so get the lock if we don't have it. */
4708+
if (!haveLock)
4709+
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
46374710

46384711
/*
46394712
* We compress the array by reading the valid values from tail to head,
@@ -4649,9 +4722,16 @@ KnownAssignedXidsCompress(bool force)
46494722
compress_index++;
46504723
}
46514724
}
4725+
Assert(compress_index==pArray->numKnownAssignedXids);
46524726

46534727
pArray->tailKnownAssignedXids=0;
46544728
pArray->headKnownAssignedXids=compress_index;
4729+
4730+
if (!haveLock)
4731+
LWLockRelease(ProcArrayLock);
4732+
4733+
/* Update timestamp for maintenance. No need to hold lock for this. */
4734+
lastCompressTs=GetCurrentTimestamp();
46554735
}
46564736

46574737
/*
@@ -4723,18 +4803,11 @@ KnownAssignedXidsAdd(TransactionId from_xid, TransactionId to_xid,
47234803
*/
47244804
if (head+nxids>pArray->maxKnownAssignedXids)
47254805
{
4726-
/* must hold lock to compress */
4727-
if (!exclusive_lock)
4728-
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);
4729-
4730-
KnownAssignedXidsCompress(true);
4806+
KnownAssignedXidsCompress(KAX_NO_SPACE,exclusive_lock);
47314807

47324808
head=pArray->headKnownAssignedXids;
47334809
/* note: we no longer care about the tail pointer */
47344810

4735-
if (!exclusive_lock)
4736-
LWLockRelease(ProcArrayLock);
4737-
47384811
/*
47394812
* If it still won't fit then we're out of memory
47404813
*/
@@ -4928,7 +5001,7 @@ KnownAssignedXidsRemoveTree(TransactionId xid, int nsubxids,
49285001
KnownAssignedXidsRemove(subxids[i]);
49295002

49305003
/* Opportunistically compress the array */
4931-
KnownAssignedXidsCompress(false);
5004+
KnownAssignedXidsCompress(KAX_TRANSACTION_END, true);
49325005
}
49335006

49345007
/*
@@ -5003,7 +5076,7 @@ KnownAssignedXidsRemovePreceding(TransactionId removeXid)
50035076
}
50045077

50055078
/* Opportunistically compress the array */
5006-
KnownAssignedXidsCompress(false);
5079+
KnownAssignedXidsCompress(KAX_PRUNE, true);
50075080
}
50085081

50095082
/*

‎src/include/storage/procarray.h

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ extern void ExpireTreeKnownAssignedTransactionIds(TransactionId xid,
3939
TransactionIdmax_xid);
4040
externvoidExpireAllKnownAssignedTransactionIds(void);
4141
externvoidExpireOldKnownAssignedTransactionIds(TransactionIdxid);
42+
externvoidKnownAssignedTransactionIdsIdleMaintenance(void);
4243

4344
externintGetMaxSnapshotXidCount(void);
4445
externintGetMaxSnapshotSubxidCount(void);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp