Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit941697c

Browse files
committed
snapshot scalability: Introduce dense array of in-progress xids.
The new array contains the xids for all connected backends / in-usePGPROC entries in a dense manner (in contrast to the PGPROC/PGXACTarrays which can have unused entries interspersed).This improves performance because GetSnapshotData() always needs toscan the xids of all live procarray entries and now there's no need togo through the procArray->pgprocnos indirection anymore.As the set of running top-level xids changes rarely, compared to thenumber of snapshots taken, this substantially increases the likelihoodof most data required for a snapshot being in l2 cache. Inread-mostly workloads scanning the xids[] array will sufficient tobuild a snapshot, as most backends will not have an xid assigned.To keep the xid array dense ProcArrayRemove() needs to move entriesbehind the to-be-removed proc's one further up in the array. Obviouslymoving array entries cannot happen while a backend sets itxid. I.e. locking needs to prevent that array entries are moved whilea backend modifies its xid.To avoid locking ProcArrayLock in GetNewTransactionId() - a fairly hotspot already - ProcArrayAdd() / ProcArrayRemove() now needs to holdXidGenLock in addition to ProcArrayLock. Adding / Removing a procarrayentry is not a very frequent operation, even taking 2PC into account.Due to the above, the dense array entries can only be read or modifiedwhile holding ProcArrayLock and/or XidGenLock. This prevents aconcurrent ProcArrayRemove() from shifting the dense array while it isaccessed concurrently.While the new dense array is very good when needing to look at allxids it is less suitable when accessing a single backend's xid. Inparticular it would be problematic to have to acquire a lock to accessa backend's own xid. Therefore a backend's xid is not just stored inthe dense array, but also in PGPROC. This also allows a backend toonly access the shared xid value when the backend had acquired anxid.The infrastructure added in this commit will be used for the remainingPGXACT fields in subsequent commits. They are kept separate to makereview easier.Author: Andres Freund <andres@anarazel.de>Reviewed-By: Robert Haas <robertmhaas@gmail.com>Reviewed-By: Thomas Munro <thomas.munro@gmail.com>Reviewed-By: David Rowley <dgrowleyml@gmail.com>Discussion:https://postgr.es/m/20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de
1 parent2ba5b2d commit941697c

File tree

11 files changed

+327
-154
lines changed

11 files changed

+327
-154
lines changed

‎src/backend/access/heap/heapam_visibility.c

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@
1111
* shared buffer content lock on the buffer containing the tuple.
1212
*
1313
* NOTE: When using a non-MVCC snapshot, we must check
14-
* TransactionIdIsInProgress (which looks in thePGXACT array)
14+
* TransactionIdIsInProgress (which looks in thePGPROC array)
1515
* before TransactionIdDidCommit/TransactionIdDidAbort (which look in
1616
* pg_xact). Otherwise we have a race condition: we might decide that a
1717
* just-committed transaction crashed, because none of the tests succeed.
1818
* xact.c is careful to record commit/abort in pg_xact before it unsets
19-
*MyPgXact->xid in thePGXACT array. That fixes that problem, but it
19+
*MyProc->xid in thePGPROC array. That fixes that problem, but it
2020
* also means there is a window where TransactionIdIsInProgress and
2121
* TransactionIdDidCommit will both return true. If we check only
2222
* TransactionIdDidCommit, we could consider a tuple committed when a
@@ -956,7 +956,7 @@ HeapTupleSatisfiesDirty(HeapTuple htup, Snapshot snapshot,
956956
* coding where we tried to set the hint bits as soon as possible, we instead
957957
* did TransactionIdIsInProgress in each call --- to no avail, as long as the
958958
* inserting/deleting transaction was still running --- which was more cycles
959-
* and more contention onthe PGXACT array.
959+
* and more contention onProcArrayLock.
960960
*/
961961
staticbool
962962
HeapTupleSatisfiesMVCC(HeapTuplehtup,Snapshotsnapshot,
@@ -1459,7 +1459,7 @@ HeapTupleSatisfiesNonVacuumable(HeapTuple htup, Snapshot snapshot,
14591459
*HeapTupleSatisfiesMVCC) and, therefore, any hint bits that can be set
14601460
*should already be set. We assume that if no hint bits are set, the xmin
14611461
*or xmax transaction is still running. This is therefore faster than
1462-
*HeapTupleSatisfiesVacuum, because wedon'tconsultPGXACT nor CLOG.
1462+
*HeapTupleSatisfiesVacuum, because we consultneither procarray nor CLOG.
14631463
*It's okay to return false when in doubt, but we must return true only
14641464
*if the tuple is removable.
14651465
*/

‎src/backend/access/transam/README

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -251,10 +251,10 @@ enforce, and it assists with some other issues as explained below.) The
251251
implementation of this is that GetSnapshotData takes the ProcArrayLock in
252252
shared mode (so that multiple backends can take snapshots in parallel),
253253
but ProcArrayEndTransaction must take the ProcArrayLock in exclusive mode
254-
while clearingMyPgXact->xidat transaction end (either commit or abort).
255-
(To reduce context switching, when multiple transactions commit nearly
256-
simultaneously, we have one backend take ProcArrayLock and clear the XIDs
257-
of multiple processes at once.)
254+
while clearingthe ProcGlobal->xids[] entryat transaction end (either
255+
commit or abort).(To reduce context switching, when multiple transactions
256+
commit nearlysimultaneously, we have one backend take ProcArrayLock and
257+
clear the XIDsof multiple processes at once.)
258258

259259
ProcArrayEndTransaction also holds the lock while advancing the shared
260260
latestCompletedXid variable. This allows GetSnapshotData to use
@@ -278,12 +278,12 @@ present in the ProcArray, or not running anymore. (This guarantee doesn't
278278
apply to subtransaction XIDs, because of the possibility that there's not
279279
room for them in the subxid array; instead we guarantee that they are
280280
present or the overflow flag is set.) If a backend released XidGenLock
281-
before storing its XID intoMyPgXact, then it would be possible for another
282-
backend to allocate and commit a later XID, causing latestCompletedXid to
283-
pass the first backend's XID, before that value became visible in the
281+
before storing its XID intoProcGlobal->xids[], then it would be possible for
282+
anotherbackend to allocate and commit a later XID, causing latestCompletedXid
283+
topass the first backend's XID, before that value became visible in the
284284
ProcArray. That would break ComputeXidHorizons, as discussed below.
285285

286-
We allow GetNewTransactionId to store the XID intoMyPgXact->xid (or the
286+
We allow GetNewTransactionId to store the XID intoProcGlobal->xids[] (or the
287287
subxid array) without taking ProcArrayLock. This was once necessary to
288288
avoid deadlock; while that is no longer the case, it's still beneficial for
289289
performance. We are thereby relying on fetch/store of an XID to be atomic,
@@ -382,12 +382,13 @@ Top-level transactions do not have a parent, so they leave their pg_subtrans
382382
entries set to the default value of zero (InvalidTransactionId).
383383

384384
pg_subtrans is used to check whether the transaction in question is still
385-
running --- the main Xid of a transaction is recorded in the PGXACT struct,
386-
but since we allow arbitrary nesting of subtransactions, we can't fit all Xids
387-
in shared memory, so we have to store them on disk. Note, however, that for
388-
each transaction we keep a "cache" of Xids that are known to be part of the
389-
transaction tree, so we can skip looking at pg_subtrans unless we know the
390-
cache has been overflowed. See storage/ipc/procarray.c for the gory details.
385+
running --- the main Xid of a transaction is recorded in ProcGlobal->xids[],
386+
with a copy in PGPROC->xid, but since we allow arbitrary nesting of
387+
subtransactions, we can't fit all Xids in shared memory, so we have to store
388+
them on disk. Note, however, that for each transaction we keep a "cache" of
389+
Xids that are known to be part of the transaction tree, so we can skip looking
390+
at pg_subtrans unless we know the cache has been overflowed. See
391+
storage/ipc/procarray.c for the gory details.
391392

392393
slru.c is the supporting mechanism for both pg_xact and pg_subtrans. It
393394
implements the LRU policy for in-memory buffer pages. The high-level routines

‎src/backend/access/transam/clog.c

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -285,15 +285,15 @@ TransactionIdSetPageStatus(TransactionId xid, int nsubxids,
285285
* updates for multiple backends so that the number of times XactSLRULock
286286
* needs to be acquired is reduced.
287287
*
288-
* For this optimization to be safe, the XIDin MyPgXact and the subxids
289-
*in MyProc must bethe same as the ones for which we're setting the
290-
*status. Check thatthis is the case.
288+
* For this optimization to be safe, the XIDand subxids in MyProc must be
289+
* the same as the ones for which we're setting the status. Check that
290+
* this is the case.
291291
*
292292
* For this optimization to be efficient, we shouldn't have too many
293293
* sub-XIDs and all of the XIDs for which we're adjusting clog should be
294294
* on the same page. Check those conditions, too.
295295
*/
296-
if (all_xact_same_page&&xid==MyPgXact->xid&&
296+
if (all_xact_same_page&&xid==MyProc->xid&&
297297
nsubxids <=THRESHOLD_SUBTRANS_CLOG_OPT&&
298298
nsubxids==MyPgXact->nxids&&
299299
memcmp(subxids,MyProc->subxids.xids,

‎src/backend/access/transam/twophase.c

Lines changed: 13 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -351,7 +351,7 @@ AtAbort_Twophase(void)
351351

352352
/*
353353
* This is called after we have finished transferring state to the prepared
354-
*PGXACT entry.
354+
*PGPROC entry.
355355
*/
356356
void
357357
PostPrepare_Twophase(void)
@@ -463,7 +463,7 @@ MarkAsPreparingGuts(GlobalTransaction gxact, TransactionId xid, const char *gid,
463463
proc->waitStatus=PROC_WAIT_STATUS_OK;
464464
/* We set up the gxact's VXID as InvalidBackendId/XID */
465465
proc->lxid= (LocalTransactionId)xid;
466-
pgxact->xid=xid;
466+
proc->xid=xid;
467467
Assert(proc->xmin==InvalidTransactionId);
468468
proc->delayChkpt= false;
469469
pgxact->vacuumFlags=0;
@@ -768,7 +768,6 @@ pg_prepared_xact(PG_FUNCTION_ARGS)
768768
{
769769
GlobalTransactiongxact=&status->array[status->currIdx++];
770770
PGPROC*proc=&ProcGlobal->allProcs[gxact->pgprocno];
771-
PGXACT*pgxact=&ProcGlobal->allPgXact[gxact->pgprocno];
772771
Datumvalues[5];
773772
boolnulls[5];
774773
HeapTupletuple;
@@ -783,7 +782,7 @@ pg_prepared_xact(PG_FUNCTION_ARGS)
783782
MemSet(values,0,sizeof(values));
784783
MemSet(nulls,0,sizeof(nulls));
785784

786-
values[0]=TransactionIdGetDatum(pgxact->xid);
785+
values[0]=TransactionIdGetDatum(proc->xid);
787786
values[1]=CStringGetTextDatum(gxact->gid);
788787
values[2]=TimestampTzGetDatum(gxact->prepared_at);
789788
values[3]=ObjectIdGetDatum(gxact->owner);
@@ -829,9 +828,8 @@ TwoPhaseGetGXact(TransactionId xid, bool lock_held)
829828
for (i=0;i<TwoPhaseState->numPrepXacts;i++)
830829
{
831830
GlobalTransactiongxact=TwoPhaseState->prepXacts[i];
832-
PGXACT*pgxact=&ProcGlobal->allPgXact[gxact->pgprocno];
833831

834-
if (pgxact->xid==xid)
832+
if (gxact->xid==xid)
835833
{
836834
result=gxact;
837835
break;
@@ -987,8 +985,7 @@ void
987985
StartPrepare(GlobalTransactiongxact)
988986
{
989987
PGPROC*proc=&ProcGlobal->allProcs[gxact->pgprocno];
990-
PGXACT*pgxact=&ProcGlobal->allPgXact[gxact->pgprocno];
991-
TransactionIdxid=pgxact->xid;
988+
TransactionIdxid=gxact->xid;
992989
TwoPhaseFileHeaderhdr;
993990
TransactionId*children;
994991
RelFileNode*commitrels;
@@ -1140,15 +1137,15 @@ EndPrepare(GlobalTransaction gxact)
11401137

11411138
/*
11421139
* Mark the prepared transaction as valid. As soon as xact.c marks
1143-
*MyPgXact as not running our XID (which it will do immediately after
1140+
*MyProc as not running our XID (which it will do immediately after
11441141
* this function returns), others can commit/rollback the xact.
11451142
*
11461143
* NB: a side effect of this is to make a dummy ProcArray entry for the
1147-
* prepared XID. This must happen before we clear the XID fromMyPgXact,
1148-
* else there is a window where the XID is not running according to
1149-
* TransactionIdIsInProgress, and onlookers would be entitled to assume
1150-
* the xact crashed. Instead we have a window where the same XID appears
1151-
* twice in ProcArray, which is OK.
1144+
* prepared XID. This must happen before we clear the XID fromMyProc /
1145+
*ProcGlobal->xids[],else there is a window where the XID is not running
1146+
*according toTransactionIdIsInProgress, and onlookers would be entitled
1147+
*to assumethe xact crashed. Instead we have a window where the same
1148+
*XID appearstwice in ProcArray, which is OK.
11521149
*/
11531150
MarkAsPrepared(gxact, false);
11541151

@@ -1404,7 +1401,6 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
14041401
{
14051402
GlobalTransactiongxact;
14061403
PGPROC*proc;
1407-
PGXACT*pgxact;
14081404
TransactionIdxid;
14091405
char*buf;
14101406
char*bufptr;
@@ -1423,8 +1419,7 @@ FinishPreparedTransaction(const char *gid, bool isCommit)
14231419
*/
14241420
gxact=LockGXact(gid,GetUserId());
14251421
proc=&ProcGlobal->allProcs[gxact->pgprocno];
1426-
pgxact=&ProcGlobal->allPgXact[gxact->pgprocno];
1427-
xid=pgxact->xid;
1422+
xid=gxact->xid;
14281423

14291424
/*
14301425
* Read and validate 2PC state data. State data will typically be stored
@@ -1726,7 +1721,7 @@ CheckPointTwoPhase(XLogRecPtr redo_horizon)
17261721
for (i=0;i<TwoPhaseState->numPrepXacts;i++)
17271722
{
17281723
/*
1729-
* Note that we are using gxact notpgxact so this works in recovery
1724+
* Note that we are using gxact notPGPROC so this works in recovery
17301725
* also
17311726
*/
17321727
GlobalTransactiongxact=TwoPhaseState->prepXacts[i];

‎src/backend/access/transam/varsup.c

Lines changed: 13 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,8 @@ VariableCache ShmemVariableCache = NULL;
3838
* Allocate the next FullTransactionId for a new transaction or
3939
* subtransaction.
4040
*
41-
* The new XID is also stored into MyPgXact before returning.
41+
* The new XID is also stored into MyProc->xid/ProcGlobal->xids[] before
42+
* returning.
4243
*
4344
* Note: when this is called, we are actually already inside a valid
4445
* transaction, since XIDs are now not allocated until the transaction
@@ -65,7 +66,8 @@ GetNewTransactionId(bool isSubXact)
6566
if (IsBootstrapProcessingMode())
6667
{
6768
Assert(!isSubXact);
68-
MyPgXact->xid=BootstrapTransactionId;
69+
MyProc->xid=BootstrapTransactionId;
70+
ProcGlobal->xids[MyProc->pgxactoff]=BootstrapTransactionId;
6971
returnFullTransactionIdFromEpochAndXid(0,BootstrapTransactionId);
7072
}
7173

@@ -190,10 +192,10 @@ GetNewTransactionId(bool isSubXact)
190192
* latestCompletedXid is present in the ProcArray, which is essential for
191193
* correct OldestXmin tracking; see src/backend/access/transam/README.
192194
*
193-
* Note that readers ofPGXACTxidfieldsshould be careful to fetch the
194-
* value only once, rather than assume they can read a value multiple
195-
* times and get the same answer each time. Note we are assuming that
196-
* TransactionId and int fetch/store are atomic.
195+
* Note that readers ofProcGlobal->xids/PGPROC->xid should be careful
196+
*to fetch thevaluefor each proconly once, rather than assume they can
197+
*read a value multipletimes and get the same answer each time. Note we
198+
*are assuming thatTransactionId and int fetch/store are atomic.
197199
*
198200
* The same comments apply to the subxact xid count and overflow fields.
199201
*
@@ -219,7 +221,11 @@ GetNewTransactionId(bool isSubXact)
219221
* answer later on when someone does have a reason to inquire.)
220222
*/
221223
if (!isSubXact)
222-
MyPgXact->xid=xid;/* LWLockRelease acts as barrier */
224+
{
225+
/* LWLockRelease acts as barrier */
226+
MyProc->xid=xid;
227+
ProcGlobal->xids[MyProc->pgxactoff]=xid;
228+
}
223229
else
224230
{
225231
intnxids=MyPgXact->nxids;

‎src/backend/commands/vacuum.c

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1724,7 +1724,7 @@ vacuum_rel(Oid relid, RangeVar *relation, VacuumParams *params)
17241724
*
17251725
* Note: these flags remain set until CommitTransaction or
17261726
* AbortTransaction. We don't want to clear them until we reset
1727-
*MyPgXact->xid/xmin, otherwise GetOldestNonRemovableTransactionId()
1727+
*MyProc->xid/xmin, otherwise GetOldestNonRemovableTransactionId()
17281728
* might appear to go backwards, which is probably Not Good.
17291729
*/
17301730
LWLockAcquire(ProcArrayLock,LW_EXCLUSIVE);

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp