Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit1213cb4

Browse files
committed
Trim TIDs during parallel GIN builds more eagerly
The parallel GIN builds perform "freezing" of TID lists when mergingchunks built earlier. This means determining what part of the list canno longer change, depending on the last received chunk. The frozen partcan be evicted from memory and written out.The code attempted to freeze items right before merging the old and newTID list, after already attempting to trim the current buffer. Thatmeans part of the data may get frozen based on the new TID list, butwill be trimmed later (on next loop). This increases memory usage.This inverts the order, so that we freeze data first (before trimming).The benefits are likely relatively small, but it's also virtually freewith no other downsides.Discussion:https://postgr.es/m/CAHLJuCWDwn-PE2BMZE4Kux7x5wWt_6RoWtA0mUQffEDLeZ6sfA@mail.gmail.com
1 parent6d2ff1d commit1213cb4

File tree

1 file changed

+36
-42
lines changed

1 file changed

+36
-42
lines changed

‎src/backend/access/gin/gininsert.c‎

Lines changed: 36 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1401,10 +1401,46 @@ GinBufferKeyEquals(GinBuffer *buffer, GinTuple *tup)
14011401
* enough TIDs to trim (with values less than "first" TID from the new tuple),
14021402
* we do the trim. By enough we mean at least 128 TIDs (mostly an arbitrary
14031403
* number).
1404+
*
1405+
* We try freezing TIDs at the beginning of the list first, before attempting
1406+
* to trim the buffer. This may allow trimming the data earlier, reducing the
1407+
* memory usage and excluding it from the mergesort.
14041408
*/
14051409
staticbool
14061410
GinBufferShouldTrim(GinBuffer*buffer,GinTuple*tup)
14071411
{
1412+
/*
1413+
* Check if the last TID in the current list is frozen. This is the case
1414+
* when merging non-overlapping lists, e.g. in each parallel worker.
1415+
*/
1416+
if ((buffer->nitems>0)&&
1417+
(ItemPointerCompare(&buffer->items[buffer->nitems-1],
1418+
GinTupleGetFirst(tup))==0))
1419+
buffer->nfrozen=buffer->nitems;
1420+
1421+
/*
1422+
* Now find the last TID we know to be frozen, i.e. the last TID right
1423+
* before the new GIN tuple.
1424+
*
1425+
* Start with the first not-yet-frozen tuple, and walk until we find the
1426+
* first TID that's higher. If we already know the whole list is frozen
1427+
* (i.e. nfrozen == nitems), this does nothing.
1428+
*
1429+
* XXX This might do a binary search for sufficiently long lists, but it
1430+
* does not seem worth the complexity. Overlapping lists should be rare
1431+
* common, TID comparisons are cheap, and we should quickly freeze most of
1432+
* the list.
1433+
*/
1434+
for (inti=buffer->nfrozen;i<buffer->nitems;i++)
1435+
{
1436+
/* Is the TID after the first TID of the new tuple? Can't freeze. */
1437+
if (ItemPointerCompare(&buffer->items[i],
1438+
GinTupleGetFirst(tup))>0)
1439+
break;
1440+
1441+
buffer->nfrozen++;
1442+
}
1443+
14081444
/* not enough TIDs to trim (1024 is somewhat arbitrary number) */
14091445
if (buffer->nfrozen<1024)
14101446
return false;
@@ -1470,48 +1506,6 @@ GinBufferStoreTuple(GinBuffer *buffer, GinTuple *tup)
14701506
buffer->key= (Datum)0;
14711507
}
14721508

1473-
/*
1474-
* Try freeze TIDs at the beginning of the list, i.e. exclude them from
1475-
* the mergesort. We can do that with TIDs before the first TID in the new
1476-
* tuple we're about to add into the buffer.
1477-
*
1478-
* We do this incrementally when adding data into the in-memory buffer,
1479-
* and not later (e.g. when hitting a memory limit), because it allows us
1480-
* to skip the frozen data during the mergesort, making it cheaper.
1481-
*/
1482-
1483-
/*
1484-
* Check if the last TID in the current list is frozen. This is the case
1485-
* when merging non-overlapping lists, e.g. in each parallel worker.
1486-
*/
1487-
if ((buffer->nitems>0)&&
1488-
(ItemPointerCompare(&buffer->items[buffer->nitems-1],
1489-
GinTupleGetFirst(tup))==0))
1490-
buffer->nfrozen=buffer->nitems;
1491-
1492-
/*
1493-
* Now find the last TID we know to be frozen, i.e. the last TID right
1494-
* before the new GIN tuple.
1495-
*
1496-
* Start with the first not-yet-frozen tuple, and walk until we find the
1497-
* first TID that's higher. If we already know the whole list is frozen
1498-
* (i.e. nfrozen == nitems), this does nothing.
1499-
*
1500-
* XXX This might do a binary search for sufficiently long lists, but it
1501-
* does not seem worth the complexity. Overlapping lists should be rare
1502-
* common, TID comparisons are cheap, and we should quickly freeze most of
1503-
* the list.
1504-
*/
1505-
for (inti=buffer->nfrozen;i<buffer->nitems;i++)
1506-
{
1507-
/* Is the TID after the first TID of the new tuple? Can't freeze. */
1508-
if (ItemPointerCompare(&buffer->items[i],
1509-
GinTupleGetFirst(tup))>0)
1510-
break;
1511-
1512-
buffer->nfrozen++;
1513-
}
1514-
15151509
/* add the new TIDs into the buffer, combine using merge-sort */
15161510
{
15171511
intnnew;

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp