Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit4e7395d

Browse files
committed
Do not decode TOAST data for table rewrites
During table rewrites (VACUUM FULL and CLUSTER), the main heap is loggedusing XLOG / FPI records, and thus (correctly) ignored in decoding.But the associated TOAST table is WAL-logged as plain INSERT records,and so was logically decoded and passed to reorder buffer.That has severe consequences with TOAST tables of non-trivial size.Firstly, reorder buffer has to keep all those changes, possibly spillingthem to a file, incurring I/O costs and disk space.Secondly, ReoderBufferCommit() was stashing all those TOAST chunks intoa hash table, which got discarded only after processing the row from themain heap. But as the main heap is not decoded for rewrites, this neverhappened, so all the TOAST data accumulated in memory, resulting eitherin excessive memory consumption or OOM.The fix is simple, as commite9edc1b already introduced infrastructure(namely HEAP_INSERT_NO_LOGICAL flag) to skip logical decoding of TOASTtables, but it only applied it to system tables. So simply use it forall TOAST data in raw_heap_insert().That would however solve only the memory consumption issue - the TOASTchanges would still be decoded and added to the reorder buffer, andspilled to disk (although without TOAST tuple data, so much smaller).But we can solve that by tweaking DecodeInsert() to just ignore suchINSERT records altogether, using XLH_INSERT_CONTAINS_NEW_TUPLE flag,instead of skipping them later in ReorderBufferCommit().Review: Masahiko SawadaDiscussion:https://www.postgresql.org/message-id/flat/1a17c643-e9af-3dba-486b-fbe31bc1823a%402ndquadrant.comBackpatch: 9.4-, where logical decoding was introduced
1 parenta0b4a63 commit4e7395d

File tree

3 files changed

+24
-24
lines changed

3 files changed

+24
-24
lines changed

‎src/backend/access/heap/rewriteheap.c

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -658,12 +658,11 @@ raw_heap_insert(RewriteState state, HeapTuple tup)
658658
options |=HEAP_INSERT_SKIP_WAL;
659659

660660
/*
661-
*The new relfilenode's relcache entrye doesn't have the necessary
662-
*information to determine whether a relation should emit data for
663-
*logical decoding. Force it to off if necessary.
661+
*While rewriting the heap for VACUUM FULL / CLUSTER, make sure data
662+
*for the TOAST table are not logically decoded. The main heap is
663+
*WAL-logged as XLOG FPI records, which are not logically decoded.
664664
*/
665-
if (!RelationIsLogicallyLogged(state->rs_old_rel))
666-
options |=HEAP_INSERT_NO_LOGICAL;
665+
options |=HEAP_INSERT_NO_LOGICAL;
667666

668667
heaptup=toast_insert_or_update(state->rs_new_rel,tup,NULL,
669668
options);

‎src/backend/replication/logical/decode.c

Lines changed: 15 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -639,13 +639,23 @@ DecodeAbort(LogicalDecodingContext *ctx, XLogRecordBuffer *buf,
639639
staticvoid
640640
DecodeInsert(LogicalDecodingContext*ctx,XLogRecordBuffer*buf)
641641
{
642+
Sizedatalen;
643+
char*tupledata;
644+
Sizetuplelen;
642645
XLogReaderState*r=buf->record;
643646
xl_heap_insert*xlrec;
644647
ReorderBufferChange*change;
645648
RelFileNodetarget_node;
646649

647650
xlrec= (xl_heap_insert*)XLogRecGetData(r);
648651

652+
/*
653+
* Ignore insert records without new tuples (this does happen when
654+
* raw_heap_insert marks the TOAST record as HEAP_INSERT_NO_LOGICAL).
655+
*/
656+
if (!(xlrec->flags&XLH_INSERT_CONTAINS_NEW_TUPLE))
657+
return;
658+
649659
/* only interested in our database */
650660
XLogRecGetBlockTag(r,0,&target_node,NULL,NULL);
651661
if (target_node.dbNode!=ctx->slot->data.database)
@@ -664,17 +674,13 @@ DecodeInsert(LogicalDecodingContext *ctx, XLogRecordBuffer *buf)
664674

665675
memcpy(&change->data.tp.relnode,&target_node,sizeof(RelFileNode));
666676

667-
if (xlrec->flags&XLH_INSERT_CONTAINS_NEW_TUPLE)
668-
{
669-
Sizedatalen;
670-
char*tupledata=XLogRecGetBlockData(r,0,&datalen);
671-
Sizetuplelen=datalen-SizeOfHeapHeader;
677+
tupledata=XLogRecGetBlockData(r,0,&datalen);
678+
tuplelen=datalen-SizeOfHeapHeader;
672679

673-
change->data.tp.newtuple=
674-
ReorderBufferGetTupleBuf(ctx->reorder,tuplelen);
680+
change->data.tp.newtuple=
681+
ReorderBufferGetTupleBuf(ctx->reorder,tuplelen);
675682

676-
DecodeXLogTuple(tupledata,datalen,change->data.tp.newtuple);
677-
}
683+
DecodeXLogTuple(tupledata,datalen,change->data.tp.newtuple);
678684

679685
change->data.tp.clear_toast_afterwards= true;
680686

‎src/backend/replication/logical/reorderbuffer.c

Lines changed: 5 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1599,17 +1599,12 @@ ReorderBufferCommit(ReorderBuffer *rb, TransactionId xid,
15991599
* transaction's changes. Otherwise it will get
16001600
* freed/reused while restoring spooled data from
16011601
* disk.
1602-
*
1603-
* But skip doing so if there's no tuple-data. That
1604-
* happens if a non-mapped system catalog with a toast
1605-
* table is rewritten.
16061602
*/
1607-
if (change->data.tp.newtuple!=NULL)
1608-
{
1609-
dlist_delete(&change->node);
1610-
ReorderBufferToastAppendChunk(rb,txn,relation,
1611-
change);
1612-
}
1603+
Assert(change->data.tp.newtuple!=NULL);
1604+
1605+
dlist_delete(&change->node);
1606+
ReorderBufferToastAppendChunk(rb,txn,relation,
1607+
change);
16131608
}
16141609

16151610
change_done:

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp