Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit3f8c8e3

Browse files
committed
Fix failure to detoast fields in composite elements of structured types.
If we have an array of records stored on disk, the individual record fieldscannot contain out-of-line TOAST pointers: the tuptoaster.c mechanisms areonly prepared to deal with TOAST pointers appearing in top-level fields ofa stored row. The same applies for ranges over composite types, nestedcomposites, etc. However, the existing code only took care of expandingsub-field TOAST pointers for the case of nested composites, not for otherstructured types containing composites. For example, given a command suchasUPDATE tab SET arraycol = ARRAY[(ROW(x,42)::mycompositetype] ...where x is a direct reference to a field of an on-disk tuple, if that fieldis long enough to be toasted out-of-line then the TOAST pointer would beinserted as-is into the array column. If the source record for x is laterdeleted, the array field value would become a dangling pointer, leadingto errors along the line of "missing chunk number 0 for toast value ..."when the value is referenced. A reproducible test case for this wasprovided by Jan Pecek, but it seems likely that some of the "missing chunknumber" reports we've heard in the past were caused by similar issues.Code-wise, the problem is that PG_DETOAST_DATUM() is not adequate toproduce a self-contained Datum value if the Datum is of composite type.Seen in this light, the problem is not just confined to arrays and ranges,but could also affect some other places where detoasting is done in thatway, for example form_index_tuple().I tried teaching the array code to apply toast_flatten_tuple_attribute()along with PG_DETOAST_DATUM() when the array element type is composite,but this was messy and imposed extra cache lookup costs whether or not anyTOAST pointers were present, indeed sometimes when the array element typeisn't even composite (since sometimes it takes a typcache lookup to findthat out). The idea of extending that approach to all the places thatcurrently use PG_DETOAST_DATUM() wasn't attractive at all.This patch instead solves the problem by decreeing that composite Datumvalues must not contain any out-of-line TOAST pointers in the first place;that is, we expand out-of-line fields at the point of constructing acomposite Datum, not at the point where we're about to insert it into alarger tuple. This rule is applied only to true composite Datums, notto tuples that are being passed around the system as tuples, so it's notas invasive as it might sound at first. With this approach, the amountof code that has to be touched for a full solution is greatly reduced,and added cache lookup costs are avoided except when there actually isa TOAST pointer that needs to be inlined.The main drawback of this approach is that we might sometimes dereferencea TOAST pointer that will never actually be used by the query, imposing arather large cost that wasn't there before. On the other side of the coin,if the field value is used multiple times then we'll come out ahead byavoiding repeat detoastings. Experimentation suggests that common SQLcoding patterns are unaffected either way, though. Applications that arevery negatively affected could be advised to modify their code to not fetchcolumns they won't be using.In future, we might consider reverting this solution in favor of detoastingonly at the point where data is about to be stored to disk, using somemethod that can drill down into multiple levels of nested structured types.That will require defining new APIs for structured types, though, so itdoesn't seem feasible as a back-patchable fix.Note that this patch changes HeapTupleGetDatum() from a macro to a functioncall; this means that any third-party code using that macro will not getprotection against creating TOAST-pointer-containing Datums until it'srecompiled. The same applies to any uses of PG_RETURN_HEAPTUPLEHEADER().It seems likely that this is not a big problem in practice: most of thetuple-returning functions in core and contrib produce outputs that couldnot possibly be toasted anyway, and the same probably holds for third-partyextensions.This bug has existed since TOAST was invented, so back-patch to allsupported branches.
1 parent65fb5ff commit3f8c8e3

File tree

16 files changed

+245
-151
lines changed

16 files changed

+245
-151
lines changed

‎src/backend/access/common/heaptuple.c

Lines changed: 43 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -617,6 +617,41 @@ heap_copytuple_with_tuple(HeapTuple src, HeapTuple dest)
617617
memcpy((char*)dest->t_data, (char*)src->t_data,src->t_len);
618618
}
619619

620+
/* ----------------
621+
*heap_copy_tuple_as_datum
622+
*
623+
*copy a tuple as a composite-type Datum
624+
* ----------------
625+
*/
626+
Datum
627+
heap_copy_tuple_as_datum(HeapTupletuple,TupleDesctupleDesc)
628+
{
629+
HeapTupleHeadertd;
630+
631+
/*
632+
* If the tuple contains any external TOAST pointers, we have to inline
633+
* those fields to meet the conventions for composite-type Datums.
634+
*/
635+
if (HeapTupleHasExternal(tuple))
636+
returntoast_flatten_tuple_to_datum(tuple->t_data,
637+
tuple->t_len,
638+
tupleDesc);
639+
640+
/*
641+
* Fast path for easy case: just make a palloc'd copy and insert the
642+
* correct composite-Datum header fields (since those may not be set if
643+
* the given tuple came from disk, rather than from heap_form_tuple).
644+
*/
645+
td= (HeapTupleHeader)palloc(tuple->t_len);
646+
memcpy((char*)td, (char*)tuple->t_data,tuple->t_len);
647+
648+
HeapTupleHeaderSetDatumLength(td,tuple->t_len);
649+
HeapTupleHeaderSetTypeId(td,tupleDesc->tdtypeid);
650+
HeapTupleHeaderSetTypMod(td,tupleDesc->tdtypmod);
651+
652+
returnPointerGetDatum(td);
653+
}
654+
620655
/*
621656
* heap_form_tuple
622657
*construct a tuple from the given values[] and isnull[] arrays,
@@ -635,7 +670,6 @@ heap_form_tuple(TupleDesc tupleDescriptor,
635670
data_len;
636671
inthoff;
637672
boolhasnull= false;
638-
Form_pg_attribute*att=tupleDescriptor->attrs;
639673
intnumberOfAttributes=tupleDescriptor->natts;
640674
inti;
641675

@@ -646,28 +680,14 @@ heap_form_tuple(TupleDesc tupleDescriptor,
646680
numberOfAttributes,MaxTupleAttributeNumber)));
647681

648682
/*
649-
* Check for nulls and embedded tuples; expand any toasted attributes in
650-
* embedded tuples. This preserves the invariant that toasting can only
651-
* go one level deep.
652-
*
653-
* We can skip calling toast_flatten_tuple_attribute() if the attribute
654-
* couldn't possibly be of composite type. All composite datums are
655-
* varlena and have alignment 'd'; furthermore they aren't arrays. Also,
656-
* if an attribute is already toasted, it must have been sent to disk
657-
* already and so cannot contain toasted attributes.
683+
* Check for nulls
658684
*/
659685
for (i=0;i<numberOfAttributes;i++)
660686
{
661687
if (isnull[i])
662-
hasnull= true;
663-
elseif (att[i]->attlen==-1&&
664-
att[i]->attalign=='d'&&
665-
att[i]->attndims==0&&
666-
!VARATT_IS_EXTENDED(DatumGetPointer(values[i])))
667688
{
668-
values[i]=toast_flatten_tuple_attribute(values[i],
669-
att[i]->atttypid,
670-
att[i]->atttypmod);
689+
hasnull= true;
690+
break;
671691
}
672692
}
673693

@@ -697,7 +717,8 @@ heap_form_tuple(TupleDesc tupleDescriptor,
697717

698718
/*
699719
* And fill in the information. Note we fill the Datum fields even though
700-
* this tuple may never become a Datum.
720+
* this tuple may never become a Datum. This lets HeapTupleHeaderGetDatum
721+
* identify the tuple type if needed.
701722
*/
702723
tuple->t_len=len;
703724
ItemPointerSetInvalid(&(tuple->t_self));
@@ -1389,7 +1410,6 @@ heap_form_minimal_tuple(TupleDesc tupleDescriptor,
13891410
data_len;
13901411
inthoff;
13911412
boolhasnull= false;
1392-
Form_pg_attribute*att=tupleDescriptor->attrs;
13931413
intnumberOfAttributes=tupleDescriptor->natts;
13941414
inti;
13951415

@@ -1400,28 +1420,14 @@ heap_form_minimal_tuple(TupleDesc tupleDescriptor,
14001420
numberOfAttributes,MaxTupleAttributeNumber)));
14011421

14021422
/*
1403-
* Check for nulls and embedded tuples; expand any toasted attributes in
1404-
* embedded tuples. This preserves the invariant that toasting can only
1405-
* go one level deep.
1406-
*
1407-
* We can skip calling toast_flatten_tuple_attribute() if the attribute
1408-
* couldn't possibly be of composite type. All composite datums are
1409-
* varlena and have alignment 'd'; furthermore they aren't arrays. Also,
1410-
* if an attribute is already toasted, it must have been sent to disk
1411-
* already and so cannot contain toasted attributes.
1423+
* Check for nulls
14121424
*/
14131425
for (i=0;i<numberOfAttributes;i++)
14141426
{
14151427
if (isnull[i])
1416-
hasnull= true;
1417-
elseif (att[i]->attlen==-1&&
1418-
att[i]->attalign=='d'&&
1419-
att[i]->attndims==0&&
1420-
!VARATT_IS_EXTENDED(values[i]))
14211428
{
1422-
values[i]=toast_flatten_tuple_attribute(values[i],
1423-
att[i]->atttypid,
1424-
att[i]->atttypmod);
1429+
hasnull= true;
1430+
break;
14251431
}
14261432
}
14271433

‎src/backend/access/common/indextuple.c

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,11 @@ index_form_tuple(TupleDesc tupleDescriptor,
158158
if (tupmask&HEAP_HASVARWIDTH)
159159
infomask |=INDEX_VAR_MASK;
160160

161+
/* Also assert we got rid of external attributes */
162+
#ifdefTOAST_INDEX_HACK
163+
Assert((tupmask&HEAP_HASEXTERNAL)==0);
164+
#endif
165+
161166
/*
162167
* Here we make sure that the size will fit in the field reserved for it
163168
* in t_info.

‎src/backend/access/heap/tuptoaster.c

Lines changed: 44 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -991,6 +991,9 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
991991
*
992992
*"Flatten" a tuple to contain no out-of-line toasted fields.
993993
*(This does not eliminate compressed or short-header datums.)
994+
*
995+
*Note: we expect the caller already checked HeapTupleHasExternal(tup),
996+
*so there is no need for a short-circuit path.
994997
* ----------
995998
*/
996999
HeapTuple
@@ -1068,59 +1071,61 @@ toast_flatten_tuple(HeapTuple tup, TupleDesc tupleDesc)
10681071

10691072

10701073
/* ----------
1071-
* toast_flatten_tuple_attribute -
1074+
* toast_flatten_tuple_to_datum -
1075+
*
1076+
*"Flatten" a tuple containing out-of-line toasted fields into a Datum.
1077+
*The result is always palloc'd in the current memory context.
1078+
*
1079+
*We have a general rule that Datums of container types (rows, arrays,
1080+
*ranges, etc) must not contain any external TOAST pointers. Without
1081+
*this rule, we'd have to look inside each Datum when preparing a tuple
1082+
*for storage, which would be expensive and would fail to extend cleanly
1083+
*to new sorts of container types.
1084+
*
1085+
*However, we don't want to say that tuples represented as HeapTuples
1086+
*can't contain toasted fields, so instead this routine should be called
1087+
*when such a HeapTuple is being converted into a Datum.
10721088
*
1073-
*If a Datum is of composite type, "flatten" it to contain no toasted fields.
1074-
*This must be invoked on any potentially-composite field that is to be
1075-
*inserted into a tuple.Doing this preserves the invariant that toasting
1076-
*goes only one level deep in a tuple.
1089+
*While we're at it, we decompress any compressed fields too. This is not
1090+
*necessary for correctness, but reflects an expectation that compression
1091+
*will be more effective if applied to the whole tuple not individual
1092+
*fields. We are not so concerned about that that we want to deconstruct
1093+
*and reconstruct tuples just to get rid of compressed fields, however.
1094+
*So callers typically won't call this unless they see that the tuple has
1095+
*at least one external field.
10771096
*
1078-
*Note that flattening does not mean expansion of short-header varlenas,
1079-
*so in one sense toasting is allowed within composite datums.
1097+
*On the other hand, in-line short-header varlena fields are left alone.
1098+
*If we "untoasted" them here, they'd just get changed back to short-header
1099+
*format anyway within heap_fill_tuple.
10801100
* ----------
10811101
*/
10821102
Datum
1083-
toast_flatten_tuple_attribute(Datumvalue,
1084-
OidtypeId,int32typeMod)
1103+
toast_flatten_tuple_to_datum(HeapTupleHeadertup,
1104+
uint32tup_len,
1105+
TupleDesctupleDesc)
10851106
{
1086-
TupleDesctupleDesc;
1087-
HeapTupleHeaderolddata;
10881107
HeapTupleHeadernew_data;
10891108
int32new_header_len;
10901109
int32new_data_len;
10911110
int32new_tuple_len;
10921111
HeapTupleDatatmptup;
1093-
Form_pg_attribute*att;
1094-
intnumAttrs;
1112+
Form_pg_attribute*att=tupleDesc->attrs;
1113+
intnumAttrs=tupleDesc->natts;
10951114
inti;
1096-
boolneed_change= false;
10971115
boolhas_nulls= false;
10981116
Datumtoast_values[MaxTupleAttributeNumber];
10991117
booltoast_isnull[MaxTupleAttributeNumber];
11001118
booltoast_free[MaxTupleAttributeNumber];
11011119

1102-
/*
1103-
* See if it's a composite type, and get the tupdesc if so.
1104-
*/
1105-
tupleDesc=lookup_rowtype_tupdesc_noerror(typeId,typeMod, true);
1106-
if (tupleDesc==NULL)
1107-
returnvalue;/* not a composite type */
1108-
1109-
att=tupleDesc->attrs;
1110-
numAttrs=tupleDesc->natts;
1111-
1112-
/*
1113-
* Break down the tuple into fields.
1114-
*/
1115-
olddata=DatumGetHeapTupleHeader(value);
1116-
Assert(typeId==HeapTupleHeaderGetTypeId(olddata));
1117-
Assert(typeMod==HeapTupleHeaderGetTypMod(olddata));
11181120
/* Build a temporary HeapTuple control structure */
1119-
tmptup.t_len=HeapTupleHeaderGetDatumLength(olddata);
1121+
tmptup.t_len=tup_len;
11201122
ItemPointerSetInvalid(&(tmptup.t_self));
11211123
tmptup.t_tableOid=InvalidOid;
1122-
tmptup.t_data=olddata;
1124+
tmptup.t_data=tup;
11231125

1126+
/*
1127+
* Break down the tuple into fields.
1128+
*/
11241129
Assert(numAttrs <=MaxTupleAttributeNumber);
11251130
heap_deform_tuple(&tmptup,tupleDesc,toast_values,toast_isnull);
11261131

@@ -1144,20 +1149,10 @@ toast_flatten_tuple_attribute(Datum value,
11441149
new_value=heap_tuple_untoast_attr(new_value);
11451150
toast_values[i]=PointerGetDatum(new_value);
11461151
toast_free[i]= true;
1147-
need_change= true;
11481152
}
11491153
}
11501154
}
11511155

1152-
/*
1153-
* If nothing to untoast, just return the original tuple.
1154-
*/
1155-
if (!need_change)
1156-
{
1157-
ReleaseTupleDesc(tupleDesc);
1158-
returnvalue;
1159-
}
1160-
11611156
/*
11621157
* Calculate the new size of the tuple.
11631158
*
@@ -1166,7 +1161,7 @@ toast_flatten_tuple_attribute(Datum value,
11661161
new_header_len= offsetof(HeapTupleHeaderData,t_bits);
11671162
if (has_nulls)
11681163
new_header_len+=BITMAPLEN(numAttrs);
1169-
if (olddata->t_infomask&HEAP_HASOID)
1164+
if (tup->t_infomask&HEAP_HASOID)
11701165
new_header_len+=sizeof(Oid);
11711166
new_header_len=MAXALIGN(new_header_len);
11721167
new_data_len=heap_compute_data_size(tupleDesc,
@@ -1178,14 +1173,16 @@ toast_flatten_tuple_attribute(Datum value,
11781173
/*
11791174
* Copy the existing tuple header, but adjust natts and t_hoff.
11801175
*/
1181-
memcpy(new_data,olddata, offsetof(HeapTupleHeaderData,t_bits));
1176+
memcpy(new_data,tup, offsetof(HeapTupleHeaderData,t_bits));
11821177
HeapTupleHeaderSetNatts(new_data,numAttrs);
11831178
new_data->t_hoff=new_header_len;
1184-
if (olddata->t_infomask&HEAP_HASOID)
1185-
HeapTupleHeaderSetOid(new_data,HeapTupleHeaderGetOid(olddata));
1179+
if (tup->t_infomask&HEAP_HASOID)
1180+
HeapTupleHeaderSetOid(new_data,HeapTupleHeaderGetOid(tup));
11861181

1187-
/*Reset thedatum length field, too */
1182+
/*Set thecomposite-Datum header fields correctly */
11881183
HeapTupleHeaderSetDatumLength(new_data,new_tuple_len);
1184+
HeapTupleHeaderSetTypeId(new_data,tupleDesc->tdtypeid);
1185+
HeapTupleHeaderSetTypMod(new_data,tupleDesc->tdtypmod);
11891186

11901187
/* Copy over the data, and fill the null bitmap if needed */
11911188
heap_fill_tuple(tupleDesc,
@@ -1202,7 +1199,6 @@ toast_flatten_tuple_attribute(Datum value,
12021199
for (i=0;i<numAttrs;i++)
12031200
if (toast_free[i])
12041201
pfree(DatumGetPointer(toast_values[i]));
1205-
ReleaseTupleDesc(tupleDesc);
12061202

12071203
returnPointerGetDatum(new_data);
12081204
}

‎src/backend/executor/execQual.c

Lines changed: 9 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -896,8 +896,6 @@ ExecEvalWholeRowFast(WholeRowVarExprState *wrvstate, ExprContext *econtext,
896896
{
897897
Var*variable= (Var*)wrvstate->xprstate.expr;
898898
TupleTableSlot*slot;
899-
HeapTupletuple;
900-
TupleDesctupleDesc;
901899
HeapTupleHeaderdtuple;
902900

903901
if (isDone)
@@ -927,32 +925,20 @@ ExecEvalWholeRowFast(WholeRowVarExprState *wrvstate, ExprContext *econtext,
927925
if (wrvstate->wrv_junkFilter!=NULL)
928926
slot=ExecFilterJunk(wrvstate->wrv_junkFilter,slot);
929927

930-
tuple=ExecFetchSlotTuple(slot);
931-
tupleDesc=slot->tts_tupleDescriptor;
932-
933928
/*
934-
* We have to make a copy of the tuple so we can safely insert the Datum
935-
* overhead fields, which are not set in on-disk tuples.
929+
* Copy the slot tuple and make sure any toasted fields get detoasted.
936930
*/
937-
dtuple= (HeapTupleHeader)palloc(tuple->t_len);
938-
memcpy((char*)dtuple, (char*)tuple->t_data,tuple->t_len);
939-
940-
HeapTupleHeaderSetDatumLength(dtuple,tuple->t_len);
931+
dtuple=DatumGetHeapTupleHeader(ExecFetchSlotTupleDatum(slot));
941932

942933
/*
943-
* If the Var identifies a named composite type, label thetuple with that
944-
* type; otherwise usewhat is inthetupleDesc.
934+
* If the Var identifies a named composite type, label thedatum with that
935+
* type; otherwisewe'lluse theslot's info.
945936
*/
946937
if (variable->vartype!=RECORDOID)
947938
{
948939
HeapTupleHeaderSetTypeId(dtuple,variable->vartype);
949940
HeapTupleHeaderSetTypMod(dtuple,variable->vartypmod);
950941
}
951-
else
952-
{
953-
HeapTupleHeaderSetTypeId(dtuple,tupleDesc->tdtypeid);
954-
HeapTupleHeaderSetTypMod(dtuple,tupleDesc->tdtypmod);
955-
}
956942

957943
returnPointerGetDatum(dtuple);
958944
}
@@ -1029,13 +1015,13 @@ ExecEvalWholeRowSlow(WholeRowVarExprState *wrvstate, ExprContext *econtext,
10291015
}
10301016

10311017
/*
1032-
* We have to make a copy of the tuple so we can safely insert the Datum
1033-
* overhead fields, which are not set in on-disk tuples.
1018+
* Copy the slot tuple and make sure any toasted fields get detoasted.
10341019
*/
1035-
dtuple= (HeapTupleHeader)palloc(tuple->t_len);
1036-
memcpy((char*)dtuple, (char*)tuple->t_data,tuple->t_len);
1020+
dtuple=DatumGetHeapTupleHeader(ExecFetchSlotTupleDatum(slot));
10371021

1038-
HeapTupleHeaderSetDatumLength(dtuple,tuple->t_len);
1022+
/*
1023+
* Reset datum's type ID fields to match the Var.
1024+
*/
10391025
HeapTupleHeaderSetTypeId(dtuple,variable->vartype);
10401026
HeapTupleHeaderSetTypMod(dtuple,variable->vartypmod);
10411027

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp