Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitfa2fa99

Browse files
committed
Permit dump/reload of not-too-large >1GB tuples
Our documentation states that our maximum field size is 1 GB, and thatour maximum row size of 1.6 TB. However, while this might be attainablein theory with enough contortions, it is not workable in practice; forstarters, pg_dump fails to dump tables containing rows larger than 1 GB,even if individual columns are well below the limit; and even if onedoes manage to manufacture a dump file containing a row that large, theserver refuses to load it anyway.This commit enables dumping and reloading of such tuples, provided twoconditions are met:1. no single column is larger than 1 GB (in output size -- for bytea this includes the formatting overhead)2. the whole row is not larger than 2 GBThere are three related changes to enable this:a. StringInfo's API now has two additional functions that allow creatinga string that grows beyond the typical 1GB limit (and "long" string).ABI compatibility is maintained. We still limit these strings to 2 GB,though, for reasons explained below.b. COPY now uses long StringInfos, so that pg_dump doesn't choketrying to emit rows longer than 1GB.c. heap_form_tuple now uses the MCXT_ALLOW_HUGE flag in its allocationfor the input tuple, which means that large tuples are accepted oninput. Note that at this point we do not apply any further limit to theinput tuple size.The main reason to limit to 2 GB is that the FE/BE protocol uses 32 bitlength words to describe each row; and because the documentation isambiguous on its signedness and libpq does consider it signed, we cannotuse the highest-order bit. Additionally, the StringInfo API uses "int"(which is 4 bytes wide in most platforms) in many places, so we'd needto change that API too in order to improve, which has lots of fallout.Backpatch to 9.5, which is the oldest that hasMemoryContextAllocExtended, a necessary piece of infrastructure. Wecould apply to 9.4 with very minimal additional effort, but any furtherthan that would require backpatching "huge" allocations too.This is the largest set of changes we could find that can beback-patched without breaking compatibility with existing systems.Fixing a bigger set of problems (for example, dumping tuples bigger than2GB, or dumping fields bigger than 1GB) would require changing the FE/BEprotocol and/or changing the StringInfo API in an ABI-incompatible way,neither of which would be back-patchable.Authors: Daniel Vérité, Álvaro HerreraReviewed by: Tomas VondraDiscussion:https://postgr.es/m/20160229183023.GA286012@alvherre.pgsql
1 parent78c8c81 commitfa2fa99

File tree

4 files changed

+74
-22
lines changed

4 files changed

+74
-22
lines changed

‎src/backend/access/common/heaptuple.c

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -741,7 +741,9 @@ heap_form_tuple(TupleDesc tupleDescriptor,
741741
* Allocate and zero the space needed. Note that the tuple body and
742742
* HeapTupleData management structure are allocated in one chunk.
743743
*/
744-
tuple= (HeapTuple)palloc0(HEAPTUPLESIZE+len);
744+
tuple=MemoryContextAllocExtended(CurrentMemoryContext,
745+
HEAPTUPLESIZE+len,
746+
MCXT_ALLOC_HUGE |MCXT_ALLOC_ZERO);
745747
tuple->t_data=td= (HeapTupleHeader) ((char*)tuple+HEAPTUPLESIZE);
746748

747749
/*

‎src/backend/commands/copy.c

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -385,7 +385,7 @@ ReceiveCopyBegin(CopyState cstate)
385385
pq_sendint(&buf,format,2);/* per-column formats */
386386
pq_endmessage(&buf);
387387
cstate->copy_dest=COPY_NEW_FE;
388-
cstate->fe_msgbuf=makeStringInfo();
388+
cstate->fe_msgbuf=makeLongStringInfo();
389389
}
390390
else
391391
{
@@ -1907,7 +1907,7 @@ CopyTo(CopyState cstate)
19071907
cstate->null_print_client=cstate->null_print;/* default */
19081908

19091909
/* We use fe_msgbuf as a per-row buffer regardless of copy_dest */
1910-
cstate->fe_msgbuf=makeStringInfo();
1910+
cstate->fe_msgbuf=makeLongStringInfo();
19111911

19121912
/* Get info about the columns we need to process. */
19131913
cstate->out_functions= (FmgrInfo*)palloc(num_phys_attrs*sizeof(FmgrInfo));
@@ -2742,8 +2742,8 @@ BeginCopyFrom(ParseState *pstate,
27422742
cstate->cur_attval=NULL;
27432743

27442744
/* Set up variables to avoid per-attribute overhead. */
2745-
initStringInfo(&cstate->attribute_buf);
2746-
initStringInfo(&cstate->line_buf);
2745+
initLongStringInfo(&cstate->attribute_buf);
2746+
initLongStringInfo(&cstate->line_buf);
27472747
cstate->line_buf_converted= false;
27482748
cstate->raw_buf= (char*)palloc(RAW_BUF_SIZE+1);
27492749
cstate->raw_buf_index=cstate->raw_buf_len=0;

‎src/backend/lib/stringinfo.c

Lines changed: 54 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,8 @@
44
*
55
* StringInfo provides an indefinitely-extensible string data type.
66
* It can be used to buffer either ordinary C strings (null-terminated text)
7-
* or arbitrary binary data. All storage is allocated with palloc().
7+
* or arbitrary binary data. All storage is allocated with palloc() and
8+
* friends.
89
*
910
* Portions Copyright (c) 1996-2016, PostgreSQL Global Development Group
1011
* Portions Copyright (c) 1994, Regents of the University of California
@@ -36,11 +37,29 @@ makeStringInfo(void)
3637
returnres;
3738
}
3839

40+
/*
41+
* makeLongStringInfo
42+
*
43+
* Same as makeStringInfo, for larger strings.
44+
*/
45+
StringInfo
46+
makeLongStringInfo(void)
47+
{
48+
StringInfores;
49+
50+
res= (StringInfo)palloc(sizeof(StringInfoData));
51+
52+
initLongStringInfo(res);
53+
54+
returnres;
55+
}
56+
57+
3958
/*
4059
* initStringInfo
4160
*
4261
* Initialize a StringInfoData struct (with previously undefined contents)
43-
* to describe an empty string.
62+
* to describe an empty string; don't enable long strings yet.
4463
*/
4564
void
4665
initStringInfo(StringInfostr)
@@ -49,9 +68,22 @@ initStringInfo(StringInfo str)
4968

5069
str->data= (char*)palloc(size);
5170
str->maxlen=size;
71+
str->long_ok= false;
5272
resetStringInfo(str);
5373
}
5474

75+
/*
76+
* initLongStringInfo
77+
*
78+
* Same as initStringInfo, plus enable long strings.
79+
*/
80+
void
81+
initLongStringInfo(StringInfostr)
82+
{
83+
initStringInfo(str);
84+
str->long_ok= true;
85+
}
86+
5587
/*
5688
* resetStringInfo
5789
*
@@ -142,7 +174,7 @@ appendStringInfoVA(StringInfo str, const char *fmt, va_list args)
142174
/*
143175
* Return pvsnprintf's estimate of the space needed. (Although this is
144176
* given as a size_t, we know it will fit in int because it's not more
145-
* than MaxAllocSize.)
177+
* thaneitherMaxAllocSize or half an int's width.)
146178
*/
147179
return (int)nprinted;
148180
}
@@ -244,15 +276,25 @@ appendBinaryStringInfo(StringInfo str, const char *data, int datalen)
244276
void
245277
enlargeStringInfo(StringInfostr,intneeded)
246278
{
247-
intnewlen;
279+
Sizenewlen;
280+
Sizelimit;
281+
282+
/*
283+
* Determine the upper size limit. Because of overflow concerns outside
284+
* of this module, we limit ourselves to 4-byte signed integer range,
285+
* even for "long_ok" strings.
286+
*/
287+
limit=str->long_ok ?
288+
(((Size)1) << (sizeof(int32)*8-1))-1 :
289+
MaxAllocSize;
248290

249291
/*
250292
* Guard against out-of-range "needed" values. Without this, we can get
251293
* an overflow or infinite loop in the following.
252294
*/
253295
if (needed<0)/* should not happen */
254296
elog(ERROR,"invalid string enlargement request size: %d",needed);
255-
if (((Size)needed) >= (MaxAllocSize- (Size)str->len))
297+
if (((Size)needed) >= (limit- (Size)str->len))
256298
ereport(ERROR,
257299
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
258300
errmsg("out of memory"),
@@ -261,7 +303,7 @@ enlargeStringInfo(StringInfo str, int needed)
261303

262304
needed+=str->len+1;/* total space required now */
263305

264-
/* Because of the above test, we now have needed <=MaxAllocSize */
306+
/* Because of the above test, we now have needed <=limit */
265307

266308
if (needed <=str->maxlen)
267309
return;/* got enough space already */
@@ -276,14 +318,14 @@ enlargeStringInfo(StringInfo str, int needed)
276318
newlen=2*newlen;
277319

278320
/*
279-
* Clamp toMaxAllocSizein case we went past it. Note we are assuming
280-
*herethatMaxAllocSize <= INT_MAX/2, else the above loop could
281-
*overflow. We willstill have newlen >= needed.
321+
* Clamp tothe limitin case we went past it. Note we are assuming here
322+
* thatlimit <= INT_MAX/2, else the above loop could overflow. We will
323+
* still have newlen >= needed.
282324
*/
283-
if (newlen>(int)MaxAllocSize)
284-
newlen=(int)MaxAllocSize;
325+
if (newlen>limit)
326+
newlen=limit;
285327

286-
str->data= (char*)repalloc(str->data,newlen);
328+
str->data= (char*)repalloc_huge(str->data, (Size)newlen);
287329

288330
str->maxlen=newlen;
289331
}

‎src/include/lib/stringinfo.h

Lines changed: 13 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,8 @@
3030
*cursoris initialized to zero by makeStringInfo or initStringInfo,
3131
*but is not otherwise touched by the stringinfo.c routines.
3232
*Some routines use it to scan through a StringInfo.
33+
*long_ok whether this StringInfo can allocate more than MaxAllocSize
34+
*bytes (but still up to 2GB).
3335
*-------------------------
3436
*/
3537
typedefstructStringInfoData
@@ -38,6 +40,7 @@ typedef struct StringInfoData
3840
intlen;
3941
intmaxlen;
4042
intcursor;
43+
boollong_ok;
4144
}StringInfoData;
4245

4346
typedefStringInfoData*StringInfo;
@@ -46,11 +49,11 @@ typedef StringInfoData *StringInfo;
4649
/*------------------------
4750
* There are two ways to create a StringInfo object initially:
4851
*
49-
* StringInfo stringptr = makeStringInfo();
52+
* StringInfo stringptr = makeStringInfo(); // or makeLongStringInfo();
5053
*Both the StringInfoData and the data buffer are palloc'd.
5154
*
5255
* StringInfoData string;
53-
* initStringInfo(&string);
56+
* initStringInfo(&string); // or initLongStringInfo();
5457
*The data buffer is palloc'd but the StringInfoData is just local.
5558
*This is the easiest approach for a StringInfo object that will
5659
*only live as long as the current routine.
@@ -67,21 +70,26 @@ typedef StringInfoData *StringInfo;
6770

6871
/*------------------------
6972
* makeStringInfo
70-
* Create an empty 'StringInfoData' & return a pointer to it.
73+
* makeLongStringInfo
74+
* Create an empty 'StringInfoData' & return a pointer to it. The former
75+
* allows up to 1 GB in size, per palloc(); the latter allows up to 2 GB.
7176
*/
7277
externStringInfomakeStringInfo(void);
78+
externStringInfomakeLongStringInfo(void);
7379

7480
/*------------------------
7581
* initStringInfo
82+
* initLongStringInfo
7683
* Initialize a StringInfoData struct (with previously undefined contents)
77-
* to describe an empty string.
84+
* to describe an empty string. Size limits as above.
7885
*/
7986
externvoidinitStringInfo(StringInfostr);
87+
externvoidinitLongStringInfo(StringInfostr);
8088

8189
/*------------------------
8290
* resetStringInfo
8391
* Clears the current content of the StringInfo, if any. The
84-
* StringInfo remains valid.
92+
* StringInfo remains valid. The long_ok flag is not reset.
8593
*/
8694
externvoidresetStringInfo(StringInfostr);
8795

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp