Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit8e10405

Browse files
committed
Avoid unnecessary out-of-memory errors during encoding conversion.
Encoding conversion uses the very simplistic rule that the outputcan't be more than 4X longer than the input, and palloc's a bufferof that size. This results in failure to convert any string longerthan 1/4 GB, which is becoming an annoying limitation.As a band-aid to improve matters, allow the allocated output buffersize to exceed 1GB. We still insist that the final result fit intoMaxAllocSize (1GB), though. Perhaps it'd be safe to relax thatrestriction, but it'd require close analysis of all callers, whichis daunting (not least because external modules might call thesefunctions). For the moment, this should allow a 2X to 4X improvementin the longest string we can convert, which is a useful gain inreturn for quite a simple patch.Also, once we have successfully converted a long string, repallocthe output down to the actual string length, returning the excessto the malloc pool. This seems worth doing since we can usuallyexpect to give back several MB if we take this path at all.This still leaves much to be desired, most notably that the assumptionthat MAX_CONVERSION_GROWTH == 4 is very fragile, and yet we have noguard code verifying that the output buffer isn't overrun. Fixingthat would require significant changes in the encoding conversionAPIs, so it'll have to wait for some other day.The present patch seems safely back-patchable, so patch all supportedbranches.Alvaro Herrera and Tom LaneDiscussion:https://postgr.es/m/20190816181418.GA898@alvherre.pgsqlDiscussion:https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
1 parentc477f3e commit8e10405

File tree

1 file changed

+56
-6
lines changed

1 file changed

+56
-6
lines changed

‎src/backend/utils/mb/mbutils.c

Lines changed: 56 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -349,23 +349,51 @@ pg_do_encoding_conversion(unsigned char *src, int len,
349349
pg_encoding_to_char(dest_encoding))));
350350

351351
/*
352-
* Allocate space for conversion result, being wary of integer overflow
352+
* Allocate space for conversion result, being wary of integer overflow.
353+
*
354+
* len * MAX_CONVERSION_GROWTH is typically a vast overestimate of the
355+
* required space, so it might exceed MaxAllocSize even though the result
356+
* would actually fit. We do not want to hand back a result string that
357+
* exceeds MaxAllocSize, because callers might not cope gracefully --- but
358+
* if we just allocate more than that, and don't use it, that's fine.
353359
*/
354-
if ((Size)len >= (MaxAllocSize / (Size)MAX_CONVERSION_GROWTH))
360+
if ((Size)len >= (MaxAllocHugeSize / (Size)MAX_CONVERSION_GROWTH))
355361
ereport(ERROR,
356362
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
357363
errmsg("out of memory"),
358364
errdetail("String of %d bytes is too long for encoding conversion.",
359365
len)));
360366

361-
result=palloc(len*MAX_CONVERSION_GROWTH+1);
367+
result= (unsignedchar*)
368+
MemoryContextAllocHuge(CurrentMemoryContext,
369+
(Size)len*MAX_CONVERSION_GROWTH+1);
362370

363371
OidFunctionCall5(proc,
364372
Int32GetDatum(src_encoding),
365373
Int32GetDatum(dest_encoding),
366374
CStringGetDatum(src),
367375
CStringGetDatum(result),
368376
Int32GetDatum(len));
377+
378+
/*
379+
* If the result is large, it's worth repalloc'ing to release any extra
380+
* space we asked for. The cutoff here is somewhat arbitrary, but we
381+
* *must* check when len * MAX_CONVERSION_GROWTH exceeds MaxAllocSize.
382+
*/
383+
if (len>1000000)
384+
{
385+
Sizeresultlen=strlen((char*)result);
386+
387+
if (resultlen >=MaxAllocSize)
388+
ereport(ERROR,
389+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
390+
errmsg("out of memory"),
391+
errdetail("String of %d bytes is too long for encoding conversion.",
392+
len)));
393+
394+
result= (unsignedchar*)repalloc(result,resultlen+1);
395+
}
396+
369397
returnresult;
370398
}
371399

@@ -682,23 +710,45 @@ perform_default_encoding_conversion(const char *src, int len,
682710
returnunconstify(char*,src);
683711

684712
/*
685-
* Allocate space for conversion result, being wary of integer overflow
713+
* Allocate space for conversion result, being wary of integer overflow.
714+
* See comments in pg_do_encoding_conversion.
686715
*/
687-
if ((Size)len >= (MaxAllocSize / (Size)MAX_CONVERSION_GROWTH))
716+
if ((Size)len >= (MaxAllocHugeSize / (Size)MAX_CONVERSION_GROWTH))
688717
ereport(ERROR,
689718
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
690719
errmsg("out of memory"),
691720
errdetail("String of %d bytes is too long for encoding conversion.",
692721
len)));
693722

694-
result=palloc(len*MAX_CONVERSION_GROWTH+1);
723+
result= (char*)
724+
MemoryContextAllocHuge(CurrentMemoryContext,
725+
(Size)len*MAX_CONVERSION_GROWTH+1);
695726

696727
FunctionCall5(flinfo,
697728
Int32GetDatum(src_encoding),
698729
Int32GetDatum(dest_encoding),
699730
CStringGetDatum(src),
700731
CStringGetDatum(result),
701732
Int32GetDatum(len));
733+
734+
/*
735+
* Release extra space if there might be a lot --- see comments in
736+
* pg_do_encoding_conversion.
737+
*/
738+
if (len>1000000)
739+
{
740+
Sizeresultlen=strlen(result);
741+
742+
if (resultlen >=MaxAllocSize)
743+
ereport(ERROR,
744+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
745+
errmsg("out of memory"),
746+
errdetail("String of %d bytes is too long for encoding conversion.",
747+
len)));
748+
749+
result= (char*)repalloc(result,resultlen+1);
750+
}
751+
702752
returnresult;
703753
}
704754

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp