Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commit8381242

Browse files
committed
Avoid unnecessary out-of-memory errors during encoding conversion.
Encoding conversion uses the very simplistic rule that the outputcan't be more than 4X longer than the input, and palloc's a bufferof that size. This results in failure to convert any string longerthan 1/4 GB, which is becoming an annoying limitation.As a band-aid to improve matters, allow the allocated output buffersize to exceed 1GB. We still insist that the final result fit intoMaxAllocSize (1GB), though. Perhaps it'd be safe to relax thatrestriction, but it'd require close analysis of all callers, whichis daunting (not least because external modules might call thesefunctions). For the moment, this should allow a 2X to 4X improvementin the longest string we can convert, which is a useful gain inreturn for quite a simple patch.Also, once we have successfully converted a long string, repallocthe output down to the actual string length, returning the excessto the malloc pool. This seems worth doing since we can usuallyexpect to give back several MB if we take this path at all.This still leaves much to be desired, most notably that the assumptionthat MAX_CONVERSION_GROWTH == 4 is very fragile, and yet we have noguard code verifying that the output buffer isn't overrun. Fixingthat would require significant changes in the encoding conversionAPIs, so it'll have to wait for some other day.The present patch seems safely back-patchable, so patch all supportedbranches.Alvaro Herrera and Tom LaneDiscussion:https://postgr.es/m/20190816181418.GA898@alvherre.pgsqlDiscussion:https://postgr.es/m/3614.1569359690@sss.pgh.pa.us
1 parent9a40720 commit8381242

File tree

1 file changed

+56
-6
lines changed

1 file changed

+56
-6
lines changed

‎src/backend/utils/mb/mbutils.c

Lines changed: 56 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -357,23 +357,51 @@ pg_do_encoding_conversion(unsigned char *src, int len,
357357
pg_encoding_to_char(dest_encoding))));
358358

359359
/*
360-
* Allocate space for conversion result, being wary of integer overflow
360+
* Allocate space for conversion result, being wary of integer overflow.
361+
*
362+
* len * MAX_CONVERSION_GROWTH is typically a vast overestimate of the
363+
* required space, so it might exceed MaxAllocSize even though the result
364+
* would actually fit. We do not want to hand back a result string that
365+
* exceeds MaxAllocSize, because callers might not cope gracefully --- but
366+
* if we just allocate more than that, and don't use it, that's fine.
361367
*/
362-
if ((Size)len >= (MaxAllocSize / (Size)MAX_CONVERSION_GROWTH))
368+
if ((Size)len >= (MaxAllocHugeSize / (Size)MAX_CONVERSION_GROWTH))
363369
ereport(ERROR,
364370
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
365371
errmsg("out of memory"),
366372
errdetail("String of %d bytes is too long for encoding conversion.",
367373
len)));
368374

369-
result=palloc(len*MAX_CONVERSION_GROWTH+1);
375+
result= (unsignedchar*)
376+
MemoryContextAllocHuge(CurrentMemoryContext,
377+
(Size)len*MAX_CONVERSION_GROWTH+1);
370378

371379
OidFunctionCall5(proc,
372380
Int32GetDatum(src_encoding),
373381
Int32GetDatum(dest_encoding),
374382
CStringGetDatum(src),
375383
CStringGetDatum(result),
376384
Int32GetDatum(len));
385+
386+
/*
387+
* If the result is large, it's worth repalloc'ing to release any extra
388+
* space we asked for. The cutoff here is somewhat arbitrary, but we
389+
* *must* check when len * MAX_CONVERSION_GROWTH exceeds MaxAllocSize.
390+
*/
391+
if (len>1000000)
392+
{
393+
Sizeresultlen=strlen((char*)result);
394+
395+
if (resultlen >=MaxAllocSize)
396+
ereport(ERROR,
397+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
398+
errmsg("out of memory"),
399+
errdetail("String of %d bytes is too long for encoding conversion.",
400+
len)));
401+
402+
result= (unsignedchar*)repalloc(result,resultlen+1);
403+
}
404+
377405
returnresult;
378406
}
379407

@@ -690,23 +718,45 @@ perform_default_encoding_conversion(const char *src, int len,
690718
returnunconstify(char*,src);
691719

692720
/*
693-
* Allocate space for conversion result, being wary of integer overflow
721+
* Allocate space for conversion result, being wary of integer overflow.
722+
* See comments in pg_do_encoding_conversion.
694723
*/
695-
if ((Size)len >= (MaxAllocSize / (Size)MAX_CONVERSION_GROWTH))
724+
if ((Size)len >= (MaxAllocHugeSize / (Size)MAX_CONVERSION_GROWTH))
696725
ereport(ERROR,
697726
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
698727
errmsg("out of memory"),
699728
errdetail("String of %d bytes is too long for encoding conversion.",
700729
len)));
701730

702-
result=palloc(len*MAX_CONVERSION_GROWTH+1);
731+
result= (char*)
732+
MemoryContextAllocHuge(CurrentMemoryContext,
733+
(Size)len*MAX_CONVERSION_GROWTH+1);
703734

704735
FunctionCall5(flinfo,
705736
Int32GetDatum(src_encoding),
706737
Int32GetDatum(dest_encoding),
707738
CStringGetDatum(src),
708739
CStringGetDatum(result),
709740
Int32GetDatum(len));
741+
742+
/*
743+
* Release extra space if there might be a lot --- see comments in
744+
* pg_do_encoding_conversion.
745+
*/
746+
if (len>1000000)
747+
{
748+
Sizeresultlen=strlen(result);
749+
750+
if (resultlen >=MaxAllocSize)
751+
ereport(ERROR,
752+
(errcode(ERRCODE_PROGRAM_LIMIT_EXCEEDED),
753+
errmsg("out of memory"),
754+
errdetail("String of %d bytes is too long for encoding conversion.",
755+
len)));
756+
757+
result= (char*)repalloc(result,resultlen+1);
758+
}
759+
710760
returnresult;
711761
}
712762

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp