Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

Commitad43447

Browse files
committed
This patch addresses some issues in TOAST compression strategy that
were discussed last year, but we felt it was too late in the 8.3 cycle tochange the code immediately. Specifically, the patch:* Reduces the minimum datum size to be considered for compression from256 to 32 bytes, as suggested by Greg Stark.* Increases the required compression rate for compressed storage from20% to 25%, again per Greg's suggestion.* Replaces force_input_size (size above which compression is forced)with a maximum size to be considered for compression. It was agreedthat allowing large inputs to escape the minimum-compression-raterequirement was not bright, and that indeed we'd rather have a knobthat acted in the other direction. I set this value to 1MB for themoment, but it could use some performance studies to tune it.* Adds an early-failure path to the compressor as suggested by Jan:if it's been unable to find even one compressible substring in thefirst 1KB (parameterizable), assume we're looking at incompressibleinput and give up. (Possibly this logic can be improved, but I'llcommit it as-is for now.)* Improves the toasting heuristics so that when we have very largefields with attstorage 'x' or 'e', we will push those out to toaststorage before considering inline compression of shorter fields.This also responds to a suggestion of Greg's, though my originalproposal for a solution was a bit off base because it didn't fixthe problem for large 'e' fields.There was some discussion in the earlier threads of exposing someof the compression knobs to users, perhaps even on a per-columnbasis. I have not done anything about that here. It seems to methat if we are changing around the parameters, we'd better get someexperience and be sure we are happy with the design before we setthings in stone by providing user-visible knobs.
1 parent1cc5290 commitad43447

File tree

3 files changed

+141
-102
lines changed

3 files changed

+141
-102
lines changed

‎src/backend/access/heap/tuptoaster.c

Lines changed: 66 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
*
99
*
1010
* IDENTIFICATION
11-
* $PostgreSQL: pgsql/src/backend/access/heap/tuptoaster.c,v 1.83 2008/02/29 17:47:41 tgl Exp $
11+
* $PostgreSQL: pgsql/src/backend/access/heap/tuptoaster.c,v 1.84 2008/03/07 23:20:21 tgl Exp $
1212
*
1313
*
1414
* INTERFACE ROUTINES
@@ -576,7 +576,8 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
576576
/* ----------
577577
* Compress and/or save external until data fits into target length
578578
*
579-
*1: Inline compress attributes with attstorage 'x'
579+
*1: Inline compress attributes with attstorage 'x', and store very
580+
* large attributes with attstorage 'x' or 'e' external immediately
580581
*2: Store attributes with attstorage 'x' or 'e' external
581582
*3: Inline compress attributes with attstorage 'm'
582583
*4: Store attributes with attstorage 'm' external
@@ -595,7 +596,8 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
595596
maxDataLen=TOAST_TUPLE_TARGET-hoff;
596597

597598
/*
598-
* Look for attributes with attstorage 'x' to compress
599+
* Look for attributes with attstorage 'x' to compress. Also find large
600+
* attributes with attstorage 'x' or 'e', and store them external.
599601
*/
600602
while (heap_compute_data_size(tupleDesc,
601603
toast_values,toast_isnull)>maxDataLen)
@@ -606,7 +608,7 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
606608
Datumnew_value;
607609

608610
/*
609-
* Search for the biggest yetuncompressed internal attribute
611+
* Search for the biggest yetunprocessed internal attribute
610612
*/
611613
for (i=0;i<numAttrs;i++)
612614
{
@@ -616,7 +618,7 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
616618
continue;/* can't happen, toast_action would be 'p' */
617619
if (VARATT_IS_COMPRESSED(toast_values[i]))
618620
continue;
619-
if (att[i]->attstorage!='x')
621+
if (att[i]->attstorage!='x'&&att[i]->attstorage!='e')
620622
continue;
621623
if (toast_sizes[i]>biggest_size)
622624
{
@@ -629,30 +631,58 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
629631
break;
630632

631633
/*
632-
* Attempt to compress it inline
634+
* Attempt to compress it inline, if it has attstorage 'x'
633635
*/
634636
i=biggest_attno;
635-
old_value=toast_values[i];
636-
new_value=toast_compress_datum(old_value);
637+
if (att[i]->attstorage=='x')
638+
{
639+
old_value=toast_values[i];
640+
new_value=toast_compress_datum(old_value);
637641

638-
if (DatumGetPointer(new_value)!=NULL)
642+
if (DatumGetPointer(new_value)!=NULL)
643+
{
644+
/* successful compression */
645+
if (toast_free[i])
646+
pfree(DatumGetPointer(old_value));
647+
toast_values[i]=new_value;
648+
toast_free[i]= true;
649+
toast_sizes[i]=VARSIZE(toast_values[i]);
650+
need_change= true;
651+
need_free= true;
652+
}
653+
else
654+
{
655+
/* incompressible, ignore on subsequent compression passes */
656+
toast_action[i]='x';
657+
}
658+
}
659+
else
639660
{
640-
/* successful compression */
661+
/* has attstorage 'e', ignore on subsequent compression passes */
662+
toast_action[i]='x';
663+
}
664+
665+
/*
666+
* If this value is by itself more than maxDataLen (after compression
667+
* if any), push it out to the toast table immediately, if possible.
668+
* This avoids uselessly compressing other fields in the common case
669+
* where we have one long field and several short ones.
670+
*
671+
* XXX maybe the threshold should be less than maxDataLen?
672+
*/
673+
if (toast_sizes[i]>maxDataLen&&
674+
rel->rd_rel->reltoastrelid!=InvalidOid)
675+
{
676+
old_value=toast_values[i];
677+
toast_action[i]='p';
678+
toast_values[i]=toast_save_datum(rel,toast_values[i],
679+
use_wal,use_fsm);
641680
if (toast_free[i])
642681
pfree(DatumGetPointer(old_value));
643-
toast_values[i]=new_value;
644682
toast_free[i]= true;
645-
toast_sizes[i]=VARSIZE(toast_values[i]);
646683
need_change= true;
647684
need_free= true;
648685
}
649-
else
650-
{
651-
/*
652-
* incompressible data, ignore on subsequent compression passes
653-
*/
654-
toast_action[i]='x';
655-
}
656686
}
657687

658688
/*
@@ -761,9 +791,7 @@ toast_insert_or_update(Relation rel, HeapTuple newtup, HeapTuple oldtup,
761791
}
762792
else
763793
{
764-
/*
765-
* incompressible data, ignore on subsequent compression passes
766-
*/
794+
/* incompressible, ignore on subsequent compression passes */
767795
toast_action[i]='x';
768796
}
769797
}
@@ -1047,16 +1075,28 @@ toast_compress_datum(Datum value)
10471075
Assert(!VARATT_IS_COMPRESSED(value));
10481076

10491077
/*
1050-
* No point in wasting a palloc cycle if value istoo short for
1051-
* compression
1078+
* No point in wasting a palloc cycle if valuesizeisout of the
1079+
*allowed range forcompression
10521080
*/
1053-
if (valsize<PGLZ_strategy_default->min_input_size)
1081+
if (valsize<PGLZ_strategy_default->min_input_size||
1082+
valsize>PGLZ_strategy_default->max_input_size)
10541083
returnPointerGetDatum(NULL);
10551084

10561085
tmp= (structvarlena*)palloc(PGLZ_MAX_OUTPUT(valsize));
1086+
1087+
/*
1088+
* We recheck the actual size even if pglz_compress() reports success,
1089+
* because it might be satisfied with having saved as little as one byte
1090+
* in the compressed data --- which could turn into a net loss once you
1091+
* consider header and alignment padding. Worst case, the compressed
1092+
* format might require three padding bytes (plus header, which is included
1093+
* in VARSIZE(tmp)), whereas the uncompressed format would take only one
1094+
* header byte and no padding if the value is short enough. So we insist
1095+
* on a savings of more than 2 bytes to ensure we have a gain.
1096+
*/
10571097
if (pglz_compress(VARDATA_ANY(value),valsize,
10581098
(PGLZ_Header*)tmp,PGLZ_strategy_default)&&
1059-
VARSIZE(tmp)<VARSIZE_ANY(value))
1099+
VARSIZE(tmp)<valsize-2)
10601100
{
10611101
/* successful compression */
10621102
returnPointerGetDatum(tmp);

‎src/backend/utils/adt/pg_lzcompress.c

Lines changed: 58 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
*This is an implementation of LZ compression for PostgreSQL.
55
*It uses a simple history table and generates 2-3 byte tags
66
*capable of backward copy information for 3-273 bytes with
7-
*anoffset of max. 4095.
7+
*a maxoffset of 4095.
88
*
99
*Entry routines:
1010
*
@@ -166,13 +166,12 @@
166166
*
167167
* Copyright (c) 1999-2008, PostgreSQL Global Development Group
168168
*
169-
* $PostgreSQL: pgsql/src/backend/utils/adt/pg_lzcompress.c,v 1.29 2008/01/01 19:45:52 momjian Exp $
169+
* $PostgreSQL: pgsql/src/backend/utils/adt/pg_lzcompress.c,v 1.30 2008/03/07 23:20:21 tgl Exp $
170170
* ----------
171171
*/
172172
#include"postgres.h"
173173

174-
#include<unistd.h>
175-
#include<fcntl.h>
174+
#include<limits.h>
176175

177176
#include"utils/pg_lzcompress.h"
178177

@@ -211,27 +210,23 @@ typedef struct PGLZ_HistEntry
211210
* ----------
212211
*/
213212
staticconstPGLZ_Strategystrategy_default_data= {
214-
256,/* Data chunks less than 256 bytes are not
215-
* compressed */
216-
6144,/* Data chunks >= 6K force compression, unless
217-
* compressed output is larger than input */
218-
20,/* Below 6K, compression rates below 20% mean
219-
* fallback to uncompressed */
220-
128,/* Stop history lookup if a match of 128 bytes
221-
* is found */
222-
10/* Lower good match size by 10% at every
223-
* lookup loop iteration */
213+
32,/* Data chunks less than 32 bytes are not compressed */
214+
1024*1024,/* Data chunks over 1MB are not compressed either */
215+
25,/* Require 25% compression rate, or not worth it */
216+
1024,/* Give up if no compression in the first 1KB */
217+
128,/* Stop history lookup if a match of 128 bytes is found */
218+
10/* Lower good match size by 10% at every loop iteration */
224219
};
225220
constPGLZ_Strategy*constPGLZ_strategy_default=&strategy_default_data;
226221

227222

228223
staticconstPGLZ_Strategystrategy_always_data= {
229-
0,/* Chunks of any size are compressed */
230-
0,
231-
0,/* It's enough to save one single byte */
232-
128,/*Stop history lookup if a match of 128 bytes
233-
* is found */
234-
6/* Look harder for a good match */
224+
0,/* Chunks of any size are compressed */
225+
INT_MAX,
226+
0,/* It's enough to save one single byte */
227+
INT_MAX,/*Never give up early */
228+
128,/* Stop history lookup if a match of 128 bytes is found */
229+
6/* Look harder for a good match */
235230
};
236231
constPGLZ_Strategy*constPGLZ_strategy_always=&strategy_always_data;
237232

@@ -491,6 +486,7 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
491486
unsignedchar*ctrlp=&ctrl_dummy;
492487
unsignedcharctrlb=0;
493488
unsignedcharctrl=0;
489+
boolfound_match= false;
494490
int32match_len;
495491
int32match_off;
496492
int32good_match;
@@ -506,11 +502,12 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
506502
strategy=PGLZ_strategy_default;
507503

508504
/*
509-
* If the strategy forbids compression (at all or if source chunktoo
510-
*small), fail.
505+
* If the strategy forbids compression (at all or if source chunksize
506+
*out of range), fail.
511507
*/
512508
if (strategy->match_size_good <=0||
513-
slen<strategy->min_input_size)
509+
slen<strategy->min_input_size||
510+
slen>strategy->max_input_size)
514511
return false;
515512

516513
/*
@@ -519,41 +516,44 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
519516
dest->rawsize=slen;
520517

521518
/*
522-
* Limit the matchsize to themaximum implementation allowed value
519+
* Limit the matchparameters to thesupported range.
523520
*/
524-
if ((good_match=strategy->match_size_good)>PGLZ_MAX_MATCH)
521+
good_match=strategy->match_size_good;
522+
if (good_match>PGLZ_MAX_MATCH)
525523
good_match=PGLZ_MAX_MATCH;
526-
if (good_match<17)
524+
elseif (good_match<17)
527525
good_match=17;
528526

529-
if ((good_drop=strategy->match_size_drop)<0)
527+
good_drop=strategy->match_size_drop;
528+
if (good_drop<0)
530529
good_drop=0;
531-
if (good_drop>100)
530+
elseif (good_drop>100)
532531
good_drop=100;
533532

534-
/*
535-
* Initialize the history lists to empty. We do not need to zero the
536-
* hist_entries[] array; its entries are initialized as they are used.
537-
*/
538-
memset((void*)hist_start,0,sizeof(hist_start));
533+
need_rate=strategy->min_comp_rate;
534+
if (need_rate<0)
535+
need_rate=0;
536+
elseif (need_rate>99)
537+
need_rate=99;
539538

540539
/*
541-
* Compute the maximum result size allowed by the strategy. If the input
542-
* size exceeds force_input_size, the max result size is the input size
543-
* itself. Otherwise, it is the input size minus the minimum wanted
544-
* compression rate.
540+
* Compute the maximum result size allowed by the strategy, namely
541+
* the input size minus the minimum wanted compression rate. This had
542+
* better be <= slen, else we might overrun the provided output buffer.
545543
*/
546-
if (slen >=strategy->force_input_size)
547-
result_max=slen;
548-
else
544+
if (slen> (INT_MAX/100))
549545
{
550-
need_rate=strategy->min_comp_rate;
551-
if (need_rate<0)
552-
need_rate=0;
553-
elseif (need_rate>99)
554-
need_rate=99;
555-
result_max=slen- ((slen*need_rate) /100);
546+
/* Approximate to avoid overflow */
547+
result_max= (slen /100)* (100-need_rate);
556548
}
549+
else
550+
result_max= (slen* (100-need_rate)) /100;
551+
552+
/*
553+
* Initialize the history lists to empty. We do not need to zero the
554+
* hist_entries[] array; its entries are initialized as they are used.
555+
*/
556+
memset(hist_start,0,sizeof(hist_start));
557557

558558
/*
559559
* Compress the source directly into the output buffer.
@@ -570,6 +570,15 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
570570
if (bp-bstart >=result_max)
571571
return false;
572572

573+
/*
574+
* If we've emitted more than first_success_by bytes without finding
575+
* anything compressible at all, fail. This lets us fall out
576+
* reasonably quickly when looking at incompressible input (such as
577+
* pre-compressed data).
578+
*/
579+
if (!found_match&&bp-bstart >=strategy->first_success_by)
580+
return false;
581+
573582
/*
574583
* Try to find a match in the history
575584
*/
@@ -586,9 +595,10 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
586595
pglz_hist_add(hist_start,hist_entries,
587596
hist_next,hist_recycle,
588597
dp,dend);
589-
dp++;/* Do not do this ++ in the line above!*/
598+
dp++;/* Do not do this ++ in the line above!*/
590599
/* The macro would do it four times - Jan.*/
591600
}
601+
found_match= true;
592602
}
593603
else
594604
{
@@ -599,7 +609,7 @@ pglz_compress(const char *source, int32 slen, PGLZ_Header *dest,
599609
pglz_hist_add(hist_start,hist_entries,
600610
hist_next,hist_recycle,
601611
dp,dend);
602-
dp++;/* Do not do this ++ in the line above!*/
612+
dp++;/* Do not do this ++ in the line above!*/
603613
/* The macro would do it four times - Jan.*/
604614
}
605615
}

0 commit comments

Comments
 (0)
[8]ページ先頭
©2009-2025 Movatter.jp