Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Commit78c0b6e

Browse files
committed
Re-allow testing of GiST buffered builds.
Commit16fa9b2 broke the ability to reliably test GiST buffered builds,because it caused sorted builds to be done instead if sortsupport isavailable, regardless of any attempt to override that. While a would-betest case could try to work around that by choosing an opclass that hasno sortsupport function, coverage would be silently lost the momentsomeone decides it'd be a good idea to add a sortsupport function.Hence, rearrange the logic in gistbuild() so that if "buffering = on"is specified in CREATE INDEX, we will use that method, sortsupport or no.Also document the interaction between sorting and the bufferingparameter, as16fa9b2 failed to do.(Note that in fact we still lack any test coverage of buffered builds,but this is a prerequisite to adding a non-fragile test.)Discussion:https://postgr.es/m/3249980.1602532990@sss.pgh.pa.us
1 parent397ea90 commit78c0b6e

File tree

3 files changed

+77
-53
lines changed

3 files changed

+77
-53
lines changed

‎doc/src/sgml/gist.sgml

Lines changed: 37 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -975,7 +975,7 @@ static char *str_param_default = "default";
975975
/*
976976
* Sample validator: checks that string is not longer than 8 bytes.
977977
*/
978-
static void
978+
static void
979979
validate_my_string_relopt(const char *value)
980980
{
981981
if (strlen(value) > 8)
@@ -987,7 +987,7 @@ validate_my_string_relopt(const char *value)
987987
/*
988988
* Sample filler: switches characters to lower case.
989989
*/
990-
static Size
990+
static Size
991991
fill_my_string_relopt(const char *value, void *ptr)
992992
{
993993
char *tmp = str_tolower(value, strlen(value), DEFAULT_COLLATION_OID);
@@ -1157,36 +1157,52 @@ my_sortsupport(PG_FUNCTION_ARGS)
11571157
<title>Implementation</title>
11581158

11591159
<sect2 id="gist-buffering-build">
1160-
<title>GiST Buffering Build</title>
1160+
<title>GiST Index Build Methods</title>
1161+
1162+
<para>
1163+
The simplest way to build a GiST index is just to insert all the entries,
1164+
one by one. This tends to be slow for large indexes, because if the
1165+
index tuples are scattered across the index and the index is large enough
1166+
to not fit in cache, a lot of random I/O will be
1167+
needed. <productname>PostgreSQL</productname> supports two alternative
1168+
methods for initial build of a GiST index: <firstterm>sorted</firstterm>
1169+
and <firstterm>buffered</firstterm> modes.
1170+
</para>
1171+
1172+
<para>
1173+
The sorted method is only available if each of the opclasses used by the
1174+
index provides a <function>sortsupport</function> function, as described
1175+
in <xref linkend="gist-extensibility"/>. If they do, this method is
1176+
usually the best, so it is used by default.
1177+
</para>
1178+
11611179
<para>
1162-
Building large GiST indexes by simply inserting all the tuples tends to be
1163-
slow, because if the index tuples are scattered across the index and the
1164-
index is large enough to not fit in cache, the insertions need to perform
1165-
a lot of random I/O. Beginning in version 9.2, PostgreSQL supports a more
1166-
efficient method to build GiST indexes based on buffering, which can
1167-
dramatically reduce the number of random I/Os needed for non-ordered data
1168-
sets. For well-ordered data sets the benefit is smaller or non-existent,
1169-
because only a small number of pages receive new tuples at a time, and
1170-
those pages fit in cache even if the index as whole does not.
1180+
The buffered method works by not inserting tuples directly into the index
1181+
right away. It can dramatically reduce the amount of random I/O needed
1182+
for non-ordered data sets. For well-ordered data sets the benefit is
1183+
smaller or non-existent, because only a small number of pages receive new
1184+
tuples at a time, and those pages fit in cache even if the index as a
1185+
whole does not.
11711186
</para>
11721187

11731188
<para>
1174-
However, buffering index build needs to call the <function>penalty</function>
1175-
function more often, which consumes some extra CPU resources. Also, the
1176-
buffers used inthebuffering build need temporary disk space, up to
1189+
The buffered method needs to call the <function>penalty</function>
1190+
function more often than the simple method does, which consumes some
1191+
extra CPU resources. Also,thebuffers need temporary disk space, up to
11771192
the size of the resulting index. Buffering can also influence the quality
11781193
of the resulting index, in both positive and negative directions. That
11791194
influence depends on various factors, like the distribution of the input
11801195
data and the operator class implementation.
11811196
</para>
11821197

11831198
<para>
1184-
By default, a GiST index build switches to the buffering method when the
1185-
index size reaches <xref linkend="guc-effective-cache-size"/>. It can
1186-
be manually turned on or off by the <literal>buffering</literal> parameter
1187-
to the CREATE INDEX command. The default behavior is good for most cases,
1188-
but turning buffering off might speed up the build somewhat if the input
1189-
data is ordered.
1199+
If sorting is not possible, then by default a GiST index build switches
1200+
to the buffering method when the index size reaches
1201+
<xref linkend="guc-effective-cache-size"/>. Buffering can be manually
1202+
forced or prevented by the <literal>buffering</literal> parameter to the
1203+
CREATE INDEX command. The default behavior is good for most cases, but
1204+
turning buffering off might speed up the build somewhat if the input data
1205+
is ordered.
11901206
</para>
11911207

11921208
</sect2>

‎doc/src/sgml/ref/create_index.sgml

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -463,11 +463,15 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ [ IF NOT EXISTS ] <replaceable class=
463463
</term>
464464
<listitem>
465465
<para>
466-
Determines whether thebuffering build technique described in
466+
Determines whether thebuffered build technique described in
467467
<xref linkend="gist-buffering-build"/> is used to build the index. With
468-
<literal>OFF</literal> it is disabled, with <literal>ON</literal> it is enabled, and
469-
with <literal>AUTO</literal> it is initially disabled, but turned on
470-
on-the-fly once the index size reaches <xref linkend="guc-effective-cache-size"/>. The default is <literal>AUTO</literal>.
468+
<literal>OFF</literal> buffering is disabled, with <literal>ON</literal>
469+
it is enabled, and with <literal>AUTO</literal> it is initially disabled,
470+
but is turned on on-the-fly once the index size reaches
471+
<xref linkend="guc-effective-cache-size"/>. The default
472+
is <literal>AUTO</literal>.
473+
Note that if sorted build is possible, it will be used instead of
474+
buffered build unless <literal>buffering=ON</literal> is specified.
471475
</para>
472476
</listitem>
473477
</varlistentry>

‎src/backend/access/gist/gistbuild.c

Lines changed: 32 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -180,9 +180,7 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
180180
MemoryContextoldcxt=CurrentMemoryContext;
181181
intfillfactor;
182182
OidSortSupportFnOids[INDEX_MAX_KEYS];
183-
boolhasallsortsupports;
184-
intkeyscount=IndexRelationGetNumberOfKeyAttributes(index);
185-
GiSTOptions*options=NULL;
183+
GiSTOptions*options= (GiSTOptions*)index->rd_options;
186184

187185
/*
188186
* We expect to be called exactly once for any index relation. If that's
@@ -192,9 +190,6 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
192190
elog(ERROR,"index \"%s\" already contains data",
193191
RelationGetRelationName(index));
194192

195-
if (index->rd_options)
196-
options= (GiSTOptions*)index->rd_options;
197-
198193
buildstate.indexrel=index;
199194
buildstate.heaprel=heap;
200195
buildstate.sortstate=NULL;
@@ -208,40 +203,46 @@ gistbuild(Relation heap, Relation index, IndexInfo *indexInfo)
208203
buildstate.giststate->tempCxt=createTempGistContext();
209204

210205
/*
211-
* Choose build strategy.If all keys support sorting, do that. Otherwise
212-
*the default strategy is switch to buffering mode whentheindex grows
213-
*too large to fit in cache.
206+
* Choose build strategy. First check whether the user specified to use
207+
*buffering mode. (The use-case for that inthefield is somewhat
208+
*questionable perhaps, but it's important for testing purposes.)
214209
*/
215-
hasallsortsupports= true;
216-
for (inti=0;i<keyscount;i++)
217-
{
218-
SortSupportFnOids[i]=index_getprocid(index,i+1,
219-
GIST_SORTSUPPORT_PROC);
220-
if (!OidIsValid(SortSupportFnOids[i]))
221-
{
222-
hasallsortsupports= false;
223-
break;
224-
}
225-
}
226-
227-
if (hasallsortsupports)
228-
{
229-
buildstate.buildMode=GIST_SORTED_BUILD;
230-
}
231-
elseif (options)
210+
if (options)
232211
{
233212
if (options->buffering_mode==GIST_OPTION_BUFFERING_ON)
234213
buildstate.buildMode=GIST_BUFFERING_STATS;
235214
elseif (options->buffering_mode==GIST_OPTION_BUFFERING_OFF)
236215
buildstate.buildMode=GIST_BUFFERING_DISABLED;
237-
else
216+
else/* must be "auto" */
238217
buildstate.buildMode=GIST_BUFFERING_AUTO;
239218
}
240219
else
241220
{
242221
buildstate.buildMode=GIST_BUFFERING_AUTO;
243222
}
244223

224+
/*
225+
* Unless buffering mode was forced, see if we can use sorting instead.
226+
*/
227+
if (buildstate.buildMode!=GIST_BUFFERING_STATS)
228+
{
229+
boolhasallsortsupports= true;
230+
intkeyscount=IndexRelationGetNumberOfKeyAttributes(index);
231+
232+
for (inti=0;i<keyscount;i++)
233+
{
234+
SortSupportFnOids[i]=index_getprocid(index,i+1,
235+
GIST_SORTSUPPORT_PROC);
236+
if (!OidIsValid(SortSupportFnOids[i]))
237+
{
238+
hasallsortsupports= false;
239+
break;
240+
}
241+
}
242+
if (hasallsortsupports)
243+
buildstate.buildMode=GIST_SORTED_BUILD;
244+
}
245+
245246
/*
246247
* Calculate target amount of free space to leave on pages.
247248
*/
@@ -852,7 +853,10 @@ gistBuildCallback(Relation index,
852853
* and switch to buffering mode if it has.
853854
*
854855
* To avoid excessive calls to smgrnblocks(), only check this every
855-
* BUFFERING_MODE_SWITCH_CHECK_STEP index tuples
856+
* BUFFERING_MODE_SWITCH_CHECK_STEP index tuples.
857+
*
858+
* In 'stats' state, switch as soon as we have seen enough tuples to have
859+
* some idea of the average tuple size.
856860
*/
857861
if ((buildstate->buildMode==GIST_BUFFERING_AUTO&&
858862
buildstate->indtuples %BUFFERING_MODE_SWITCH_CHECK_STEP==0&&

0 commit comments

Comments
 (0)

[8]ページ先頭

©2009-2025 Movatter.jp