Movatterモバイル変換

Table 64.3. Built-inGIN Operator Classes

Datum *extractValue(Datum itemValue, int32 *nkeys, bool **nullFlags)

Returns a palloc'd array of keys given an item to be indexed. The number of returned keys must be stored into*nkeys. If any of the keys can be null, also palloc an array of*nkeysbool fields, store its address at*nullFlags, and set these null flags as needed.*nullFlags can be leftNULL (its initial value) if all keys are non-null. The return value can beNULL if the item contains no keys.

Datum *extractQuery(Datum query, int32 *nkeys, StrategyNumber n, bool **pmatch, Pointer **extra_data, bool **nullFlags, int32 *searchMode)

Returns a palloc'd array of keys given a value to be queried; that is,query is the value on the right-hand side of an indexable operator whose left-hand side is the indexed column.n is the strategy number of the operator within the operator class (seeSection 36.16.2). Often,extractQuery will need to consultn to determine the data type ofquery and the method it should use to extract key values. The number of returned keys must be stored into*nkeys. If any of the keys can be null, also palloc an array of*nkeysbool fields, store its address at*nullFlags, and set these null flags as needed.*nullFlags can be leftNULL (its initial value) if all keys are non-null. The return value can beNULL if thequery contains no keys.

searchMode is an output argument that allowsextractQuery to specify details about how the search will be done. If*searchMode is set toGIN_SEARCH_MODE_DEFAULT (which is the value it is initialized to before call), only items that match at least one of the returned keys are considered candidate matches. If*searchMode is set toGIN_SEARCH_MODE_INCLUDE_EMPTY, then in addition to items containing at least one matching key, items that contain no keys at all are considered candidate matches. (This mode is useful for implementing is-subset-of operators, for example.) If*searchMode is set toGIN_SEARCH_MODE_ALL, then all non-null items in the index are considered candidate matches, whether they match any of the returned keys or not. (This mode is much slower than the other two choices, since it requires scanning essentially the entire index, but it may be necessary to implement corner cases correctly. An operator that needs this mode in most cases is probably not a good candidate for a GIN operator class.) The symbols to use for setting this mode are defined inaccess/gin.h.

pmatch is an output argument for use when partial match is supported. To use it,extractQuery must allocate an array of*nkeysbools and store its address at*pmatch. Each element of the array should be set to true if the corresponding key requires partial match, false if not. If*pmatch is set toNULL then GIN assumes partial match is not required. The variable is initialized toNULL before call, so this argument can simply be ignored by operator classes that do not support partial match.

extra_data is an output argument that allowsextractQuery to pass additional data to theconsistent andcomparePartial methods. To use it,extractQuery must allocate an array of*nkeys pointers and store its address at*extra_data, then store whatever it wants to into the individual pointers. The variable is initialized toNULL before call, so this argument can simply be ignored by operator classes that do not require extra data. If*extra_data is set, the whole array is passed to theconsistent method, and the appropriate element to thecomparePartial method.

bool consistent(bool check[], StrategyNumber n, Datum query, int32 nkeys, Pointer extra_data[], bool *recheck, Datum queryKeys[], bool nullFlags[])

Returns true if an indexed item satisfies the query operator with strategy numbern (or might satisfy it, if the recheck indication is returned). This function does not have direct access to the indexed item's value, sinceGIN does not store items explicitly. Rather, what is available is knowledge about which key values extracted from the query appear in a given indexed item. Thecheck array has lengthnkeys, which is the same as the number of keys previously returned byextractQuery for thisquery datum. Each element of thecheck array is true if the indexed item contains the corresponding query key, i.e., if (check[i] == true) the i-th key of theextractQuery result array is present in the indexed item. The originalquery datum is passed in case theconsistent method needs to consult it, and so are thequeryKeys[] andnullFlags[] arrays previously returned byextractQuery.extra_data is the extra-data array returned byextractQuery, orNULL if none.

WhenextractQuery returns a null key inqueryKeys[], the correspondingcheck[] element is true if the indexed item contains a null key; that is, the semantics ofcheck[] are likeIS NOT DISTINCT FROM. Theconsistent function can examine the correspondingnullFlags[] element if it needs to tell the difference between a regular value match and a null match.

On success,*recheck should be set to true if the heap tuple needs to be rechecked against the query operator, or false if the index test is exact. That is, a false return value guarantees that the heap tuple does not match the query; a true return value with*recheck set to false guarantees that the heap tuple does match the query; and a true return value with*recheck set to true means that the heap tuple might match the query, so it needs to be fetched and rechecked by evaluating the query operator directly against the originally indexed item.

GinTernaryValue triConsistent(GinTernaryValue check[], StrategyNumber n, Datum query, int32 nkeys, Pointer extra_data[], Datum queryKeys[], bool nullFlags[])

triConsistent is similar toconsistent, but instead of Booleans in thecheck vector, there are three possible values for each key:GIN_TRUE,GIN_FALSE andGIN_MAYBE.GIN_FALSE andGIN_TRUE have the same meaning as regular Boolean values, whileGIN_MAYBE means that the presence of that key is not known. WhenGIN_MAYBE values are present, the function should only returnGIN_TRUE if the item certainly matches whether or not the index item contains the corresponding query keys. Likewise, the function must returnGIN_FALSE only if the item certainly does not match, whether or not it contains theGIN_MAYBE keys. If the result depends on theGIN_MAYBE entries, i.e., the match cannot be confirmed or refuted based on the known query keys, the function must returnGIN_MAYBE.

When there are noGIN_MAYBE values in thecheck vector, aGIN_MAYBE return value is the equivalent of setting therecheck flag in the Booleanconsistent function.

int compare(Datum a, Datum b)

Compares two keys (not indexed items!) and returns an integer less than zero, zero, or greater than zero, indicating whether the first key is less than, equal to, or greater than the second. Null keys are never passed to this function.

int comparePartial(Datum partial_key, Datum key, StrategyNumber n, Pointer extra_data)

Compare a partial-match query key to an index key. Returns an integer whose sign indicates the result: less than zero means the index key does not match the query, but the index scan should continue; zero means that the index key does match the query; greater than zero indicates that the index scan should stop because no more matches are possible. The strategy numbern of the operator that generated the partial match query is provided, in case its semantics are needed to determine when to end the scan. Also,extra_data is the corresponding element of the extra-data array made byextractQuery, orNULL if none. Null keys are never passed to this function.

void options(local_relopts *relopts)

Defines a set of user-visible parameters that control operator class behavior.

Theoptions function is passed a pointer to alocal_relopts struct, which needs to be filled with a set of operator class specific options. The options can be accessed from other support functions using thePG_HAS_OPCLASS_OPTIONS() andPG_GET_OPCLASS_OPTIONS() macros.

Since both key extraction of indexed values and representation of the key inGIN are flexible, they may depend on user-specified parameters.

Figure 64.1. GIN Internals

Updating aGIN index tends to be slow because of the intrinsic nature of inverted indexes: inserting or updating one heap row can cause many inserts into the index (one for each key extracted from the indexed item).GIN is capable of postponing much of this work by inserting new tuples into a temporary, unsorted list of pending entries. When the table is vacuumed or autoanalyzed, or whengin_clean_pending_list function is called, or if the pending list becomes larger thangin_pending_list_limit, the entries are moved to the mainGIN data structure using the same bulk insert techniques used during initial index creation. This greatly improvesGIN index update speed, even counting the additional vacuum overhead. Moreover the overhead work can be done by a background process instead of in foreground query processing.

The main disadvantage of this approach is that searches must scan the list of pending entries in addition to searching the regular index, and so a large list of pending entries will slow searches significantly. Another disadvantage is that, while most updates are fast, an update that causes the pending list to become“too large” will incur an immediate cleanup cycle and thus be much slower than other updates. Proper use of autovacuum can minimize both of these problems.

If consistent response time is more important than update speed, use of pending entries can be disabled by turning off thefastupdate storage parameter for aGIN index. SeeCREATE INDEX for details.

GIN can support“partial match” queries, in which the query does not determine an exact match for one or more keys, but the possible matches fall within a reasonably narrow range of key values (within the key sorting order determined by thecompare support method). TheextractQuery method, instead of returning a key value to be matched exactly, returns a key value that is the lower bound of the range to be searched, and sets thepmatch flag true. The key range is then scanned using thecomparePartial method.comparePartial must return zero for a matching index key, less than zero for a non-match that is still within the range to be searched, or greater than zero if the index key is past the range that could match.

Create vs. insert

Insertion into aGIN index can be slow due to the likelihood of many keys being inserted for each item. So, for bulk insertions into a table it is advisable to drop the GIN index and recreate it after finishing bulk insertion.

Whenfastupdate is enabled forGIN (seeSection 64.4.4.1 for details), the penalty is less than when it is not. But for very large updates it may still be best to drop and recreate the index.

maintenance_work_mem

Build time for aGIN index is very sensitive to themaintenance_work_mem setting; it doesn't pay to skimp on work memory during index creation.

gin_pending_list_limit

During a series of insertions into an existingGIN index that hasfastupdate enabled, the system will clean up the pending-entry list whenever the list grows larger thangin_pending_list_limit. To avoid fluctuations in observed response time, it's desirable to have pending-list cleanup occur in the background (i.e., via autovacuum). Foreground cleanup operations can be avoided by increasinggin_pending_list_limit or making autovacuum more aggressive. However, enlarging the threshold of the cleanup operation means that if a foreground cleanup does occur, it will take even longer.

gin_pending_list_limit can be overridden for individual GIN indexes by changing storage parameters, which allows each GIN index to have its own cleanup threshold. For example, it's possible to increase the threshold only for the GIN index which can be updated heavily, and decrease it otherwise.

gin_fuzzy_search_limit

The primary goal of developingGIN indexes was to create support for highly scalable full-text search inPostgreSQL, and there are often situations when a full-text search returns a very large set of results. Moreover, this often happens when the query contains very frequent words, so that the large result set is not even useful. Since reading many tuples from the disk and sorting them could take a lot of time, this is unacceptable for production. (Note that the index search itself is very fast.)

To facilitate controlled execution of such queries,GIN has a configurable soft upper limit on the number of rows returned: thegin_fuzzy_search_limit configuration parameter. It is set to 0 (meaning no limit) by default. If a non-zero limit is set, then the returned set is a subset of the whole result set, chosen at random.

“Soft” means that the actual number of returned results could differ somewhat from the specified limit, depending on the query and the quality of the system's random number generator.

From experience, values in the thousands (e.g., 5000 — 20000) work well.

btree_gin

B-tree equivalent functionality for several data types

hstore

Module for storing (key, value) pairs

intarray

Enhanced support forint[]

pg_trgm

Text similarity using trigram matching