Movatterモバイル変換

12.3.1. Parsing Documents

12.3.2. Parsing Queries

12.3.3. Ranking Search Results

12.3.4. Highlighting Results

PostgreSQL provides the functionto_tsvector for converting a document to thetsvector data type.

to_tsvector([configregconfig,]documenttext) returnstsvector

to_tsvector parses a textual document into tokens, reduces the tokens to lexemes, and returns atsvector which lists the lexemes together with their positions in the document. The document is processed according to the specified or default text search configuration. Here is a simple example:

SELECT to_tsvector('english', 'a fat  cat sat on a mat - it ate a fat rats');                  to_tsvector----------------------------------------------------- 'ate':9 'cat':3 'fat':2,11 'mat':7 'rat':12 'sat':4

In the example above we see that the resultingtsvector does not contain the wordsa,on, orit, the wordrats becamerat, and the punctuation sign- was ignored.

Theto_tsvector function internally calls a parser which breaks the document text into tokens and assigns a type to each token. For each token, a list of dictionaries (Section 12.6) is consulted, where the list can vary depending on the token type. The first dictionary thatrecognizes the token emits one or more normalizedlexemes to represent the token. For example,rats becamerat because one of the dictionaries recognized that the wordrats is a plural form ofrat. Some words are recognized asstop words (Section 12.6.1), which causes them to be ignored since they occur too frequently to be useful in searching. In our example these area,on, andit. If no dictionary in the list recognizes the token then it is also ignored. In this example that happened to the punctuation sign- because there are in fact no dictionaries assigned for its token type (Space symbols), meaning space tokens will never be indexed. The choices of parser, dictionaries and which types of tokens to index are determined by the selected text search configuration (Section 12.7). It is possible to have many different configurations in the same database, and predefined configurations are available for various languages. In our example we used the default configurationenglish for the English language.

The functionsetweight can be used to label the entries of atsvector with a givenweight, where a weight is one of the lettersA,B,C, orD. This is typically used to mark entries coming from different parts of a document, such as title versus body. Later, this information can be used for ranking of search results.

Becauseto_tsvector(NULL) will returnNULL, it is recommended to usecoalesce whenever a field might be null. Here is the recommended method for creating atsvector from a structured document:

UPDATE tt SET ti =    setweight(to_tsvector(coalesce(title,'')), 'A')    ||    setweight(to_tsvector(coalesce(keyword,'')), 'B')  ||    setweight(to_tsvector(coalesce(abstract,'')), 'C') ||    setweight(to_tsvector(coalesce(body,'')), 'D');

Here we have usedsetweight to label the source of each lexeme in the finishedtsvector, and then merged the labeledtsvector values using thetsvector concatenation operator||. (Section 12.4.1 gives details about these operations.)

PostgreSQL provides the functionsto_tsquery andplainto_tsquery for converting a query to thetsquery data type.to_tsquery offers access to more features thanplainto_tsquery, but is less forgiving about its input.

to_tsquery([configregconfig,]querytexttext) returnstsquery

to_tsquery creates atsquery value fromquerytext, which must consist of single tokens separated by the Boolean operators& (AND),| (OR) and! (NOT). These operators can be grouped using parentheses. In other words, the input toto_tsquery must already follow the general rules fortsquery input, as described inSection 8.11. The difference is that while basictsquery input takes the tokens at face value,to_tsquery normalizes each token to a lexeme using the specified or default configuration, and discards any tokens that are stop words according to the configuration. For example:

SELECT to_tsquery('english', 'The & Fat & Rats');  to_tsquery   --------------- 'fat' & 'rat'

As in basictsquery input, weight(s) can be attached to each lexeme to restrict it to match onlytsvector lexemes of those weight(s). For example:

SELECT to_tsquery('english', 'Fat | Rats:AB');    to_tsquery    ------------------ 'fat' | 'rat':AB

Also,* can be attached to a lexeme to specify prefix matching:

SELECT to_tsquery('supern:*A & star:A*B');        to_tsquery        -------------------------- 'supern':*A & 'star':*AB

Such a lexeme will match any word in atsvector that begins with the given string.

to_tsquery can also accept single-quoted phrases. This is primarily useful when the configuration includes a thesaurus dictionary that may trigger on such phrases. In the example below, a thesaurus contains the rulesupernovae stars : sn:

SELECT to_tsquery('''supernovae stars'' & !crab');  to_tsquery--------------- 'sn' & !'crab'

Without quotes,to_tsquery will generate a syntax error for tokens that are not separated by an AND or OR operator.

plainto_tsquery([configregconfig,]querytexttext) returnstsquery

plainto_tsquery transforms unformatted textquerytext totsquery. The text is parsed and normalized much as forto_tsvector, then the& (AND) Boolean operator is inserted between surviving words.

Example:

SELECT plainto_tsquery('english', 'The Fat Rats'); plainto_tsquery ----------------- 'fat' & 'rat'

Note thatplainto_tsquery cannot recognize Boolean operators, weight labels, or prefix-match labels in its input:

SELECT plainto_tsquery('english', 'The Fat & Rats:C');   plainto_tsquery   --------------------- 'fat' & 'rat' & 'c'

Here, all the input punctuation was discarded as being space symbols.

Ranking attempts to measure how relevant documents are to a particular query, so that when there are many matches the most relevant ones can be shown first.PostgreSQL provides two predefined ranking functions, which take into account lexical, proximity, and structural information; that is, they consider how often the query terms appear in the document, how close together the terms are in the document, and how important is the part of the document where they occur. However, the concept of relevancy is vague and very application-specific. Different applications might require additional information for ranking, e.g., document modification time. The built-in ranking functions are only examples. You can write your own ranking functions and/or combine their results with additional factors to fit your specific needs.

The two ranking functions currently available are:

For both these functions, the optionalweights argument offers the ability to weigh word instances more or less heavily depending on how they are labeled. The weight arrays specify how heavily to weigh each category of word, in the order:

{D-weight, C-weight, B-weight, A-weight}

If noweights are provided, then these defaults are used:

{0.1, 0.2, 0.4, 1.0}

Typically weights are used to mark words from special areas of the document, like the title or an initial abstract, so they can be treated with more or less importance than words in the document body.

Since a longer document has a greater chance of containing a query term it is reasonable to take into account document size, e.g., a hundred-word document with five instances of a search word is probably more relevant than a thousand-word document with five instances. Both ranking functions take an integernormalization option that specifies whether and how a document's length should impact its rank. The integer option controls several behaviors, so it is a bit mask: you can specify one or more behaviors using| (for example,2|4).

If more than one flag bit is specified, the transformations are applied in the order listed.

It is important to note that the ranking functions do not use any global information, so it is impossible to produce a fair normalization to 1% or 100% as sometimes desired. Normalization option 32 (rank/(rank+1)) can be applied to scale all ranks into the range zero to one, but of course this is just a cosmetic change; it will not affect the ordering of the search results.

Here is an example that selects only the ten highest-ranked matches:

SELECT title, ts_rank_cd(textsearch, query) AS rankFROM apod, to_tsquery('neutrino|(dark & matter)') queryWHERE query @@ textsearchORDER BY rank DESCLIMIT 10;                     title                     |   rank-----------------------------------------------+---------- Neutrinos in the Sun                          |      3.1 The Sudbury Neutrino Detector                 |      2.4 A MACHO View of Galactic Dark Matter          |  2.01317 Hot Gas and Dark Matter                       |  1.91171 The Virgo Cluster: Hot Plasma and Dark Matter |  1.90953 Rafting for Solar Neutrinos                   |      1.9 NGC 4650A: Strange Galaxy and Dark Matter     |  1.85774 Hot Gas and Dark Matter                       |   1.6123 Ice Fishing for Cosmic Neutrinos              |      1.6 Weak Lensing Distorts the Universe            | 0.818218

This is the same example using normalized ranking:

SELECT title, ts_rank_cd(textsearch, query, 32 /* rank/(rank+1) */ ) AS rankFROM apod, to_tsquery('neutrino|(dark & matter)') queryWHERE  query @@ textsearchORDER BY rank DESCLIMIT 10;                     title                     |        rank-----------------------------------------------+------------------- Neutrinos in the Sun                          | 0.756097569485493 The Sudbury Neutrino Detector                 | 0.705882361190954 A MACHO View of Galactic Dark Matter          | 0.668123210574724 Hot Gas and Dark Matter                       |  0.65655958650282 The Virgo Cluster: Hot Plasma and Dark Matter | 0.656301290640973 Rafting for Solar Neutrinos                   | 0.655172410958162 NGC 4650A: Strange Galaxy and Dark Matter     | 0.650072921219637 Hot Gas and Dark Matter                       | 0.617195790024749 Ice Fishing for Cosmic Neutrinos              | 0.615384618911517 Weak Lensing Distorts the Universe            | 0.450010798361481

Ranking can be expensive since it requires consulting thetsvector of each matching document, which can be I/O bound and therefore slow. Unfortunately, it is almost impossible to avoid since practical queries often result in large numbers of matches.

To present search results it is ideal to show a part of each document and how it is related to the query. Usually, search engines show fragments of the document with marked search terms.PostgreSQL provides a functionts_headline that implements this functionality.

ts_headline([configregconfig,]documenttext,querytsquery [,optionstext]) returnstext

ts_headline accepts a document along with a query, and returns an excerpt from the document in which terms from the query are highlighted. The configuration to be used to parse the document can be specified byconfig; ifconfig is omitted, thedefault_text_search_config configuration is used.

If anoptions string is specified it must consist of a comma-separated list of one or moreoption=value pairs. The available options are:

Any unspecified options receive these defaults:

StartSel=<b>, StopSel=</b>,MaxWords=35, MinWords=15, ShortWord=3, HighlightAll=FALSE,MaxFragments=0, FragmentDelimiter=" ... "

For example:

SELECT ts_headline('english',  'The most common type of searchis to find all documents containing given query termsand return them in order of their similarity to thequery.',  to_tsquery('query & similarity'));                        ts_headline                         ------------------------------------------------------------ containing given <b>query</b> terms and return them in order of their <b>similarity</b> to the <b>query</b>.SELECT ts_headline('english',  'The most common type of searchis to find all documents containing given query termsand return them in order of their similarity to thequery.',  to_tsquery('query & similarity'),  'StartSel = <, StopSel = >');                      ts_headline                      ------------------------------------------------------- containing given <query> terms and return them in order of their <similarity> to the <query>.

ts_headline uses the original document, not atsvector summary, so it can be slow and should be used with care. A typical mistake is to callts_headline forevery matching document when only ten documents are to be shown.SQL subqueries can help; here is an example:

SELECT id, ts_headline(body, q), rankFROM (SELECT id, body, q, ts_rank_cd(ti, q) AS rank      FROM apod, to_tsquery('stars') q      WHERE ti @@ q      ORDER BY rank DESC      LIMIT 10) AS foo;