US20090083243A1 - Cross-language search - Google Patents

Abstract

Methods, systems, and apparatus, including computer program products, in which a cross-language search can be performed. A search query can be examined to determine whether it is a candidate for cross-language searching. Upon identifying a candidate, the search can be performed using a translated search query associated with the search query. The results of the translated search query can be served to the client device submitting the search query.

Description

CROSS-REFERENCE
• This Application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Application Ser. No. 60/974,332, filed on Sep. 21, 2007.
BACKGROUND
• This disclosure is related to cross-language searching.
• The number of web pages written in certain languages (e.g., English) may be much larger than the number of web pages written in other languages. This disparity is due in large part to the number of speakers of the certain languages and the widespread availability of technology in the countries where those languages are spoken. Moreover, such pages are often better maintained and capable of providing more accurate and updated information about many subjects than are pages in other languages. Accordingly, people who speak a language that is not widely used on the web might nevertheless be interested in the information found on a web page available in other languages (e.g., English). Thus, searching the web pages written in the language used to submit a search query may not always satisfy a searcher's query, because relevant results might not even exist in the searcher's language.
• For example, people in Egypt are sometimes interested in information about traveling to places that are outside of the Middle East, studying in foreign universities, or learning about famous non-Egyptian people. However, there might not be many pages related to these topics in the language used to submit the query, i.e., the user's native language. Moreover, the pages that are in the user's native language may be of relatively low quality, since the authors of the page may not have ample access to such topics. Additionally, such users may not be able to issue queries in another language because the users do not speak another language. Thus, these users may be unable to find relevant and/or quality content.
SUMMARY
• In general, one aspect of the subject matter described in this specification can be embodied in a system that includes an input module, an evaluation module and an output module. The input module can receive an original search query in a first language, the original search query being a request for documents relevant to the original search. The evaluation module can receive a first set of relevance scores associated with results from the original search query and a second set of relevance scores associated with results from a translated search query in a second language. The evaluation module can compare the first and second sets of relevance scores to determine whether the original search query or the translated search query produces the most relevant results. The output module can provide search results including at least a portion of one or both of the results from the original search query or the results from the translated search query to a user based on the whether the original search query or the translated query has the most relevant results. Other embodiments of this aspect include corresponding methods, apparatus, and computer program products.
• An example method for performing cross-language searching can include: receiving an original search query in a first language; obtaining a translated search query associated with the original search query, the translated search query being in a second language; evaluating the translated search query to determine whether the translated search query is a candidate for a cross-language search based on: a relevancy score associated with the results of the translated search query, a translation confidence score associated with the translation, or a search query including an entity associated with the first or second languages; and serving a search results page based on the result of the evaluation.
• The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.
BRIEF DESCRIPTION OF DRAWINGS
• FIG. 1A is a block diagram depicting an architecture of a cross-language search system.
• FIG. 1B is a block diagram illustrating an example cross-language search system.
• FIG. 1C is a block diagram illustrating an example implementation of the query evaluation module.
• FIG. 2 is a block diagram illustrating a query evaluation module.
• FIG. 3 is a block diagram illustrating another example cross-language search system.
• FIGS. 4-7 are block diagrams illustrating some implementations of an evaluation module.
• FIGS. 8-10 are flowcharts illustrating example processes of providing a cross-language search.
• Like reference numbers and designations in the various drawings indicate like elements.
DETAILED DESCRIPTION
• In some implementations of this specification, systems and methods are realized that receive a search query from a client device and determine if a web page in a foreign language is a good match to the query. For example, if a user in Egypt enters a query in Arabic for a subject such as American football, there might not be many web pages in the query language that provide relevant results for the query. However, there may be relevant pages in other languages. For example, the Arabic query from the previous example can be translated to an English query, and compared to an English index of web pages. If there are more relevant results found among the web pages in the English language in this example, a list of relevant pages can be translated and provided to the user. In some implementations, a relevant web page can be translated and provided to the user based on the user selecting a web page from among the list of relevant pages. In some implementations, the list of relevant results can include both translated results and non-translated results.
• FIG. 1A is a block diagram depicting an architecture of across-language search system100. Thecross-language search system100 can receive a search query (Q)10ain a first language. In some implementations, the first language (e.g., the search language) can be identified based on an input mode associated with a client device. In other implementations, the first language can be identified based on a particular country code top-level domain being accessed. For example, Google, of Mountain View, Calif., provides many different web pages to many different countries, and the country code top-level domain can be used to identify from which language (or languages) the client expects to receive input. In such examples, if the user is accessing the “google.ru” country code top level domain, the client expects to receive input in Russian. In still further implementations, the user can specify the first language using the client. For example, the user can use a drop down menu to select one of a number of supported languages in which the user would like to enter the search query.
• In some implementations, thecross-language search system100 can send the search query to asearch engine20. Thesearch engine20 can receive queries and identify search results. For example, thesearch engine20 can retrieve relevant search results from an index of documents (e.g., from an index of web pages). Search results can include, for example, lists of web page titles, snippets of text extracted from those web pages, and hypertext links to those web pages, and may be grouped into a predetermined number of (e.g., ten) search results.
• In some implementations, thesearch engine20 can parse the search query (Q)10ainto component tokens (Q₁, Q₂, . . . , Q_n)10b.In some implementations, the tokens are words (or groups of words where the group of words forms a single idea, such as, e.g., “New York City”) that form the query. The component token (Q₁, Q₂, . . . , Q_n)10bcan be used to identify web pages in a first language that are relevant to the search query (Q)10a.In some implementations, the relevant web pages are those in the first language that include more than one of the component tokens (Q₁, Q₂, . . . , Q_n)10b.Such pages can be found, for example, in an index of web pages that is limited to thefirst language30. In some implementations, thesearch engine20 can derive relevance scores associated with the most relevant pages. In further implementations, thesearch engine20 can derive page rank scores associated with the quality of a web page as measured by the number of other sites linking to a page. Thesearch engine20 can also derive a score related to the search results (e.g., an information retrieval (IR) score). In some implementations, IR scores can be computed from, for example, dot products of feature vectors corresponding to a query and a document relevance, page rank scores, and/or combinations of IR scores and page rank scores, etc.
• In some implementations, thecross-language search system100 determines whether any of the component tokens (Q₁, Q₂, . . . , Q_n)10barelocal entities40 and excludes any search query (Q)10athat includes a local entity from cross-language search. Entities are those words or word phrases that identify persons, places, or things. Local entities are those entities that are associated with the language in which the query is input, and the person, place or thing is indicative of a place of origin where the language is primarily spoken. There can be physical indications of places of origin in which the language is primarily spoken, logical indication of places of origin, or historical indications of places of origin. For example, the Eiffel Tower is located in France, and thus the query “la Tour Eiffel” is a local entity for the French language. In another example, The Continental Army has both a logical and historical indication of a place of origin in the United States and is thus a local entity for the English language. In further examples, the term “MocκBá” is a local entity for the Russian language because it identifies Moscow, which is a city located in Russia.
• Lists of such entities can be derived from encyclopedias, location servers, public filings, etc., or can be provided from third-parties. Lists of local entities typically exclude individual generic terms, even if a generic term is a constituent word in a brand name. However, a multi-word brand name that includes generic terms can be included in the list of local entities (e.g., “Auburn University”).
• Lists of local entities also typically exclude foreign entities from inclusion. In one implementations, foreign entities are those entities that are not local entities to the language used to enter the query. In other implementations, foreign entities can be entities and translations of those entities that are local to another language. An entity can be local to one language and foreign to another language. For example, the token “Vladimir Putin” defines a local entity to the Russian language, while the same token would define a foreign entity to the English language. Thus, if a Russian user entered “Vladimir Putin” (in Russian) as part of a search query20a,the entity “Vladimir Putin” would be found in the local entities to the Russian language and the search can be excluded from performing a cross-language search. In another example, “MocκBá” (e.g., “Moscow”) is a local entity to the Russian language.
• In some implementations, search queries that include a token that matches a local entity are not eligible for a cross-language search and the search can be served in the language used to submit the search query. For example, if a search “Al-Qahira” is received (e.g., correlating to an Egyptian entity “Cairo”), the search query can be served from an index of web pages in the Arabic language because the token “Al-Qahira” matches a local entity to the Arabic language. Thus, the web pages authored in Arabic can be assumed to be more relevant to the search query than web pages written in another language (e.g., the web pages resulting from the translated search query “Cairo” in English). Moreover, the web pages written in another language may be difficult to understand after translation. For example, web pages written in English that are translated to Arabic can be more difficult to read for native Arabic speakers than web pages that are written in Arabic.
• In some implementations, the component tokens (Q₁, Q₂, . . . , Q_n)10bcan be compared to thelocal entities40 to determine the extent of any intersection of the set of component tokens (Q₁, Q₂, . . . , Q_n)10bwith the collection oflocal entities40. The extent of an intersection between thecomponent tokens10band the collection oflocal entities40 can be used to determine whether to perform a cross-language search.
• In other implementations, subsets of the search query (e.g., ngrams) can be compared to the collection oflocal entities40. In such implementations, thecross-language search system100 can compare each of the subsets of the search query to the collection oflocal entities40 to determine whether any of the subsets included in the search query match a local entity from the collection oflocal entities60. For example, if thecross-language search system100 received the search query “vacation homes finger lakes,” thecross-language search system100 could search the collection oflocal entities40 for inclusion of any of the subsets: “vacation,” “vacation homes,” “vacation homes finger,” “vacation homes finger lakes,” “homes,” “homes finger,” “homes finger lakes,” “finger,” and “finger lakes.” In some implementations, thecross-language search system100 can stop searching an subpart string if the first word of the subpart string contains no matches in the collection oflocal entities40. In additional implementations, thecross-language search system100 can discontinue the search after finding a subset that matches an entity from the collection oflocal entities40. For example, if thecross-language search system100 received the query “Manhattan gourmet restaurants,” thecross-language search system100 might immediately find the term “Manhattan” as matching a local entity from the collection oflocal entities40. Thus, searching for further matches can be considered unnecessary.
• In those instances where a cross-language search is to be performed, thecross-language search system100 can send the search query (Q)10ato atranslator25. Thetranslator25 uses the search query (Q)10ato derive a translated search query (Q′)50ain a second language. In some implementations, the translator can be an automated machine translator operable to receive a search query (Q)10ain a first language and to output a translated query (Q′)50ain a second language.
• The translated search query (Q′)50acan be divided into component tokens (Q′₁, Q′₂, Q′_n)50bby asearch engine20. In some implementations, thesearch engine20 or thecross-language search system100 can compare the component tokens (Q′₁, Q′₂, . . . , Q′_n)50bto a collection offoreign entities60. The collection offoreign entities60 can include those entities that are associated with the second language. For example, if the second language is Russian, the foreign entities could include locations (e.g., Red Square, the Kremlin, Moscow, or Siberia), as well as people (e.g., Lenin, Stalin, Gorbachev, or Yeltsin), or Russian brand names, products and companies associated with the Russian language. In those instances where the set of component tokens in the second language (Q′₁, Q′₂, . . . , Q′_n)50bincludes some intersection withforeign entities60, the translated query (Q′)50acan be considered a good candidate to perform a cross-language search. In other implementations, the translated query (Q′) can be separated into subsets and compared to the entities in the collection offoreign entities60 to determine whether a term included in the search query matches a foreign entity.
• In further implementations, the translator can produce atranslation confidence score70 associated with the translated search query (Q′)50a.Thetranslation confidence score70 associated with the translated query (Q′)50acan be used to determine whether the translation is likely to produce relevant results. In such implementations, thetranslation confidence70 can be compared to a translation confidence threshold to determine whether the translated query (Q′)50ais a candidate for the cross-language search. For example, a search query (Q)10acan have potentially a hundred or more translations. A greater number of potential translations can produce greater uncertainty with regard to a translated search query (Q′) chosen from the possible translations. Thus, there is some likelihood that the chosen translation is incorrect, and that any results generated using such a search query would be irrelevant to theoriginal search query10a.Translated search queries (Q′)50awhose translation confidence scores70 do not meet a translation confidence threshold can therefore be considered poor candidates for performing a cross-language search.
• In some implementations, thetranslator25 can provide multiple translations to thesearch engine20, based on multiple translations exceeding a translation confidence threshold. In other implementations, thetranslator25 can provide a single translation (e.g., the translated search query (Q′)50a) to thecross-language search system100, and the cross-language system can determine whether the translation confidence scores associated with that translated search query (Q′)50ameet a translation confidence threshold. Translated search queries (Q′)50athat do not meet the translation confidence threshold are excluded from cross-language search.
• If the translated search query (Q′) meets the confidence threshold, the translated search query (Q′)50acan be sent to asearch engine20 to search for documents (e.g., web pages) relevant to the translated search query (Q′)50a.Thesearch engine20 can use component tokens (Q′₁, Q′₂, . . . , Q′_n)50bof the translated search query (Q′)50ato find relevant web pages in thesecond language80. As described previously, thesearch engine20 can provide thecross-language search system100 with IR scores associated with the relevant web pages in thesecond language80.
• In some implementations, the respective relevance scores associated with the results of the translated search query (Q′)50acan be compared to a relevancy score threshold to determine whether to serve a translated web page as a result to the search query (Q)20a.For example, in some instances the results of the translated query (Q′)50amight have such low relevance scores that they will not be served to the searcher, even if the results of the translated search query (Q′)50ahave a higher relevance scores than the results of the search query (Q)20a.
• In some implementations, the relevance scores for the results of the translated search query (Q′)50acan be compared to the relevance scores for the results of the search query (Q)10a.The more relevant results can be served to the user as results90. In those instances in which the results of the translated search query (Q′)50ahave a higher relevance score than the relevance score of the results of the search query (Q)10a,the relevant web pages in the second language can be translated into the first language and served asresults90. In some implementations, the relevance scores can be biased to favor the results from the search query (Q)10abased on the likelihood that a translated web page is more difficult to read than a web page originally authored in the first language.
• In some implementations, a translated web page result of the translated search query can include a translation confidence score. In such implementations, the translation confidence score or quality associated with the translation of a translated web page can be compared to a translation confidence threshold to determine whether a translated web page should be served as aresult90 to thesearch query10a.For example, in some instances it may be determined that a translation of a relevant web page or an associated portion or snippet in the second language would be practically unreadable to the searcher. In such examples, thecross-language search system100 might serve only those results which are more relevant than local results and meet the translation confidence threshold.
• FIG. 1B is a block diagram illustrating an examplecross-language search system100. In some implementations, thesystem100 can include aninput module110, aquery evaluation module120, anentities data store130 and anoutput module140. Theinput module110 can receive a search query from a client device. In some implementations, theinput module110 can include search engine web page code that is provided to a user device, e.g., a computer device, and which renders a web page on the user device that enables the user to input a search query. In other implementations, theinput module110 can include a communications interface enabling the system to receive a query from a client device.
• In some implementations, thequery evaluation module120 can receive the search query from theinput module110 and can evaluate the search query to determine whether the search query is a candidate for performing a cross-language search. In these implementations, thequery evaluation module120 can filter the search queries to identify those queries that are more likely to have more relevant results from web pages in another language than would be found on web pages authored in the language used to input the query.
• In some implementations, thequery evaluation module120 can communicated with atranslation service122. Thetranslation service122 can receive a search query in a first language and can translate the search query into a second language and generate a translation confidence score associated with the translated search query. The translated search query and the translation confidence score can be provided to thequery evaluation module120.
• In some implementations, tokens from the search query that match entities (e.g., local entities) associated with the first language (e.g., search query language) can be identified by thequery evaluation module120 in conjunction with theentities data store130. In some implementations, local entities can be proper nouns (e.g., cities, places, etc.) that are associated with an area local to the language used to enter the search query.
• In other implementations, search queries that include a token that matches a local entity can nevertheless be eligible for cross-language search based on the relevance of web pages in the first language. In such implementations, when a search query includes a token that matches a local entity, the results associated with the local entity in the language of the query can be compared to a threshold relevancy score to determine whether the results in the first language are of poor quality. For example, a search query might include a token that matches a local entity; however, web pages in the language associated with that local entity might be sparse. Thus, it can be useful to determine whether web pages in another language might have more relevant results.
• In some implementations, thecross-language search system100 can identify when a token from a translated search query matches a foreign entity. A foreign entity is any entity associated with language other than the first language (e.g., “Cairo, Egypt” is a foreign entity to the English language). In some examples, an encyclopedia for the first or second language can be used to determine whether the translated query would be a good candidate for a cross-language search. In these examples, the tokens included in a search query or a translated search query that match an entity (e.g., local or foreign) can be identified using an encyclopedia website associated with the first language or the second language, respectively. In other examples, location servers can be used to identify locations associated with the search query or the translated search query.
• In compiling theentities data store130, in some implementations, generic words can be identified and removed from theentities data store130, even where the words represent brands or cities associated with the first or second language. Generic words, for example, are those words that do not have any particular association with that language. An example, of a generic word that also has a city or brand associated with it is “auburn.” In some instances, “auburn” generically refers to a color, in other instances it could refer to the city of Auburn, Ala., or the brand “Auburn University.” In another example, the term “apple” can generically refer to the fruit, while in other instances it could refer to a company. These words are removed from theentities data store130 because they can lead to irrelevant results if the user is attempting to enter the generic term instead of the city or brand.
• In some implementations, thequery evaluation module120 can retrieve historical statistics related to entities included in the encyclopedia. Based on the statistics associated with the entities, thequery evaluation module120 can determine a frequency associated with an entity's inclusion in previously received search queries. For example, the more frequently an entity appears in a search query, the more popular a topic related to that entity. More popular topics are more likely to have relevant web pages associated with the topics because interest as indicated by user queries is often mirrored by interest from publishers. Moreover, when there is no relevant content for a query, that query can become unpopular because it does not result in relevant results. Entities that are queried with a threshold frequency can be included in theentities data store130, because such entities can be assumed to have a relevant topic corpus in associated web pages.
• In yet further implementations,query evaluation module120 can determine a quality of translation of the search query. The quality of the translation can be used to determine whether the search query is a good candidate for cross-language searching. For example, if a translation is poor, e.g., a translation confidence indicates that the translation has relatively a high probability of being incorrect, then the results from the poorly translated query might also be poor, or may not be at all relevant to the original search query. Various translation services provide confidence scores for a translation to determine which word is most likely to be implicated by usage. In some implementations, the translation service can provide a confidence score for the overall translation of the search query based on a confidence associated with the translation of each word in the search query. Thus, a search query might be difficult to translate accurately, and a confidence value or quality value can be used to filter search queries that use such words from being used for cross-language searches. For example, a particular word might have multiple translations into a foreign language or might have no translation at all. Such words would be difficult to translate, and result in a lower confidence score related to the translation.
• In some implementations, the translated search query can be tested to determine whether the translated search query results in relevant web page matches. In such implementations, thequery evaluation module120 can send the translated search query to asearch engine124. Thesearch engine124 can use the translated search query to identify relevant documents in document corpora126 (e.g., a index in the second language). The identified documents and the translated search query can be used to derive relevance scores associated with results of the translated search query. In some implementations, the relevance score can include both a relevance component and a quality component. Upon receiving a relevance score associated with the results from a translated search query received from thesearch engine124, thequery evaluation module120 can compare the relevance score associated with the results of the search query to a relevance threshold to determine whether or not the translated search query is likely to be relevant to the search query. For example, in some instances, the results from the translated search query might be so irrelevant that the results are not even offered to the searcher.
• In further implementations, the activity of users with respect to the results of previous queries matching the translated query can be used to determine whether the results from similar queries were satisfactory to other users, thereby indicating that similar search queries have produced relevant results. For example, thesearch engine124 can log when the client device requests a result, e.g., clicks on a search result and fetches a landing page associated with the result. If the client device returns to the search page to enter a new query immediately after visiting the landing page, it is likely that the results were not relevant. Conversely, if the client device dwells on one or more of the result pages for a period of time, it is more likely that the results were relevant to the search query. Thequery evaluation module120 can thus determine whether the translated search query is likely to yield quality results based on the activity by other users that used a search query that matched the translated query.
• In further implementations, thequery evaluation module120 can determine whether the search query in the first language has more relevant results than the translated search query in a second language. In such implementations, thequery evaluation module120 can send both the search query and the translated search query to asearch engine124. Thesearch engine124 can derive first set of relevance scores associated with a first set of results from the original search query, and a second set of relevance scores associated with the second set of results from the translated search query. In some implementations, if the search query yields better results than any of the results from the translated search query as indicated by the first and second relevance scores, respectively, the results from the search query can be served to the searcher. Alternatively, if the results from the translated search query are more relevant than the results from the search query as indicated by the respective relevance scores, the results from the translated search query can be served to the searcher. Such a comparison assumes that the relevance measures for different languages are commensurable across the respective corpora.
• In some implementations, a web page (or multiple web pages) in a second language can be translated to produce a translated version of the web page in a first language. The translated version of the web page(s) in the first language, for example, can be compared to the search query in the first language to derive a relevance score. For example, a search query received in Spanish can be translated to English. If a highest ranked English web page is more relevant to the translated English query than a highest ranked Spanish web page is to the Spanish query, the English web page can be translated into Spanish. In this example, after the English web page is translated into Spanish, the Spanish query can be compared to the translated English web page. If the translated English web page is still more relevant to the Spanish query than the Spanish web page is to the Spanish query, the translated English web page can be provided as a result. If the translated English web page is not more relevant to the Spanish query than the Spanish web page is to the Spanish query, the Spanish web page can be provided to the search as a result. Thus, the relevance score of a translated search page to the original query can be compared to the relevance score of a web page written in the first language to the original search query.
• In some implementations, the results served to the client device can include both translated web pages and web pages written in the first language based on the relevance scores associated with the respective results. For example, if the results from an English query have relevance scores of 10, 9, and 2, while the results from a Spanish query have relevance scores of 8, 8 and 6, the cross-language search system could provide the two most relevant results from the English query, followed by the results of the Spanish query, and ending with the least relevant result from the English query. In other implementations, the relevance scores of the results of the original search query can be aggregated to produce a first aggregate relevance score and the relevance score of the original search query can be aggregated to produce a second aggregate relevance score. In some implementations, the aggregation can include summing the respective relevance scores to produce the first and second aggregate relevance score. The first aggregate relevance score can be compared to the second aggregate relevance score to determine whether to serve the results associated with the translated search query.
• Theoutput module140 is operable to receive the results from thequery evaluation module120 and serve the results to a client device. In some implementations, theoutput module140 can assemble a web page containing search results for transmission to a client device. In other implementations, theoutput module140 can serve the results to another module operable to format and transmit the results to a client device.
• FIG. 1C is a block diagram illustrating an example implementation of thequery evaluation module120. Thequery evaluation module120 can receive a search query (Q) in a first language. Thequery evaluation module120 can send the search query (Q) to asearch engine124. Thesearch engine124 can perform a search ofdocument corpora126 associated with the first language and based on the search query (Q) to produce a set of search results (R_Q).
• Thequery evaluation module120 can also provide the received search query to anentity filter154. Theentity filter154 can compare the tokens (e.g., words, or groups of words that form a single concept) to entities in a localentities data store156 and to entities in a foreignentities data store158. If theentity filter154 determines that the search query (Q) includes a token that matches an entity from the localentities data store156, thequery evaluation module120 can determine that the search query (Q) is not eligible for cross-language search. Accordingly, only the set of search results (R_Q)152 is served to the searcher.
• If the search query (Q) does not include a token that matches an entity from the localentities data store156, tokens from the search query (Q) can be compared to a list of foreign entities in a foreignentities data store158. If theentity filter154 determines that the search query (Q) includes a token that matches a foreign entity, thequery evaluation module120 can respond to the received search query (Q) with results from a translated search query (Q′)162 using atranslation service122 and thesearch engine124 as indicated bypaths159 and163, thereby avoiding further evaluation by thequery evaluation filter120. However, if theentity filter154 determines that the search query (Q) does not include a token that matches a foreign entity from the foreignentities data store158, theentity filter154 can determine that the query should undergo further evaluation and instruct thetranslation service122 to translate the search query (Q) to produce a translated search query (Q′).
• While this application describes a list of local entities that is derived from an encyclopedias and other reference sources and excludes certain generic terms and terms that are local to other languages, the list of local entities and list(s) of foreign entities, in some implementations, can be generated by another system and can be provided to the cross-language search system for use in determining whether a search term includes a token that matches a local or foreign entity from the lists provided by such other system(s).
• Atranslation evaluation filter160 can receive the translated search query (Q′) from thetranslation service122. Thetranslation evaluation filter160 can also receive a translation confidence score from thetranslation service122. Thetranslation evaluation filter160 can compare the translation confidence score from thetranslation service122 to a threshold confidence score to determine whether the translation is of high enough quality to be a good candidate for performing a cross-language search. If the translated search query (Q′) meets the confidence threshold, thetranslation evaluation filter160 can transmit the translated search query (Q′) to asearch engine124. In some implementations, the most likely translation of the original search query can be returned to thequery evaluation module120, and a translation confidence score associated with this translation can be compared to the translation confidence threshold. In other implementations, all translations of the original search query that exceed the translation confidence threshold can be returned to thequery evaluation module120.
• Thesearch engine124 can perform a search ofdocument corpora126 associated with the second language and based on the translated search query (Q′) produce a set of search results (R_Q′)162 in the second language. Thesearch engine124 can provide the results (R_Q′)162 of the translated search query (Q′) to a relevancy results filter164. Thesearch engine124 can also provide a relevancy score (e.g., IR score) associated with the results (R_Q′)162 of the translated search query (Q′) to the relevancy results filter164.
• The relevancy results filter164 can determine whether the results (R_Q′)162 of the translated search query (Q′) are relevant enough to be served to a searcher. The relevancy results filter164 can compare the results (R_Q′)162 of the translated search query (Q′) to a threshold relevancy to ensure that the results (R_Q′)162 of the translated search query (Q′) are relevant enough to be served to the searcher. In other examples, the relevancy results filter164 can retrieve statistics associated with the translated search query (Q′) and determine the relevancy of the results based on click analysis associated with the results (R_Q′)162. In those instances where the results (R_Q′)162 are determined not to meet a threshold relevancy, only the results (R_Q)152 of the original query (Q) are served to the searcher. If the results (R_Q′)162 of the translated search query (Q′) are determined to meet a threshold relevancy, the results (R_Q′)162 of the translated search query (Q′) are provided to arelevancy comparison filter168.
• Therelevancy comparison filter168 can receive the results (R_Q)152 of the original search query (Q) and the results (R_Q′)162 of the translated search query (Q′). Therelevancy comparison filter168 can also receive first relevancy scores (e.g., IR score) of the results (R_Q)152 of the original search query (Q) and second relevancy scores of the results (R_Q′)162 of the translated search query (Q′). Therelevancy comparison filter168 can compare the first and second relevancy scores to determine whether to serve the results (R_Q)152 of the original search query and/or the results (R_Q′)162 of the translated search query. For example, if the results (R_Q′)162 of a translated search query (Q′) have a higher aggregate relevance score than the results (R_Q)152 of the original search query, the results (R_Q′)162 of the translated search query can be offered assearch results170 to the searcher. Otherwise, the results (R_Q)152 of the original search query are served assearch results170 to the user. Other serving schemes based on relevancy can also be used.
• The search results170 can be served to the client in a variety of ways. In some implementations, the search results170 can include links to an alternative result page (e.g., including translated web page results) identified by thequery evaluation module120 if a cross-language search has been performed by thequery evaluation module120. In other implementations, the search results170 can include an aggregation of the results identified by thequery evaluation module120. For example, the search results170 can include both results (R_Q)152 of the original search query (Q) and results (R_Q′)162 of the translated search query (Q′). In still other implementations, the search results170 can include partitionedresults176. For example, the search results170 can include only results (R_Q)152 of the original search query or results (R_Q′)162 of the translated search query, or a link to one set of results from a page presenting the other. In yet further implementations, the search results170 can include translatedresults178. For example, if the search results170 include results (R_Q′)162 of the translated query, the results (R_Q′) can be translated before presentation to the searcher.
• FIG. 2 is a block diagram illustrating aquery evaluation module120. In some implementations, thequery evaluation module120 can include alocal entity filter210. Thelocal entity filter210 can determine whether the search query is related to a local entity. Thelocal entity filter210 can compare tokens included in the search query to a local entities data store130ato determined whether the search query is related to a local entity (e.g., includes a token that matches a local entity). In those instances that the search query is related to a local entity, it is likely that the results from the web pages written in the first language will be relevant to the search query. In such instances, thequery evaluation module120 can exclude the search query from a cross-language search.
• In some implementations, thequery evaluation module120 can include aforeign entity filter220. Theforeign entity filter220 can determine whether the search query or translated search query is related to a foreign entity (e.g., includes a token that matches a foreign entity). In some implementations, theforeign entity filter220 can include a database of foreign entities, against which the tokens associated with the search query or the translated query can be compared. In those instances when the search query is related to a foreign entity (e.g., includes a token that matches a foreign entity), it is likely that the results from the web pages written in the second language will be relevant to the search query. In such instances, thequery evaluation module120 can serve the search results based on search engine results124 from a translated search query associated with the original search query.
• In some examples, the local entities data store130aand the foreignentities data store130bcan be compiled to include proper nouns associated with a first language (e.g., the query language) or second language (e.g., the translated query language), respectively. For example, the local entities and foreign entities can include brand names used in the respective languages, locations associated with the languages, celebrities, television shows, products, etc. Such entities can be presumed to have better results in their respective language than they do in another language. In some implementations, the local entities data store130aand the foreignentities data store130bcan be compiled, for example, using encyclopedias (e.g., electronic encyclopedias) and/or location servers, such as those used to provide maps. In some implementations, generic terms and/or locations local to the language used to submit the query can be removed from the local entities data store130aand/or the foreignentities data store130b.
• In some implementations, the foreign and local entities included in the database can be compared to query statistics (e.g., how many times the entity has previously been queried) to ensure that each foreign or local entity from the encyclopedia should be included in the list of entities. For example, an entity that has not appeared in many previous queries would probably not result in relevant web pages being found based on the implication that if no searchers are interested in the entity, there are likewise few publishers interested in the entity. Alternatively, the fact that an entity has not appeared in many previous queries can be an indication that the entity is a spelling or grammar mistake, and should be eliminated from consideration.
• In some implementations, thequery evaluation module120 can be coupled to atranslation service122. Thetranslation service122, in some examples, can receive a search query and translate the search query to produce a translated search query. In some implementations, thetranslation service122 can also produce a confidence value associated with the translated query. The confidence value, for example, can provide a confidence associated with a translation of the search query.
• In some implementations, thequery evaluation module120 can include atranslation quality filter230. Thetranslation quality filter230 can determine the likelihood of whether a translated search query has been accurately translated. For example, if a word in a first language (e.g., the query language) has many translations in a second language, it can be difficult to determine which of the multiple words is the proper word to use in the translation. The more potential translations associated with a word, the less likely that the chosen translation is correct. Thus, in those instances where the translated web query does not meet a confidence/quality threshold, thetranslation quality filter230 can determine that the translated search query received from thetranslation service122 will not result in relevant results to the original query. In such instances, thequery evaluation module120 can exclude the translated search query from eligibility for the cross-language search and serve the results to the original search query.
• In some implementations, thequery evaluation module120 can include athreshold relevancy filter240. Thethreshold relevancy filter240 can send the translated search query to asearch engine124. The search engine can identify relevant web pages from adocument corpora126 and provide the relevant web pages and a relevancy score associated with relevant web pages. Thethreshold relevancy filter240 can compare the relevancy score of the results received from thesearch engine124 associated with the translated search query to a threshold relevancy score to determine whether or not the translated search query produces results that are relevant enough to be served to the user. For example, a user might enter a German search query. The German search query can be translated to produce an English search query. The English search query can be used to identify English web pages related to the English search query based on a relevance score associated with each of the English web pages. The relevance score of the English web pages can be compared to a threshold relevance score to determine whether the English web pages are relevant enough to be returned to the searcher. For example, if the English pages are relatively irrelevant, the English pages would not be translated for presentation as a result of the German search query. In those instances where the relevance scores associated with English web pages do not meet a threshold relevance score, German web pages that result from the German search query can be served to the searcher.
• In some implementations, thequery evaluation module120 can include arelevancy comparison filter250. Therelevancy comparison filter250 transmit the original search query and the translated query to asearch engine124. Thesearch engine124 can identify relevant web pages from adocument corpora126 and provide the relevant web pages to the search query and translated search query. Thesearch engine124 can also provide first relevancy scores associated with relevant web pages to the original search query and second relevancy scores associated with relevant web pages to the translated search query. In some implementations, therelevancy comparison filter250 can compare the first relevancy scores to the second relevancy scores. If the second relevancy scores are higher than the first relevancy scores, the translated search query can be used. Otherwise, the original search query can be used. In some implementations, the second relevancy scores associated with the results from the translated search query can be discounted. For example, a web page in English could be translated to German, but it would probably be difficult to read. Thus, in these implementations, a readability factor associated with the translated web pages can be considered in the comparison, thereby favoring web pages in the language in which the search query was submitted.
• In some implementations, such as shown inFIG. 2, each of the filters must be satisfied for a search query to be eligible for cross-language searching. In other implementations, any of the filters may be satisfied for the search query to be eligible for cross-language searching.
• Other implementations in addition to the implementations ofFIGS. 1A-C and2 can also be used to facilitate cross-language searching. For example,FIGS. 3-7 outline different implementations in which only a single filter can be included in the query evaluation module. Combinations of any subset of the filters can also be used.
• FIG. 3 is a block diagram illustrating another examplecross-language searching system300. In the implementation ofFIG. 3,entities databases340,350 can be provided by a system remote to thecross-language searching system300. Thecross-language searching system300 can include aninput module310. Theinput module310 can be operable to receive a search query. In some implementations, theinput module310 can receive the search query from a client device. In other implementations, theinput module310 can receive the search query through a search engine interface.
• Thecross-language searching system300 communicates with atranslation service320. Thetranslation service320 can receive the search query from theinput module310. In some implementations, thetranslation service320 can translate the search query from a first language to produce a translated search query in a second language. In some implementations, the translation module can translate the search query to produce multiple translated search queries that can be used to perform a cross-language search. In some implementations, thetranslation module320 can also produce a translation confidence score associated with the translation. The translation confidence score can identify a confidence associated with the translated search query.
• In some implementations, the cross-language searching system can include aquery evaluation module330. Thequery evaluation module330 can evaluate the search query to determine whether the search query is a candidate for performing a cross-language search. In some implementations, the query evaluation module can identify those queries that would result in relevant results when translated to language. For example, thequery evaluation module330 can compare a confidence score associated with the translation to a threshold confidence score. In such examples, thequery evaluation module330 can exclude translated search queries with confidence scores below the threshold from cross-language searching, because translated search queries whose translation confidence scores below a threshold will likely not result in relevant results.
• In other examples, thequery evaluation module330 can determine whether the translated search query is associated with an entity (or entities) that are local to the language associated with the translated search query. In such examples, a user might enter “Al Qahira,” and the query evaluation module can determine that “Al Qahira” is a local entity to the Arabic language. Thus, the query evaluation module can determine that results in Arabic are likely to be more relevant to the search query than results from English web pages.
• In still further examples, thequery evaluation module330 can determine if the results from the translated search query meet a minimum relevance threshold. In such examples, thequery evaluation module330 can transmit the translated search query to asearch service360. Thesearch service360 can provide thequery evaluation module330 with results relevant to the translated search query and a relevance score associated with results from the translated search query.
• In some examples, thequery evaluation module330 can transmit the original search query and a translated search query to asearch service360. Thesearch service360 can provide a first set of results relevant to the original search query and a second set of results relevant to the translated search query to thequery evaluation module360. Thesearch service360 can also provide first relevance scores associated with the first set of results and second relevance scores associated with the second set of results. The query evaluation module can determine if the first relevance scores are higher than the second relevance scores. In some implementations, the first relevance scores and second relevance scores can be computed similarly, such that the scores from similarly relative results in separate languages are comparable. In other implementations, the first relevance scores or second relevance scores can undergo a transformation to a universal relevance scale.
• FIG. 4 is a block diagram illustrating an implementation of aquery evaluation module330a. In some implementations, thequery evaluation module330acan include a localentity filter module410 operable to determine whether the original search query is related to an entity local to the language of the original search query. In these implementations, the localentity filter module410 can determine whether any of the tokens included in the original search query match any entities included in a localentities data store340. If one or more of the tokens of the search query are local to the language of the original search query, the query evaluation module can send the original search query (and/or results from the original search query) to an output module (e.g.,output module370 ofFIG. 3).
• Alternatively, if the tokens of the original search query are not local to the language of the original search query, the query evaluation module, in some implementations, can include a foreign entities filtermodule430. The foreign entities filtermodule430 can determine if the search query is related to a foreign entity (e.g., an entity foreign to the language of the original search query). For example, if a searcher entered the equivalent of “Yankee Stadium” in Arabic, it can be assumed that the results from English web pages would be more relevant to the search query than results from Arabic web pages. In these implementations, the foreign entities filtermodule430 can identify whether any of the tokens included in the search query match any of the entities included in a foreignentities data store350. In some implementations, if the search query is not related to any of the foreign entities included in the foreign entities database440, the original search query can be transmitted to an output module (e.g.,output module370 ofFIG. 3). In other implementations, if the search query is not related to any of thelocal entities340 and the search query is not related to any of theforeign entities350, thequery evaluation module330 can perform further evaluation of the search query and/or translated search query.
• In some implementations, when the search query is associated with one of theforeign entities350, thequery evaluation module330acan include aquery frequency module450 operable to determine whether the translated search query is used frequently enough that it is likely to produce relevant results. For example, if a translated search query is not used with some frequency, it is likely that there is not much interest in the subject by either searchers or publishers. Thus, it is likely that there are not many web pages relevant to the translated search query, and the translated search query would be unlikely to yield better results than the original search query. In some implementations, thequery frequency module450 can derive the popularity of a search query based on query statistics retrieved from thesearch service360. The query statistics can include the number of times that the search query has previously been issued by clients. If the query statistics associated with the translated search query meet a threshold frequency, the translated search query can be sent to an output module (e.g.,output module370 ofFIG. 3). If the query statistics associated with the translated search query do not meet a threshold frequency, the original search query can be sent to the output module. In other implementations, if a threshold frequency is not found by thequery frequency module450, thequery evaluation module330acan perform further evaluation of the search query and/or translated search query.
• FIG. 5 is a block diagram illustrating an implementation of anotherquery evaluation module330b. In some implementations, thequery evaluation module330bcan include atranslation evaluation module510. Thetranslation evaluation module510 can receive a confidence score associated with the translated search query from atranslation service320. Thetranslation service320 can derive a translation confidence score based on the confidence associated with the translation. The confidence can be based on a likelihood that the translation is correct, for example, based on a number of alternative translations available for the search query.
• In some implementations, thetranslation evaluation module510 can compare the confidence score associated with the translated search query to a threshold confidence score. In such implementations, the threshold confidence score can identify a minimum confidence associated with the translated search query that is likely to produce relevant results. If the translation confidence score is below the threshold confidence score, the original search query can be sent to an output module (e.g.,output module370 ofFIG. 3). Otherwise the translated search query can be sent to the output module.
• FIG. 6 is a block diagram illustrating an implementation of anotherquery evaluation module330c.In some implementations, thequery evaluation module330ccan include arelevance module610 operable transmit the translated search query to asearch engine360. Thesearch engine360 can derive relevance score associated with the results identifying web pages relevant to translated search query and provide the relevance scores of the results to therelevance module610.
• In some implementations, thequery evaluation module330ccan include acomparison module620 operable to compare the relevance score received fromsearch engine360 through therelevance module610 to a threshold relevance score. The threshold relevance score can be a measure of the minimum relevance of a translated search query that is likely to produce results that would be of interest to a searcher. If the relevance score associated with the translated query is below the threshold relevance score, thequery evaluation module330ccan send the original search query to an output module (e.g.,output module370 ofFIG. 3). Alternatively, if the relevance score associated with the translated search query is above the threshold relevance score, thequery evaluation module330ccan send the translated search query to the output module.
• FIG. 7 is a block diagram illustrating an implementation of anotherquery evaluation module330d.In some implementations, thequery evaluation module330dcan include afirst relevance module710, and asecond relevance module720. Thefirst relevance module710 can send the original search query to asearch service360. Thesearch service360 can identify first set of relevance scores associated with an original search query and provide the first set of relevance scores and a first set of results including relevant web pages in the first language to thefirst relevance module710.
• Thesecond relevance module720 can transmit the translated search query to thesearch service360. Thesearch service360 can identify a second set of relevance scores associated with a translated search query. Thesearch service360 can provide the second set of relevance scores and a second set of results including relevant web pages in the second language to thesecond relevance module720.
• In some implementations, thequery evaluation module330dcan include acomparison module730. Thecomparison module730 can compare the first set of relevance score and the second relevance score to determine whether an original search query or a translated search query result in more relevant search results. In some implementations, if a first relevance score is higher than a second relevance score, a first set of result associated with the original search query can be sent to an output module (e.g.,output module370 ofFIG. 3). If the second relevance score is higher than the first relevance score, a second set of results associated with the translated search query can be sent to the output module. In some implementations, the comparison module can be biased to select the results associated with the original search query if the second relevance score is not significantly higher than the first relevance score.
• In some implementations, any of the implementations of thequery evaluation module330a-ddisclosed with respect toFIGS. 4-7 can be combined with any one or more of the otherquery evaluation modules330a-d.
• FIG. 8 is a flowchart illustrating anexample process800 of providing a cross-language search. Theprocess800 receives a search query (805). The search query can be received, for example, by an input module (e.g.,input module110 ofFIG. 1B). In some implementations, the search query can be received from a search engine. For example, a cross-language search system can be an add-on component or a third party search component operable to receive the search query through an application programming interface (API). In other implementations, a cross-language search system can be integrated into the search engine, such that the input module is a search engine interface web page. In still further implementations, the input module can include a client software component operating on a client device operable to intercept a search query.
• The process obtains a translated search query (810). The translated search query can be obtained, for example, from a translation service (e.g.,translation service122 ofFIG. 1B) by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the translation service can produce a translated search query and a confidence score associated with the translated search query. The confidence score can indicate a confidence that the translation is correct.
• The process transmits the search query and translated search query to a search engine (815). The search query and translated search query can be transmitted to the search engine, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the translated search query can include multiple translations of the original search query, each of which are different translations of the original search query (e.g., in the same language or different languages).
• The process receives results and relevance scores (820). The results and relevance scores can be received, for example, from the search engine (e.g.,search engine124 ofFIG. 1B) by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). The results and relevance scores can include a first set of results and a first set of relevance scores associated with the original search query. The results and relevance scores can also include a second set of results and a second set of relevance scores associated with the translated search query.
• The process evaluates the results and relevance scores from the search query and translated search query (825). The results and relevance scores can be evaluated, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the search query can be compared to a list of local and foreign entities, the translation quality can be tested using a confidence score, a relevance score of translated search query results can be tested against a minimum relevance threshold, and/or the original search query results can be compared to results from the translated search query based on a first and second relevance score respectively associated with the results of the queries.
• The process serves results based on the results of the evaluation (830). The results can be served, for example, by an output module (e.g.,output module140 ofFIG. 1B). In some implementations the results can include results from the original search query and/or the translated search query. In other implementations the results can include a list of web pages that are most relevant to the original search query or to the translated search query.
• FIG. 9 is a flowchart illustrating anexample process900 of providing a cross-language search. The process receives a search query (905). The search query can be received, for example, by an input module (e.g.,input module110 ofFIG. 1B). In some implementations, the search query can be received from a search engine interface. For example, a cross-language search system can be an add-on component or a third party search component operable to receive the search query from a search engine interface. In other implementations, a cross-language search system can be integrated into the search engine, such that the input module is a search engine interface web page. In still further implementations, the input module can include a client software component operating on a client device operable to intercept a search query.
• The process obtains a translated search query (910). The translated search query can be obtained, for example, from a translation service (e.g.,translation service122 ofFIG. 1B) by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the translation service can produce a translated search query and a confidence score associated with the translated search query. The confidence score can indicate a confidence that the translation is correct.
• The process obtains relevance scores of results from the original search query and translated search query (915). Relevance scores for the original search query and translated search query results can be obtained, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B) from a search engine (e.g.,search engine124 ofFIG. 1B). In some implementations, the original search query and translated search query can be transmitted to a search engine. The search engine can provide a first set of results and a first set of relevance scores associated with the original search query and a second set of results and second set of relevance scores associated with the translated search query.
• The process can compare the results of the search query and the translated search query (920). The results of the search query and the translated search query can be compared, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, comparing the results of the original search query and the translated search query can include comparing the first set of relevance scores and to the second set of relevance scores. In some implementations, the first set of relevance scores and second set of relevance scores can be compared to determine whether to use the original search query or the translated search query based on which search query has more relevant results as indicated by the respective relevance scores.
• If the process determines that the original search query yields more relevant results, the process serves the results of the original search query (925). The original search query results can be served, for example, by an output module (e.g.,output module140 ofFIG. 1B) in conjunction with a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the results of the search query can be served to the client device that submitted the search query.
• If the process determines that the translated search query yields more relevant results, the process serves translated search query (930). The translated search query can be served, for example, by an output module (e.g.,output module140 ofFIG. 1B) in conjunction with a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the results of the translated search query can be served to the client device that submitted the search query.
• FIG. 10 is a flowchart illustrating another example method of providing a cross-language search. The process receives a search query (1005). The search query can be received, for example, by an input module (e.g.,input module110 ofFIG. 1B). In some implementations, the search query can be received from a search engine interface. For example, a cross-language search system can be an add-on component or a third party search component operable to receive the search query from a search engine interface. In other implementations, a cross-language search system can be integrated into the search engine, such that the input module is a search engine interface web page. In still further implementations, the input module can include a client software component operating on a client device operable to intercept a search query.
• The process obtains a translated search query (1010). The translated search query can be obtained, for example, from a translation module (e.g.,translation service122 ofFIG. 1B) by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the translation service can provide a translated search query and a confidence score associated with the translated search query. The confidence score can indicate a confidence that the translation is correct.
• The process obtains results from the translated search query (1015). The translated search query results can be obtained, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, the translated search query can be transmitted to a search engine (e.g.,search engine124 ofFIG. 1B). The search engine can identify a set of web pages relevant to the translated search query and a set of relevance scores associated with the set of relevant web pages. The search engine can provide the results (e.g., relevant web pages and associated relevance scores) to the query evaluation module.
• The process determines whether the results of the translated search query are satisfactory (1020). The determination of whether the translated search query results are satisfactory, for example, can be performed by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B). In some implementations, determination of whether the results from translated search query are satisfactory can be based on the set of relevance scores (e.g., individually or in aggregate) meeting a minimum relevance score threshold, the relevance scores of the results associated with the translated search query being better than a relevance score of the results associated with an original search query, the translation confidence score associated with the translated search query meeting a minimum confidence threshold, or an token from the search query or translated search query matching an entity from an entity data store.
• If the process determines that the results of translated search query are not satisfactory, the process serves the results of the original search query (1025). The results of original search query can be served, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B) in conjunction with an output module (e.g.,output module140 ofFIG. 1B). In some implementations, the results of the search query can be served to the client device that submitted the search query.
• If the process determines that the results of the translated search query are satisfactory, the process serves the results of the translated search query (1030). The results of the translated search query can be served, for example, by a query evaluation module (e.g.,query evaluation module120 ofFIG. 1B) in conjunction with an output module (e.g.,output module140 ofFIG. 1B). In some implementations, the results of the translated search query can be served to the client device that submitted the search query.
• Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a tangible program carrier for execution by, or to control the operation of, data processing apparatus. The tangible program carrier can be a propagated signal or a computer readable medium. The propagated signal is an artificially generated signal, e.g., a machine generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a computer. The computer readable medium can be a machine readable storage device, a machine readable storage substrate, a memory device, a composition of matter effecting a machine readable propagated signal, or a combination of one or more of them.
• The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
• A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
• The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
• Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, to name just a few.
• Computer readable media suitable for storing computer program instructions and data include all forms of non volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
• To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input.
• Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described is this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.
• The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client server relationship to each other.
• While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.
• Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
• Particular embodiments of the subject matter described in this specification have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims (23)

1. A computer implemented method comprising:

receiving a search query from a client device, the search query being submitted in a first language;

obtaining a translated search query, the translated search query being a translation of the search query to a second language;

receiving a first set of results responsive to the search query and a second set of results responsive to the translated search query, the first set of results including a first set of web pages and a first set of relevance scores and the second set of results including a second set of web pages and a second set of relevance scores;

comparing the first set of relevance scores with the second set of relevance scores; and

serving a results page comprising at least a portion of one or both of the first set of results or the second set of results in response to the comparison of the first set of relevance scores and second set of relevance scores.

2. The computer implemented method ofclaim 1, further comprising serving a first portion of the first set of web pages and a second portion of the second set of web pages, the first portion and the second portion comprising the most relevant results using the relevance scores associated with the first and second portions.

3. The computer implemented method ofclaim 1, wherein comparing the first set of relevance scores with the second set of relevance scores comprises:

aggregating the first set of relevance scores;

aggregating the second set of relevance scores;

comparing the aggregation of the second set of relevance scores is greater than the aggregation of the first set of relevance scores;

wherein results from the second set of results are served if the aggregation of the second set of relevance scores is greater than the aggregation of the second set of relevance scores, otherwise results from the first set of search results are served.

4. The computer implemented method ofclaim 1, further comprising obtaining a translated web page if the comparison of the first set of relevance scores and the second set of relevance scores identifies that the second set of relevance scores are more relevant and serving the translated web page based on receiving a selection of the web page from the client device.

5. The computer implemented method ofclaim 1, further comprising:

comparing an aggregation of relevance scores from the second set of relevance scores to a threshold relevance score; and

if the aggregation of relevance scores exceeds the threshold relevance score, serving search results.

6. The computer implemented method ofclaim 1, further comprising:

obtaining a number of previous queries that correspond to the translated search query;

comparing the number of previous queries to a threshold number of queries; and

wherein serving the second set of results is based on the comparison of the number of previous queries to the threshold number of queries.

7. The computer implemented method ofclaim 1, wherein the first set of relevance scores and the second set of relevance scores each comprise a relevance factor and a ranking factor, the relevance factor measuring a degree to which the results from the search query or the translated search query matches the search query, and the ranking factor measuring a frequency with which a web page is linked by other web pages.

8. The computer implemented method ofclaim 1, wherein the first set of relevance scores is based on an analysis of first click through rates of results from the first search query, and the second set of relevance scores is based on an analysis of second click through rates of results from the translated search query.

9. Software stored in one or more computer readable media and comprising instructions executable by a processing system and upon execution, causing the processing system to perform operations, comprising:

receiving a search query from a client device, the search query being submitted in a first language;

obtaining a translated search query, the translated search query being a translation of the search query to a second language;

receiving a first set of results responsive to the search query and a second set of results responsive to the translated search query, the first set of results including a first set of web pages and a first set of relevance scores and the second set of results including a second set of web pages and a second set of relevance scores;

comparing the first set of relevance scores with the second set of relevance scores; and

serving a results page comprising at least a portion of one or both of the first set of results or the second set of results based on the comparison of the first set of relevance scores and second set of relevance scores.

10. The software ofclaim 9, wherein the instructions are further operable to cause the processing system to perform an operation comprising serving a first portion of the first set of web pages and a second portion of the second set of web pages, the first portion and the second portion comprising the most relevant results using the relevance scores associated with the first and second portions.

11. The software ofclaim 9, wherein the instructions are further operable to cause the processing system to perform an operation comprising obtaining a translated web page if the comparison of the first set of relevance scores and the second set of relevance scores identifies that the second set of relevance scores are more relevant and serving the translated web page based on receiving a selection of the web page from the client device.

12. The software ofclaim 9, wherein the instructions are further operable to cause the processing system to perform operations comprising:

comparing an aggregation of relevance scores from the second set of relevance scores to a threshold relevance score; and

if the aggregation of relevance scores exceeds the threshold relevance score, serving search results.

13. The software ofclaim 9, wherein the instructions are further operable to cause the processing system to perform operations comprising:

obtaining a number of previous queries that correspond to the translated search query;

comparing the number of previous queries to a threshold number of queries; and

wherein serving the second set of results is based on the comparison of the number of previous queries to the threshold number of queries.

14. The software ofclaim 9, wherein the first set of relevance scores and the second set of relevance scores each comprise a relevance factor and a ranking factor, the relevance factor measuring a degree to which the results from the search query or the translated search query matches the search query, and the ranking factor measuring a frequency with which a web page is linked by other web pages.

15. The software ofclaim 9, wherein the first set of relevance scores is based on an analysis of first click through rates of results from the first search query, and the second set of relevance scores is based on an analysis of second click through rates of results from the translated search query.

16. A computer implemented method comprising:

receiving a search query from a client device, the search query being submitted in a first language;

obtaining a translated search query, the translated search query being a translation of the search query to a second language;

receiving first results responsive to the search query, the first results including first web pages having first relevance scores;

receiving second results responsive to the translated search query, the second results including second web pages having second relevance scores;

combining the first set of results and the second set of results into an ordinal list of search results;

deriving an order of most relevant to least relevant from the first and second relevance scores, the order comprising the order of the ordinal list of search results;

serving a results page comprising a specified number of the ordinal list of search results.

17. The computer implemented method ofclaim 16, further comprising obtaining a translated web page if the results page includes a web page from the second set of search results.

18. The computer implemented method ofclaim 16, further comprising:

comparing respective relevance scores from the ordinal list of search results to a threshold relevance score; and

if a respective relevance score does not exceed the threshold relevance score, eliminating the result from the ordinal list of search results.

19. The computer implemented method ofclaim 16, further comprising:

obtaining a number of previous queries that correspond to the translated search query;

comparing the number of previous queries to a threshold number of queries; and

wherein results from the second set of search results are included in the ordinal list of search results if the number of previous queries for the translated search query exceeds the threshold number of queries.

20. The computer implemented method ofclaim 16, wherein the first set of relevance scores is based on an analysis of first click through rates of results from the first search query, and the second set of relevance scores is based on an analysis of second click through rates of results from the translated search query.

21. A computer implemented method, comprising:

receiving a search query from a client device, the search query being submitted in a first language;

obtaining a translated search query, the translated search query being a translation of the search query to a second language;

receiving a first result responsive to the search query, the first result including a first web page having a first relevance score;

receiving a second result responsive to the translated search query, the second result including a second web page having a second relevance score;

comparing the first relevance score to the second relevance score; and

serving a results page comprising the first result if the first relevance score exceeds the second relevance score, the results page comprising the second result if the second relevance score exceeds the first relevance score.

22-41. (canceled)

42. A computer implemented method comprising:

receiving an original search query in a first language;

obtaining a translated search query associated with the original search query, the translated search query being in a second language;

evaluating the translated search query to determine whether the translated search query is a candidate for a cross-language search; and

if the translated search query is a candidate for the cross-language search:

receiving a first set of results responsive to the search query and a second set of results responsive to the translated search query, the first set of results including a first set of web pages and a first set of relevance scores and the second set of results including a second set of web pages and a second set of relevance scores;

comparing the first set of relevance scores with the second set of relevance scores; and

serving a results page comprising at least a portion of one or both of the first set of results or the second set of results in response to the comparison of the first set of relevance scores and second set of relevance scores; and

if the translated search query is not a candidate for the cross-language search serving search results for the original search query.

Images