CN113743107A - Entity word extraction method and device and electronic equipment - Google Patents

Abstract

The embodiment of the disclosure discloses a method and a device for extracting entity words and electronic equipment. One embodiment of the method comprises: acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set; extracting word characteristics of each candidate entity word in the candidate entity word set; and selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word. The embodiment improves the accuracy of entity word extraction.

Description

Entity word extraction method and device and electronic equipment

Technical Field

The embodiment of the disclosure relates to the technical field of computers, in particular to a method and a device for extracting entity words and electronic equipment.

Background

In a carrier for exchanging information with text information, such as Instant Messaging (IM) software, a document editing application, a mail application, etc., various abbreviations, product nouns, project nouns, enterprise dedications, terms, etc. are usually included, and these terms may be referred to as entity words. Since entity words generally belong to a particular subject area, certain difficulties may be brought to the user's understanding of the text. Therefore, mining these entity words and giving corresponding word interpretations may facilitate the user's understanding of the text.

Disclosure of Invention

This disclosure is provided to introduce concepts in a simplified form that are further described below in the detailed description. This disclosure is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

The embodiment of the disclosure provides an entity word extraction method, an entity word extraction device and electronic equipment, wherein candidate entity words are screened by using word characteristics, and the accuracy of entity word extraction is improved.

In a first aspect, an embodiment of the present disclosure provides an entity word extraction method, including: acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set; extracting word characteristics of each candidate entity word in the candidate entity word set; and selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word.

In a second aspect, an embodiment of the present disclosure provides an entity word extraction apparatus, including: the acquisition unit is used for acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set; the extraction unit is used for extracting word characteristics of each candidate entity word in the candidate entity word set; and the selecting unit is used for selecting the target entity words from the candidate entity word set based on the word characteristics and outputting the target entity words.

In a third aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device for storing one or more programs, which when executed by one or more processors, cause the one or more processors to implement the entity word extraction method as in the first aspect.

In a fourth aspect, the disclosed embodiments provide a computer-readable medium, on which a computer program is stored, which when executed by a processor, implements the steps of the entity word extraction method according to the first aspect.

According to the entity word extraction method, the entity word extraction device and the electronic equipment, the text to be processed is obtained firstly, and the text to be processed is subjected to preset processing to obtain a candidate entity word set; then, extracting word characteristics of each candidate entity word in the candidate entity word set; and finally, selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word. By adopting the method, the candidate entity words are screened by utilizing the word characteristics, and the accuracy of entity word extraction is improved.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.

FIG. 1 is an exemplary system architecture diagram in which various embodiments of the present disclosure may be applied;

FIG. 2 is a flow diagram of one embodiment of a method of entity word extraction according to the present disclosure;

FIG. 3 is a flow diagram of yet another embodiment of a method of entity word extraction according to the present disclosure;

FIG. 4 is a flow diagram of one embodiment of updating a set of candidate entity words in a method of entity word extraction according to the present disclosure;

FIG. 5 is a schematic diagram of one embodiment of a method of entity word extraction, according to the present disclosure;

FIG. 6 is a flow diagram of yet another embodiment of a method of entity word extraction according to the present disclosure;

FIG. 7 is a schematic diagram of yet another embodiment of a method of entity word extraction according to the present disclosure;

FIG. 8 is a schematic structural diagram illustrating one embodiment of an entity word extraction apparatus according to the present disclosure;

FIG. 9 is a schematic block diagram of a computer system suitable for use in implementing an electronic device of an embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

Fig. 1 illustrates anexemplary system architecture 100 to which embodiments of the text processing method of the present disclosure may be applied.

As shown in fig. 1, thesystem architecture 100 may includeterminal devices 1011, 1012, 1013, anetwork 102, and aserver 103.Network 102 is the medium used to provide communication links betweenterminal devices 1011, 1012, 1013 andserver 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

The user may interact with theserver 103 via thenetwork 102 using theterminal devices 1011, 1012, 1013 to send or receive messages or the like, for example, the user may send text to be processed to theserver 103 using theterminal devices 1011, 1012, 1013. Various communication client applications, such as an image processing application, an instant messaging software, and the like, may be installed on theterminal devices 1011, 1012, 1013.

Theterminal devices 1011, 1012, 1013 may be hardware or software. When theterminal devices 1011, 1012, 1013 are hardware, they may be various electronic devices having a display screen and supporting information interaction, including but not limited to smart phones, tablet computers, laptop computers, and the like. When theterminal devices 1011, 1012, 1013 are software, they may be installed in the electronic devices listed above. It may be implemented as multiple pieces of software or software modules (e.g., multiple pieces of software or software modules to provide distributed services) or as a single piece of software or software module. And is not particularly limited herein.

Theserver 103 may be a server that provides various services. For example, theserver 103 may obtain a text to be processed from theterminal devices 1011, 1012, 1013, and perform preset processing on the text to be processed to obtain a candidate entity word set; then, the word characteristics of each candidate entity word in the candidate entity word set can be extracted; finally, a target entity word may be selected from the candidate entity word set based on the word feature, and the target entity word may be output, for example, the target entity word may be output to theterminal devices 1011, 1012, 1013, or the target entity word may be locally output.

Theserver 103 may be hardware or software. When theserver 103 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When theserver 103 is software, it may be implemented as multiple pieces of software or software modules (e.g., to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.

It should be further noted that the entity word extraction method provided by the embodiment of the present disclosure may be executed by theserver 103, and in this case, the entity word extraction device is generally disposed in theserver 103.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

With continued reference to FIG. 2, aflow 200 of one embodiment of a method of entity word extraction in accordance with the present disclosure is shown. The entity word extraction method comprises the following steps:

step 201, acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set.

In this embodiment, an execution subject (for example, a server shown in fig. 1) of the entity word extraction method may acquire a text to be processed. The text to be processed may be a text extracted from the target text, or may be a search term (query) input by the user. Here, the target text may be text input by the user.

And then, the execution main body can perform preset processing on the text to be processed to obtain a candidate entity word set. Specifically, the execution main body may perform language identification on the text to be processed to obtain a language identification result. The language identification result may include an english text and a chinese text. It should be noted that the chinese-english mixed text may be processed as a chinese text.

If the text to be processed is a Chinese text, the execution main body may perform word segmentation on the text to be processed, for example, a Chinese word segmentation method may be used to perform word segmentation on the text to be processed. The stop word may then be deleted from the word results obtained from the word segmentation. Then, part-of-speech tagging can be performed on each word after the stop word is deleted, and then part-of-speech screening can be performed on each word to obtain a candidate entity word set. By way of example, words including, but not limited to, at least one of the following parts of speech may be retained: english, IT technology related vocabulary, academic vocabulary, mathematics related vocabulary, institution related vocabulary, education related vocabulary, government institution vocabulary, factory name, company name, bank, place name, hotel and abbreviation.

If the text to be processed is an english text, the execution main body may perform a normalization process on punctuations and abbreviations in the text to be processed, for example, converting's into is. Then, the normalized text to be processed may be parsed (for example, the text may be parsed using benepar (deep learning training based parser)), and noun blocks meeting a preset rule may be extracted according to the part of speech to obtain a candidate entity word set, where the rule may be a reserved noun and an abbreviation.

Step 202, extracting word characteristics of each candidate entity word in the candidate entity word set.

In this embodiment, the execution subject may extract a word feature of each candidate entity word in the candidate entity word set. Here, for each candidate entity word in the candidate entity word set, the execution subject may input the candidate entity word into a feature extraction model trained in advance, so as to obtain a word feature of the candidate entity word. By way of example, the above-mentioned word characteristics may include, but are not limited to, at least one of: word length of a word, part of speech of a word, word vector of a word, and whether a word is a common word.

And 203, selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word.

In this embodiment, the execution subject may select a target entity word from the candidate entity word set based on the word feature. Specifically, the execution subject may determine a score of each candidate entity word in the candidate entity word set based on the word feature. As an example, for each entity word candidate in the entity word candidate set, the executing entity may input the word feature of the entity word candidate into a first score prediction model trained in advance to obtain a score of the entity word candidate. Then, a preset first number (for example, 20) of candidate entity words may be selected from the candidate entity word set in the order of scores from large to small and output as the target entity words.

Here, the executing entity may output the target entity word to a terminal device of a target user (e.g., a reviewer), and if the target user passes the review of the target entity word, the target entity word may be added to an existing entity word set.

The method provided by the embodiment of the disclosure obtains a text to be processed, and performs preset processing on the text to be processed to obtain a candidate entity word set; then, extracting word characteristics of each candidate entity word in the candidate entity word set; and finally, selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word. By adopting the method, the candidate entity words are screened by utilizing the word characteristics, and the accuracy of entity word extraction is improved.

In some optional implementation manners, the executing main body may perform preset processing on the text to be processed in the following manner to obtain a candidate entity word set: the execution main body can identify the language of the text to be processed. Here, the execution subject may input the text to be processed into a language identification model trained in advance, so as to obtain the language of the text to be processed. The language herein may include, but is not limited to, at least one of the following: chinese and english. If the text to be processed is a Chinese text or a mixed Chinese and English text, the execution main body may recognize the Entity word in the text to be processed by using a Named Entity Recognition (NER) technique. Named entity recognition, which may also be referred to as "proper name recognition," generally refers to recognizing entities in text that have a particular meaning, including, primarily, names of people, places, organizations, proper nouns, and the like. Simply, the boundaries and categories of entity designations in natural text are identified. Here, the celebrity entity words may be removed. And then, deleting the entity words existing in the target entity word set from the identified entity words to obtain a candidate entity word set. The target set of entity words may be a maintained set of mined entity words.

In some optional implementation manners, the executing main body may perform preset processing on the text to be processed in the following manner to obtain a candidate entity word set: the execution main body can identify the language of the text to be processed. If the text to be processed is an English text, the execution main body can identify the entity words in the text to be processed by using a text matching technology. Here, a regular matching technique may be employed, for example, a matching pair is (a sentence to be matched: "i is writing a relevant article of an enterprise entity word", and a word to be matched: "enterprise entity word"), and the regular matching may result in whether the "enterprise entity word" exists in the original sentence and the position of the occurrence, for example, returning [5,9] to be the position of the word to be matched in the sentence to be matched. Here, the celebrity entity words may be removed. And then, deleting the entity words existing in the target entity word set from the identified entity words to obtain a candidate entity word set. The target set of entity words may be a maintained set of mined entity words.

With further reference to fig. 3, aflow 300 of yet another embodiment of an entity word extraction method is shown. Theprocess 300 of the entity word extraction method includes the following steps:

step 301, acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set.

Step 302, extracting word characteristics of each candidate entity word in the candidate entity word set.

In the present embodiment, the steps 301-302 can be performed in a similar manner to the steps 201-202, and will not be described herein again.

Step 303, determining the word weight of each candidate entity word in the candidate entity word set based on the position information of the entity word in the text to be processed.

In this embodiment, an executing subject (for example, the server shown in fig. 1) of the entity word extraction method may determine a word weight of each candidate entity word in the candidate entity word set based on the position information of the entity word in the text to be processed.

Specifically, for each entity word candidate in the entity word candidate set, the execution main body may determine a position of the entity word candidate in the text to be processed, and the execution main body may store a correspondence between the position and the weight, and if the entity word candidate is located at a plurality of positions in the text to be processed, the execution main body may add weights corresponding to the positions to obtain a word weight of the entity word candidate.

As an example, if the correspondence between word positions and weights is as follows: the text is as follows: 1; title: 0.5; h1 label 0.3; h2 label 0.2; h3 tag: 0.1, and if the candidate entity word "TCE" appears in the text, title, h1 tag, and h3 tag at the same time, the word weight of the candidate entity word "TCE" is 1.9.

Step 304, determining the score of each candidate entity word in the candidate entity word set based on the word characteristics and the word weight of the candidate entity word.

In this embodiment, for each candidate entity word in the candidate entity word set, the execution subject may determine a score of the candidate entity word based on the word feature and the word weight of the candidate entity word. Specifically, the executing agent may input the word feature and the word weight of the candidate entity word into a second score prediction model trained in advance, so as to obtain the score of the candidate entity word.

And 305, selecting a target entity word from the candidate entity word set based on the scores of all candidate entity words in the candidate entity word set, and outputting the target entity word.

In this embodiment, the execution subject may select a target entity word from the set of candidate entity words and output the target entity word based on the score of each candidate entity word in the set of candidate entity words.

As an example, the executing entity may select a preset second number of candidate entity words from the set of candidate entity words as target entity words in an order of scores decreasing from high to low.

As another example, the execution subject may select a candidate entity word with a score greater than a preset first score threshold from the candidate entity word set as the target entity word.

As can be seen from fig. 3, compared with the embodiment corresponding to fig. 2, theprocess 300 of the entity word extraction method in this embodiment embodies a step of scoring candidate entity words by using word features of words and weights of words. Therefore, the scheme described in the embodiment can further improve the accuracy of entity word extraction.

In some optional implementations, the text to be processed may be a chinese text or a mixed chinese and english text. At this time, the term characteristics may include an inverse document frequency of the term, a term frequency inverse document frequency of the term, a ratio of N-Gram scores of the term in the text to be processed and the target corpus, and a ratio of confusion of the term in the text to be processed and the target corpus. The term frequency of the term generally refers to the number of times the term appears in the text to be processed. The Inverse Document Frequency (IDF) of the above words is commonly used to describe the category distinction ability of the words, which is a measure of the general importance of the words. If fewer documents contain terms, the greater the IDF of the terms, and the more documents contain terms, the smaller the IDF of the terms. The inverse document frequency of a term can be obtained by dividing the total number of files (the number of preset external corpora) by the number of files containing the term, and taking the obtained quotient to be a base-10 logarithm. The Term Frequency Inverse Document Frequency (TF-IDF) of the Term generally refers to a product of the Term Frequency of the Term and the Inverse Document Frequency of the Term. The main idea of word frequency inverse document frequency of the words is as follows: if a word appears in a piece of article with a high frequency TF and rarely appears in other articles, the word is considered to have a good classification capability and is suitable for classification. The ratio of the N-Gram scores of the words in the text to be processed and the target corpus generally refers to the ratio of the N-Gram scores of the words in the text to be processed and the N-Gram scores of the words in the target corpus (e.g., corpus in wikipedia). Here, the N-Gram score is a score that can be calculated inferentially for the input text (here, words) based on the N-Gram language model, and represents how common a word is on a corpus, and the value is negative, and the smaller the value is, the more rare it is, for example, -100; the larger, the more common, e.g., -1.0. The calculation of the N-Gram score can be supported by a KenLM tool, a model is trained on a specified corpus, and then words can be input into the trained model to be calculated to obtain the score. The ratio of the confusion of the word in the text to be processed and the target corpus is usually referred to as the ratio of the confusion of the word in the text to be processed and the confusion of the word in the target corpus. The degree of confusion is typically used to measure how well a probability distribution or probability model predicts a sample. The confusion degree can be obtained by a pre-trained confusion degree prediction model, namely, the confusion degree of the words in the text is obtained by inputting the words and the text into the confusion degree prediction model.

It should be noted that the inverse document frequency of a word and the word frequency of the word are generally used to measure the importance of the word, and the ratio of the N-Gram scores of the word in the to-be-processed text and the target corpus and the ratio of the confusion degree of the word in the to-be-processed text and the target corpus are generally used to measure the specificity of the word.

In some optional implementations, the execution subject may determine the score of the candidate entity word based on the word feature and the word weight of the candidate entity word by: the execution main body may perform weighted summation on the inverse document frequency of the candidate entity word, the word frequency inverse document frequency of the candidate entity word, the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, and the ratio of the confusion degree of the candidate entity word in the text to be processed and the target corpus, and multiply the summation result by the word weight of the candidate entity word to obtain the score of the candidate entity word. Here, the execution subject described above may determine the score of the candidate entity word by the following formula (1):

wherein Score is the Score of the candidate entity word, tfidf is the word frequency inverse document frequency of the candidate entity word, idf is the inverse document frequency of the candidate entity word, ngram _ Score is the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, perlexity _ Score is the ratio of the confusion of the candidate entity word in the text to be processed and the target corpus, k1 is the coefficient corresponding to tfidf of the candidate entity word, k2 is the coefficient corresponding to idf of the candidate entity word, k3 is the coefficient corresponding to ngram _ Score of the candidate entity word, k4 is the coefficient corresponding to perlexity _ Score of the candidate entity word, and string _ weight is the word weight of the candidate entity word.

In some optional implementations, the text to be processed may be an english text. At this time, the term features may include a keyword extraction score of the term, a ratio of N-Gram scores of the term in the text to be processed and the target corpus, and a ratio of confusion of the term in the text to be processed and the target corpus. The keyword extraction score of the above words generally refers to calculating the RAKE score of the entity word using the RAKE algorithm. The RAKE algorithm is used to extract key words, which are actually key phrases and tend to be longer, and in english, keywords usually include many words but rarely include punctuation and stop words, such as and, the, of, etc., and other words that do not contain semantic information. The RAKE algorithm first divides a document into several clauses using punctuation marks (e.g., half-corner periods, question marks, exclamation marks, commas, etc.), and then for each clause, divides the clause into several phrases using stop words as delimiters, which serve as candidate words for the finally extracted keywords. The ratio of the N-Gram scores of the words in the text to be processed and the target corpus is generally referred to as the ratio of the N-Gram scores of the words in the text to be processed and the N-Gram scores of the words in the target corpus. The ratio of the confusion of the word in the text to be processed and the target corpus is usually referred to as the ratio of the confusion of the word in the text to be processed and the confusion of the word in the target corpus. The degree of confusion is typically used to measure how well a probability distribution or probability model predicts a sample.

It should be noted that, the keyword extraction score of a word is generally used to measure the importance of the word, and the ratio of the N-Gram score of the word in the to-be-processed text and the target corpus and the ratio of the confusion degree of the word in the to-be-processed text and the target corpus are generally used to measure the specificity of the word.

In some optional implementations, the execution subject may determine the score of the candidate entity word based on the word feature and the word weight of the candidate entity word by: the execution main body may perform weighted summation on the keyword extraction score of the candidate entity word, the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, and the ratio of the confusion degree of the candidate entity word in the text to be processed and the target corpus, and multiply the summation result by the word weight of the candidate entity word to obtain the score of the candidate entity word. Here, the execution subject described above may determine the score of the candidate entity word by the following formula (2):

wherein, Score is the Score of the candidate entity word, rake _ Score is the keyword extraction Score of the candidate entity word, ngram _ Score is the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, perlexity _ Score is the ratio of the confusion of the candidate entity word in the text to be processed and the target corpus, k1 is the coefficient corresponding to rake _ Score of the candidate entity word, k2 is the coefficient corresponding to ngram _ Score of the candidate entity word, k3 is the coefficient corresponding to perlexity _ Score of the candidate entity word, and structure _ weight is the word weight of the candidate entity word.

In some optional implementations, the text to be processed may be a chinese text or a mixed chinese and english text. The execution subject may select a target entity word from the set of candidate entity words based on the score of each candidate entity word in the set of candidate entity words as follows: the execution subject may update the set of candidate entity words based on the scores and word features of each candidate entity word in the set of candidate entity words. Here, the execution main body may select, from the candidate entity word set, an entity word whose ratio of N-Gram scores of words in the text to be processed and the target corpus is greater than a preset first ratio threshold, whose ratio of confusion degrees of words in the text to be processed and the target corpus is greater than a preset second ratio threshold, and whose score is greater than a preset second score threshold, and generate an updated candidate entity word set; then, the target entity word can be selected from the updated candidate entity word set. Here, a preset third number of candidate entity words may be selected from the updated candidate entity word set as the target entity words according to the order of the word frequency of the words in the text to be processed from large to small.

With further reference to FIG. 4, aflow 400 of one embodiment of updating a set of candidate entity words in an entity word extraction method is shown. Theupdate process 400 for updating the candidate entity word set includes the following steps:

step 401, based on the candidate entity word set, executing the following entity word selecting steps: selecting entity words meeting preset conditions from the candidate entity word set, and combining the entity words meeting the conditions to obtain at least one word combination; determining word combinations appearing in the text to be processed in at least one word combination as candidate compound entity words; for each candidate compound entity word, determining the score of the candidate compound entity word based on the scores of the candidate entity words forming the candidate compound entity word and the word characteristics of the candidate entity word; updating the candidate entity word set based on the scores of the candidate compound entity words, the word characteristics of the candidate compound entity words, and the scores and the word characteristics of all candidate entity words in the candidate entity word set; determining whether the updated set of candidate entity words is the same as the set of candidate entity words.

In this embodiment, step 401 may include sub-steps 4011, 4012, 4013, 4014, and 4015. Wherein:

step 4011, selecting entity words meeting preset conditions from the candidate entity word set, and combining the entity words meeting the conditions to obtain at least one word combination.

In this embodiment, an execution subject (for example, a server shown in fig. 1) of the entity word extraction method may select entity words meeting a preset condition from the candidate entity word set, and combine the entity words meeting the condition to obtain at least one word combination. The condition may include that a preset character length threshold is not exceeded, and at this time, the execution subject may select a candidate entity word having a character length not exceeding the character length threshold from the candidate entity word set. For example, if the candidate entity word is chinese, the character length threshold may be set to 4, and the execution subject may select, from the candidate entity word set, a candidate entity word whose word includes no more than 4 characters. If the candidate entity word is english, the character length threshold may be set to 15, and the execution main body may select a candidate entity word whose number of letters included in a word does not exceed 15 from the candidate entity word set. Then, the selected candidate entity words may be combined pairwise. For example, if the selected candidate entity words are a and b, two combinations of ab and ba can be obtained.

Step 4012, determining the word combination appearing in the text to be processed in the at least one word combination as a candidate compound entity word, and adding the candidate compound entity word into the candidate entity word set.

In this embodiment, the execution subject may determine, as the candidate compound entity word, a word combination appearing in the text to be processed in the at least one word combination. As an example, if the word combination is ab and ba, the execution subject may determine whether ab or ba appears in the text to be processed. If the word combination ab appears in the text to be processed and the word combination ba does not appear in the text to be processed, the word combination ab may be determined as a candidate compound entity word.

It should be noted that, if the words a and b are english, the execution body needs to determine whether a + '+ b and b +' + a appear in the text to be processed.

Thereafter, the executing entity may add the candidate compound entity word to a set of candidate entity words.

Step 4013, for each candidate compound entity word, determining a score of the candidate compound entity word based on the scores of the candidate entity words constituting the candidate compound entity word.

In this embodiment, for each candidate compound entity word, the execution subject may determine a score of the candidate compound entity word based on the scores of the candidate entity words constituting the candidate compound entity word. Specifically, the execution subject may determine an average value of scores of two candidate entity words constituting the candidate compound entity word as the score of the candidate compound entity word.

And step 4014, updating the added candidate entity word set based on the scores of the candidate compound entity words, the word characteristics of the candidate compound entity words, and the scores and the word characteristics of each candidate entity word in the candidate entity word set.

In this embodiment, the execution subject may update the added candidate entity word set based on the score of the candidate compound entity word, the word feature of the candidate compound entity word, and the score and the word feature of each candidate entity word in the candidate entity word set.

Here, the execution main body may select, from the added candidate entity word set, candidate entity words with a score greater than a preset third score threshold, and then may select, from the selected candidate entity words, a preset fourth number of candidate entity words as target entity words according to a descending order of word frequencies of the words in the text to be processed.

Step 4015, determine whether the updated set of candidate entity words is the same as the set of candidate entity words.

In this embodiment, the execution subject may determine whether the updated set of candidate entity words is the same as the set of candidate entity words. I.e., determining whether no new candidate compound entity word has been added to the set of candidate entity words.

If the updated set of candidate entity words is not the same as the set of candidate entity words, the executing entity may executestep 402.

And 402, if the updated candidate entity word set is different from the candidate entity word set, taking the updated candidate entity word set as the candidate entity word set, and continuing to execute the entity word selecting step.

In this embodiment, if it is determined instep 4015 that the updated candidate entity word set is not the same as the candidate entity word set, the execution main body may use the updated candidate entity word set as the candidate entity word set, and continue to execute the entity word selection step (i.e.,step 4011 and 4015).

The method provided by the above embodiment of the present disclosure obtains the candidate compound entity words by recombining the candidate entity words, determines the scores of the candidate compound entity words, and updates the candidate entity word set based on the word features of the candidate entity words and the candidate compound entity words and the scores of the candidate entity words and the candidate compound entity words, thereby avoiding the situation that the entity words cannot be mined due to word segmentation.

In some optional implementation manners, if it is determined instep 4015 that the updated set of candidate entity words is the same as the set of candidate entity words, the execution subject may select a target entity word from the updated set of candidate entity words based on a score of each candidate entity word in the updated set of candidate entity words. Here, the execution subject may select a preset fifth number of candidate entity words from the updated candidate entity word set as the target entity words in the order of scores decreasing from high to low.

In some alternative implementations, the term characteristics may include a term frequency of the term. Here, the word frequency of the candidate entity word may be the number of occurrences of the candidate entity word in the text to be processed. The executing agent may determine the score of the candidate compound entity word based on the scores of the candidate entity words constituting the candidate compound entity word by: the executing body may perform weighted summation on the scores of the two candidate entity words constituting the candidate compound entity word to obtain the score of the candidate compound entity word. Here, for each of the two candidate entity words constituting the candidate compound entity word, the weight corresponding to the candidate entity word may be a ratio of a word frequency of the candidate entity word to a total word frequency, and the total word frequency may be a sum of the word frequencies of the two candidate entity words constituting the candidate compound entity word.

Here, the execution subject may determine the score of the candidate compound entity word by the following formula (3):

wherein Score is the Score of the candidate compound entity word, a_countAnd b_countThe word frequencies, a, of the two words (a and b) constituting the candidate compound entity word, respectively_scoreAnd b_scoreRespectively, the scores of the two words (a and b) that make up the candidate compound entity word.

Here, the weight of the word a is a ratio of the word frequency of the word a to the total word frequency (the sum of the word frequency of the word a and the word frequency of the word b), and the weight of the word b is a ratio of the word frequency of the word b to the total word frequency (the sum of the word frequency of the word a and the word frequency of the word b).

In some optional implementations, the executing entity may update the added candidate entity word set based on the score of the candidate compound entity word, the word feature of the candidate compound entity word, the score and the word feature of each candidate entity word in the candidate entity word set, as follows: the execution main body may filter the candidate entity words in the added candidate entity word set based on the scores of the candidate entity words to generate a candidate entity word subset. Specifically, the executing agent may sort the candidate entity words except the compound entity word in the candidate entity word set according to an order of scores from small to large, that is, only the fine-grained entity words are processed. Then, a preset percentage (e.g., 20%) of the candidate entity words may be deleted from the candidate entity word set, so as to obtain a candidate entity word subset. Then, for each candidate compound entity word, the executing entity may determine whether at least two candidate entity words constituting the candidate compound entity word exist in the subset of candidate entity words. If so, the executing agent may delete a candidate entity word with a low score from the subset of candidate entity words, among the at least two candidate entity words constituting the candidate compound entity word.

As an example, for a candidate compound entity word ab, the execution subject may determine whether there are two fine-grained entity words a and b constituting ab in the candidate entity word subset. If the candidate entity word subset includes the fine-grained entity words a and b, the execution subject may compare the scores of the fine-grained entity words a and b, and if the score of the fine-grained entity word b is lower than the score of the fine-grained entity word a, the execution subject may delete the fine-grained entity word b from the candidate entity word subset.

In some alternative implementations, the term feature may include a ratio of N-Gram scores of terms in the pending text and the target corpus and a ratio of confusion of terms in the pending text and the target corpus. After determining whether at least two candidate entity words forming the candidate compound entity word exist in the candidate entity word subset for each candidate compound entity word, if one candidate entity word forming the candidate compound entity word exists in the candidate entity word subset, the execution main body may determine whether a first ratio corresponding to the candidate compound entity word is greater than a first ratio corresponding to the candidate entity word forming the candidate compound entity word, and determine whether a second ratio corresponding to the candidate compound entity word is greater than a second ratio corresponding to the candidate entity word forming the candidate compound entity word. The first ratio may be a ratio of N-Gram scores of words in the text to be processed and the target corpus, and the second ratio may be a ratio of confusion degrees of words in the text to be processed and the target corpus.

If the first ratio corresponding to the candidate compound entity word is greater than the first ratio corresponding to the candidate entity word constituting the candidate compound entity word and/or the second ratio corresponding to the candidate compound entity word is greater than the second ratio corresponding to the candidate entity word constituting the candidate compound entity word, the execution main body may delete the candidate entity word constituting the candidate compound entity word from the candidate entity word subset.

If the first ratio corresponding to the candidate compound entity word is smaller than the first ratio corresponding to the candidate entity word constituting the candidate compound entity word and the second ratio corresponding to the candidate compound entity word is smaller than the second ratio corresponding to the candidate entity word constituting the candidate compound entity word, the execution main body may simultaneously retain the candidate entity word constituting the candidate compound entity word and the candidate compound entity word.

For example, if the subset of candidate entity words is { a, b, c, d, ab }, then { a, b, ab } needs to be integrated. If the ratio of the scores of a, b and ab and the N-Gram score to the confusion is as shown in the following table:

the execution main body determines that two fine-grained entity words a and b forming ab exist in the candidate entity word subset, the scores of the fine-grained entity words a and b can be compared, and the execution main body can delete the fine-grained entity words b from the candidate entity word subset because the score of the fine-grained entity words b is 5.0 which is lower than the score of the fine-grained entity words a by 10.0. Then, the execution subject may compare the ratio of the N-Gram scores corresponding to the fine-grained entity word a with the ratio of the N-Gram scores corresponding to the candidate compound entity word ab, and compare the ratio of the confusion degree corresponding to the fine-grained entity word a with the ratio of the confusion degree corresponding to the candidate compound entity word ab. Because the ratio of the N-Gram scores corresponding to the candidate compound entity word ab and the ratio of the confusion degree are both larger than the ratio of the N-Gram scores corresponding to the fine-grained entity word a and the ratio of the confusion degree, the fine-grained entity word a can be deleted from the candidate entity word subset, and the updated candidate entity word set is { c, d, ab }.

With continuing reference to FIG. 5, a schematic diagram of one embodiment of a method for entity word extraction is shown. In fig. 5, an executing body of the entity word extracting method may obtain user authorization data, and then may perform sentence classification on the user authorization data to obtain a plurality of texts to be processed, where the plurality of texts to be processed include a chinese/chinese-english mixed text and an english text.

Aiming at the Chinese/Chinese-English mixed text, the execution main body can carry out Chinese preprocessing on the Chinese/Chinese-English mixed text, such as word segmentation, part of speech tagging, stop word filtering and part of speech screening on the text. And then, extracting the Chinese characteristics of the words, and scoring and sequencing the candidate entity words by using the Chinese characteristics. And then, recombining the candidate entity words to obtain candidate compound entity words, extracting compound word characteristics of the candidate compound entity words, and grading and sequencing the candidate compound entity words by using the compound word characteristics. Finally, the candidate entity words and the candidate compound entity words may be ranked using the scores of the candidate entity words and the scores of the candidate compound entity words.

Aiming at the English text, the execution main body can perform English preprocessing on the English text, for example, the punctuations and the abbreviations in the text are normalized, the text to be processed after the normalization processing is analyzed, and noun blocks meeting the preset rules are extracted according to the part of speech to obtain a candidate entity word set. And then, extracting the English features of the candidate entity words in the candidate entity word set, and then scoring and sequencing the candidate entity words by using the English features.

And finally, integrating the candidate entity words obtained by the Chinese/English entity word mining module and the candidate entity words obtained by the pure English entity word mining module to obtain the target entity words.

With further reference to FIG. 6, aflow 600 of yet another embodiment of a method for entity word extraction is shown. Theupdate process 400 for updating the candidate entity word set includes the following steps:

step 601, acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set.

Step 602, extracting word characteristics of each candidate entity word in the candidate entity word set.

In the present embodiment, the steps 601-602 can be performed in a similar manner to the steps 201-202, and will not be described herein again.

Step 603, extracting text features of the text to be processed.

In this embodiment, an execution subject (for example, a server shown in fig. 1) of the entity word extraction method may extract text features of the text to be processed. Here, the text to be processed is usually a search sentence (query) input by a user, and in this case, the text feature may include, but is not limited to, at least one of the following: text retrieval frequency, number of relevant users of the text, user stickiness of the text, and user permeability of the text. The text retrieval frequency generally refers to the number of times the text is retrieved, the number of users associated with the text generally refers to the number of users who have retrieved the text, the user stickiness of the text generally refers to the ratio of the number of times the text has been retrieved to the number of users who have retrieved the text, and the user permeability of the text generally refers to the ratio of the number of users who have retrieved the text to the total number of users who have retrieved the text.

And step 604, extracting word characteristics of the entity words in the target entity word set.

In this embodiment, the execution subject may extract word features of the entity words in the target entity word set. Here, the target entity word set may be a mined entity word set maintained. The above word features may include, but are not limited to, at least one of: word length, word part of speech, word vector of word, whether word is a common word, word frequency of word in the preset internal corpus, word vector of word in the preset internal corpus, word frequency of word in the preset external corpus, and word vector of word in the preset external corpus.

It should be further noted that the text corresponding to the entity word in the target entity word set may be the entity word itself. At this time, the word feature may further include: the frequency of retrieval of the terms, the number of related users of the terms, the user stickiness of the terms, and the user permeability of the terms. The term retrieval frequency generally refers to the number of times the term is retrieved, the number of users related to the term generally refers to the number of users who have retrieved the term, the term user viscosity generally refers to the ratio of the number of times the term is retrieved to the number of users who have retrieved the term, and the term user permeability generally refers to the ratio of the number of users who have retrieved the term to the total number of users who have retrieved the term.

Step 605, constructing a feature space by using the word features of the entity words in the candidate entity word set, the word features of the entity words in the target entity word set, and the text features.

In this embodiment, the execution subject may construct a feature space by using the word features of the entity words in the candidate entity word set, the word features of the entity words in the target entity word set, and the text features. The feature space is a space where the feature vector is located, and in the feature space, each feature corresponds to a one-dimensional coordinate in the feature space. Here, the coordinate system of the feature space may be constructed by the word feature and the text feature, that is, the physical meanings of the coordinate axes of the feature space form a one-to-one correspondence relationship with the word feature or the text feature.

As an example, if the word feature includes: the word length, word part of speech, the word vector of word, whether the word is the commonly used word, the word frequency of word in predetermineeing inside corpus, the word vector of word in predetermineeing inside corpus, the word frequency of word in predetermineeing outside corpus and the word vector of word in predetermineeing outside corpus are eight, and the text characteristic includes: the execution subject may construct 12 coordinate axes to form a 12-dimensional feature space, and then set, in the constructed feature space, a position point representing an entity word in the candidate entity word set and a position point representing an entity word in the target entity word set according to a word feature and a text feature of each entity word in the candidate entity word set and a word feature of each entity word in the target entity word set.

And 606, selecting a target entity word from the candidate entity word set based on the feature space, and outputting the target entity word.

In this embodiment, the execution subject may select a target entity word from the candidate entity word set based on the feature space, and output the target entity word. Here, for each entity word candidate in the entity word candidate set, the execution subject may search, from the feature space, an entity word that exists in the target entity word set and is closest to the entity word candidate, and then may determine a distance between the searched entity word and the entity word candidate, and if the distance is greater than the target distance, may determine the entity word candidate as the target entity word.

As can be seen from fig. 6, compared with the embodiment corresponding to fig. 2, theprocess 600 of the entity word extraction method in this embodiment embodies steps of constructing a feature space and selecting a target entity word from the feature space. Therefore, the scheme described in the embodiment can further improve the accuracy of entity word extraction.

In some optional implementations, the executing entity may select a target entity word from the candidate entity word set based on the feature space by: the execution body may classify the words in the feature space using a K-Nearest Neighbor (KNN) classification algorithm. The idea of the K nearest neighbor classification algorithm is as follows: in feature space, if the majority of the k nearest (i.e., nearest neighbor in feature space) samples in the vicinity of a sample belong to a certain class, then the sample also belongs to this class. Then, for each candidate entity word in the candidate entity word set, the execution subject may determine, in the feature space, the number of entity words existing in the target entity word set, for which a distance from the candidate entity word is smaller than a preset distance threshold; then, it may be determined whether the number is greater than a preset number threshold (e.g., 10); if so, the execution subject may determine the candidate entity word as the target entity word.

With continuing reference to FIG. 7, a schematic diagram of yet another embodiment of a method for entity word extraction is shown. In fig. 7, an execution main body of the entity word extraction method may obtain query data, and then may classify the query data to obtain a chinese query and an english query.

For a Chinese query, the execution subject may identify entity words in the Chinese query by using NER (named entity identification technology), and then remove named entity words from the identified entity words to obtain candidate Chinese entity words.

For an english query, the executing body may identify an entity word in the english query by using an english enterprise entity word identification method (text matching technology), and then remove a celebrity entity word and a common word in the identified entity word to obtain a candidate english entity word.

And then, the execution main body can merge the candidate Chinese entity words and the candidate English entity words, and filter the candidate Chinese entity words and the candidate English entity words by comparing with the existing entity word list to remove the entity words which are already recorded. Extracting characteristics aiming at candidate entity words which are not in the existing entity word list, identifying the candidate entity words by using the extracted characteristics, and adding the identified entity words into the enterprise entity words.

Here, when extracting features of a candidate entity word, it is necessary to use the internal and external corpora of the enterprise to extract word features such as a word frequency of the candidate entity word in the internal expectation, a word frequency of the candidate entity word in the external expectation, a word vector of the candidate entity word in the internal expectation, and a word vector of the candidate entity word in the external expectation.

With further reference to fig. 8, as an implementation of the method shown in the above-mentioned figures, the present disclosure provides an embodiment of an entity word extracting apparatus, where the embodiment of the apparatus corresponds to the embodiment of the method shown in fig. 2, and the apparatus may be specifically applied to various electronic devices.

As shown in fig. 8, the entityword extracting apparatus 800 of the present embodiment includes: anacquisition unit 801, afirst extraction unit 802, and aselection unit 803. The acquiringunit 801 is configured to acquire a text to be processed, and perform preset processing on the text to be processed to obtain a candidate entity word set; thefirst extraction unit 802 is configured to extract word features of each candidate entity word in the candidate entity word set; the selectingunit 803 is configured to select a target entity word from the candidate entity word set based on the word characteristics, and output the target entity word.

In this embodiment, the specific processing of the acquiringunit 801, the first extractingunit 802 and the selectingunit 803 of the entityword extracting apparatus 800 may refer to step 201,step 202 and step 203 in the corresponding embodiment of fig. 2.

In some optional implementations, the selectingunit 803 may be further configured to select the target entity word from the candidate entity word set based on the word feature by: the selectingunit 803 may determine the word weight of each candidate entity word in the candidate entity word set based on the position information of the entity word in the text to be processed; then, for each candidate entity word in the candidate entity word set, determining a score of the candidate entity word based on the word feature and the word weight of the candidate entity word; then, the target entity word may be selected from the candidate entity word set based on the score of each candidate entity word in the candidate entity word set.

In some optional implementations, the text to be processed is a chinese text or a mixed chinese and english text, and the word characteristics include an inverse document frequency of the word, a word frequency inverse document frequency of the word, a ratio of N-Gram scores of the word in the text to be processed and the target corpus, and a ratio of confusion of the word in the text to be processed and the target corpus; and the selectingunit 803 may be further configured to determine the score of the candidate entity word based on the word feature and the word weight of the candidate entity word by: the selectingunit 803 may perform weighted summation on the inverse document frequency of the candidate entity word, the word frequency inverse document frequency of the candidate entity word, the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, and the ratio of the confusion degree of the candidate entity word in the text to be processed and the target corpus, and multiply the summation result by the word weight of the candidate entity word to obtain the score of the candidate entity word.

In some optional implementation manners, the text to be processed is an english text, and the term features include a keyword extraction score of the term, a ratio of N-Gram scores of the term in the text to be processed and the target corpus, and a ratio of confusion of the term in the text to be processed and the target corpus; and the selectingunit 803 may be further configured to determine the score of the candidate entity word based on the word feature and the word weight of the candidate entity word by: the selectingunit 803 may perform weighted summation on the keyword extraction score of the candidate entity word, the ratio of the N-Gram scores of the candidate entity word in the text to be processed and the target corpus, and the ratio of the confusion degree of the candidate entity word in the text to be processed and the target corpus, and multiply the summation result by the word weight of the candidate entity word to obtain the score of the candidate entity word.

In some optional implementations, the text to be processed is a chinese text or a mixed chinese and english text; and the selectingunit 803 may be further configured to select a target entity word from the candidate entity word set based on the scores of the candidate entity words in the candidate entity word set as follows: the selectingunit 803 may update the candidate entity word set based on the score and the word feature of each candidate entity word in the candidate entity word set, and select the target entity word from the updated candidate entity word set.

In some optional implementation manners, the selecting unit 803 may be further configured to update the set of candidate entity words based on the scores and word features of each candidate entity word in the set of candidate entity words, and select the target entity word from the updated set of candidate entity words by: the selecting unit 803 may perform the following entity word selecting step based on the candidate entity word set: selecting entity words meeting preset conditions from the candidate entity word set, and combining the entity words meeting the conditions to obtain at least one word combination; determining a word combination appearing in the text to be processed in at least one word combination as a candidate compound entity word, and adding the candidate compound entity word into a candidate entity word set; for each candidate compound entity word, determining the score of the candidate compound entity word based on the scores of the candidate entity words forming the candidate compound entity word; updating the added candidate entity word set based on the scores of the candidate compound entity words, the word characteristics of the candidate compound entity words, and the scores and the word characteristics of all candidate entity words in the candidate entity word set; determining whether the updated candidate entity word set is the same as the candidate entity word set; if not, taking the updated candidate entity word set as a candidate entity word set, and continuing to execute the entity word selecting step.

In some optional implementation manners, if the updated set of candidate entity words is the same as the set of candidate entity words, the selectingunit 803 may select a target entity word from the updated set of candidate entity words based on the score of each candidate entity word in the updated set of candidate entity words.

In some alternative implementations, the term characteristics include a term frequency of the term; and the selectingunit 803 may be further configured to determine the score of the candidate compound entity word based on the scores of the candidate entity words constituting the candidate compound entity word by: the selectingunit 803 may perform weighted summation on the scores of the two candidate entity words forming the candidate compound entity word to obtain the score of the candidate compound entity word, where for each candidate entity word of the two candidate entity words forming the candidate compound entity word, the weight corresponding to the candidate entity word is the ratio of the word frequency of the candidate entity word to the total word frequency, and the total word frequency is the sum of the word frequencies of the two candidate entity words forming the candidate compound entity word.

In some optional implementation manners, the selectingunit 803 may be further configured to update the added candidate entity word set based on the score of the candidate compound entity word, the word feature of the candidate compound entity word, and the score and word feature of each candidate entity word in the candidate entity word set, as follows: the selectingunit 803 may screen candidate entity words in the added candidate entity word set based on the scores of the candidate entity words to generate a candidate entity word subset; then, for each candidate compound entity word, determining whether at least two candidate entity words forming the candidate compound entity word exist in the candidate entity word subset; and if so, deleting the candidate entity words with low scores in at least two candidate entity words forming the candidate compound entity words from the candidate entity word subset.

In some optional implementation manners, the word characteristics include a ratio of N-Gram scores of the words in the text to be processed and the target corpus and a ratio of confusion degrees of the words in the text to be processed and the target corpus; the selecting unit 803 may be further configured to update the added candidate entity word set based on the scores of the candidate compound entity words, the word features of the candidate compound entity words, and the scores and the word features of each candidate entity word in the candidate entity word set, as follows: if a candidate entity word forming the candidate compound entity word exists in the candidate entity word subset, determining whether a first ratio corresponding to the candidate compound entity word is larger than a first ratio corresponding to the candidate entity word forming the candidate compound entity word, and determining whether a second ratio corresponding to the candidate compound entity word is larger than a second ratio corresponding to the candidate entity word forming the candidate compound entity word, wherein the first ratio is a ratio of N-Gram scores of the word in the text to be processed and the target corpus, and the second ratio is a ratio of confusion of the word in the text to be processed and the target corpus; and if the first ratio corresponding to the candidate compound entity word is larger than the first ratio corresponding to the candidate entity word forming the candidate compound entity word and/or the second ratio corresponding to the candidate compound entity word is larger than the second ratio corresponding to the candidate entity word forming the candidate compound entity word, deleting the candidate entity word forming the candidate compound entity word from the candidate entity word subset.

In some optional implementation manners, the obtainingunit 801 may be further configured to perform preset processing on the text to be processed to obtain a candidate entity word set by: the obtainingunit 801 may perform language identification on the text to be processed; and if the text to be processed is a Chinese text or a Chinese-English mixed text, identifying the entity words in the text to be processed by using a named entity identification technology, and deleting the entity words existing in the target entity word set from the identified entity words to obtain a candidate entity word set.

In some optional implementation manners, the obtainingunit 801 may be further configured to perform preset processing on the text to be processed to obtain a candidate entity word set by: the obtainingunit 801 may perform language identification on the text to be processed; and if the text to be processed is an English text, identifying entity words in the text to be processed by using a text matching technology, and removing entity words existing in the target entity word set from the identified entity words to obtain a candidate entity word set.

In some optional implementations, the entityword extraction apparatus 800 may further include a second extraction unit (not shown in the figure) and a third extraction unit (not shown in the figure). The second extraction unit may extract text features of the text to be processed; the third extraction unit may extract word features of the entity words in the target entity word set. The selectingunit 803 may be further configured to select a target entity word from the candidate entity word set based on the word features as follows: the selectingunit 803 may construct a feature space by using the word features of the entity words in the candidate entity word set, the word features of the entity words in the target entity word set, and the text features; then, the target entity word can be selected from the candidate entity word set based on the feature space.

In some optional implementations, the selectingunit 803 may be further configured to select the target entity word from the candidate entity word set based on the feature space by: for each candidate entity word in the candidate entity word set, the selectingunit 803 may determine, in the feature space, the number of entity words existing in the target entity word set, where a distance between the entity word and the candidate entity word is smaller than a preset distance threshold; thereafter, it may be determined whether the number is greater than a preset number threshold; if yes, the candidate entity word may be determined as the target entity word.

Referring now to FIG. 9, shown is a schematic diagram of an electronic device (e.g., the server of FIG. 1) 900 suitable for use in implementing embodiments of the present disclosure. The electronic device shown in fig. 9 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 9, theelectronic device 900 may include a processing means (e.g., a central processing unit, a graphics processor, etc.) 901 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)902 or a program loaded from a storage means 908 into a Random Access Memory (RAM) 903. In the RAM903, various programs and data necessary for the operation of theelectronic apparatus 900 are also stored. The processing apparatus 901, the ROM 902, and the RAM903 are connected to each other through abus 904. An input/output (I/O) interface 905 is also connected tobus 904.

Generally, the following devices may be connected to the I/O interface 905:input devices 906 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; anoutput device 907 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like;storage 908 including, for example, magnetic tape, hard disk, etc.; and acommunication device 909. Thecommunication device 909 may allow theelectronic apparatus 900 to perform wireless or wired communication with other apparatuses to exchange data. While fig. 9 illustrates anelectronic device 900 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 9 may represent one device or may represent multiple devices as desired.

In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network through thecommunication device 909, or installed from thestorage device 908, or installed from the ROM 902. The computer program, when executed by the processing apparatus 901, performs the above-described functions defined in the methods of the embodiments of the present disclosure. It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set; extracting word characteristics of each candidate entity word in the candidate entity word set; and selecting a target entity word from the candidate entity word set based on the word characteristics, and outputting the target entity word.

Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, an extraction unit, and a selection unit. Where the names of these units do not in some cases constitute a limitation on the unit itself, for example, the first extraction unit may also be described as a "unit that extracts word features of individual candidate entity words in the set of candidate entity words".

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims (17)

Translated fromChinese

1.一种实体词提取方法，其特征在于，包括：1. an entity word extraction method, is characterized in that, comprises:获取待处理文本，对所述待处理文本进行预设处理，得到候选实体词集合；Obtaining the text to be processed, and performing preset processing on the text to be processed to obtain a candidate entity word set;提取所述候选实体词集合中的各个候选实体词的词语特征；extracting word features of each candidate entity word in the candidate entity word set;基于所述词语特征，从所述候选实体词集合中选取出目标实体词，输出所述目标实体词。Based on the word feature, a target entity word is selected from the candidate entity word set, and the target entity word is output.2.根据权利要求1所述的方法，其特征在于，所述基于所述词语特征，从所述候选实体词集合中选取出目标实体词，包括：2. The method according to claim 1, wherein, based on the word feature, selecting a target entity word from the candidate entity word set, comprising:基于实体词在所述待处理文本中的位置信息，确定所述候选实体词集合中的各个候选实体词的词语权重；Determine the word weight of each candidate entity word in the candidate entity word set based on the location information of the entity word in the to-be-processed text;针对所述候选实体词集合中的每个候选实体词，基于该候选实体词的词语特征和词语权重，确定该候选实体词的分数；For each candidate entity word in the candidate entity word set, determine the score of the candidate entity word based on the word feature and word weight of the candidate entity word;基于所述候选实体词集合中的各个候选实体词的分数，从所述候选实体词集合中选取出目标实体词。Based on the scores of each candidate entity word in the candidate entity word set, a target entity word is selected from the candidate entity word set.3.根据权利要求2所述的方法，其特征在于，所述待处理文本为中文文本或中英文混合文本，所述词语特征包括词语的逆文档频率、词语的词频逆文档频率、词语在所述待处理文本和目标语料中的N-Gram分数的比值和词语在所述待处理文本和目标语料中的困惑度的比值；以及3. The method according to claim 2, wherein the text to be processed is a Chinese text or a Chinese-English mixed text, and the word features include the inverse document frequency of the word, the word frequency inverse document frequency of the word, and the inverse document frequency of the word The ratio of the N-Gram scores in the text to be processed and the target corpus and the ratio of the perplexity of words in the text to be processed and the target corpus; and所述基于该候选实体词的词语特征和词语权重，确定该候选实体词的分数，包括：Determining the score of the candidate entity word based on the word feature and word weight of the candidate entity word includes:对该候选实体词的逆文档频率、该候选实体词的词频逆文档频率、该候选实体词在所述待处理文本和目标语料中的N-Gram分数的比值以及该候选实体词在所述待处理文本和目标语料中的困惑度的比值进行加权求和，将求和结果与该候选实体词的词语权重相乘，得到该候选实体词的分数。The inverse document frequency of the candidate entity word, the inverse document frequency of the candidate entity word, the ratio of the N-Gram score of the candidate entity word in the text to be processed and the target corpus, and the candidate entity word in the to-be-processed text. The ratio of the perplexity in the processing text and the target corpus is weighted and summed, and the summation result is multiplied by the word weight of the candidate entity word to obtain the score of the candidate entity word.4.根据权利要求2所述的方法，其特征在于，所述待处理文本为英文文本，所述词语特征包括词语的关键词提取分数、词语在所述待处理文本和目标语料中的N-Gram分数的比值和词语在所述待处理文本和目标语料中的困惑度的比值；以及4. The method according to claim 2, wherein the text to be processed is an English text, and the word features include a keyword extraction score of the word, the N- of the word in the text to be processed and the target corpus. The ratio of Gram scores to the perplexity of words in the text to be processed and the target corpus; and所述基于该候选实体词的词语特征和词语权重，确定该候选实体词的分数，包括：Determining the score of the candidate entity word based on the word feature and word weight of the candidate entity word includes:对该候选实体词的关键词提取分数、该候选实体词在所述待处理文本和目标语料中的N-Gram分数的比值以及该候选实体词在所述待处理文本和目标语料中的困惑度的比值进行加权求和，将求和结果与该候选实体词的词语权重相乘，得到该候选实体词的分数。The keyword extraction score of the candidate entity word, the ratio of the N-Gram score of the candidate entity word in the text to be processed and the target corpus, and the perplexity of the candidate entity word in the text to be processed and the target corpus A weighted sum is performed on the ratio of , and the summation result is multiplied by the word weight of the candidate entity word to obtain the score of the candidate entity word.5.根据权利要求2所述的方法，其特征在于，所述待处理文本为中文文本或中英文混合文本；以及5. The method according to claim 2, wherein the text to be processed is a Chinese text or a Chinese-English mixed text; and所述基于所述候选实体词集合中的各个候选实体词的分数，从所述候选实体词集合中选取出目标实体词，包括：Described based on the score of each candidate entity word in the candidate entity word set, selecting the target entity word from the candidate entity word set, including:基于所述候选实体词集合中的各个候选实体词的分数和词语特征，对所述候选实体词集合进行更新，从更新后的候选实体词集合中选取出目标实体词。Based on the scores and word features of each candidate entity word in the candidate entity word set, the candidate entity word set is updated, and a target entity word is selected from the updated candidate entity word set.6.根据权利要求5所述的方法，其特征在于，所述基于所述候选实体词集合中的各个候选实体词的分数和词语特征，对所述候选实体词集合进行更新，从更新后的候选实体词集合中选取出目标实体词，包括：6. The method according to claim 5, wherein, based on the scores and word features of each candidate entity word in the candidate entity word set, the candidate entity word set is updated, from the updated candidate entity word set. The target entity word is selected from the candidate entity word set, including:基于所述候选实体词集合，执行如下实体词选取步骤：从候选实体词集合中选取符合预设条件的实体词，对符合所述条件的实体词进行组合，得到至少一个词语组合；将所述至少一个词语组合中出现在所述待处理文本中的词语组合确定为候选复合实体词，将所述候选复合实体词添加到候选实体词集合中；针对每个候选复合实体词，基于组成该候选复合实体词的候选实体词的分数，确定该候选复合实体词的分数；基于所述候选复合实体词的分数、所述候选复合实体词的词语特征、候选实体词集合中各个候选实体词的分数和词语特征，对添加后的候选实体词集合进行更新；确定更新后的候选实体词集合与候选实体词集合是否相同；Based on the candidate entity word set, the following entity word selection steps are performed: select entity words that meet preset conditions from the candidate entity word set, and combine the entity words that meet the conditions to obtain at least one word combination; A word combination that appears in the text to be processed in at least one word combination is determined as a candidate compound entity word, and the candidate compound entity word is added to the candidate entity word set; for each candidate compound entity word, based on the composition of the candidate compound entity word The score of the candidate entity word of the compound entity word, to determine the score of the candidate compound entity word; based on the score of the candidate compound entity word, the word feature of the candidate compound entity word, and the score of each candidate entity word in the candidate entity word set and word features, update the added candidate entity word set; determine whether the updated candidate entity word set is the same as the candidate entity word set;若否，则将更新后的候选实体词集合作为候选实体词集合，继续执行所述实体词选取步骤。If not, take the updated candidate entity word set as the candidate entity word set, and continue to perform the entity word selection step.7.根据权利要求6所述的方法，其特征在于，在所述确定更新后的候选实体词集合与候选实体词集合是否相同之后，所述方法还包括：7. The method according to claim 6, wherein after determining whether the updated candidate entity word set is the same as the candidate entity word set, the method further comprises:若是，则基于更新后的候选实体词集合中的各个候选实体词的分数，从更新后的候选实体词集合中选取出目标实体词。If so, the target entity word is selected from the updated candidate entity word set based on the scores of each candidate entity word in the updated candidate entity word set.8.根据权利要求6所述的方法，其特征在于，所述词语特征包括词语的词频；以及8. The method of claim 6, wherein the word features comprise word frequencies of words; and所述基于组成该候选复合实体词的候选实体词的分数，确定该候选复合实体词的分数，包括：The described score of the candidate compound entity word is determined based on the score of the candidate entity word that constitutes the candidate compound entity word, including:对组成该候选复合实体词的两个候选实体词的分数进行加权求和，得到该候选复合实体词的分数，其中，针对组成该候选复合实体词的两个候选实体词中的每个候选实体词，该候选实体词对应的权重为该候选实体词的词频与总词频的比值，所述总词频为组成该候选复合实体词的两个候选实体词的词频之和。Weighted summation is performed on the scores of the two candidate entity words that make up the candidate compound entity word to obtain the score of the candidate compound entity word, wherein, for each candidate entity in the two candidate entity words that make up the candidate compound entity word The weight corresponding to the candidate entity word is the ratio of the word frequency of the candidate entity word to the total word frequency, and the total word frequency is the sum of the word frequencies of the two candidate entity words that form the candidate compound entity word.9.根据权利要求6所述的方法，其特征在于，所述基于所述候选复合实体词的分数、所述候选复合实体词的词语特征、候选实体词集合中各个候选实体词的分数和词语特征，对添加后的候选实体词集合进行更新，包括：9 . The method according to claim 6 , wherein the method based on the score of the candidate compound entity word, the word feature of the candidate compound entity word, the score and the word of each candidate entity word in the candidate entity word set feature, update the added candidate entity word set, including:基于各个候选实体词的分数，对添加后的候选实体词集合中的候选实体词进行筛选，生成候选实体词子集合；Based on the scores of each candidate entity word, filter the candidate entity words in the added candidate entity word set to generate a candidate entity word subset;针对每个候选复合实体词，确定所述候选实体词子集合中是否存在组成该候选复合实体词的至少两个候选实体词；For each candidate compound entity word, determine whether there are at least two candidate entity words that form the candidate compound entity word in the candidate entity word subset;若是，则将组成该候选复合实体词的至少两个候选实体词中分数低的候选实体词从所述候选实体词子集合中删除。If so, delete the candidate entity word with a low score among the at least two candidate entity words composing the candidate compound entity word from the candidate entity word subset.10.根据权利要求9所述的方法，其特征在于，所述词语特征包括词语在所述待处理文本和目标语料中的N-Gram分数的比值和词语在所述待处理文本和目标语料中的困惑度的比值；以及10. The method according to claim 9, wherein the word features include the ratio of the N-Gram scores of words in the text to be processed and the target corpus and the words in the text to be processed and the target corpus the perplexity ratio of ; and在所述针对每个候选复合实体词，确定所述候选实体词子集合中是否存在组成该候选复合实体词的至少两个候选实体词之后，所述方法还包括：After determining, for each candidate compound entity word, whether there are at least two candidate entity words composing the candidate compound entity word in the candidate entity word subset, the method further includes:若所述候选实体词子集合中存在组成该候选复合实体词的一个候选实体词，则确定该候选复合实体词对应的第一比值是否大于组成该候选复合实体词的候选实体词对应的第一比值，以及确定该候选复合实体词对应的第二比值是否大于组成该候选复合实体词的候选实体词对应的第二比值，其中，所述第一比值为词语在所述待处理文本和所述目标语料中的N-Gram分数的比值，所述第二比值为词语在所述待处理文本和目标语料中的困惑度的比值；If there is one candidate entity word composing the candidate compound entity word in the candidate entity word subset, determine whether the first ratio corresponding to the candidate compound entity word is greater than the first ratio corresponding to the candidate entity word composing the candidate compound entity word ratio, and determining whether the second ratio corresponding to the candidate compound entity word is greater than the second ratio corresponding to the candidate entity words constituting the candidate compound entity word, wherein the first ratio is the word in the text to be processed and the The ratio of N-Gram scores in the target corpus, and the second ratio is the ratio of the perplexity of words in the text to be processed and the target corpus;若该候选复合实体词对应的第一比值大于组成该候选复合实体词的候选实体词对应的第一比值和/或该候选复合实体词对应的第二比值大于组成该候选复合实体词的候选实体词对应的第二比值，则将组成该候选复合实体词的候选实体词从所述候选实体词子集合中删除。If the first ratio corresponding to the candidate compound entity word is greater than the first ratio corresponding to the candidate entity words constituting the candidate compound entity word and/or the second ratio corresponding to the candidate compound entity word is greater than the candidate entity constituting the candidate compound entity word If the second ratio corresponding to the word is selected, the candidate entity words forming the candidate compound entity word are deleted from the candidate entity word subset.11.根据权利要求1所述的方法，其特征在于，所述对所述待处理文本进行预设处理，得到候选实体词集合，包括：11. The method according to claim 1, characterized in that, performing preset processing on the to-be-processed text to obtain a candidate entity word set, comprising:对所述待处理文本进行语种识别；Perform language identification on the text to be processed;若所述待处理文本为中文文本或中英文混合文本，则利用命名实体识别技术识别所述待处理文本中的实体词，在识别出的实体词中删除存在于目标实体词集合中的实体词，得到候选实体词集合。If the to-be-processed text is Chinese text or Chinese-English mixed text, use named entity recognition technology to identify entity words in the to-be-processed text, and delete the entity words existing in the target entity word set from the identified entity words , get the candidate entity word set.12.根据权利要求1所述的方法，其特征在于，所述对所述待处理文本进行预设处理，得到候选实体词集合，包括：12. The method according to claim 1, characterized in that, performing preset processing on the to-be-processed text to obtain a candidate entity word set, comprising:对所述待处理文本进行语种识别；Perform language identification on the text to be processed;若所述待处理文本为英文文本，则利用文本匹配技术识别所述待处理文本中的实体词，在识别出的实体词中去除存在于目标实体词集合中的实体词，得到候选实体词集合。If the text to be processed is English text, use text matching technology to identify entity words in the text to be processed, remove entity words existing in the target entity word set from the identified entity words, and obtain a candidate entity word set .13.根据权利要求1所述的方法，其特征在于，在所述基于所述词语特征，从所述候选实体词集合中选取出目标实体词之前，所述方法还包括：13. The method according to claim 1, wherein, before the target entity word is selected from the candidate entity word set based on the word feature, the method further comprises:提取所述待处理文本的文本特征；extracting text features of the text to be processed;提取目标实体词集合中的实体词的词语特征；以及extracting word features of entity words in the target entity word set; and所述基于所述词语特征，从所述候选实体词集合中选取出目标实体词，包括：Described based on the word feature, from the candidate entity word set, the target entity word is selected, including:利用所述候选实体词集合中的实体词的词语特征、所述目标实体词集合中的实体词的词语特征和所述文本特征，构建特征空间；Using the word feature of the entity words in the candidate entity word set, the word feature of the entity word in the target entity word set, and the text feature to construct a feature space;基于所述特征空间，从所述候选实体词集合中选取出目标实体词。Based on the feature space, a target entity word is selected from the candidate entity word set.14.根据权利要求13所述的方法，其特征在于，所述基于所述特征空间，从所述候选实体词集合中选取出目标实体词，包括：14. The method according to claim 13, wherein the selecting a target entity word from the candidate entity word set based on the feature space comprises:针对所述候选实体词集合中的每个候选实体词，在所述特征空间中确定与该候选实体词之间的距离小于预设距离阈值的、存在于所述目标实体词集合中的实体词的数目；For each candidate entity word in the candidate entity word set, determine the entity word in the target entity word set whose distance from the candidate entity word is less than a preset distance threshold in the feature space Number of;确定所述数目是否大于预设数目阈值；determining whether the number is greater than a preset number threshold;若是，则将该候选实体词确定为目标实体词。If so, the candidate entity word is determined as the target entity word.15.一种实体词提取装置，其特征在于，包括：15. A device for extracting entity words, comprising:获取单元，用于获取待处理文本，对所述待处理文本进行预设处理，得到候选实体词集合；an acquisition unit, configured to acquire the text to be processed, perform preset processing on the text to be processed, and obtain a candidate entity word set;第一提取单元，用于提取所述候选实体词集合中的各个候选实体词的词语特征；a first extraction unit, used for extracting word features of each candidate entity word in the candidate entity word set;选取单元，用于基于所述词语特征，从所述候选实体词集合中选取出目标实体词，输出所述目标实体词。The selecting unit is configured to select a target entity word from the candidate entity word set based on the word feature, and output the target entity word.16.一种电子设备，其特征在于，包括：16. An electronic device, characterized in that, comprising:一个或多个处理器；one or more processors;存储装置，其上存储有一个或多个程序，a storage device on which one or more programs are stored,当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如权利要求1-14中任一所述的方法。The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-14.17.一种计算机可读介质，其上存储有计算机程序，其特征在于，该程序被处理器执行时实现如权利要求1-14中任一所述的方法。17. A computer-readable medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method according to any one of claims 1-14 is implemented.

Images