Innatural language processing, aword embedding is a representation of a word. Theembedding is used intext analysis. Typically, the representation is areal-valued vector that encodes the meaning of the word in such a way that the words that are closer in the vector space are expected to be similar in meaning.^[1] Word embeddings can be obtained usinglanguage modeling andfeature learning techniques, where words or phrases from the vocabulary are mapped tovectors ofreal numbers.

Methods to generate this mapping includeneural networks,^[2]dimensionality reduction on the wordco-occurrence matrix,^[3][4][5] probabilistic models,^[6] explainable knowledge base method,^[7] and explicit representation in terms of the context in which words appear.^[8]

Word and phrase embeddings, when used as the underlying input representation, have been shown to boost the performance in NLP tasks such assyntactic parsing^[9] andsentiment analysis.^[10]

Indistributional semantics, a quantitative methodological approach for understanding meaning in observed language, word embeddings or semanticfeature space models have been used as a knowledge representation for some time.^[11] Such models aim to quantify and categorize semantic similarities between linguistic items based on their distributional properties in large samples of language data. The underlying idea that "a word is characterized by the company it keeps" was proposed in a 1957 article byJohn Rupert Firth,^[12] but also has roots in the contemporaneous work on search systems^[13] and in cognitive psychology.^[14]

The notion of a semantic space with lexical items (words or multi-word terms) represented as vectors or embeddings is based on the computational challenges of capturing distributional characteristics and using them for practical application to measure similarity between words, phrases, or entire documents. The first generation of semantic space models is thevector space model for information retrieval.^[15][16][17] Such vector space models for words and their distributional data implemented in their simplest form results in a very sparse vector space of high dimensionality (cf.curse of dimensionality). Reducing the number of dimensions using linear algebraic methods such assingular value decomposition then led to the introduction oflatent semantic analysis in the late 1980s and therandom indexing approach for collecting word co-occurrence contexts.^{[18][19][20][21]} In 2000,Bengio et al. provided in a series of papers titled "Neural probabilistic language models" to reduce the high dimensionality of word representations in contexts by "learning a distributed representation for words".^[22][23][24]

A study published inNeurIPS (NIPS) 2002 introduced the use of both word and document embeddings applying the method of kernel CCA to bilingual (and multi-lingual) corpora, also providing an early example ofself-supervised learning of word embeddings.^[25]

Word embeddings come in two different styles, one in which words are expressed as vectors of co-occurring words, and another in which words are expressed as vectors of linguistic contexts in which the words occur; these different styles are studied in Lavelli et al., 2004.^[26] Roweis and Saul published inScience how to use "locally linear embedding" (LLE) to discover representations of high dimensional data structures.^[27] Most new word embedding techniques after about 2005 rely on aneural network architecture instead of more probabilistic and algebraic models, after foundational work done by Yoshua Bengio^{[28][circular reference]} and colleagues.^[29][30]

The approach has been adopted by many research groups after theoretical advances in 2010 had been made on the quality of vectors and the training speed of the model, as well as after hardware advances allowed for a broaderparameter space to be explored profitably. In 2013, a team atGoogle led byTomas Mikolov createdword2vec, a word embedding toolkit that can train vector space models faster than previous approaches. The word2vec approach has been widely used in experimentation and was instrumental in raising interest for word embeddings as a technology, moving the research strand out of specialised research into broader experimentation and eventually paving the way for practical application.^[31]

Historically, one of the main limitations of static word embeddings or wordvector space models is that words with multiple meanings are conflated into a single representation (a single vector in the semantic space). In other words,polysemy andhomonymy are not handled properly. For example, in the sentence "The club I tried yesterday was great!", it is not clear if the termclub is related to the word sense of aclub sandwich,clubhouse,golf club, or any other sense thatclub might have. The necessity to accommodate multiple meanings per word in different vectors (multi-sense embeddings) is the motivation for several contributions in NLP to split single-sense embeddings into multi-sense ones.^[32][33]

Most approaches that produce multi-sense embeddings can be divided into two main categories for their word sense representation, i.e., unsupervised and knowledge-based.^[34] Based onword2vec skip-gram, Multi-Sense Skip-Gram (MSSG)^[35] performs word-sense discrimination and embedding simultaneously, improving its training time, while assuming a specific number of senses for each word. In the Non-Parametric Multi-Sense Skip-Gram (NP-MSSG) this number can vary depending on each word. Combining the prior knowledge of lexical databases (e.g.,WordNet,ConceptNet,BabelNet), word embeddings andword sense disambiguation, Most Suitable Sense Annotation (MSSA)^[36] labels word-senses through an unsupervised and knowledge-based approach, considering a word's context in a pre-defined sliding window. Once the words are disambiguated, they can be used in a standard word embeddings technique, so multi-sense embeddings are produced. MSSA architecture allows the disambiguation and annotation process to be performed recurrently in a self-improving manner.^[37]

The use of multi-sense embeddings is known to improve performance in several NLP tasks, such aspart-of-speech tagging, semantic relation identification,semantic relatedness,named entity recognition and sentiment analysis.^[38][39]

As of the late 2010s, contextually-meaningful embeddings such asELMo andBERT have been developed.^[40] Unlike static word embeddings, these embeddings are at the token-level, in that each occurrence of a word has its own embedding. These embeddings better reflect the multi-sense nature of words, because occurrences of a word in similar contexts are situated in similar regions of BERT’s embedding space.^[41][42]

Word embeddings forn-grams in biological sequences (e.g. DNA, RNA, and Proteins) forbioinformatics applications have been proposed by Asgari and Mofrad.^[43] Named bio-vectors (BioVec) to refer to biological sequences in general with protein-vectors (ProtVec) for proteins (amino-acid sequences) and gene-vectors (GeneVec) for gene sequences, this representation can be widely used in applications of deep learning inproteomics andgenomics. The results presented by Asgari and Mofrad^[43] suggest that BioVectors can characterize biological sequences in terms of biochemical and biophysical interpretations of the underlying patterns.

Word embeddings with applications ingame design have been proposed by Rabii and Cook^[44] as a way to discoveremergent gameplay using logs of gameplay data. The process requires transcribing actions that occur during a game within aformal language and then using the resulting text to create word embeddings. The results presented by Rabii and Cook^[44] suggest that the resulting vectors can capture expert knowledge about games likechess that are not explicitly stated in the game's rules.

The idea has been extended to embeddings of entire sentences or even documents, e.g. in the form of thethought vectors concept. In 2015, some researchers suggested "skip-thought vectors" as a means to improve the quality ofmachine translation.^[45] A more recent and popular approach for representing sentences is Sentence-BERT, or SentenceTransformers, which modifies pre-trainedBERT with the use of siamese and triplet network structures.^[46]

Software for training and using word embeddings includesTomáš Mikolov'sWord2vec, Stanford University'sGloVe,^[47] GN-GloVe,^[48] Flair embeddings,^[38] AllenNLP'sELMo,^[49]BERT,^[50]fastText,Gensim,^[51] Indra,^[52] andDeeplearning4j.Principal Component Analysis (PCA) andT-Distributed Stochastic Neighbour Embedding (t-SNE) are both used to reduce the dimensionality of word vector spaces and visualize word embeddings andclusters.^[53]

For instance, the fastText is also used to calculate word embeddings fortext corpora inSketch Engine that are available online.^[54]

Word embeddings may contain the biases and stereotypes contained in the trained dataset, as Bolukbasi et al. points out in the 2016 paper “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings” that a publicly available (and popular) word2vec embedding trained on Google News texts (a commonly used data corpus), which consists of text written by professional journalists, still shows disproportionate word associations reflecting gender and racial biases when extracting word analogies.^[55] For example, one of the analogies generated using the aforementioned word embedding is “man is to computer programmer as woman is to homemaker”.^[56][57]

Research done by Jieyu Zhou et al. shows that the applications of these trained word embeddings without careful oversight likely perpetuates existing bias in society, which is introduced through unaltered training data. Furthermore, word embeddings can even amplify these biases .^[58][59]

• Embedding (machine learning)
• Brown clustering
• Distributional–relational database

• Language modeling
• Artificial neural networks
• Natural language processing
• Computational linguistics
• Semantic relations

• CS1: long volume value
• Articles with short description
• Short description matches Wikidata
• All articles lacking reliable references
• Articles lacking reliable references from May 2024