Topic models are statistical tools for discovering the hidden semantic structure in a collection of documents (Blei et al.,2003; Blei,2012). Topic models and their extensions have been applied to many fields, such as marketing, sociology, political science, and the digital humanities. Boyd-Graber et al. (2017) provide a review.

Most topic models build on latent Dirichlet allocation (lda) (Blei et al.,2003).lda is a hierarchical probabilistic model that represents each topic as a distribution over terms and represents each document as a mixture of the topics. When fit to a collection of documents, the topics summarize their contents, and the topic proportions provide a low-dimensional representation of each document.lda can be fit to large datasets of text by using variational inference and stochastic optimization (Hoffman et al.,2010,s).

lda is a powerful model and it is widely used. However, it suffers from a pervasive technical problem—it fails in the face of large vocabularies. Practitioners must severely prune their vocabularies in order to fit good topic models—namely, those that are both predictive and interpretable. This is typically done by removing the most and least frequent words. On large collections, this pruning may remove important terms and limit the scope of the models. The problem of topic modeling with large vocabularies has yet to be addressed in the research literature.

In parallel with topic modeling came the idea of word embeddings. Research in word embeddings begins with the neural language model of Bengio et al. (2003), published in the same year and journal as Blei et al. (2003). Word embeddings eschew the “one-hot” representation of words—a vocabulary-length vector of zeros with a single one—to learn a distributed representation, one where words with similar meanings are close in a lower-dimensional vector space (Rumelhart and Abrahamson,1973; Bengio et al.,2006). As for topic models, researchers scaled up embedding methods to large datasets (Mikolov et al.,2013a,s; Pennington et al.,2014; Levy and Goldberg,2014; Mnih and Kavukcuoglu,2013). Word embeddings have been extended and developed in many ways. They have become crucial in many applications of natural language processing (Maas et al.,2011; Li and Yang,2018), and they have also been extended to datasets beyond text (Rudolph et al.,2016).

In this paper, we develop theembedded topic model(etm), a document model that marrieslda and word embeddings. Theetm enjoys the good properties of topic models and the good properties of word embeddings. As a topic model, it discovers an interpretable latent semantic structure of the documents; as a word embedding model, it provides a low-dimensional representation of the meaning of words. Theetm robustly accommodates large vocabularies and the long tail of language data.

Figure 1 illustrates the advantages. This figure shows the ratio between the perplexity on held-out documents (a measure of predictive performance) and the topic coherence (a measure of the quality of the topics), as a function of the size of the vocabulary. (The perplexity has been normalized by the vocabulary size.) This is for a corpus of 11.2K articles from the20NewsGroup and for 100 topics. The red line islda; its performance deteriorates as the vocabulary size increases—the predictive performance and the quality of the topics get worse. The blue line is theetm; it maintains good performance, even as the vocabulary size become large.

Likelda, theetm is a generative probabilistic model: Each document is a mixture of topics and each observed word is assigned to a particular topic. In contrast tolda, the per-topic conditional probability of a term has a log-linear form that involves a low-dimensional representation of the vocabulary. Each term is represented by an embedding and each topic is a point in that embedding space. The topic’s distribution over terms is proportional to the exponentiated inner product of the topic’s embedding and each term’s embedding.Figures 2 and3 show topics from a 300-topicetm ofThe New York Times. The figures show each topic’s embedding and its closest words; these topics are about Christianity and sports.

Representing topics as points in the embedding space allows theetm to be robust to the presence of stop words, unlike most topic models. When stop words are included in the vocabulary, theetm assigns topics to the corresponding area of the embedding space (we demonstrate this inSection 6).

As for most topic models, the posterior of the topic proportions is intractable to compute. We derive an efficient algorithm for approximating the posterior with variational inference (Jordan et al.,1999; Hoffman et al.,2013; Blei et al.,2017) and additionally use amortized inference to efficiently approximate the topic proportions (Kingma and Welling,2014; Rezende et al.,2014). The resulting algorithm fits theetm to large corpora with large vocabularies. This algorithm can either use previously fitted word embeddings, or fit them jointly with the rest of the parameters. (In particular,Figures 1 to3 were made using the version of theetm that uses pre-fitted skip-gram word embeddings.)

We compared the performance of theetm tolda, the neural variational document model (nvdm) (Miao et al.,2016), andprodlda (Srivastava and Sutton,2017).¹ Thenvdm is a form of multinomial matrix factorization andprodlda is a modern version oflda that uses a product of experts to model the distribution over words. We also compare to a document model that combinesprodlda with pre-fitted word embeddings. Theetm yields better predictive performance, as measured by held-out log-likelihood on a document comple tion task (Wallach et al.,2009b). It also discovers more meaningful topics, as measured by topic coherence (Mimno et al.,2011) and topic diver sity. The latter is a metric we introduce in this paper that, together with topic coherence, gives a better indication of the quality of the topics. Theetm is especially robust to large vocabularies.

This work develops a new topic model that extendslda.lda has been extended in many ways, and topic modeling has become a subfield of its own. For a review, see Blei (2012) and Boyd-Graber et al. (2017).

A broader set of related works are neural topic models. These mainly focus on improving topic modeling inference through deep neural networks (Srivastava and Sutton,2017; Card et al.,2017; Cong et al.,2017; Zhang et al.,2018). Specifically, these methods reduce the dimension of the text data through amortized inference and the variational auto-encoder (Kingma and Welling,2014; Rezende et al.,2014). To perform inference in theetm, we also avail ourselves of amortized inference methods (Gershman and Goodman,2014).

As a document model, theetm also relates to works that learn per-document representations as part of an embedding model (Le and Mikolov,2014; Moody,2016; Miao et al.,2016; Li et al.,2016). In contrast to these works, the document variables in theetm are part of a larger probabilistic topic model.

One of the goals in developing theetm is to incorporate word similarity into the topic model, and there is previous research that shares this goal. These methods either modify the topic priors (Petterson et al.,2010; Zhao et al.,2017b; Shi et al.,2017; Zhao et al.,2017a) or the topic assignment priors (Xie et al.,2015). For example, Petterson et al. (2010) use a word similarity graph (as given by a thesaurus) to biaslda towards assigning similar words to similar topics. As another example, Xie et al. (2015) model the per-word topic assignments oflda using a Markov random field to account for both the topic proportions and the topic assignments of similar words. These methods use word similarity as a type of “side information” about language; in contrast, theetm directly models the similarity (via embeddings) in its generative process of words.

However, a more closely related set of works directly combine topic modeling and word embeddings. One common strategy is to convert the discrete text into continuous observations of embeddings, and then adaptlda to generate real-valued data (Das et al.,2015; Xun et al.,2016; Batmanghelich et al.,2016; Xun et al.,2017). With this strategy, topics are Gaussian distributions with latent means and covariances, and the likelihood over the embeddings is modeled with a Gaussian (Das et al.,2015) or a Von-Mises Fisher distribution (Batmanghelich et al.,2016). Theetm differs from these approaches in that it is a model of categorical data, one that goes through the embeddings matrix. Thus it does not require pre-fitted embeddings and, indeed, can learn embeddings as part of its inference process. Theetm also differs from these approaches in that it is amenable to large datasets with large vocabularies.

There are few other ways of combininglda and embeddings. Nguyen et al. (2015) mix the likelihood defined bylda with a log-linear model that uses pre-fitted word embeddings; Bunk and Krestel (2018) randomly replace words drawn from a topic with their embeddings drawn from a Gaussian; Xu et al. (2018) adopt a geometric perspective, using Wasserstein distances to learn topics and word embeddings jointly; and Keya et al. (2019) propose the neural embedding allocation (NEA), which has a similar generative process to theetm but is fit using a pre-fittedlda model as a target distribution. Because it requireslda, thenea suffers from the same limitation aslda. These models often lack scalability with respect to the vocabulary size and are fit using Gibbs sampling, limiting their scalability to large corpora.

Theetm builds on two main ideas,lda and word embeddings. Consider a corpus ofD documents, where the vocabulary containsV distinct terms. Letw_dn ∈{1,…,V } denote then^th word in thed^th document.

lda is a probabilistic generative model of documents (Blei et al.,2003). It positsK topicsβ_1:K, each of which is a distribution over the vocabulary.lda assumes each document comes from a mixture of topics, where the topics are shared across the corpus and the mixture proportions are unique for each document. The generative process for each document is the following:

1. Draw topic proportionθ_d ∼Dirichlet(α_θ).

2. For each wordn in the document:

   • Draw topic assignmentz_dn ∼Cat(θ_d).

   • Draw word$wdn∼Cat(βzdn)$⁠.

Word embeddings provide models of language that use vector representations of words (Rumelhart and Abrahamson,1973; Bengio et al.,2003). The word representations are fitted to relate to meaning, in that words with similar meanings will have representations that are close. (In embeddings, the “meaning” of a word comes from the contexts in which it is used [Harris,1954].)

Theetm is a topic model that uses embedding representations of both words and topics. It contains two notions of latent dimension. First, it embeds the vocabulary in anL-dimensional space. These embeddings are similar in spirit to classical word embeddings. Second, it represents each document in terms ofK latent topics.

In traditional topic modeling, each topic is a full distribution over the vocabulary. In theetm, however, thek^th topic is a vectorα_k ∈ℝ^L in the embedding space. We callα_k atopic embedding— it is a distributed representation of thek^th topic in the semantic space of words.

In its generative process, theetm uses the topic embedding to form a per-topic distribution over the vocabulary. Specifically, theetm uses a log-linear model that takes the inner product of the word embedding matrix and the topic embedding. With this form, theetm assigns high probability to a wordv in topick by measuring the agreement between the word’s embedding and the topic’s embedding.

Denote theL ×V word embedding matrix byρ; the columnρ_v is the embedding of termv. Under theetm, the generative process of thed^th document is the following:

1. Draw topic proportions$θd∼LN(0,I)$⁠.

2. For each wordn in the document:

   • Draw topic assignmentz_dn ∼Cat(θ_d).

   • Draw the wordw_dn ∼softmax(ρ^⊤$αzdn)$⁠.

Steps 1 and 2a are standard for topic modeling: They represent documents as distributions over topics and draw a topic assignment for each observed word. Step 2b is different; it uses the embeddings of the vocabularyρ and the assigned topic embedding$αzdn$ to draw the observed word from the assigned topic, as given byz_dn.

The topic distribution in Step 2b mirrors thecbow likelihood in Eq.1. Recallcbow uses the surrounding words to form the context vectorα_dn. In contrast, theetm uses the topic embedding$αzdn$ as the context vector, where the assigned topicz_dn is drawn from the per-document variableθ_d. Theetm draws its words from a document context, rather than from a window of surrounding words.

Theetm likelihood uses a matrix of word embeddingsρ, a representation of the vocabulary in a lower dimensional space. In practice, it can either rely on previously fitted embeddings or learn them as part of its overall fitting procedure. When theetm learns the embeddings as part of the fitting procedure, it simultaneously finds topics and an embedding space.

When theetm uses previously fitted embeddings, it learns the topics of a corpus in a particular embedding space. This strategy is particularly useful when there are words in the embedding that are not used in the corpus. Theetm can hypothesize how those words fit in to the topics because it can calculate$ρv⊤αk$ even for wordsv that do not appear in the corpus.

We are given a corpus of documents {w₁,…,w_D}, where thed^th documentw_d is a collection ofN_d words. How do we fit theetm to this corpus?

We sidestep the intractable integral in Eq. eq:integral with variational inference (Jordan et al.,1999; Blei et al.,2017). Variational inference optimizes a sum of per-document bounds on the log of the marginal likelihood of Eq.4.

To begin, posit a family of distributions of the untransformed topic proportionsq(δ_d ;w_d,ν). This family of distributions is parameterized byν. We use amortized inference, whereq(δ_d ;w_d,ν) (called avariational distribution) depends on both the documentw_d and shared parametersν. In particular,q(δ_d ;w_d,ν) is a Gaussian whose mean and variance come from an “inference network,” a neural network parameterized byν (Kingma and Welling,2014). The inference network ingests a bag-of-words representation of the documentw_d and outputs the mean and covariance ofδ_d. (To accommodate documents of varying length, we form the input of the inference network by normalizing the bag-of-word representation of the document by the number of wordsN_d.)

We study the performance of theetm and compare it to other unsupervised document models. A good document model should provide both coherent patterns of language and an accurate distribution of words, so we measure performance in terms of both predictive accuracy and topic interpretability. We measure accuracy with log-likelihood on a document completion task (Rosen-Zvi et al.,2004; Wallach et al.,2009b); we measure topic interpretability as a blend of topic coherence and diversity. We find that, of the interpretable models, theetm is the one that provides better predictions and topics.

In a separate analysis (Section 6.1), we study the robustness of each method in the presence of stop words. Standard topic models fail in this regime—because stop words appear in many documents, every learned topic includes some stop words, leading to poor topic interpretability. In contrast, theetm is able to use the information from the word embeddings to provide interpretable topics.

We study the20Newsgroups corpus and theNew York Times corpus; the statistics of both corpora are summarized inTable 1.

The20Newsgroup corpus is a collection of newsgroup posts. We preprocess the corpus by filtering stop words, words with document frequency above 70%, and tokenizing. To form the vocabulary, we keep all words that appear in more than a certain number of documents, and we vary the threshold from 100 (a smaller vocabulary, whereV = 3,102) to 2 (a larger vocabulary, whereV = 52,258). After preprocessing, we further remove one-word documents from the validation and test sets. We split the corpus into a training set of 11,260 documents, a test set of 7,532 documents, and a validation set of 100 documents.

TheNew York Times corpus is a larger collection of news articles. It contains more than 1.8 million articles, spanning the years 1987–2007. We follow the same preprocessing steps as for20Newsgroups. We form versions of this corpus with vocabularies ranging fromV = 9,842 toV = 212,237. After preprocessing, we use 85% of the documents for training, 10% for testing, and 5% for validation.

We compare the performance of theetm against several document models. We briefly describe each below.

We consider latent Dirichlet allocation (lda) (Blei et al.,2003), a standard topic model that posits Dirichlet priors for the topicsβ_k and topic proportionsθ_d. (We set the prior hyperparameters to 1.) It is a conditionally conjugate model, amenable to variational inference with coordinate ascent. We considerlda because it is the most commonly used topic model, and it has a similar generative process as theetm.

We also consider the neural variational document model (nvdm) (Miao et al.,2016). Thenvdm is a multinomial factor model of documents; it posits the likelihoodw_dn ∼softmax(β^⊤θ_d), where theK-dimensional vector$θd∼N(0,IK)$ is a per-document variable, andβ is a real-valued matrix of sizeK ×V. Thenvdm uses a per-document real-valued latent vectorθ_d to average over the embedding matrixβ in the logit space. Like theetm, thenvdm uses amortized variational inference to jointly learn the approximate posterior over the document representationθ_d and the model parameterβ.

nvdm is not interpretable as a topic model; its latent variables are unconstrained. We study a more interpretable variant of thenvdm which constrainsθ_d to lie in the simplex, replacing its Gaussian prior with a logistic normal (Aitchison and Shen,1980). (This can be thought of as a semi-nonnegative matrix factorization.) We call this document model Δ-nvdm.

We also considerprodlda (Srivastava and Sutton,2017). It posits the likelihoodw_dn ∼softmax(β^⊤θ_d) where the topic proportionsθ_d are from the simplex. Contrary tolda, the topic-matrixβ s unconstrained.

prodlda shares the generative model with Δ-nvdm but it is fit differently.prodlda uses amortized variational inference with batch normalization (Ioffe and Szegedy,2015) and dropout (Srivastava et al.,2014).

Finally, we consider a document model that combinesprodlda with pre-fitted word embeddingsρ, by using the likelihoodw_dn ∼softmax(ρ^⊤θ_d). We call this document modelprodlda-PWE, where PWE stands for Pre-fitted Word Embeddings.

We study two variants of theetm, one where the word embeddings are pre-fitted and one where they are learned jointly with the rest of the parameters. The variant with pre-fitted embeddings is called theetm-PWE.

Forprodlda-PWE and theetm-PWE, we first obtain the word embeddings (Mikolov et al.,2013b) by training skip-gram on each corpus. (We reuse the same embeddings across the experiments with varying vocabulary sizes.)

Given a corpus, each model comes with an approximate posterior inference problem. We use variational inference for all of the models and employsvi (Hoffman et al.,2013) to speed up the optimization. The minibatch size is 1,000 documents. Forlda, we set the learning rate as suggested by Hoffman et al. (2013): the delay is 10 and the forgetting factor is 0.85.

Withinsvi,lda enjoys coordinate ascent variational updates; we use five inner steps to optimize the local variables. For the other models, we use amortized inference over the local variablesθ_d. We use 3-layer inference networks and we set the local learning rate to 0.002. We useℓ₂ regularization on the variational parameters (the weight decay parameter is 1.2 × 10⁻⁶).

We first examine the embeddings. Theetm,nvdm, Δ-nvdm, andprodlda all learn word embeddings. We illustrate them by fixing a set of terms and showing the closest words in the embedding space (as measured by cosine distance). For comparison, we also illustrate word embeddings learned by the skip-gram model.

Table 2 illustrates the embeddings of the different models. All the methods provide interpretable embeddings—words with related meanings are close to each other. Theetm, thenvdm, andprodlda learn embeddings that are similar to those from the skip-gram. The embeddings of Δ-nvdm are different; the simplex constraint on the local variable and the inference procedure change the nature of the embeddings.

We next look at the learned topics.Table 3 displays the seven most used topics for all methods, as given by the average of the topic proportionsθ_d.lda and both variants of theetm provide interpretable topics. The rest of the models do not provide interpretable topics; their matricesβ are unconstrained and thus are not interpretable as distributions over the vocabulary that mix to form documents. Δ-nvdm also suffers from this effect although it is less apparent (see, e.g., the fifth listed topic for Δ-nvdm).

We next study the models quantitatively. We measure the quality of the topics and the predictive performance of the model. We found that among the models with interpretable topics, theetm provides the best predictions.

The idea behind topic coherence is that a coherent topic will display words that tend to occur in the same documents. In other words, the most likely words in a coherent topic should have high mutual information. Document models with higher topic coherence are more interpretable topic models.

We combine coherence with a second metric, topic diversity. We define topic diversity to be the percentage of unique words in the top 25 words of all topics. Diversity close to 0 indicates redundant topics; diversity close to 1 indicates more varied topics.

We define the overall quality of a model’s topics as the product of its topic diversity and topic coherence.

A good topic model also provides a good distribution of language. To measure predictive power, we calculate log likelihood on a document completion task (Rosen-Zvi et al.,2004; Wallach et al.,2009b). We divide each test document into two sets of words. The first half is observed: it induces a distribution over topics which, in turn, induces a distribution over the next words in the document. We then evaluate the second half under this distribution. A good document model should provide high log-likelihood on the second half. (For all methods, we approximate the likelihood by settingθ_d to the variational mean.)

We study both corpora and with different vocabularies.Figures 4 and5 show interpretability of the topics as a function of predictive power. (To ease visualization, we exponentiate topic quality and normalize all metrics by subtracting the mean and dividing by the standard deviation across methods.) The best models are on the upper right corner.

lda predicts worst in almost all settings. On the20NewsGroups, thenvdm’s predictions are in general better thanlda but worse than for the other methods; on theNew York Times, thenvdm gives the best predictions. However, topic quality for thenvdm is far below the other methods. (It does not provide “topics”, so we assess the interpretability of itsβ matrix.) In prediction, both versions of theetm are at least as good as the simplex-constrained Δ-nvdm. More importantly, both versions of theetm outperform theprodlda-PWE; signaling theetm provides a better way of integrating word embeddings into a topic model.

These figures show that, of the interpretable models, theetm provides the best predictive performance while keeping interpretable topics. It is robust to large vocabularies.

We now study a version of theNew York Times corpus that includes all stop words. We remove infrequent words to form a vocabulary of size 10,283. Our goal is to show that theetm-PWE provides interpretable topics even in the presence of stop words, another regime where topic models typically fail. In particular, given that stop words appear in many documents, traditional topic models learn topics that contain stop words, regardless of the actual semantics of the topic. This leads to poor topic interpretability. There are extensions of topic models specifically designed to cope with stop words (Griffiths et al.,2004; Chemudugunta et al.,2006; Wallach et al.,2009a); our goal here is not to establish comparisons with these methods but to show the performance of theetm-PWE in the presence of stop words.

We fitlda, the Δ-nvdm, theprodlda-PWE, and theetm-PWE withK = 300 topics. (We do not report thenvdm because it does not provide interpretable topics.)Table 4 shows the topic quality (the product of topic coherence and topic diversity). Overall, theetm-PWE gives the best performance in terms of topic quality.

While theetm has a few “stop topics” that are specific for stop words (see, e.g.,Figure 6), Δ-nvdm andlda have stop words in almost every topic. (The topics are not displayed here for space constraints.) The reason is that stop words co-occur in the same documents as every other word; therefore traditional topic models have difficulties telling apart content words and stop words. Theetm-PWE recognizes the location of stop words in the embedding space; its sets them off on their own topic.

We developed theetm, a generative model of documents that marrieslda with word embeddings. Theetm assumes that topics and words live in the same embedding space, and that words are generated from a categorical distribution whose natural parameter is the inner product of the word embeddings and the embedding of the assigned topic.

Theetm learns interpretable word embeddings and topics, even in corpora with large vocabularies. We studied the performance of theetm against several document models. Theetm learns both coherent patterns of language and an accurate distribution of words.

DB and AD are supported by ONR N00014-17-1-2131, ONR N00014-15-1-2209, NIH 1U01MH115727-01, NSF CCF-1740833, DARPA SD2 FA8750-18-C-0130, Amazon, NVIDIA, and the Simons Foundation. FR received funding from the EU’s Horizon 2020 R&I programme under the Marie Skłodowska-Curie grant agreement 706760. AD is supported by a Google PhD Fellowship.

Work done while at Columbia University and the University of Cambridge.