Top-p sampling, also known asnucleus sampling, is astochastic decoding strategy for generating sequences fromautoregressive probabilistic models. It was originally proposed byAri Holtzman,Yejin Choi and colleagues in 2019 fornatural language generation to address the issue of repetitive and nonsensical text generated by other common decoding methods likebeam search.^[1] The technique has since been applied in other scientific fields, such asprotein engineering^[2] andgeophysics.^[3]

In top-p sampling, a probability thresholdp is set, and the next item in a sequence is sampled only from the smallest possible set of high-probability candidates whose cumulative probability exceedsp. This method adapts the size of the candidate pool based on the model's certainty, making it more flexible thantop-k sampling, which samples from a fixed number of candidates. Due to its effectiveness, top-p sampling is a widely used technique in manylarge language model applications.^[4]

At each step of the text generation process, a language model calculates aprobability distribution over its entire vocabulary for the next token. While simply picking the token with the highest probability (greedy search) or a limited set of high-probability sequences (beam search) is possible, these deterministic methods often produce text that is dull, repetitive, or nonsensical.^[1] Top-p sampling introduces randomness to avoid these issues while maintaining quality.

The core idea is to sample from a smaller, more credible set of tokens at each step, called thenucleus. This nucleus contains the most likely next tokens whose combined, orcumulative probability, just exceeds the thresholdp. By sampling only from this dynamically-sized group, the model can adapt to different situations. When the model is confident about the next token (e.g., one token has a very high probability), the nucleus will be small. When the model is uncertain (the probabilities are more evenly distributed), the nucleus will be larger, allowing for more diversity.

The process at each step is as follows:

1. The model calculates the probabilities for all possible next tokens.
2. The tokens are sorted by their probability in descending order.
3. The nucleus is formed by selecting tokens from the top of the list until their cumulative probability exceeds the predefined threshold,p.
4. The probabilities of tokens within this nucleus are then rescaled so that they sum to 1. All tokens outside the nucleus are discarded (given a probability of 0).
5. The final next token is randomly sampled from this new, smaller distribution.

Formally, the nucleus,${\displaystyle V^{(p)}\subseteq V}$, is defined as the smallest set of tokens satisfying:$${\displaystyle \sum _{x\in V^{(p)}}P(x|x_1,\dots ,x_{t-1})\ge p}$$In this formula,${\displaystyle P(x|x_1,\dots ,x_{t-1})}$ represents the probability of a token${\displaystyle x}$ given the preceding tokens${\displaystyle x_1,\dots ,x_{t-1}}$.

Imagine at a certain step, a language model has a vocabulary of five words: `[the, a, cat, dog, eats]` and produces the following probabilities:

• the: 0.5
• a: 0.2
• cat: 0.1
• dog: 0.1
• eats: 0.1

If we set${\displaystyle p=0.8}$:

1. The tokens are sorted by probability: [the, a, cat, dog, eats].
2. The cumulative probability is calculated:
   • the: 0.5
   • the + a: 0.5 + 0.2 = 0.7
   • the + a + cat: 0.7 + 0.1 = 0.8
3. The nucleus is the smallest set with cumulative probability ≥ 0.8, which is${\displaystyle V^{(0.8)}=\{\text{the, a, cat}\}}$.
4. The probabilities for this set are rescaled to sum to 1:
   • P(the) = 0.5 / 0.8 = 0.625
   • P(a) = 0.2 / 0.8 = 0.25
   • P(cat) = 0.1 / 0.8 = 0.125
5. The next token is then sampled from this new distribution, meaning dog and eats have a 0% chance of being chosen.

Top-k sampling is a similar technique where the pool of candidate tokens is restricted to the${\displaystyle k}$ most likely tokens. The main advantage of top-p is its adaptability. When the model is very certain about the next token (a peaked distribution), the nucleus${\displaystyle V^{(p)}}$ can be very small. When the model is uncertain (a flat distribution), the nucleus can be much larger, allowing for more diversity. In contrast, top-k always samples from a fixed number of tokens, which may be too restrictive or too broad depending on the context.^[1]

While top-p sampling is most famously used as a decoding strategy for large language models, the technique has also been adapted for use in other scientific domains that involve generating or analyzing sequential data from probabilistic models.

In its original domain of natural language generation, top-p sampling is valued for its ability to produce more diverse and coherent text compared to deterministic methods. It has been shown to be beneficial in tasks like automatic question generation, where sample diversity is important for creating effective training data forquestion answering models.^[5]

Top-p sampling is used incomputational biology to generate novel molecular and protein sequences from specialized language models. Inde novodrug design, chemical language models trained on molecular structures use nucleus sampling to generate focused libraries of new, valid drug candidates. By combining this generation with a predictive model forbioactivity, researchers have identified novel, potentkinase inhibitors.^[6] Similarly, inprotein engineering, the technique is used to sample protein language models to explore the vast space of possibleamino acid sequences to find novel, functional candidates for use intherapeutics or new materials.^[2]

The technique has also been applied ingeophysics for denoising audiomagnetotelluric (AMT) data. In one method, nucleus sampling is integrated into an attention mechanism to help identify and remove complex anthropogenic noise from geophysical signals. This improves the accuracy of AMT in interpreting the Earth's subsurface resistivity structure, which is critical for applications likemineral exploration.^[3]

While top-p and top-k sampling address many of the issues found in deterministic methods likebeam search, they are not without shortcomings. Research has shown that these stochastic methods can produce text with undesirable repetitions and may not fully capture the statistical properties of human language.^[7][8][9]

A range of alternative sampling strategies have been proposed to address these limitations.

• Factual-nucleus sampling was proposed to counter the tendency for the "uniform randomness" applied to the nucleus to harm thefactuality of the generated text. It dynamically adapts the level of randomness to improve factual accuracy while maintaining text quality.^[10]
• Locally typical sampling frames text generation in aninformation-theoretic light. Instead of selecting only the highest-probability tokens, it samples from a set of tokens that are "locally typical" in an information-theoretic sense, which has been shown to reduce repetition and improve quality.^[7]
• Priority sampling is a deterministic alternative designed to address the issue of repeated or incoherent samples. It produces a set of unique samples ordered by the model's confidence and has been shown to outperform nucleus sampling in somecompiler optimization tasks.^[11]

• Beam search

• Natural language generation
• Machine learning algorithms
• Large language models
• 2019 in artificial intelligence

• Articles with short description
• Short description is different from Wikidata