Google

An auto-regressive language model generates text one token at a time, where each new token is predicted based on the sequence of tokens that came before it. The overall probability of the generated sequence is calculated by multiplying the conditional probabilities of each token. The following table illustrates this process for generating the three tokens $$b$$, $$c$$, and $$d$$ given the prefix $$\langle s \rangle\ a$$:

| Context | Predicted Token | Decision Rule | Cumulative Sequence Probability |
| :--- | :--- | :--- | :--- |
| $$\langle s \rangle\ a$$ | $$b$$ | $$\argmax_{x_2 \in V} \Pr(x_{2}|\langle s \rangle\ a)$$ | $$\Pr(\langle s \rangle) \cdot \Pr(a|\langle s \rangle) \cdot \Pr(b|\langle s \rangle\ a)$$ |
| $$\langle s \rangle\ a\ b$$ | $$c$$ | $$\argmax_{x_3 \in V} \Pr(x_{3}|\langle s \rangle\ a\ b)$$ | $$\Pr(\langle s \rangle) \cdot \Pr(a|\langle s \rangle) \cdot \Pr(b|\langle s \rangle\ a) \cdot \Pr(c|\langle s \rangle\ a\ b)$$ |
| $$\langle s \rangle\ a\ b\ c$$ | $$d$$ | $$\argmax_{x_4 \in V} \Pr(x_{4}|\langle s \rangle\ a\ b\ c)$$ | $$\Pr(\langle s \rangle) \cdot \Pr(a|\langle s \rangle) \cdot \Pr(b|\langle s \rangle\ a) \cdot \Pr(c|\langle s \rangle\ a\ b) \cdot \Pr(d|\langle s \rangle\ a\ b\ c)$$ |

At each step, the model selects a token $$x_i$$ from the vocabulary $$V$$ so that the conditional probability $$\Pr(x_{i}|x_0,...,x_{i-1})$$ is maximized. This predicted token is then appended to the end of the context sequence for the next step.

Step-by-Step Example of Auto-Regressive Sequence Generation

A language model is tasked with generating the three-token sequence 'The quick brown' using a greedy, auto-regressive approach. Arrange the following actions in the correct chronological order that the model would take.

An auto-regressive language model is generating a sequence. After two steps, it has produced the sequence `<s> The cat` with a cumulative probability of `0.15`. In the third step, it adds the token `sat`, and the new cumulative probability for the sequence `<s> The cat sat` becomes `0.09`. What was the conditional probability the model assigned to the token `sat` at this step, i.e., what is `Pr(sat | <s> The cat)`? Show your calculation.

Calculating Conditional Probability in a Generation Step

An auto-regressive language model using a greedy decision rule has generated the sequence 'The cat sat' with a cumulative probability of 0.2. At the next step, the model calculates the following conditional probabilities for the next token: P('on' | 'The cat sat') = 0.6, P('by' | 'The cat sat') = 0.3, and P('under' | 'The cat sat') = 0.1. What will be the newly generated sequence and its updated cumulative probability?

Learn Before

Related