Google

This schematic illustrates the sequential probability calculation in Causal Language Modeling, a type of auto-regressive model. For a sequence $$x_0, x_1, \dots, x_4$$, the model predicts each token based on the embeddings of the tokens that came before it. The process begins by setting the probability of the first token, $$\Pr(x_0)$$, to 1. Each subsequent token's probability is then conditioned on the embeddings of all prior tokens, as shown in the diagram below. This unidirectional, step-by-step dependency is a core feature of causal language models.

```text
Token:       x0       x1        x2            x3                x4
             ↓        ↓         ↓             ↓                 ↓
Probability: Pr(x0)=1 Pr(x1|e0) Pr(x2|e0, e1) Pr(x3|e0, e1, e2) Pr(x4|e0, e1, e2, e3)
```

Schematic of Probability Calculation in Causal Language Modeling

An auto-regressive language model is calculating the probability of the three-token sequence `x_0, x_1, x_2`. Arrange the following probability calculations in the order they would be performed by the model.

Based on the principles of sequential probability calculation in an auto-regressive model, what is the fundamental error in the model's observed behavior described in the case study below? Explain why this approach violates the core assumption of this type of model.

Debugging a Language Model's Probability Calculation

An auto-regressive model processes a sequence of four tokens: `token_0, token_1, token_2, token_3`. The model calculates the probability of each token based on the numerical representations (embeddings) of all preceding tokens. Which of the following expressions correctly represents how the model would calculate the probability of `token_2`?

Learn Before

Related