Google

In a Transformer language model comprising $$L$$ stacked blocks, the probability distribution for the next token is generated by applying a Softmax layer to the output of the final block. This involves multiplying the final block's output, $$\mathbf{H}^{L}$$, by a parameter weight matrix, $$\mathbf{W}^{o} \in \mathbb{R}^{d \times |V|}$$. This operation produces a sequence of probability distributions over the vocabulary, representing the conditional probability of the next token given the preceding sequence: $$\begin{bmatrix} \Pr(\cdot | x_0,...,x_{m-1}) \\ \vdots \\ \Pr(\cdot | x_0,x_{1}) \\ \Pr(\cdot | x_0) \end{bmatrix} = \mathrm{Softmax}(\mathbf{H}^{L} \mathbf{W}^{o})$$.

Output Probability Calculation in Transformer Language Models

Based on the case study, describe the two essential, sequential operations that must be applied to the model's final hidden state matrix to convert it into the desired set of probability distributions over the vocabulary. For each operation, specify its purpose and the resulting shape of the data.

Diagnosing a Language Model's Output Layer

A decoder-only language model has just computed the final hidden state vector for the last token in an input sequence. Describe, in detail, the sequence of two critical operations that are applied to this vector to generate a probability distribution over the entire vocabulary for predicting the *next* token. For each operation, identify any associated parameter matrices and explain their purpose and dimensional relationship to the model's hidden size and vocabulary size.

From Hidden State to Probability Distribution

A decoder-only language model has an internal hidden dimension of 768 and a vocabulary of 30,000 unique tokens. After processing an input sequence, the model's final layer of hidden states is multiplied by a weight matrix to produce logits, which are then passed to a final activation function. What must be the dimensions of this weight matrix and what is its primary role in this process?

Learn Before

Related