Google

In a causal multi-head attention mechanism, the output for a single head `j` at a specific token position `i` is computed using the standard Query-Key-Value (QKV) attention function. This calculation is restricted to the current and preceding tokens to maintain the autoregressive property. The formula is: $$\text{head}_j = \text{Att}_{\text{qkv}}(\mathbf{q}_i^{[j]}, \mathbf{K}_{\leq i}^{[j]}, \mathbf{V}_{\leq i}^{[j]})$$ Here, $$\mathbf{q}_i^{[j]}$$ is the query vector for the i-th token projected for head `j`, while $$\mathbf{K}_{\leq i}^{[j]}$$ and $$\mathbf{V}_{\leq i}^{[j]}$$ are the key and value matrices for head `j`, containing information from tokens 0 up to `i`.

Causal Attention Output for a Single Head and Token

True or False: In a causal attention mechanism, when a single attention head is calculating the output for the 4th token in a sequence, the query vector for that 4th token (q_4) will interact with the key vector from the 6th token (k_6) to compute an attention score.

An autoregressive language model is processing a sequence of tokens. To compute the output for the 5th token (at index 4), what are the specific inputs (Query, Key, and Value) that the 3rd attention head (head 2) will use in its attention function? Describe each input in terms of the token indices and the head index.

Causal Attention Inputs

An autoregressive language model is in the process of generating a sequence of tokens. When a single attention head calculates its output for the 4th token in the sequence, which set of key and value vectors does it use to ensure it only relies on previously generated information?

Learn Before

Related