Google

A set representing a sequence of Key-Value pairs, indexed from 1 to $\tau$. Each element in the set is a tuple $(\mathbf{K}_{\le i}^{[t]}, \mathbf{V}_{\le i}^{[t]})$, where $t$ is the index (e.g., representing a layer or head). The bold letters $\mathbf{K}$ and $\mathbf{V}$ typically denote Key and Value matrices or vectors, and the subscript ${\le i}$ indicates that these keys and values are for all positions up to and including $i$ in a sequence. The full expression is given by: $\{ (\mathbf{K}_{\le i}^{[1]}, \mathbf{V}_{\le i}^{[1]}), \dots, (\mathbf{K}_{\le i}^{[\tau]}, \mathbf{V}_{\le i}^{[\tau]}) \}$.

Set of Indexed Key-Value Pairs

This formula calculates the output for a single attention head, `head_j`, in a transformer model that implements Grouped-Query Attention (GQA) with causal masking. The formula is:

$$\text{head}_j = \text{Att}_{\text{qkv}}(\mathbf{q}_i^{[j]}, \mathbf{K}_{\le i}^{[g(j)]}, \mathbf{V}_{\le i}^{[g(j)]})$$

In this equation, $$\text{Att}_{\text{qkv}}$$ represents the attention function, and $$\mathbf{q}_i^{[j]}$$ is the query vector for the current token `i` and head `j`. The keys $$\mathbf{K}$$ and values $$\mathbf{V}$$ are shared among a group of query heads, as determined by the function $$g(j)$$. The subscript $${\le i}$$ signifies that the attention is causal, meaning it only considers tokens up to the current position `i`.

Attention Head Output with Grouped Queries and Causal Masking

The output computation for a specific attention head $$j$$ in a Grouped-Query Attention (GQA) model depends on its assigned key-value group. If $$g(j)$$ represents the group ID for the $$j$$-th head, the head's output is calculated using the formula: $$\mathrm{head}_j = \mathrm{Att}_{\mathrm{qkv}}(\mathbf{q}_{i}^{[j]}, \mathbf{K}_{\le i}^{[g(j)]}, \mathbf{V}_{\le i}^{[g(j)]})$$ In this expression, the unique query vector $$\mathbf{q}_{i}^{[j]}$$ for the current token attends to the keys $$\mathbf{K}_{\le i}^{[g(j)]}$$ and values $$\mathbf{V}_{\le i}^{[g(j)]}$$ that are shared within its respective group $$g(j)$$.

Attention Head Output in Grouped-Query Attention (GQA)

A computational system processes the sentence 'The quick brown fox' one word at a time. At the third word ('brown'), the system maintains a collection of data structured as follows:

$ \{ (\mathbf{K}_{\le i}^{[1]}, \mathbf{V}_{\le i}^{[1]}), \dots, (\mathbf{K}_{\le i}^{[	au]}, \mathbf{V}_{\le i}^{[	au]}) \} $

In this specific instance, the sequence position is `i=3` and the total number of pairs in the set is `τ=4`.

Describe what the specific element $(\mathbf{K}_{\le 3}^{[2]}, \mathbf{V}_{\le 3}^{[2]})$ represents within this collection. Your explanation must address the meaning of both the subscript `≤ 3` and the superscript `[2]`.

Interpreting a Set of Indexed Key-Value Pairs

Based on the notation \{ (\mathbf{K}_{\le i}^{[1]}, \mathbf{V}_{\le i}^{[1]}), \dots, (\mathbf{K}_{\le i}^{[	au]}, \mathbf{V}_{\le i}^{[	au]}) \}, which describes a collection of key-value pairs, analyze the provided scenario. What are the specific values for the indices `i` and `τ`, and what does the complete set represent at this moment?

State of Key-Value Cache During Generation

A computational model processes sequences and, at a specific step `i`, maintains a collection of data represented as: 

$ \{ (\mathbf{K}_{\le i}^{[1]}, \mathbf{V}_{\le i}^{[1]}), \dots, (\mathbf{K}_{\le i}^{[	au]}, \mathbf{V}_{\le i}^{[	au]}) \} $ 

In this set, each $(\mathbf{K}, \mathbf{V})$ is a pair of matrices, the subscript ${\le i}$ indicates that the matrices contain information for all sequence positions from the start up to position `i`, and the superscript `[t]` is an index ranging fr

Learn Before

Related