Google

In autoregressive models, a new set of vectors is generated for the current token at position `i'`: a query vector (`q'` or `q_{i'}`), a key vector (`k'` or `k_{i'}`), and a value vector (`v'` or `v_{i'}`). The new query `q'` interacts with all previous key vectors to compute attention scores. The new key `k'` and value `v'` are then appended to the Key-Value cache, making them available for subsequent tokens.

Notation for Current Query, Key, and Value Vectors (`q'`, `k'`, `v'`)

Based on the scenario below, explain the specific roles of the newly created `q'` and `k'` vectors in the model's process of determining the *next* token.

Applying Vector Notation in Autoregressive Generation

During a single step of autoregressive generation for a new token, the newly computed value vector (`v'`) is immediately used to calculate the output for that same token.

The notation $$\mathbf{v}_{i'}$$ represents a new value vector associated with the index $$i'$$. In the context of transformer models during inference, this vector is generated for a new token. It is then paired with its corresponding new key vector, $$\mathbf{k}_{i'}$$, and this pair is added to the Key-Value cache. The bold font for $$\mathbf{v}$$ signifies that it is a vector quantity.

Notation for a New Value Vector (v_i')

During a single step of autoregressive generation in a transformer-based model, a new query vector (`q'`) and a new key vector (`k'`) are computed for the token being generated. What is the immediate role of these two new vectors in the self-attention mechanism?

Learn Before

Related