Google

The notation $$\mathbf{v}_{i'}$$ represents a new value vector associated with the index $$i'$$. In the context of transformer models during inference, this vector is generated for a new token. It is then paired with its corresponding new key vector, $$\mathbf{k}_{i'}$$, and this pair is added to the Key-Value cache. The bold font for $$\mathbf{v}$$ signifies that it is a vector quantity.

Notation for a New Value Vector (v_i')

In the context of a transformer model generating text one token at a time, a new vector represented by the notation Vi' is created for the most recently generated token. Briefly explain what this vector represents and what happens to it immediately after it is computed.

Role of the New Value Vector in Inference

In a transformer model generating text token-by-token, the newly computed value vector for the current position *i'*, denoted as Vi', is immediately weighted by attention scores to produce the output for that same position *i'*.

During the process of generating text one token at a time, a model computes a new vector denoted as Vi' for the most recent token. To efficiently generate the next token, what is the primary, immediate purpose of this Vi' vector?

Learn Before

Related