Google

During the decoding phase of autoregressive generation, the Key-Value (KV) cache state corresponds to the sequence of tokens generated so far. The model uses this cache to predict the next token. This sequential process, where each prefix and its associated cache are used to generate the subsequent token, can be represented as: $$ x_0 \quad (\text{predicts } x_1) \Rightarrow \text{cache}_1 $$ $$ x_0x_1 \quad (\text{predicts } x_2) \Rightarrow \text{cache}_2 $$ $$ \dots $$ $$ x_0x_1\dots x_{m-1} \quad (\text{predicts } x_m) \Rightarrow \text{cache}_m $$ Here, $$\text{cache}_m$$ is the KV cache state containing the key-value pairs for the prefix $$x_0x_1\dots x_{m-1}$$.

Formula for Cache State Evolution during Autoregressive Decoding

An autoregressive language model is given a starting token and generates a three-token sequence: 'A', 'B', 'C'. Arrange the following states of the key-value (KV) cache in the chronological order they occur during this generation process.

Based on the provided scenario, what sequence of tokens do the key-value pairs currently stored in the KV cache correspond to? Explain your reasoning.

KV Cache State during Generation

An autoregressive language model is generating a sequence of tokens one by one. As the length of the generated sequence increases from 10 tokens to 100 tokens, what is the primary impact of the evolving key-value cache on the computation required to generate the *next* token?

Learn Before

Related