Google

During inference, the space complexity of the Key-Value (KV) cache is directly proportional to the number of tokens for which keys and values are stored. This relationship is captured by the formula $$O(L \cdot \tau \cdot d_h \cdot m)$$, where $$L$$ is the number of layers, $$\tau$$ is the number of attention heads, $$d_h$$ is the head dimension, and $$m$$ is the number of tokens being cached.

Space Complexity of the KV Cache

The space complexity of the standard Key-Value (KV) cache, which grows linearly with the number of tokens $$m$$ as $$O(L \cdot \tau \cdot d_h \cdot m)$$, can be reduced by caching fewer tokens. For instance, sliding window attention utilizes a fixed-size window $$m_w$$ to store keys and values only for the local context. This restricts the caching mechanism's space complexity to a constant $$O(L \cdot \tau \cdot d_h \cdot m_w)$$, making it more manageable regardless of the overall sequence length.

Reducing KV Cache Complexity via Windowed Caching

Based on the provided model specifications, determine which model will require a larger key-value cache to process an input sequence of 1000 tokens and justify your reasoning.

KV Cache Memory Footprint Comparison

An autoregressive model uses a mechanism to store key and value vectors for previously processed tokens to speed up inference. For a sequence of 100 tokens, this storage mechanism consumes 200 MB of memory. Assuming all other model parameters remain constant, how much memory would this mechanism consume for a sequence of 400 tokens? Explain your reasoning.

Calculating Memory Growth for Token Caching

The memory footprint of the Key-Value (KV) cache can be decreased not only by reducing the number of tokens cached (represented by the sequence length, $$m$$) but also along other architectural dimensions. A widely adopted approach to achieve this is by enabling the sharing of keys and values across the various attention heads within a multi-head self-attention mechanism.

Reducing KV Cache Complexity via Head Sharing

The memory footprint of the Key-Value (KV) cache for a specific context window size can be quantified. The total size is proportional to the product of four key parameters: the number of layers in the model ($$L$$), the number of attention heads per layer ($$\tau$$), the dimensionality of each head's key/value vectors ($$d_h$$), and the size of the context window ($$m_w$$). The overall memory complexity is therefore given by the formula: $$O(L \cdot \tau \cdot d_h \cdot m_w)$$.

Formula for KV Cache Memory Size

An engineer is deploying a large autoregressive model for a chatbot. They observe that as a conversation with a user gets longer, the model's memory consumption increases steadily, eventually leading to performance issues. This is because the model stores key and value vectors for every token in the conversation history to speed up the generation of the next token. Based on this mechanism, what is the fundamental relationship between the length of the conversation history (in tokens) and the amo

Learn Before

Related