Google

An alternative to using only the current sequence for context is to populate the k-NN datastore with key-value pairs from a larger collection of sequences, such as an entire training dataset. This approach enables a Large Language Model to leverage a more generalized context for making predictions.

Extending k-NN Datastore Context with a Training Dataset

A significant drawback of using an entire collection of sequences to populate a k-NN datastore is the high computational cost. As the number of sequences and corresponding key-value pairs in the datastore increases, the process of searching for nearest neighbors becomes computationally intensive.

Computational Challenge of Large-Scale k-NN Datastores

The datastore for a k-NN Language Model is a collection of key-value tuples. Each tuple links a context representation (key) to its corresponding ground-truth next token (value). A set of such tuples is represented as $\{(\mathbf{Z}_1, w_1), ..., (\mathbf{Z}_k, w_k)\}$. In this structure, each key $\mathbf{Z}_i$ is the final hidden state vector from the LLM's Transformer at a specific position $i$, and the value $w_i$ is the actual token that follows in the sequence. The datastore is populated by processing a large training corpus and collecting these $(\mathbf{Z}_i, w_i)$ pairs for every token position.

Datastore Composition in k-NN Language Models

A legal tech company is building a language model to assist lawyers in drafting contracts. They are using a memory system that stores examples of 'context -> next phrase' to improve generation quality. They are considering two strategies for populating this memory system:

Designing a Memory-Augmented Legal AI

A language model can use an external memory datastore to improve its predictions. One strategy is to populate this datastore with context-to-next-word examples drawn from a massive, general-purpose training corpus. Analyze the primary benefit of this approach compared to using only examples from the current text being generated. Then, identify and explain a significant potential drawback that a system designer would need to consider when implementing this strategy.

Trade-offs in k-NN Datastore Population

Consider two language models that use an external datastore of (context -> next word) examples to help generate text. 

- **Model X** populates its datastore only with examples from the specific document it is currently generating.
- **Model Y**'s datastore is pre-filled with millions of examples from a vast and diverse library of texts before it begins generating any new document.

When asked to complete a sentence about a niche historical fact not mentioned earlier in the current document, whi

Learn Before

Related