Google

The cumulative average of the keys and values for a memory component ($$\mathrm{Mem}$$) is formulated as a tuple containing the cumulative average of key vectors ($$\mathbf{k}_j$$) and value vectors ($$\mathbf{v}_j$$) up to a given index $$i$$. This is achieved by summing all vectors from the start of the sequence (index $$j=0$$) to the current index $$i$$, and then normalizing by the total number of elements, $$i+1$$. The mathematical representation is: $$\mathrm{Mem} = \Big( \frac{\sum_{j=0}^{i} \mathbf{k}_j}{i+1}, \frac{\sum_{j=0}^{i} \mathbf{v}_j}{i+1} \Big)$$

Formula for Memory as a Cumulative Average of Keys and Values

A language model is generating text for a travel blog. Despite having a mechanism that penalizes the repeated use of specific words, it produces the following output: 'The journey was incredible. The trip was amazing. The expedition was wonderful.' Analyze why the existing penalty mechanism is failing and propose a more effective approach to mitigate this specific type of repetition, explaining the underlying principle of your proposed solution.

Diagnosing Semantic Repetition

A memory mechanism updates its state after each token by maintaining a tuple of the cumulative average of all key vectors and all value vectors seen so far. Given the following sequence of 2-dimensional key (k) and value (v) vectors, calculate the final memory state `Mem` after processing the token at index `i=2`.

- `i=0`: `k_0 = [2, 4]`, `v_0 = [1, 0]`
- `i=1`: `k_1 = [6, 2]`, `v_1 = [3, 2]`
- `i=2`: `k_2 = [1, 0]`, `v_2 = [5, 1]`

Express your answer as a tuple of two vectors, e.g., `([x1, y1], [x2, y2])`.

Calculating a Memory State

The cumulative average for the memory component can be calculated efficiently using a recursive formula. The memory state at the current step, $$\mathrm{Mem}_i$$, is computed based on the current key-value pair $$(\mathbf{k}_i, \mathbf{v}_i)$$ and the memory state from the previous step, $$\mathrm{Mem}_{i-1}$$. The recursive formula is expressed as: $$\mathrm{Mem}_i = \frac{(\mathbf{k}_i, \mathbf{v}_i) + i \cdot \mathrm{Mem}_{i-1}}{i + 1}$$ This approach prevents the need to recalculate the sum of all historical key-value pairs at each step, significantly increasing computational efficiency.

Recursive Formula for Memory as a Cumulative Average

A memory mechanism calculates its final state by taking the cumulative average of all key vectors and all value vectors in a sequence of 100 tokens. How does the influence of the key vector from the first token (at index 0) on the final memory state compare to the influence of the key vector from the last token (at index 99)?

Learn Before

Related