Google

The notation $$\mathbf{V}_{[i-n_c+1,i]}$$ represents a matrix created by vertically stacking a sequence of value vectors within a sliding window. This matrix contains the value vectors from index $$i-n_c+1$$ to the current index $$i$$, capturing the $$n_c$$ most recent values. It is formally defined as: $$\mathbf{V}_{[i-n_c+1,i]} = \begin{bmatrix} \mathbf{v}_{i-n_c+1} \\ \vdots \\ \mathbf{v}_i \end{bmatrix}$$

Value Matrix from a Sliding Window

To construct the memory component, denoted as $$\mathrm{Mem}$$, in a fixed-size window attention mechanism, a continuous subset of the most recent key and value vectors is extracted. Specifically, this slice spans from index $$i - n_c + 1$$ up to the current processing step $$i$$, with $$n_c$$ defining the capacity of the context window. The relationship is mathematically expressed as: $$\mathrm{Mem} = (\mathbf{K}_{[i - n_c + 1, i]}, \mathbf{V}_{[i - n_c + 1, i]})$$.

Formula for Fixed-Size Window Memory

A system is processing a long sequence of value vectors, denoted as $\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3, ...$. At a certain point in time, it constructs the matrix shown below by stacking the most recent vectors within a sliding window. Based on the structure of this matrix, determine the current processing index (`i`) and the window size (`n_c`) that were used.

Inferring Window Parameters

Imagine you have a long sequence of 128-dimensional value vectors. You are processing this sequence using a sliding window that considers the 16 most recent vectors at any given time. What will be the dimensions (rows × columns) of the matrix formed by vertically stacking the value vectors contained within this window?

Determining Matrix Dimensions from a Sliding Window

Consider a sequence of 2-dimensional value vectors, where:

*   **v**_1 = [10, 11]
*   **v**_2 = [12, 13]
*   **v**_3 = [14, 15]
*   **v**_4 = [16, 17]
*   **v**_5 = [18, 19]

Given a current processing index of *i* = 5 and a context window size of *n*<sub>c</sub> = 3, which matrix below correctly represents the structure formed by vertically stacking the value vectors from the corresponding sliding window, from index *i* - *n*<sub>c</sub> + 1 to *i*?

Learn Before

Related