Google

The notation $$\left\{ \mathbf{k}_{i-n_c+1}, \dots, \mathbf{k}_i \right\}$$ represents a set of key vectors, denoted by the bold letter $$\mathbf{k}$$. This set comprises all key vectors within a specific context window, starting from the index $$i-n_c+1$$ and ending at the index $$i$$. The size of this context window is determined by $$n_c$$.

Context Window of Key Vectors Notation

A model is processing a sequence and is currently at position `i=10`. It is using a context window of size `n_c=7`. According to the notation \{ \mathbf{k}_{i-n_c+1}, \dots, \mathbf{k}_i \}, the first key vector in this specific context window would be at index ____.

A language model's attention mechanism uses the notation \{ \mathbf{k}_{i-n_c+1}, \dots, \mathbf{k}_i \} to represent the set of key vectors in its context window. If, at a particular step, the model is attending to the specific set \{ \mathbf{k}_{21}, \mathbf{k}_{22}, \mathbf{k}_{23}, \mathbf{k}_{24}, \mathbf{k}_{25} \}, what are the numerical values for the current position `i` and the context window size `n_c`? Explain your reasoning.

Determining Context Window Parameters

An attention mechanism is processing a sequence of inputs. At the current position, indexed as `i=8`, the model needs to attend to a context window of key vectors from the last `n_c=5` positions (including the current position). Which of the following notations correctly represents this specific set of key vectors?

Learn Before

Related