Google

In a causal self-attention mechanism, a query at a given position `i` can only interact with keys at positions `j` where $$j \le i$$. For a sequence of length 7 (indexed 0 to 6), this results in the following explicit set of query-key dot products being computed:
- $$q_0$$: $$q_0 k_0^T$$
- $$q_1$$: $$q_1 k_0^T$$, $$q_1 k_1^T$$
- $$q_2$$: $$q_2 k_0^T$$, $$q_2 k_1^T$$, $$q_2 k_2^T$$
- $$q_3$$: $$q_3 k_0^T$$, $$q_3 k_1^T$$, $$q_3 k_2^T$$, $$q_3 k_3^T$$
- $$q_4$$: $$q_4 k_0^T$$, $$q_4 k_1^T$$, $$q_4 k_2^T$$, $$q_4 k_3^T$$, $$q_4 k_4^T$$
- $$q_5$$: $$q_5 k_0^T$$, $$q_5 k_1^T$$, $$q_5 k_2^T$$, $$q_5 k_3^T$$, $$q_5 k_4^T$$, $$q_5 k_5^T$$
- $$q_6$$: $$q_6 k_0^T$$, $$q_6 k_1^T$$, $$q_6 k_2^T$$, $$q_6 k_3^T$$, $$q_6 k_4^T$$, $$q_6 k_5^T$$, $$q_6 k_6^T$$

Explicit Enumeration of Causal Self-Attention Dot Products

In a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`, how many total query-key dot product computations are performed for an input sequence of length 5 (indexed 0 to 4)?

In a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`, list all the query-key dot products that are computed for the query vector at position 4 (`q_4`) in a sequence of length 8 (indexed 0 to 7).

Causal Self-Attention Dot Product Enumeration

For a sequence of 10 tokens (indexed 0 to 9), which of the following query-key dot product computations would be invalid in a self-attention mechanism where a query at a given position `i` can only interact with keys at positions `j` where `j <= i`?

Learn Before

Related