Sparse Attention

Self-attention layers' first approach

In a general attention mechanism, the output is calculated as a weighted sum of the Value vectors, where the weights are determined by the interaction between Query and Key vectors. The standard formula is: $Att(\textbf{Q}, \textbf{K}, \textbf{V}) = \alpha(\textbf{Q}, \textbf{K})\textbf{V}$. Consider a scenario where this formula is mistakenly altered to be: $Att_{modified} = \alpha(\textbf{Q}, \textbf{K})\textbf{K}$. What is the most significant consequence of this modification?

Dimensional Analysis of the Attention Formula

Applying the Attention Mechanism Roles

Self-Attention Output Formula

Attention Weight Matrix ($\alpha$)

In a self-attention mechanism, the raw attention scores (β) for a single query vector with respect to three key vectors are calculated as [2.0, 1.0, 0.5]. To convert these scores into a probability distribution, a normalization function is applied. What is the resulting normalized attention weight (α) corresponding to the first key vector (score of 2.0)?

In a self-attention mechanism, a set of raw, unnormalized attention scores for a specific query are [1.5, 0.5, -1.0]. If a constant value of 10 is added to each of these scores, resulting in a new set of scores [11.5, 10.5, 9.0], how will the final normalized attention weights (the probability distribution) calculated from the new scores compare to the weights calculated from the original scores?

Calculating and Interpreting Attention Weights

Computing Attention Weights in Sequence Parallelism

Distributed Attention Weight Formula