Google

In the ALiBi framework, the positional bias term is defined as the offset between the query and key positions, which is then multiplied by a negative scalar. This method effectively imposes a linear penalty on the attention score that increases with the distance between tokens.

ALiBi Bias Term Definition

In Attention with Linear Biases (ALiBi), the positional bias term is calculated as the negative scaled difference between the query position and the key position. Specifically, for a query at position $$i$$ and a key at position $$j$$, the positional embedding bias $$\mathrm{PE}(i,j)$$ is defined by the equation $$\mathrm{PE}(i,j) = -\beta \cdot (i - j)$$. This can be equivalently formulated by distributing the negative sign to yield $$\mathrm{PE}(i,j) = \beta \cdot (j - i)$$, where $$\beta$$ is a scaling factor.

ALiBi Bias Term Formula

In a self-attention mechanism that incorporates a linear bias based on token distance, the bias term added to the attention score is a positive value that decreases as the distance between the query and key increases.

A self-attention mechanism is modified to include a bias term. This term is calculated by taking the distance between a query token and a key token and multiplying it by a fixed negative number. Explain how this modification influences the attention scores for tokens that are close together compared to tokens that are far apart.

Effect of Distance-Based Attention Penalty

Imagine a self-attention mechanism where a modification adds a penalty to the attention score between any two words. This penalty is designed to increase in a straight, consistent line as the distance between the words' positions in the sequence grows. What is the most likely behavioral outcome of this modification?

Learn Before

Related