Google

The Kerple method for positional bias can be implemented using a logarithmic function to penalize attention based on token distance. For a query at position $$i$$ and a key at position $$j$$, the bias is calculated with the formula: $$\mathrm{PE}(i,j) = -\beta_1 \log(1 + \beta_2(i - j))$$. Here, $$\beta_1$$ and $$\beta_2$$ are hyperparameters controlling the scale and shape of the logarithmic penalty.

Kerple Logarithmic Bias Formula

A positional bias mechanism uses the formula `Bias = -βlog(1 + β)` to penalize attention between tokens, where `β` is a positive value that increases with the distance between the tokens. Describe how the *rate of increase* of this penalty changes as the distance between tokens grows larger. In other words, is the additional penalty for one extra unit of distance greater for nearby tokens or for very distant tokens? Justify your answer based on the properties of the function.

Analysis of Positional Penalty Growth

Based on the team's observations, would implementing a positional bias using the formula `Bias = -βlog(1 + β)` (where `β` is a positive value that increases with token distance) be a suitable strategy? Justify your decision by explaining how the properties of this formula relate to the specific problem the team is facing.

Selecting a Positional Bias Strategy

A model designer implements a positional bias using the formula `Bias = -βlog(1 + β)`, where `β` is a positive value that increases with token distance. The goal is to penalize attention to more distant tokens. By mistake, the designer forgets the leading negative sign, implementing the formula as `Bias = βlog(1 + β)`. What is the most likely effect of this error on the model's behavior?

Learn Before

Related