Learn Before
Visual Representation of T5 Bias Application (nb=3, distmax=5)
This image visualizes the application of T5 relative position bias in a causal self-attention setting, using hyperparameters and . The grid shows the matrix of query-key dot products (), where the lower triangular shape enforces causality, meaning a query at position attends only to keys at positions . The bucketing rules for these hyperparameters are as follows: relative position offsets of 0 and 1 are mapped directly to buckets 0 () and 1 (); offsets 2 and 3 are grouped into bucket 2 (); and all offsets of 4 and greater are consolidated into bucket 3 (). A shared bias parameter from the corresponding bucket is added to each dot product to compute the final attention score.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Unified Formula for T5 Bias Bucketing
Example of T5 Bias Bucketing
A model designer is implementing a mechanism to account for the relative distance between tokens in a sequence. The proposed strategy uses a unique, learnable value for each of the first few relative distances (e.g., 1, 2, 3...), but then groups larger distances into a smaller set of shared values, with the size of these groups increasing as the distance grows. What is the primary trade-off this combined approach is designed to optimize?
Analysis of a Hybrid Positional Bucketing System
Formula for Applying T5 Relative Position Bias
Generalization Advantage of T5 Positional Bias
A model uses a hybrid strategy to handle relative positional distances between tokens, assigning each distance to one of a limited number of 'buckets'. The rules are:
- For small distances (e.g., 0-15), each distance is assigned to its own unique bucket.
- For medium distances, the ranges of distances assigned to a single bucket grow progressively larger as the distance increases.
- For very large distances (e.g., beyond 512), all are assigned to a single, final bucket.
Based on this system,
Learn After
In a causal self-attention mechanism, a relative position bias is added to the dot product of each query-key pair. The bias is determined by bucketing the relative position offset, which is calculated as (query index - key index). Given the following bucketing rules:
- Offset 0 → Bucket 0
- Offset 1 → Bucket 1
- Offsets 2 or 3 → Bucket 2
- Offsets 4 or greater → Bucket 3
Match each query-key pair below to the correct bias bucket that would be applied.
A causal self-attention mechanism uses a relative position bias. The bias is determined by bucketing the relative position offset (query index - key index) according to these rules:
- Offset 0 → Bucket 0
- Offset 1 → Bucket 1
- Offsets 2 or 3 → Bucket 2
- Offsets 4 or greater → Bucket 3
The following grid shows the calculated bias bucket index for each query-key pair in a sequence. One of the bucket indices is incorrect. Identify the query-key pair with the incorrectly calculated bias bucket.
In a causal self-attention mechanism, a relative position bias is added to the dot product of each query-key pair. The bias is determined by bucketing the relative position offset (query index - key index) according to these rules:
- Offset 0 → Bucket 0
- Offset 1 → Bucket 1
- Offsets 2 or 3 → Bucket 2
- Offsets 4 or greater → Bucket 3
Which of the following statements accurately describes a structural property of the resulting bias matrix?