Learn Before
Connecting all pairs of input and output positions using a stack of convolutional layers with contiguous kernels of width k requires a maximum path length of ___.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Complexity and Path Lengths in Self-Attention - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Match each layer architecture to its corresponding maximum path length between arbitrary input and output positions.
Order these layer architectures from shortest to longest maximum path length between arbitrary input and output positions (assuming kernel width 1 < k < n):
Based on path length analysis, explain how adopting restricted self-attention changes the maximum path length compared to full self-attention, and describe the resulting effect on gradient flow and long-range dependency learning.