Explain the sequence length extrapolation capability of fixed sinusoidal positional encodings in the Transformer architecture. In your response, define what this capability entails and explain why it served as the deciding factor in selecting sinusoidal encodings over learned positional embeddings.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Sinusoidal and Learned Positional Encodings - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor