During the initial phase defined by warmup_steps, the Transformer schedule modulates the learning rate such that it increases ___ with respect to the step number.
0
1
Tags
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.2 Transformer Training and Evaluation - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Transformer Training, Regularization, and Optimization - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Match each optimizer hyperparameter or schedule parameter to its specific value used in the reported Transformer experiments.
Order the trajectory of the learning rate across training steps from earliest to latest.
Explain why Model B has a lower learning rate than Model A throughout training according to the scheduling formula, and calculate the exact ratio of Model B's learning rate to Model A's learning rate (lrate_B / lrate_A).