Theory
Momentum Convergence on a Scalar Quadratic
To analyze the convergence of the momentum method on a scalar quadratic function , the update equations for the position and velocity can be formulated as a coupled system: . The convergence behavior is entirely governed by the eigenvalues of this transition matrix. Analysis of this matrix reveals that the velocity converges when the hyperparameters satisfy . This feasible range is substantially larger than the constraint required for standard gradient descent, confirming that large momentum coefficients () safely permit much larger learning rates without divergence.
0
1
Updated 2026-06-25
Tags
D2L
Dive into Deep Learning @ D2L