University of Michigan - Ann Arbor

On CIFAR-10, the standard deviations of layer responses are generally smaller in ResNets than in corresponding plain networks. In a residual block, the learned branch represents $$\mathcal{F}(\mathbf{x}) := \mathcal{H}(\mathbf{x}) - \mathbf{x}$$. If the desired mapping $$\mathcal{H}$$ is close to the identity mapping, then the residual mapping $$\mathcal{F}$$ is close to zero: the identity shortcut carries $$\mathbf{x}$$ directly while the residual branch learns a perturbation. The smaller measured responses are therefore consistent with, but do not by themselves prove, the identity-mapping motivation for residual learning.

Smaller ResNet Layer Responses and the Identity-Mapping Motivation

According to the foundational premise of residual learning, why do the residual function responses stay close to zero and produce smaller response standard deviations compared to the plain network?

Learn Before

Related