University of Michigan - Ann Arbor

A Transformer position-wise feed-forward network applies the same two-layer transformation independently at every token position within a layer, while its parameters $$W_1, b_1, W_2, b_2$$ differ across layers. This position-wise transformation can be interpreted as two consecutive convolutions with kernel size $$1$$. Combining self-attention with a point-wise feed-forward layer has the same computational complexity as a separable convolution whose kernel size $$k$$ equals the sequence length $$n$$. In the encoder and decoder, the feed-forward sub-layer is wrapped by a residual connection and layer normalization, producing $$\operatorname{LayerNorm}(x + \operatorname{Sublayer}(x))$$.

Layer-Specific Parameters and Convolutional Interpretation of Transformer FFNs

Evaluate the engineer's implementation against the standard design of the position-wise feed-forward network. Identify the two errors in how parameters are distributed across token positions and layers, and explain how the weights must actually be configured.

Learn Before

Related