Google

The Feed-Forward Network (FFN) sub-layers within Transformer models feature a hidden layer with a specific size denoted as $$d_{\textrm{ffn}}$$. This dimension is typically designed to be larger than the standard hidden size, $$d$$. A common architectural setup sets $$d_{\textrm{ffn}} = 4d$$. For more recent, larger-scale Transformers, $$d_{\textrm{ffn}}$$ can be assigned to an even larger value to boost capacity.

FFN Hidden Size in Transformers

Evaluate the two strategies described in the case study. Which one is more likely to preserve the model's performance, and why? Justify your answer based on the role of the feed-forward network's intermediate layer.

Optimizing Transformer Model Size

A model's architecture includes a feed-forward sub-layer composed of two linear transformations with an intermediate expansion. The model's main hidden size, $d$, is 512. The intermediate layer's size, $d_{ffn}$, is initially set to 2048. If an engineer increases this intermediate size to 3072 to improve model capacity, how many additional parameters are introduced into this specific sub-layer? (You can ignore bias terms in your calculation).

Calculating Parameter Impact of FFN Expansion

A team of engineers is designing a large neural network for a complex language task. Within each block of their model, they use a sub-network composed of two linear transformations with a non-linearity in between. They are debating whether to make the dimensionality of the intermediate layer in this sub-network significantly larger (e.g., four times larger) than the model's primary embedding and hidden state dimension. What is the primary trade-off they must consider when making this decision?

Learn Before

Related