Google

The expressive power of Transformer networks can be effectively enhanced by increasing the model depth, denoted by $$L$$, which represents the total number of stacked processing layers. In standard BERT architectures, the depth $$L$$ is typically configured to either 12 or 24. However, employing networks with even greater depth is a viable strategy to achieve further performance enhancements.

Model Depth in Transformers

In a BERT model, input tokens are initially represented as embeddings—calculated as the sum of their corresponding token, positional, and segment embeddings. This combined embedding sequence is then processed by the core architecture, which is a deep, multi-layer Transformer network formed by stacking numerous Transformer layers. Each layer in this stack is composed of a self-attention sub-layer and a feed-forward network (FFN) sub-layer, both of which utilize a post-norm architecture. In this structure, the output is calculated as $$\mathrm{output} = \mathrm{LNorm}(F(\mathrm{input}) + \mathrm{input})$$, where $$F(\cdot)$$ represents the sub-layer's main function (self-attention or FFN) and $$\mathrm{LNorm}(\cdot)$$ is layer normalization. The final output produced by the network's last Transformer layer is a sequence of real-valued vectors, with one vector corresponding to each position in the input sequence.

BERT's Core Architecture

In a Transformer language model comprising $$L$$ stacked blocks, the probability distribution for the next token is generated by applying a Softmax layer to the output of the final block. This involves multiplying the final block's output, $$\mathbf{H}^{L}$$, by a parameter weight matrix, $$\mathbf{W}^{o} \in \mathbb{R}^{d \times |V|}$$. This operation produces a sequence of probability distributions over the vocabulary, representing the conditional probability of the next token given the preceding sequence: $$\begin{bmatrix} \Pr(\cdot | x_0,...,x_{m-1}) \\ \vdots \\ \Pr(\cdot | x_0,x_{1}) \\ \Pr(\cdot | x_0) \end{bmatrix} = \mathrm{Softmax}(\mathbf{H}^{L} \mathbf{W}^{o})$$.

Output Probability Calculation in Transformer Language Models

A research team is developing a new language model. They are deciding between a version with 12 layers and a version with 48 layers. Describe one significant advantage and one significant disadvantage of choosing the 48-layer version over the 12-layer version.

Trade-offs of Model Depth

Analyze the observed performance changes as the number of layers increases from 4 to 48. Explain the likely reasons for the initial performance gains and the subsequent plateau and degradation.

Analyzing the Impact of Increasing Model Layers

An AI team is developing solutions for two distinct tasks: Task A, which involves classifying short customer reviews as positive or negative, and Task B, which requires generating concise summaries of long, complex legal documents. They have two available models: Model X with 6 stacked processing layers and Model Y with 24 stacked processing layers. Based on the relationship between model depth and capability, which of the following strategies is most appropriate?

Learn Before

Related