Google

In a more general vector-level formulation of multi-head attention (Eq. 11.5.2), the final layer output is obtained by stacking the $$h$$ individual head outputs $$\mathbf{h}_1, \ldots, \mathbf{h}_h$$—each lying in $$\mathbb{R}^{p_v}$$—into a single concatenated vector of dimensionality $$hp_v$$, and then multiplying by a learnable output projection matrix $$\mathbf{W}_o \in \mathbb{R}^{p_o 	imes hp_v}$$:

$$\mathbf{W}_o \begin{bmatrix} \mathbf{h}_1 \ \vdots \ \mathbf{h}_h \end{bmatrix} \in \mathbb{R}^{p_o}$$

Unlike the matrix-level formulation that fixes the output projection to $$\mathbb{R}^{d 	imes d}$$, this parameterization allows the output dimensionality $$p_o$$ to differ from both the input dimensionality and the per-head value dimensionality $$p_v$$, providing additional architectural flexibility.

Multi-Head Attention Output Formula (General Vector Form)

A multi-head attention layer in a model has 8 parallel attention heads. For a single input token, the output from each of these 8 heads is a vector with 64 dimensions. The mechanism's next step is to concatenate these 8 vectors into a single, larger vector. This larger vector is then multiplied by a final weight matrix to produce the layer's final output vector for that token. What is the dimensionality of the single vector that results from the concatenation step, before the final matrix multip

A multi-head attention layer is configured with 12 parallel attention heads. The output of each head is a 64-dimensional vector. After concatenating the outputs from all heads, the resulting vector is multiplied by a final weight matrix to produce the layer's final 768-dimensional output vector. What must be the dimensions of this final weight matrix?

Determining Weight Matrix Dimensions in Multi-Head Attention

After each parallel attention head has computed its individual output vector, what is the correct sequence of operations to produce the final output of the multi-head attention layer?

Learn Before

Related