Google

The basic form of a Gated Linear Unit (GLU) activation function is expressed by the equation:

$$\sigma_{\mathrm{glu}}(\mathbf{h}) = \sigma(\mathbf{h}\mathbf{W}_1 + \mathbf{b}_1) \odot (\mathbf{W}_2 + \mathbf{b}_2)$$

In this formulation, $$\mathbf{W}_1, \mathbf{W}_2 \in \mathbb{R}^{d \times d}$$ and $$\mathbf{b}_1, \mathbf{b}_2 \in \mathbb{R}^{d}$$ denote the model parameters. The function $$\sigma(\cdot)$$ represents an internal non-linear activation function, where different choices of $$\sigma(\cdot)$$ lead to different versions of GLU functions.

Gated Linear Unit (GLU) Formula

An activation function is defined by the formula: `Output = σ(hW₁ + b₁) ⊙ (hW₂ + b₂)`. Your task is to calculate the final output vector given the specific inputs below. Show the results of your intermediate calculations.

Calculating the Output of a Gated Activation

In the formula for a gated activation, $\sigma(\mathbf{hW}_1 + \mathbf{b}_1) \odot (\mathbf{hW}_2 + \mathbf{b}_2)$, what is the primary reason that the two resulting vectors, one from each parallel path, must have the same dimensions?

The GeGLU (GELU-based Gated Linear Unit) activation function is defined by the following formula: $$\sigma_{\text{geglu}}(\mathbf{h}) = \sigma_{\text{gelu}}(\mathbf{hW}_1 + \mathbf{b}_1) \odot (\mathbf{hW}_2 + \mathbf{b}_2)$$ In this equation, $$\mathbf{h}$$ represents the input, while $$\mathbf{W}_1$$, $$\mathbf{W}_2$$, $$\mathbf{b}_1$$, and $$\mathbf{b}_2$$ are learnable model parameters (weights and biases). The function $$\sigma_{\text{gelu}}$$ is the Gaussian Error Linear Unit (GELU) activation, and $$\odot$$ signifies the element-wise product.

GeGLU (GELU-based Gated Linear Unit) Formula

An activation function is defined by the formula: `Output = σ(Input ⋅ W₁ + b₁) ⊙ (Input ⋅ W₂ + b₂)` where `Input` is a vector, `W₁`, `W₂`, `b₁`, `b₂` are learnable parameters, `σ` is a non-linear function (such as the sigmoid function), and `⊙` denotes the element-wise product. What is the primary functional role of the `σ(Input ⋅ W₁ + b₁)` component in this architecture?

Learn Before

Related