Google

The Gaussian Error Linear Unit (GeLU) is a prominent alternative to the ReLU activation function in Large Language Models (LLMs), effectively acting as a smoothed version of it. Instead of gating outputs strictly by the sign of the input, the GeLU function operates by weighting its input using the percentile $$\Pr(h \le \mathbf{h})$$. In this formulation, $$h$$ represents a $$d$$-dimensional vector where each entry is sampled from the standard normal distribution, denoted as $$\mathrm{Gaussian}(0,1)$$, producing a vector of percentiles corresponding to the elements of the input $$\mathbf{h}$$.

Gaussian Error Linear Unit (GELU)

The Gaussian Error Linear Unit (GeLU) function is defined by the formula:

$$\sigma_{\mathrm{gelu}}(\mathbf{h}) = \mathbf{h} \Pr(h \le \mathbf{h}) = \mathbf{h} \Phi(\mathbf{h})$$

Here, $$\mathbf{h}$$ is the input vector, and $$h$$ represents a $$d$$-dimensional vector with entries drawn from the standard normal distribution $$\mathrm{Gaussian}(0,1)$$. The informal notation $$\Pr(h \le \mathbf{h})$$ refers to a vector where each entry represents the percentile for the corresponding entry of $$\mathbf{h}$$, which is mathematically equivalent to applying the cumulative distribution function $$\Phi(\mathbf{h})$$.

GELU (Gaussian Error Linear Unit) Formula

The Gaussian Error Linear Unit (GELU) activation function has been widely adopted in the architecture of several influential Large Language Models. Notable examples of models that utilize GELU include BERT, GPT-3, and BLOOM.

Applications of GELU in Large Language Models

An activation function is defined by its behavior of weighting an input value by that value's corresponding cumulative probability from a standard normal distribution (mean=0, variance=1). Given two inputs, `x = -3` and `y = 3`, which statement best describes their respective outputs, `f(x)` and `f(y)`?

The 2016 paper by Hendrycks and Gimpel is the original source that introduced the Gaussian Error Linear Unit (GELU) activation function. It provides the foundational theory and discusses convenient implementation methods for the function.

Hendrycks and Gimpel [2016] on GELU

Based on the provided case study, explain why the alternative activation function might address the engineer's concern about inactive neurons for negative inputs.

Activation Function Selection for a Language Model

You are reviewing a teammate’s proposed change to your company’s in-house Transformer block used for a customer-support LLM. They want to (a) replace standard LayerNorm with RMSNorm and (b) replace the FFN activation from GELU to SwiGLU. After the change, early training becomes less stable: loss occasionally spikes, and activation magnitudes in the FFN show a persistent positive mean shift across features (measured per token) even though the overall scale seems controlled.

Write an analysis that explains, using the LayerNorm formula and the defining behavior of RMSNorm, how removing mean-centering can allow a non-zero mean to persist even when the vector is rescaled. Then connect that to how the choice of FFN nonlinearity (GELU vs a gated activation like SwiGLU) can interact with this mean shift to amplify or dampen instability. Conclude with one concrete, technically justified adjustment you would recommend (e.g., where to place/parameterize normalization, whether to keep/modify bias terms, or which activation to use) and explain the tradeoff your recommendation makes.

Diagnosing Training Instability When Changing Normalization and FFN Activations

You are leading an LLM platform team that must standardize a Transformer block for multiple internal products. Two candidate designs are on the table for the FFN sublayer and its surrounding normalization:

Design A: Standard LayerNorm applied to the FFN input, using LNorm(h) = α * (h − μ) / (σ + ε) + β, followed by a 2-layer FFN with GELU activation.

Design B: RMSNorm applied to the FFN input (i.e., no mean subtraction; scale by the vector’s root-mean-square magnitude), followed by a gated FFN using SwiGLU: swish(hW1 + b1) ⊙ (hW2 + b2).

In production, you observe a recurring issue: for certain customer domains, the pre-FFN hidden states develop a persistent positive mean shift (most features become biased positive) while their overall magnitude varies widely across requests. You are not allowed to change the optimizer or add extra normalization layers, and you must pick either Design A or Design B as the company standard.

Write an essay that argues which design you would choose and why, explicitly connecting (1) what mean-centering vs non-centering normalization will do to a positively shifted hidden-state distribution, and (2) how the chosen activation (GELU vs SwiGLU’s swish-gated multiplicative form) will interact with that normalized distribution to affect information flow and stability. Your answer should make at least one concrete prediction about how the two designs would differ in behavior on small negative vs small positive feature values after normalization, and how that would matter for downstream model behavior.

Choosing an FFN Activation and Normalization Pair Under Deployment Constraints

You are reviewing a teammate’s change to a Transformer block in an internal LLM. They made two simultaneous edits: (1) replaced standard LayerNorm with RMSNorm (i.e., removed mean subtraction and normalized only by the root-mean-square magnitude), and (2) replaced the FFN activation from GELU to SwiGLU (a gated FFN where one linear branch is passed through a Swish nonlinearity and then multiplied element-wise by a second linear branch). After the change, offline evaluation shows a consistent increase in the average activation mean (positive bias) entering the FFN output projection, and occasional saturation-like behavior in the gate (many near-zero or very large gate values), even though overall activation magnitudes look similar.

Write an analysis that (a) uses the LayerNorm formula to explain what statistical property standard LayerNorm enforces that RMSNorm does not, (b) connects that difference to why the input distribution seen by a smooth activation like GELU versus a multiplicative gate like SwiGLU can change in qualitatively different ways, and (c) proposes one concrete, minimal modification (e.g., to normalization parameters, placement, or FFN parameterization) that would most directly test whether the observed mean shift is the root cause of the gating behavior. Justify your proposal with a clear causal chain rather than general statements.

Explaining a Distribution Shift Caused by Swapping LayerNorm for RMSNorm and GELU for SwiGLU

You are reviewing a production LLM refactor where the team changed two things in the Transformer block: (1) the FFN activation was changed from GELU to SwiGLU, and (2) standard LayerNorm was replaced with RMSNorm. After the change, offline eval shows a consistent degradation on tasks sensitive to subtle token-level biases (e.g., sentiment and toxicity), even though perplexity is nearly unchanged. A quick probe on a representative hidden-state vector h (per token) shows that before the FFN, the mean of features is no longer close to 0, but the overall magnitude (RMS) is similar to before. The team also notes that with SwiGLU, the multiplicative gate sometimes strongly suppresses or amplifies channels depending on the sign and size of its pre-activation.

As the on-call ML engineer, write a short diagnosis that (a) explains how the difference between standard LayerNorm (re-centering + re-scaling) and RMSNorm (re-scaling only) can interact with the gating behavior of SwiGLU versus the smoother, non-gated behavior of GELU to produce a systematic output drift, and (b) proposes one concrete, minimal change (either to normalization parameters/placement or to the FFN activation choice) that would most directly test your hypothesis without reverting the entire refactor.

Root-Cause Analysis of FFN Output Drift After Swapping Normalization and Activation

You are leading an LLM inference optimization effort. After switching the model to 8-bit weight-only quantization, you observe a consistent quality regression that correlates with occasional activation outliers in the feed-forward network (FFN) blocks. You are allowed to change (A) the FFN activation (currently GELU) and/or (B) the normalization used before the FFN (currently standard LayerNorm). You must keep the model’s parameter count roughly the same and you cannot add extra normalization layers.

Standard LayerNorm is: LNorm(h) = α * (h − μ) / (σ + ε) + β, where μ and σ are the mean and standard deviation over features of h.

RMSNorm is: RMSNorm(h) = α * h / (rms(h) + ε) + β, where rms(h) = sqrt((1/d) * Σ_k h_k^2) and it does NOT subtract the mean.

SwiGLU is: swish(hW1 + b1) ⊙ (hW2 + b2), i.e., a gated FFN variant where the gate uses the Swish nonlinearity.

Case study: In a diagnostic run, you log the pre-FFN normalized vectors for a problematic layer and find that many tokens have a large positive mean across features (μ is strongly > 0), while their per-feature spread is moderate. The quantization team reports that the worst outliers appear after the FFN nonlinearity, not before it.

Which single change is the most defensible first experiment to reduce post-activation outliers while preserving model quality, and why? Choose one option and justify it by explicitly linking (i) the mean-subtraction vs no-mean-subtraction behavior of the normalization formula and (ii) the gating/smoothness behavior of GELU vs SwiGLU in how they can amplify or dampen large positive shifts.

Options:
1) Keep LayerNorm, switch GELU → SwiGLU
2) Switch LayerNorm → RMSNorm, keep GELU
3) Switch LayerNorm → RMSNorm and switch GELU → SwiGLU
4) Keep LayerNorm and keep GELU (change nothing)

Selecting a Normalization + FFN Activation Change After Quantization Regressions

You are on an LLM platform team reviewing a regression after a refactor of the Transformer feed-forward network (FFN). The refactor changed two things at once: (1) the FFN activation was changed from GELU to a gated variant (SwiGLU), and (2) the normalization layer was changed from standard LayerNorm to RMSNorm. No other hyperparameters were intentionally changed.

You have the following telemetry collected on the same held-out batch, measured at the input to the FFN (right after normalization) and at the FFN output (right before the residual add):

Before refactor (LayerNorm + GELU):
- Normalized FFN input: per-token feature mean ≈ 0.00, per-token feature std ≈ 1.00
- FFN output: ~48% of elements are negative; output mean ≈ 0.00

After refactor (RMSNorm + SwiGLU):
- Normalized FFN input: per-token feature mean ≈ +0.35, per-token RMS ≈ 1.00
- FFN output: ~8% of elements are negative; output mean ≈ +0.60

Assume RMSNorm is implemented as y = α * h / (rms(h) + ε) + β (no mean subtraction), and standard LayerNorm is y = α * (h − μ) / (σ + ε) + β. Also assume SwiGLU is implemented as swish(hW1 + b1) ⊙ (hW2 + b2).

As the reviewer, write a concise root-cause analysis that explains how the combination of (a) removing mean-centering in normalization and (b) switching from GELU to a gated activation could plausibly produce the observed shift toward positive FFN outputs. Your answer must explicitly connect the normalization formulas to the gating behavior (element-wise product) and explain why the effect is directional (more positive) rather than just a change in scale.

Interpreting Activation/Normalization Interactions from FFN Telemetry

You are reviewing a teammate’s proposed Transforme...

In a transformer feed-forward block, your team is ...

You’re debugging a transformer FFN refactor where ...

You’re reviewing a PR that changes a transformer b...

For a fixed scalar input x, the paper motivates GELU using a Bernoulli mask M with Pr(M=1)=Φ(x). The gate outputs xM: x when M=1 and zero otherwise. Averaging over M gives E_M[xM]=xΦ(x). GELU uses this expected transformation as a deterministic nonlinearity; evaluating it does not require sampling M. The random gate motivates the activation rather than adding a sampling step to its definition. Keep x fixed when taking this expectation.

Source: https://arxiv.org/pdf/1606.08415v5

GELU as a deterministic Bernoulli-gate expectation

An activation function is designed to scale its input value by the probability that a randomly drawn value from a standard normal distribution (mean=0, variance=1) is less than or equal to that input. How does this function's output for a small negative input (e.g., -0.1) compare to the output of a function that simply sets all negative inputs to zero?

Learn Before

Related