Google

A modified Rotary Positional Embedding (RoPE) function, denoted $$\mathrm{Ro}'$$, can be defined through its equivalence to the original $$\mathrm{Ro}$$ function. Applying the modified function $$\mathrm{Ro}'$$ to a token embedding $$\mathbf{x}_i$$ with position parameters $$i\theta$$ is identical to applying the original function $$\mathrm{Ro}$$ to the same embedding but with a transformed set of position parameters, $$i\theta'$$. This relationship is formally stated as: $$ \mathrm{Ro}'(\mathbf{x}_i, i\theta) = \mathrm{Ro}(\mathbf{x}_i, i\theta') $$

General Equivalence Formula for Modified RoPE

The implementation of linear positional interpolation in Rotary Positional Embeddings (RoPE) modifies the input to the embedding model. The new model, denoted as $$\mathrm{Ro}'$$, scales the position parameters by a factor of $$m_l / m$$, and is given by: $$\mathrm{Ro}'(\mathbf{x}_i, i\theta) = \mathrm{Ro}(\mathbf{x}_i, \frac{m_l}{m} i\theta)$$ where $$m_l$$ is the original sequence length and $$m$$ is the new extended sequence length.

Formula for RoPE with Linear Positional Interpolation

The scaling of Rotary Positional Embeddings (RoPE) can be conceptualized as a transformation of the rotation angle. A scaled rotation function, $$\mathrm{Ro}'$$, applied to an embedding $$\mathbf{x}_i$$ with an original angle $$\theta$$, is equivalent to applying the original rotation function, $$\mathrm{Ro}$$, with a transformed angle $$\theta'$$. This equivalence is captured by the formula: $$\mathrm{Ro}'(\mathbf{x}_i, i\theta) = \mathrm{Ro}(\mathbf{x}_i, i\theta')$$ This principle demonstrates that adapting RoPE for different sequence lengths is achieved by adjusting the rotation angles applied to the embeddings.

RoPE Scaling Transformation Equivalence

An LLM developer is experimenting with a new way to handle positional information. They have two potential implementation strategies:

Strategy A: Create a completely new function, `Ro_new`, which takes a token embedding and a position index. Internally, this function first doubles the position index and then applies the standard rotational transformation.

Strategy B: Use the original, unmodified RoPE function, `Ro`. However, before passing the position index to this function, they preprocess it by doubling its value.

Based on the principle of general equivalence for modified rotary embeddings, are these two strategies functionally identical? Explain your reasoning.

Equivalence of RoPE Modification Strategies

A key principle for modifying rotary positional embeddings states that the effect of a new function, `Ro'`, can be understood as applying the original function, `Ro`, with a transformed position parameter, `iθ'`. This is expressed as: $$ \text{Ro}'(\mathbf{x}_i, i\theta) = \text{Ro}(\mathbf{x}_i, i\theta') $$ Based on this principle, analyze the following scenario.

Analysis of a Flawed RoPE Modification

A researcher defines a new rotary position embedding function, `Ro_new`, for a token `x_i` at position `i`. The new function is defined as `Ro_new(x_i, iθ) = Ro(x_i, (i+c)θ)`, where `Ro` is the original function and `c` is a constant offset. According to the general equivalence principle, this can be written as `Ro_new(x_i, iθ) = Ro(x_i, iθ')`. What is the correct expression for the transformed position parameter `iθ'`?

Learn Before

Related