Google

The state of a model at a specific layer, often denoted by a capital letter like $$\mathbf{H}^l$$, is represented as an ordered sequence of vectors. For example, $$\mathbf{H}^l = \mathbf{h}_0^l \mathbf{h}_1^l \dots \mathbf{h}_m^l$$. Each bolded element $$\mathbf{h}_i^l$$ is a vector corresponding to the `i`-th position in the sequence at layer `l`. This collection of vectors encapsulates the full set of representations for that layer.

Representation of a Layer's State as a Sequence of Vectors

The input to a Transformer layer at a given depth $$l$$ is represented as a sequence of hidden states, denoted by $$\mathbf{H}^l = \mathbf{h}_0^l \mathbf{h}_1^l ... \mathbf{h}_m^l$$. In this notation, $$\mathbf{H}^l$$ is the sequence containing all hidden states for the layer.

Input Representation in a Transformer Layer

A computational model is processing the three-word sentence 'cat sat mat'. Using the notation where a layer's state is an ordered sequence of vectors, write out the full representation for the state of this model's 4th layer, which we can denote as $\mathbf{H}^4$.

Representing a Layer's State for a Sequence

A computational model is processing a short sentence. Based on the scenario described below, construct the formal representation for the state of the specified layer.

Applying Layer State Representation

A computational model is processing a sequence of three items (at positions 0, 1, and 2). The model's internal state at its 5th processing stage is represented by the collection of vectors $\mathbf{S}^5 = [\mathbf{v}_0^5, \mathbf{v}_1^5, \mathbf{v}_2^5]$. What does the single vector $\mathbf{v}_1^5$ represent in this context?

Learn Before

Related