Google

For a concatenated sequence `[x, y]`, the initial input representation for the Transformer stack is generated on a per-token basis. For each position `i'` in the sequence, the token is converted into an embedding vector. This vector becomes the initial representation for that specific position and is then fed into the first Transformer layer.

Initial Representation for Concatenated [x, y] Sequences

Consider two token sequences, `x` and `y`. A language model calculates two separate quantities: one by processing the sequence `x` followed by `y` (concatenated as `[x, y]`), and another by processing the sequence `y` by itself. Explain why the initial input vector for the first token of `y` will be different in these two scenarios. Your explanation should identify the specific component of the input vector that changes and the reason for this change.

Positional Context in Concatenated Sequences

Analyze the following proposed method for generating the initial input representation for a combined token sequence `[x, y]`. Is this method correct? Justify your answer by explaining the fundamental principle of how representations are generated for concatenated sequences.

Evaluating a Representation Generation Method

A language model is given two token sequences: sequence `x` with 10 tokens (at positions 0 through 9) and sequence `y` with 5 tokens. To process them together, they are concatenated into a single sequence `[x, y]`. How is the initial input vector for the very first token of the original sequence `y` calculated before being passed to the first processing layer?

Learn Before

Related