Google

The final input embedding for a sequence token, denoted as $$\mathbf{e}$$, is calculated by summing three distinct component vectors: the token embedding $$\mathbf{x}$$, the positional embedding $$\mathbf{e}_{\text{pos}}$$, and the segment embedding $$\mathbf{e}_{\text{seg}}$$. This composition is represented by the formula:
$$\mathbf{e} = \mathbf{x} + \mathbf{e}_{\text{pos}} + \mathbf{e}_{\text{seg}}$$

Formula for Input Embedding Composition

Based on the provided case study, calculate the final input embedding vector for the specified token.

Calculating a Final Input Embedding

A common method for creating the final input representation for a token in a sequence involves summing three distinct vectors. Match each vector component to its specific function in this process.

A researcher designs a language model where the final input representation for each word is created by summing a vector for the word's identity and a vector for the sentence it belongs to. However, they intentionally omit the vector that encodes the word's specific position in the sequence. What is the most likely deficiency this model will exhibit?

Learn Before

Related