Google

In sentence-pair tasks such as Next Sentence Prediction (NSP), two sentences are combined into a single input sequence. A special start token, `[CLS]`, is prepended, while a `[SEP]` token separates the two sentences. For example, a training sample formatted as `[CLS] It is raining . [SEP] I need an umbrella . [SEP]` is classified with the label `IsNext`, indicating that the second sentence logically follows the first.

Example of Next Sentence Prediction (NSP) Input Formatting

A junior data scientist has prepared the following input sequence for a model that needs to understand the relationship between two sentences: `The sun is bright. [SEP] The sky is blue. [CLS]`. Identify two distinct formatting errors in this sequence and explain the correct placement and purpose of the special tokens involved.

Input Sequence Formatting Analysis

Given two sentences, Sentence A: 'The model was pre-trained.' and Sentence B: 'It requires fine-tuning.', construct the complete, single input sequence that combines them for a sentence-pair task. Your answer should include all necessary special tokens in their correct positions.

Constructing a Sentence-Pair Input Sequence

The step-by-step processing of a sentence pair through a Transformer encoder can be visualized to understand sequence classification. Given a concatenated input sequence such as $$[\mathrm{CLS}]$$ It is raining . $$[\mathrm{SEP}]$$ I need an umbrella . $$[\mathrm{SEP}]$$, the procedure unfolds as follows: First, each input token $$x_i$$ is mapped to its corresponding embedding vector $$\mathbf{e}_i$$. Next, the entire sequence of embeddings ($$\mathbf{e}_0, \dots, \mathbf{e}_{11}$$) is fed into the encoder. The encoder then generates a corresponding sequence of contextualized output vectors ($$\mathbf{h}_0, \dots, \mathbf{h}_{11}$$). Finally, because the initial hidden state $$\mathbf{h}_0$$ acts as the aggregate representation of the entire sequence, a Softmax classification layer is applied directly to it to yield a binary prediction, such as 'Is Next or Not?'.

Illustration of Transformer Encoding for Sequence Classification

A language model is being prepared for a task that involves understanding the relationship between two sentences. Given Sentence A: 'The model learns patterns.' and Sentence B: 'It then makes predictions.', which of the following represents the correctly formatted single input sequence for the model, using special tokens to delineate the structure?

Learn Before

Related