Google

In encoder pre-training and related tasks, the special $$[\mathrm{CLS}]$$ token is conventionally used as the start symbol for an input sequence. It is typically the first token, denoted as $$x_0$$, and serves as a generic start-of-sequence marker.

CLS Token as a Start Symbol in Encoder Pre-training

A data scientist is preparing a sentence for a sequence model. After tokenization, the sentence is represented as the list: `['The', 'model', 'processes', 'sequences']`. The data scientist's script then appends a special token, resulting in the final input: `['The', 'model', 'processes', 'sequences', '[CLS]']`. Based on the conventional role of this specific special token in sequence models, identify the error in this final input structure and explain the token's correct placement and purpose.

Analyzing a Preprocessing Step

When preparing an input sequence for certain neural network architectures, a special token is conventionally placed at the very beginning to serve as a start-of-sequence marker. This token is denoted as ____.

A researcher is preparing the sentence 'Neural networks learn patterns' for input into a sequence model. The model's design specifies that a special token must be placed at the very beginning of every input sequence to act as a start-of-sequence marker. Given this requirement, how should the researcher format the tokenized input?

Learn Before

Related