Google

In machine translation, fine-tuning data can be formulated as a concatenated $$(\mathbf{x}, \mathbf{y})$$ pair, functioning as a complete training sample for a Large Language Model. The input sequence, denoted mathematically as $$\mathbf{x}$$, is constructed by joining the task instruction and the specific source text. The target output, denoted as $$\mathbf{y}$$, provides the correct translation. For example, an English-to-Chinese translation sample is represented as:

- $$\mathbf{x}$$ = "Translate the text from English to Chinese.\n How's the weather today?"
- $$\mathbf{y}$$ = "今天天气怎么样？"

This structured $$(\mathbf{x}, \mathbf{y})$$ pair is then used directly to adjust the model's parameters during fine-tuning.

Example of a Concatenated $$(\mathbf{x}, \mathbf{y})$$ Sample for Machine Translation Fine-Tuning

A data scientist is preparing a dataset to fine-tune a model for translating German news headlines to Japanese. Analyze the following training sample they created. Identify the primary error in how the data is structured according to the concatenated input method, and briefly explain the correction.

Error Analysis in Fine-Tuning Data Preparation

You are preparing a dataset to fine-tune a language model for translating from Spanish to Japanese. You have the following components:

- Instruction: `Translate the following text from Spanish to Japanese.`
- Source Text: `¿Cuál es tu nombre?`
- Target Translation: `お名前は何ですか？`

Based on the standard concatenated `(x, y)` format for a training sample, write out the exact content for the input `x` and the output `y`.

Constructing a Fine-Tuning Sample for Translation

A developer is preparing a dataset to fine-tune a language model for translating French legal text to English. The task instruction is 'Translate the following legal text from French to English.' and a specific French sentence to be translated is 'Le défendeur a plaidé non coupable.'. The correct English translation is 'The defendant pleaded not guilty.'. Which of the following options correctly formats this single data sample as a concatenated input (`x`) and output (`y`) pair for training?

Learn Before

Related