Google

When Supervised Fine-Tuning (SFT) is framed as a standard language model training task, the objective is to find the parameters $$\tilde{\theta}$$ that maximize the sum of the log-probabilities of the concatenated input-output sequences across the entire dataset $$\mathcal{D}$$. This is formally expressed as: $$ \tilde{\theta} = \arg\max_{\theta} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}} \log \mathrm{Pr}_{\theta}(\mathrm{seq}_{\mathbf{x},\mathbf{y}}) $$. By taking $$\log \mathrm{Pr}_{\theta}(\mathrm{seq}_{\mathbf{x},\mathbf{y}})$$ as the objective function, SFT can be implemented using standard LLMs, treating the combined input $$\mathbf{x}$$ and output $$\mathbf{y}$$ as a single sequence for the model to process.

SFT Objective as Maximizing Joint Log-Probability of Concatenated Sequences

Based on the standard formulation of the supervised fine-tuning objective, is the scenario described in the case study plausible? Justify your answer by explaining how the components of the objective function relate to the overall training goal.

Interpreting the SFT Objective Function

When fine-tuning a language model with the objective to maximize the sum of log-probabilities across all concatenated input-output sequences in a dataset, which of the following statements accurately describes the training dynamics?

A language model is being fine-tuned on a dataset `D` containing two input-output pairs: `(x1, y1)` and `(x2, y2)`. The training objective is to find the model parameters that maximize the sum of the log-probabilities of the concatenated input-output sequences across the entire dataset.

Two candidate models, Model A and Model B, produce the following log-probabilities for the concatenated sequences:

- **Model A:**
  - `log Pr(seq_x1,y1) = -1.2`
  - `log Pr(seq_x2,y2) = -0.8`
- **Model B:**
  -

Learn Before

Model B:

Related