Google

A fundamental difference exists between standard language modeling and supervised fine-tuning objectives. Standard language modeling minimizes the loss over all tokens of a concatenated input-output sequence $$\mathrm{seq}_{\mathbf{x},\mathbf{y}} = [\mathbf{x},\mathbf{y}]$$, optimizing the joint log-probability $$\log \mathrm{Pr}_{\theta}(\mathbf{x},\mathbf{y})$$. In contrast, fine-tuning focuses on the conditional log-probability of the output. By applying the chain rule, the joint sequence probability is decomposed into the probability of the input and the conditional probability of the output: $$\log \mathrm{Pr}_{\theta}(\mathrm{seq}_{\mathbf{x},\mathbf{y}}) = \log \mathrm{Pr}_{\theta}(\mathbf{x}) + \log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x})$$. In the fine-tuning context, the loss computation over the input $$\mathbf{x}$$ is set to $${}0$$, meaning the loss is computed exclusively for the output tokens $$\mathbf{y}$$.

Conditional vs. Joint Probability Objectives in Language Modeling

The relationship between the joint, conditional, and marginal probabilities of token sequences is defined by the chain rule of probability. In logarithmic form, this allows the conditional log-probability of an output sequence `y` given an input `x` to be expressed in terms of the joint and marginal log-probabilities. The formula is: $$ \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) = \log \text{Pr}_{\theta}([\mathbf{x}, \mathbf{y}]) - \log \text{Pr}_{\theta}(\mathbf{x}) $$ This identity can be rearranged to show how the joint log-probability of the concatenated sequence `[x, y]` is composed of the marginal and conditional parts: $$ \log \text{Pr}_{\theta}([\mathbf{x}, \mathbf{y}]) = \log \text{Pr}_{\theta}(\mathbf{x}) + \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) $$ This relationship is fundamental for training language models, where the objective is often to maximize the conditional log-probability `log Pr_θ(y|x)`.

Relationship Between Joint, Conditional, and Marginal Log-Probabilities of Sequences

A researcher is training a language model for a summarization task using `[article, summary]` pairs. They are considering two different methods for calculating the training loss:

*   **Method 1:** The loss is calculated based on the model's predictions for all tokens in the concatenated `[article, summary]` sequence.
*   **Method 2:** The loss is calculated based only on the model's predictions for the tokens in the `summary` part of the sequence.

For each method, identify whether it corresponds to optimizing a joint probability objective or a conditional probability objective. Then, explain the key difference in what the model is being trained to accomplish with each objective.

Analysis of Language Model Training Objectives

A machine learning team is working on two separate projects involving a large language model. Analyze the two scenarios described below and determine which training objective is more appropriate for each. Justify your reasoning for both scenarios.

Selecting an Appropriate Language Model Training Objective

A developer is fine-tuning a language model on a dataset of `[instruction, response]` pairs. Initially, the training process calculated the prediction loss across all tokens in both the `instruction` and the `response`. The developer then modifies the process to calculate loss *only* on the tokens in the `response`. What is the primary effect of this change on the model's training objective?

Learn Before

Related