Google

In the context of fine-tuning, a common objective is to adjust the model's parameters, $$\theta$$, to maximize the likelihood of observing the true responses given the prompts in a dataset $$\mathcal{D} = \{(\mathbf{x}, \mathbf{y})\}$$. This is achieved by maximizing the sum of the log-likelihoods for all pairs in the dataset, which is mathematically equivalent to minimizing the negative log-likelihood loss.

Fine-Tuning as Maximum Likelihood Estimation

A popular method for fine-tuning a model is to find the optimal parameters, $$\hat{\theta}$$, by maximizing the total conditional log-likelihood over a dataset $$\mathcal{D}$$ of prompt-response pairs. This approach, equivalent to minimizing the negative log-likelihood loss, seeks parameters that make the observed outputs $$\mathbf{y}$$ most probable given the inputs $$\mathbf{x}$$. In some cases, the prompt $$\mathbf{x}$$ is decomposed into an instruction $$\mathbf{c}$$ and a user input $$\mathbf{z}$$, such that $$\mathbf{x} = (\mathbf{c}, \mathbf{z})$$. The formal expression is: $$ \hat{\theta} = \arg \max_{\theta} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}} \log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) = \arg \max_{\theta} \sum_{(\mathbf{x},\mathbf{y}) \in \mathcal{D}} \log \mathrm{Pr}_{\theta}(\mathbf{y}|\mathbf{c}, \mathbf{z}) $$ where $$\mathrm{Pr}_{\theta}(\cdot|\cdot)$$ is the probability predicted by an LLM with the parameters $$\theta$$.

Fine-Tuning Objective as Log-Likelihood Maximization

A training objective for a model can be formulated as maximizing the joint log-likelihood of concatenated input-output sequences. For a dataset `D` of input-output pairs `(x, y)`, the optimal parameters `˜θ` are found by maximizing the sum of the log-probabilities of the combined sequence `seq_{x,y}`. The formula is: $$˜θ = \arg \max_{\theta} \sum_{(x,y)∈D} \log \text{Pr}_{\theta}(\text{seq}_{x,y})$$ This approach is equivalent to maximizing the conditional log-likelihood `log Pr(y|x)` when the input distribution `Pr(x)` is not dependent on the model parameters `θ`.

Training Objective as Joint Log-Likelihood Maximization of Concatenated Sequences

A machine learning engineer is fine-tuning a pre-trained language model on a specialized dataset of question-answer pairs. The chosen training objective is to adjust the model's parameters to maximize the sum of the log-probabilities of the ground-truth answers, conditioned on their corresponding questions. Which statement best analyzes the direct effect of this training objective on the model's behavior?

During the fine-tuning of a language model on a dataset of prompt-response pairs, an engineer uses the negative log-likelihood as the loss function. The engineer observes that this loss value is steadily decreasing over training epochs. In the context of probability, explain what this decreasing loss indicates about the model's predictions for the correct responses in the dataset.

Interpreting Fine-Tuning Loss

Analyze the following scenario through the lens of fine-tuning as a process of maximum likelihood estimation. Explain why the observed model behavior is a direct consequence of this training objective.

Analyzing Fine-Tuning Behavior

When fine-tuning a language model, the objective of maximizing the sum of the log-likelihoods of the true responses given the prompts is mathematically equivalent to minimizing the mean squared error loss over the dataset.

Learn Before

Related