Google

In standard supervised training, the objective for a Large Language Model is to maximize the probability of generating a correct 'gold-standard' output sequence, $$y$$, given an input, $$x$$. This is achieved through Maximum Likelihood Estimation (MLE), where the model, which produces a series of token distributions, is trained to align these predictions with the one-hot distributions representing the target sequence. The formal objective is to maximize the conditional probability: $$\max \Pr(y|x)$$.

Maximum Likelihood Estimation (MLE) Objective in Supervised Language Model Training

A team is training a language model for a customer support chatbot. They are using a standard supervised learning approach where the model's goal is to maximize the probability of generating the exact responses found in their training dataset. The dataset consists of thousands of real chat logs between human agents and customers. However, the team discovers that in about 5% of these logs, the human agent provided a factually incorrect or unhelpful answer.

Analyze how the training objective will likely affect the model's responses when it is deployed and encounters questions similar to those that were answered incorrectly in the training data. Explain the underlying mechanism of the training objective that leads to this outcome.

Analyzing Model Training with Flawed Data

While supervised fine-tuning with explicit instruction-response mappings is effective for teaching Large Language Models to perform specific tasks, it is often insufficient for achieving full alignment. A major limitation is that standard supervised learning struggles to capture and encode ethical nuances and complex contextual considerations into a fine-tuning dataset. Furthermore, humans themselves frequently cannot precisely express their own preferences, making it difficult to create comprehensive labeled data for complex behavioral alignment.

Limitations of Supervised Fine-Tuning for LLM Alignment

Imagine a language model is being trained on the following example: 
Input: 'The capital of France is' 
Target next token: 'Paris' 

During this training step, the model's current prediction for the next token gives 'London' a probability of 0.3, 'Paris' a probability of 0.2, and 'Berlin' a probability of 0.1 (with other words having the remaining probability). 

Based on the standard objective of maximizing the likelihood of the correct output, describe how the model's internal parameters will be adjusted in response to this specific training example.

Parameter Updates in Supervised LLM Training

A language model is being trained with a supervised objective to maximize the probability of the correct output. Given the input 'The largest city in the US is', the target output is the two-token sequence 'New York'. Two different models are evaluated on this single instance.

*   **Model A** predicts the first token 'New' with a probability of 0.6, and then predicts the second token 'York' with a probability of 0.8.
*   **Model B** predicts the first token 'New' with a probability of 0.9, and 

Learn Before

Related