Google

To efficiently train a model on multi-turn dialogues in a single run, the entire alternating conversation is treated as a single concatenated sequence, $$\mathrm{seq} = [\mathbf{x}^1, \mathbf{y}^1, \dots, \mathbf{x}^K, \mathbf{y}^K]$$. Its overall log-probability is decomposed into conditional probabilities for each turn. A key trick in supervised fine-tuning (SFT) for conversational models is that loss computation is applied exclusively to the model's responses, while the loss terms for generating the user's inputs are ignored (set to $${}0$$). The decomposed log-probability is: $$ \log \mathrm{Pr}_{\theta}(\mathrm{seq}) = \log \mathrm{Pr}_{\theta}(\mathbf{x}^1) + \log \mathrm{Pr}_{\theta}(\mathbf{y}^1/\mathbf{x}^1) + \dots + \log \mathrm{Pr}_{\theta}(\mathbf{x}^K/\mathbf{x}^1,\mathbf{y}^1,\dots,\mathbf{y}^{K-1}) + \log \mathrm{Pr}_{\theta}(\mathbf{y}^K/\mathbf{x}^1,\mathbf{y}^1,\dots,\mathbf{x}^K) $$. In this sum, terms predicting user inputs like $$\log \mathrm{Pr}_{\theta}(\mathbf{x}^1)$$ are masked to $${}0$$, and only terms predicting responses like $$\log \mathrm{Pr}_{\theta}(\mathbf{y}^k/\dots)$$ contribute to the training loss.

Log-Probability Decomposition for Efficient Multi-Turn Dialogue Training

A data scientist is training a dialogue model by representing conversations as single, concatenated sequences of user inputs and model responses. They observe that the training loss converges very quickly to a low value, but the model's generated responses are generic and unhelpful. Upon reviewing the implementation, you discover that the loss is being calculated based on the model's ability to predict *all* tokens in the sequence, including the user's inputs. Explain why this approach is incorrect and how it leads to the observed poor model performance.

Debugging a Dialogue Model Training Loop

A machine learning team is training a multi-turn dialogue model. The standard approach is to process an entire conversation (e.g., `user_input_1, model_response_1, user_input_2, model_response_2`) as a single sequence in one forward pass, but only calculate the training loss on the tokens corresponding to the model's responses.

A junior engineer proposes an alternative method. For the same conversation, they suggest creating separate training examples for each model turn:
1.  Input: `user_input_1` -> Target: `model_response_1`
2.  Input: `user_input_1, model_response_1, user_input_2` -> Target: `model_response_2`

They argue this is more conceptually straightforward. As the senior engineer on the team, evaluate this proposal. What is the most significant disadvantage of the proposed alternative compared to the standard single-pass method, particularly for training on large datasets with long conversations?

Evaluating Dialogue Model Training Strategies

The training of a multi-round conversational model can be achieved by maximizing the log-likelihood over an entire training dataset $$\mathcal{D}$$. Because the loss for generating user inputs is masked (set to $${}0$$), the sum of the conditional log-probabilities of the model's responses is mathematically equal to the log-probability of the entire concatenated conversational sequence, $$\log \mathrm{Pr}_{\theta}(\mathrm{seq})$$. Therefore, the training objective across the dataset is formulated as: $$ \tilde{\theta} = \argmax_{\theta} \sum_{\mathrm{seq} \in \mathcal{D}} \log \mathrm{Pr}_{\theta}(\mathrm{seq}) $$. This formulation simplifies the implementation of multi-round Supervised Fine-Tuning (SFT) by treating it fundamentally the same as standard language model training on independent sequences.

Dataset-Level Objective for Multi-Round Conversational Models

A dialogue model is trained by processing entire multi-turn conversations as single, concatenated sequences of text. To make this process efficient, the training loss is calculated based only on the model's ability to predict certain parts of the sequence, while the log-probabilities of other parts are ignored. Given the following two-turn conversation, which parts of the sequence would be used to calculate the training loss?

*   Turn 1 (User): 'What is the weather like'
*   Turn 1 (Model): 'In

Learn Before

Related