In the Stage I preference loss formulation $\mathcal{L}_{\text{I}}^{\text{pref}}(\theta)$, what is the function of the hyperparameter $\beta_{\text{pref}}$?

In the Stage I preference loss $\mathcal{L}_{\text{I}}^{\text{pref}}(\theta)$, the reference model $p_{\text{ref}}$ is actively updated alongside the policy parameters $\theta$.

What specific generation capability does the supervised fine-tuning loss $\mathcal{L}_{\text{I}}^{\text{gen}}(\theta)$ instill in the model?

Describe the two primary criteria that candidate scaffolds are biased toward by the preference loss $\mathcal{L}_{\text{I}}^{\text{pref}}(\theta)$, and explain the function of the weighting factor $\Delta_{\text{val}}$.

The training objective for Stage I in JIT-Agent couples reference-anchored preference optimization with supervised ___ learning into a combined loss.

Arrange the mathematical operations in the order they are evaluated within the preference loss expectation to compute the weighted argument of the loss.