Post-training alignment (such as PPO reinforcement learning) can significantly degrade the calibration of large language models. In pre-trained models such as GPT-4, predicted probabilities (logprobs) across multiple-choice options track actual task accuracy closely, yielding near-perfect calibration with an Expected Calibration Error (ECE) of $0.007$ on benchmarks like MMLU. However, subsequent post-training alignment impairs this correspondence, substantially inflating calibration error to an ECE of $0.074$ and causing the model's confidence to diverge from its true probability of being correct.

Degradation of Model Calibration from Post-Training Alignment

Empirical evaluation demonstrates that GPT-4's performance across standardized exam benchmarks originates primarily in the pre-training stage rather than post-training alignment. When tested on multiple-choice exam sections, the base pre-trained GPT-4 model achieves an average score of $73.7\%$ while the post-RLHF model scores $74.0\%$, indicating that reinforcement learning from human feedback does not substantially alter the fundamental capabilities acquired during pre-training.

Origin of GPT-4 Exam Capabilities in Pre-training

Evaluating pre-trained base models against post-RLHF models on an equal footing is difficult for free-response tasks compared to multiple-choice benchmarks. Free-response answer generation and sampling methodologies rely heavily on the model's capacity to follow open-ended instructions accurately, an attribute developed during instruction-tuning and RLHF post-training that raw pre-trained base models inherently lack.

Evaluation Asymmetry in Base and RLHF Free-Response Comparison

University of Michigan - Ann Arbor

Ch.2 Post-Training Alignment and Calibration - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor

Safety Alignment and Rule-Based Reward Models - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor

RLHF Effects on Capability and Calibration - Frontier Foundation Models, Capability Evaluation, and Just-In-Time Agent Harnesses @ University of Michigan - Ann Arbor

Learn Before

Related

Learn After