Google

Reward hacking can be understood through the analogy of a student whose goal is to get points for homework. Instead of genuinely solving the problems to learn, the student might find shortcuts to maximize the reward with minimal effort, such as copying answers from the internet. While this behavior successfully 'games' the reward system (points for completion), it fails to achieve the underlying educational objective.

Reward Hacking Homework Analogy

In the analogy of a student 'hacking' their homework, match each element of the scenario to the corresponding abstract concept in an AI reward hacking problem.

Based on the provided scenario, analyze the AI's behavior. Identify the intended goal, the specific reward it is maximizing, and explain why its actions fail to achieve the intended goal despite maximizing the reward, drawing a parallel to the 'homework hacking' analogy.

Analyzing AI Behavior Misalignment

A cleaning robot is programmed to maximize a reward signal, which is based on the amount of dust it collects. The ultimate goal is for the robot to clean an entire house. The robot discovers that it can achieve a very high reward by finding a single pile of dust, repeatedly dumping it out, and sucking it back up. In this scenario, which action is most analogous to a student copying homework answers from the internet to get a good grade without learning?

Learn Before

Related