Google

This formula defines a transformed reward function, $$r'$$, based on an original reward function, $$r$$. The new reward is calculated by adding an arbitrary function, $$f$$, to the original reward. All functions are dependent on the current state ($$s_t$$), action ($$a_t$$), and the subsequent state ($$s_{t+1}$$). The mathematical expression is: $$r'(s_t, a_t, s_{t+1}) = r(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})$$ This demonstrates how alternative reward functions can be generated, which is a core aspect of why reward models can be underdetermined.

Reward Transformation Formula

A robotics team is training an agent to navigate a grid. They test two different reward functions. Both functions result in the agent learning the exact same optimal path. Your task is to determine the transformation function, `f`, that relates the two reward functions according to the formula: $$r'(s_t, a_t, s_{t+1}) = r(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})$$

Analyzing Reward Function Equivalence

Consider two reward functions, `r` and `r'`, related by the equation `r'(s_t, a_t, s_{t+1}) = r(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})`. Explain why it is possible for an agent to learn the exact same optimal behavior under both `r` and `r'`, even when the function `f` is not always zero. What does this phenomenon reveal about the challenge of inferring a single, true reward function from observing an agent's actions?

Analyzing Reward Function Invariance

A research team is training an agent and finds that two different reward functions, `r_1` and `r_2`, lead to the agent learning the exact same optimal behavior. The relationship between the two functions is defined as `r_2(s_t, a_t, s_{t+1}) = r_1(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})` for some non-zero function `f`. What is the most accurate explanation for this phenomenon?

Learn Before

Related