Google

In reward shaping, a new reward signal, known as the transformed reward function $$r'(s_t, a_t, s_{t+1})$$, is created by adding a shaping reward function $$f(s_t, a_t, s_{t+1})$$ to the environment's original reward function $$r(s_t, a_t, s_{t+1})$$. This relationship is expressed by the formula: $$r'(s_t, a_t, s_{t+1}) = r(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})$$ This technique provides an agent with additional feedback, where all functions depend on the current state $$s_t$$, action $$a_t$$, and next state $$s_{t+1}$$.

Transformed Reward Function Formula

When using a transformed reward function, the choice of the shaping reward function is critical for maintaining the original optimal policy. To ensure that the agent's optimal behavior is not altered, the shaping function must be defined in a specific, constrained form rather than being an arbitrary addition.

Condition for Policy Invariance in Reward Shaping

An AI agent is being trained to navigate a grid world to reach a goal square. The original reward function, $r$, provides +10 for reaching the goal and -0.1 for every other step. To encourage faster learning, a new, transformed reward function, $r'$, is implemented. For a specific step that moves the agent from a square 5 units away from the goal to a square 4 units away, the agent receives a total transformed reward $r'$ of +0.9. Based on the general formula for reward shaping, $r'(s_t, a_t, s_{t+1}) = r(s_t, a_t, s_{t+1}) + f(s_t, a_t, s_{t+1})$, what are the numerical values for the original reward $r$ and the shaping function $f$ for this specific step? Explain the purpose of the shaping function in this context.

Deconstructing a Shaped Reward Function

Given the following scenario, identify the part that represents the original environment reward function and the part that represents the shaping reward function, according to the standard formulation of reward shaping.

Analyzing Reward Components in a Maze Navigation Task

A reinforcement learning agent is operating in an environment where taking a specific action in a given state results in a transition to a new state. The environment's original reward for this transition is -0.5. To guide the agent more effectively, a shaping function is added, which provides an additional reward value of +2.0 for this same transition. According to the standard formulation for reward shaping, what is the total transformed reward the agent receives?

Learn Before

Related