Markov Decision Processes – Engineering AI Agents

Author

Pantelis Monogioudis

Bellman Expectation Backup

In this section we describe how to calculate the value functions by establishing a recursive relationship similar to the one we did for the return. We replace the…

Bellman Optimality Backup

Now that we can calculate the value functions efficiently via the Bellman expectation recursions, we can now solve the MDP which requires maximize either of the two…

Markov Decision Processes

Many of the algorithms presented here like policy and value iteration have been developed in older repos such as this and this. This site is being migrated to be compatible…

Policy Evaluation (Prediction)

The policy \(\pi\) is evaluated when we have produced the state-value function \(v_\pi(s)\) for all states. In other words when we know the expected discounted returns that…

Policy Improvement (Control)

In the policy improvement step we are given the value function and simply apply the greedy heuristic to it.

Policy Iteration

In this section we develop dynamic programming algorithms for the so called planning problem (which is RL without learning) where we are dealing with a perfectly known MDP.

The recycling robot.

Finite State Machine of a a recycling robot and MDP dynamics LUT