AI: Reinforcement Learning

Overview

Reinforcement Learning (RL) is, in the words of the course, a melting pot where the computational techniques of dynamic programming, heuristic search in problem spaces, and the functional representation power of neural networks have converged to build a highly scalable discipline — one that lets us efficiently attack a wide range of problems previously considered practically unapproachable. An agent acts in an environment; the environment returns a state and a reward; the agent's job is to learn a policy π(a|s) that maximises the expected cumulative, discounted return.

Learning Objectives

The main objective is to introduce students to the field of RL and to set up a framework of knowledge that lets them make informed analysis of the opportunities and challenges for its successful application in business. The course aims for a sound understanding of RL techniques and their variations (model-free, model-based, online/offline RL, behavioural cloning, …) and a good intuition for how to apply them to tasks of one's own design. It encompasses knowledge about:

Teaching Methodology

IE University's teaching method is collaborative, active, and applied: students participate throughout the process to build their knowledge and sharpen their skills, while the professor leads and guides toward the learning objectives. The course is both lecture- and example-based, with frequent in-class group discussions. Knowledge blocks are organised by integrating the conceptual/theoretical background first, then moving into practice/tutorial examples, and finally framing assignments of assorted complexity. The course has six main elements:

Effort distribution (150 hours total)

The learning-activity weighting reflects how a student is expected to spend the 150 hours of the 6-ECTS workload.

Assessment & Evaluation

Roughly one third of the grade rewards attendance/participation, one third the individual assignments, and the final third the group work and presentation plus the test/exam. The official weighting is:

Note: the ongoing-grading scoring matrix in the syllabus distributes session points across Participation (20%), Individual Assignment (30%), Group Assignment (20%), and Test/Exam (30%) of the continuous-evaluation total; the criteria table above states the headline grade weights.

Attendance, passing & re-sit rules

• Attendance. Students are expected to attend all lectures and tutorials — material covered in class is not always in the readings. The 80% attendance rule is binding.
• Test-exam. The Session-28 test-exam is a concept-summary review, not a final exam; it adds to the rest of the items and has no minimum passing grade.
• Four calls. Each student has four chances to pass a course over two consecutive academic years (ordinary + extraordinary June/July re-sits).
• Attendance failure. Students who do not comply with the 80% rule fail both calls for the year and must re-enrol the next academic year — with no re-sit opportunity.
• Re-sit (June/July). A single comprehensive exam, in-person at the enrolled campus (Segovia or Madrid). The final grade depends on this exam only; continuous evaluation is not counted. Pass mark 5, maximum 8.0 ("notable").
• Retakers (3rd call). Must check the assigned professor's syllabus and contact them about specific criteria; maximum grade 10.0.
• Review session. Grade appeals require having attended the post-exam review session.
• Program rule. Failing more than 18 ECTS in the year after the June/July re-sits leads to being asked to leave the program.

Per-component rubric, deliverable format & tips

The criteria below interpret the syllabus's review-criteria language into concrete expectations. The professor reports the exact structure and review criteria for each specific job; treat this as the default standard.

Program — All 30 Sessions

The program is organised here into eight thematic modules that group the 30 sessions by the arc of the field: foundations → tabular planning & learning → integration & evolution → frameworks → approximation & deep policy methods → advanced paradigms → frontiers & applications → assessment. Sessions flagged assignment, group, or exam mark deliverables. The disclaimer in the syllabus applies: the coverage is tentative and pace adapts to the group.

Key Concepts — Glossary

A quick-reference glossary of the ~50 core terms used throughout the course, ordered roughly from foundations to advanced methods. Notation: s state, a action, r reward, γ discount, π policy, α step size, θ/w parameters.

Agent / Environment

The learner/decision-maker and everything it interacts with. They exchange actions, states, and rewards in a loop.

Policy π(a|s)

A mapping from states to a probability distribution over actions — the agent's behaviour.

Reward & Return

Reward r is the immediate scalar signal; return G_t = Σ γ^kr is the discounted cumulative reward the agent maximises.

Discount factor γ

A value in [0,1] weighting future rewards; smaller γ makes the agent myopic, larger γ far-sighted.

State-value V(s)

Expected return starting from state s and following policy π thereafter.

Action-value Q(s,a)

Expected return after taking action a in state s, then following π — the basis of value-based control.

Markov property

The future depends only on the current state, not the history that led there — the present state is a sufficient statistic.

MDP ⟨S,A,P,R,γ⟩

Markov Decision Process: the formal model of a sequential decision problem with stochastic transitions and rewards.

Bellman equations

Recursive consistency conditions for value functions; the expectation form evaluates π, the optimality form defines v*/q*.

Dynamic Programming

Exact solution of a known MDP by iterating Bellman backups — value iteration and policy iteration.

Generalized Policy Iteration

The interplay of policy evaluation and policy improvement converging to the optimal policy.

Exploration vs. exploitation

The trade-off between trying uncertain actions to gain information and choosing the current best to gain reward.

ε-greedy / UCB / Thompson

Strategies for managing exploration in bandits: random slack, optimism under uncertainty, and posterior sampling.

Monte Carlo methods

Estimate values by averaging complete observed returns — unbiased but high-variance, episodic only.

Temporal-Difference (TD)

Learn from incomplete episodes by bootstrapping toward r + γV(s′); the workhorse of practical RL.

Bootstrapping

Updating an estimate toward another estimate rather than a true sampled target — trades variance for bias.

SARSA vs. Q-learning

On-policy TD control (learns the policy it follows) vs. off-policy (learns the greedy optimal policy).

Expected SARSA

Uses the expected next-action value under the policy, reducing variance versus plain SARSA.

TD(λ) & eligibility traces

A spectrum between TD(0) and Monte Carlo; traces credit recently-visited states for new errors.

On-policy / Off-policy

Whether the agent learns about the policy it is following, or about a different (e.g. greedy) target policy.

Importance sampling

Reweighting returns from a behaviour policy to estimate values under a target policy — enables off-policy learning.

Function approximation

Representing V, Q, or π with parameters (e.g. a neural net) so RL scales to large/continuous spaces via generalisation.

Deadly triad

The instability that can arise when bootstrapping, off-policy learning, and function approximation are combined.

Policy Gradient Theorem

Expresses ∇J(θ) as an expectation of ∇log π · return, enabling direct optimisation of parameterised policies.

REINFORCE

Monte Carlo policy gradient: update θ by ∇log π(a|s) scaled by the episode return.

Baseline & Advantage

Subtracting V(s) (a baseline) gives A(s,a)=Q−V, reducing gradient variance without bias.

Actor-Critic / A2C

An actor (policy) trained with a critic's (value) low-variance advantage estimate.

DQN

Deep Q-Network: a neural Q-function stabilised by experience replay and a target network.

Experience replay

A buffer of past transitions sampled randomly to decorrelate updates and improve data efficiency.

Target network

A periodically-frozen copy of the Q-network used to compute stable regression targets.

Model-based vs. model-free

Whether the agent learns/uses a model of dynamics to plan, or learns values/policy directly from experience.

Inverse RL / Imitation

Recovering the reward function (IRL) or copying behaviour (behavioural cloning) from observed demonstrations.

Pareto frontier / NEAT

The set of non-dominated trade-offs in multi-objective optimisation; NEAT evolves neural-net weights and topology.

Multi-Agent RL (MARL)

RL with several simultaneously-learning agents, making the environment non-stationary; analysed via game theory.

Reward shaping

Adding auxiliary reward terms to guide learning toward sparse-reward goals without changing the optimal policy (potential-based shaping is provably policy-preserving).

Trajectory / Episode

A trajectory is a sequence s₀,a₀,r₁,s₁,…; an episode is a trajectory that ends in a terminal state. Episodic tasks reset; continuing tasks never terminate.

Horizon

How far into the future returns are accumulated — finite (fixed length), infinite-discounted (via γ), or average-reward.

Stationary vs. non-stationary

Whether the environment's dynamics/rewards change over time; non-stationarity favours constant step sizes and is intrinsic to multi-agent settings.

Stochastic vs. deterministic policy

A stochastic policy outputs a distribution over actions (needed for exploration and continuous control); a deterministic one outputs a single action.

Greedy / ε-greedy action

Greedy picks argmax_a Q(s,a); ε-greedy does so with probability 1−ε and explores at random with probability ε.

Softmax / Boltzmann policy

π(a|s) ∝ exp(preference/τ): a temperature-controlled smooth alternative to ε-greedy, and the policy form used by policy-gradient methods.

Bellman optimality equation

v*(s) = max_a Σ p(s′,r|s,a)[r+γv*(s′)] — the fixed-point characterising the best achievable value.

Value vs. policy iteration

Two DP schemes: value iteration applies the optimality backup directly; policy iteration alternates full evaluation with greedy improvement.

Backup diagram

A small tree picturing how value flows from successor states/actions back to the current node — the visual form of a Bellman update.

TD error δ

δ = r + γV(s′) − V(s): the surprise between a bootstrapped target and the current estimate; the learning signal for TD and the critic.

n-step return

A target using n real rewards then bootstrapping: interpolates between one-step TD (n=1) and Monte Carlo (n=∞).

Maximisation bias

The systematic overestimation introduced by the max operator in Q-learning/DQN; mitigated by Double Q-learning.

Semi-gradient methods

Function-approximation updates that ignore the target's dependence on the parameters — efficient but a source of instability.

Tile coding / RBF features

Classic fixed feature representations for linear function approximation, partitioning the state space into overlapping regions.

Generalized Advantage Estimation

A λ-weighted blend of n-step advantage estimates that trades bias for variance — the modern policy-gradient standard.

PPO / TRPO

Trust-region / clipped-objective policy-gradient methods that bound how far the policy may move per update for stable training.

A3C / A2C

Asynchronous (A3C) and synchronous (A2C) advantage actor-critic — parallel actors feeding a shared advantage-based update.

Rainbow

A DQN agent combining six orthogonal improvements (Double, Dueling, Prioritised Replay, Distributional, Noisy Nets, n-step).

Offline (batch) RL

Learning a policy purely from a fixed dataset with no further interaction; the central challenge is distributional shift.

Behavioural cloning

Supervised imitation that maps expert states to actions; simple but brittle to compounding distribution shift.

Hierarchical RL / options

Temporally-extended sub-policies ("options") that let an agent plan over abstract, multi-step actions toward sub-goals.

Reward hacking

An agent maximising a flawed proxy reward in unintended ways instead of the designer's true objective.

Sample efficiency

How much environment interaction a method needs to reach a given performance — a key axis distinguishing RL algorithms.

Sim-to-real transfer

Carrying a policy trained in simulation over to the real world despite the "reality gap"; aided by domain randomisation.

Credit assignment

Attributing a delayed outcome to the earlier actions responsible for it — the core difficulty eligibility traces and advantages address.

Bibliography

Explore the Demos

Every technique above that has a live visualisation in this repo, in course order: