MDPs, rewards, value functions, Bellman equations, Q-learning, policy gradients, exploration versus exploitation and DQN.
40 cards · basic cards · AI-written, checked twice. Edit anything.
- What is a Markov Decision Process?
Show answer
A tuple (S, A, P, R, gamma) where S is the state space, A is the action space, P describes transition probabilities, R is the reward function, and gamma is the discount factor.- What does the Markov property state?
Show answer
The future state depends only on the current state and action, not on the history of states and actions that preceded it.- What is a state space in an MDP?
Show answer
The set S of all possible states an agent can be in within an environment.- What is an action space in an MDP?
Show answer
The set A of all possible actions an agent can take in the environment.- What is a transition probability function in an MDP?
Show answer
P(s'|s, a) describes the probability of reaching state s' from state s by taking action a.- What is a reward function in an MDP?
Show answer
R(s, a) or R(s, a, s') specifies the immediate numerical reward the agent receives for taking action a in state s or after reaching s'.- What is the discount factor gamma?
Show answer
A number between 0 and 1 that weighs immediate rewards more heavily than future rewards; it controls the agent's planning horizon.- What is the return (cumulative reward) in reinforcement learning?
Show answer
The sum of all future rewards from a starting time step, typically G_t = R_t+1 + gamma * R_t+2 + gamma^2 * R_t+3 + ...- What is a state value function V(s)?
Show answer
The expected return starting from state s, V(s) = E[G_t | S_t = s], representing how good a state is under a given policy.- What is an action value function Q(s, a)?
Show answer
The expected return starting from state s, taking action a, then following the policy; Q(s, a) = E[G_t | S_t = s, A_t = a].- What is the relationship between V(s) and Q(s, a)?
Show answer
V(s) equals the expected value of Q(s, a) over all actions weighted by the policy: V(s) = E_a[Q(s, a)] under the current policy.- State the Bellman expectation equation for V(s).
Show answer
V(s) = E[R_t+1 + gamma * V(s') | S_t = s], expressing the value of a state as the expected immediate reward plus the discounted value of the next state.- State the Bellman expectation equation for Q(s, a).
Show answer
Q(s, a) = E[R_t+1 + gamma * Q(s', a') | S_t = s, A_t = a], where a' is the next action sampled from the policy.- State the Bellman optimality equation for V(s).
Show answer
V*(s) = max_a E[R_t+1 + gamma * V*(s') | S_t = s, A_t = a], expressing optimal state value as the maximum expected return over all actions.- State the Bellman optimality equation for Q(s, a).
Show answer
Q*(s, a) = E[R_t+1 + gamma * max_a' Q*(s', a') | S_t = s, A_t = a], expressing the optimal action value recursively.
FlashKeepers