Robotics Control and Reinforcement Learning

An autonomous delivery robot rolling through an Indian airport terminal or hospital corridor — the kind of two-wheeled "last-mile" bot that startups such as Ottonomy build — is really running two controllers at once, on two different clocks. One controller fires a thousand times a second: it reads the tilt and wheel-speed sensors and decides how much current to send to each motor so the robot does not topple over or drift off its wheels. The other controller fires maybe ten times a second: it looks at where the robot is in the corridor, where the obstacles and doorways are, and decides which direction to roll next. The first controller is solved with seventy-year-old control theory and never "learns" anything — its parameters are fixed at design time. The second controller, in modern robots, is often learned from millions of trial-and-error episodes in simulation using reinforcement learning (RL). This chapter builds both halves from first principles, connects them formally, and works through the exact arithmetic of how an RL agent updates its estimate of what a good decision looks like.

Two control loops, two timescales

Every physical robot separates "how do I execute this motion accurately" from "what motion should I attempt." The first question is a classical control problem: given a target joint angle, wheel speed, or balance point, compute a motor command that drives the physical error toward zero, using a mathematical model of how the robot's motors and links respond to current. The second question is a decision-making problem: given the robot's current state and a map or sensor read of its surroundings, choose the next waypoint, grasp target, or path segment that best serves a goal. Classical control assumes you already know (or can estimate) how the plant behaves, so it can compute the right correction directly from a formula. Decision-making in a cluttered, partially unpredictable environment — where the "cost" of an action might only become clear several steps later — is exactly where reinforcement learning replaces hand-derived formulas with a value learned from experience.

Classical feedback control: the PID loop

The workhorse of the low-level loop is the PID controller (Proportional-Integral-Derivative). Let e(t) be the error at time t — the difference between the setpoint (target angle, target speed) and the measured value. The controller computes a motor command:

u(t) = Kp·e(t) + Ki·∫e(t)dt + Kd·de(t)/dt

The proportional term pushes harder the further you are from the target. The integral term accumulates small persistent errors (useful when, say, friction or gravity means a pure proportional term never quite reaches the setpoint) and eventually eliminates that steady-state offset. The derivative term looks at how fast the error is closing and brakes the command as the target approaches, which is what prevents the arm from slamming past the setpoint and oscillating.

Worked example. A robot arm's shoulder joint has a setpoint of 90°. Two sensor samples 0.1 s apart show the joint at 70° and then 75°, so the error has gone from e(t−1) = 90 − 70 = 20° to e(t) = 90 − 75 = 15°. The numerical derivative is de/dt = (15 − 20) / 0.1 = −50°/s (negative because the error is shrinking — the arm is closing in). For this single step, approximate the integral term as e(t)·Δt = 15 × 0.1 = 1.5. With gains Kp = 2.0, Ki = 0.5, Kd = 0.02:

u(t) = 2.0(15) + 0.5(1.5) + 0.02(−50) = 30 + 0.75 − 1 = 29.75

The proportional term alone would have commanded 30 units of torque; the derivative term shaves 1 unit off because the error is already closing quickly, a small pre-emptive brake against overshoot. If Kd were raised to 0.5 instead, the same de/dt gives a derivative contribution of 0.5 × (−50) = −25, and the total command drops to 30 + 0.75 − 25 = 5.75 — a much gentler push. That is the real engineering trade-off in tuning Kd: too small and the arm overshoots and oscillates around 90°; too large and the arm becomes sluggish, and because a numerical derivative amplifies sensor noise, an aggressive Kd can also make the motor command jittery on a real, noisy sensor.

Crucially, nothing here is "learned." Kp, Ki, Kd are fixed numbers chosen by an engineer (or tuned once via a method like Ziegler–Nichols) based on a model of the arm's mass and friction. Feed the same error and derivative into the same PID controller a thousand times and it outputs the same command every time — there is no reward, no trial and error, no update rule. This distinction matters for the misconception addressed later in this chapter.

Figure: the two-timescale control hierarchy

When the model is unknown: control as a Markov Decision Process

PID control needed a model: it assumed the relationship between motor current and joint acceleration was known well enough to make de/dt meaningful. Path-level decisions in an unfamiliar corridor, a cluttered warehouse aisle, or a construction site have no such clean model — friction with an unknown floor, the unpredictable path of a pedestrian, the exact grip needed on an oddly-shaped package are not things an engineer can write down as a differential equation. Reinforcement learning formalizes this decision problem as a Markov Decision Process (MDP): a state space S (the robot's position, sensor readings, or a discretized summary of them), an action space A (the choices available — turn left, turn right, apply a grasp), a reward function r(s, a, s′) that scores how good a transition was, and — critically — the assumption that the next state depends only on the current state and action, not on the full history (the Markov property).

The object the agent wants to learn is the Q-function: Q(s, a) is the total discounted future reward the agent can expect if it takes action a in state s and then behaves optimally afterward. It obeys the Bellman optimality equation:

Q*(s, a) = E[ r(s, a, s′) + γ · maxₐ′ Q*(s′, a′) ]

where γ ∈ [0, 1) is a discount factor that values a reward received sooner more than the same reward received later (this also keeps the sum finite over an unbounded number of future steps). If you already know the transition probabilities, you can solve this equation directly by dynamic programming — that is called value iteration, and it is model-based. Q-learning instead estimates Q(s, a) purely from sampled experience, without ever writing down a transition model — it is model-free, which is exactly why it applies to a real robot whose floor friction and motor wear are not known in closed form. Each time the agent observes a transition (s, a, r, s′), it nudges its estimate toward the sampled Bellman target:

Q(s, a) ← Q(s, a) + α · [ r + γ·maxₐ′ Q(s′, a′) − Q(s, a) ]

The bracketed quantity is called the temporal-difference (TD) error: the gap between what the agent currently believes Q(s, a) is worth and what one real observed step plus its own downstream estimate suggests it should be worth. α is the learning rate, controlling how much of that gap gets corrected on each visit.

Worked example: Q-learning in a hospital corridor

Model a delivery robot in a five-cell corridor, cells 0 through 4, with the delivery point (goal) at cell 4. Two actions are available at every cell: Left (L) and Right (R), and motion is deterministic — R moves the robot one cell right (clamped at 4), L moves it one cell left (clamped at 0). The reward is −1 for every non-terminal step (a shaping term that penalizes wasted time and therefore favors the shortest path) and +10 for the step that reaches the goal, which ends the episode. Set the learning rate α = 0.5 and discount γ = 0.9, and initialize every Q(s, a) = 0.

Suppose the robot starts each episode at cell 0 and, for these first three episodes, happens to take the action sequence R, R, R, R (it reaches the goal in four steps). Because Q-learning's update rule does not depend on the exploration policy that generated the data, tracing this fixed sequence is enough to see the core learning behavior. Episode 1, updating Q(s, R) in order:

step 1: s=0→1, r=-1, max(Q[1,·])=0
        Q(0,R) = 0 + 0.5·(-1 + 0.9·0 - 0)        = -0.500
step 2: s=1→2, r=-1, max(Q[2,·])=0
        Q(1,R) = 0 + 0.5·(-1 + 0.9·0 - 0)        = -0.500
step 3: s=2→3, r=-1, max(Q[3,·])=0
        Q(2,R) = 0 + 0.5·(-1 + 0.9·0 - 0)        = -0.500
step 4: s=3→4 (goal), r=+10, terminal so no bootstrap
        Q(3,R) = 0 + 0.5·(10 + 0.9·0 - 0)        =  5.000

Only Q(3, R) — the cell adjacent to the goal — picked up a large positive value. The other three updates only registered the local −1 step cost, because at the time they were computed, nothing downstream had any value yet to bootstrap from. This is the seed of the misconception addressed below. Episode 2 replays the same actions, but now Q(3, R) = 5.0 is available to bootstrap from at step 3:

step 1: max(Q[1,·])=max(0,-0.5)=0
        Q(0,R) = -0.5 + 0.5·(-1+0.9·0-(-0.5))    = -0.750
step 2: max(Q[2,·])=max(0,-0.5)=0
        Q(1,R) = -0.5 + 0.5·(-1+0.9·0-(-0.5))    = -0.750
step 3: max(Q[3,·])=max(0,5.0)=5.0
        Q(2,R) = -0.5 + 0.5·(-1+0.9·5.0-(-0.5))  =  1.500
step 4: terminal, r=+10
        Q(3,R) = 5.0 + 0.5·(10-5.0)               =  7.500

Q(2, R) jumped from −0.5 to +1.5 — the goal's value has propagated back one more cell. Episode 3 repeats the pattern:

step 1: max(Q[1,·])=max(0,-0.75)=0
        Q(0,R) = -0.75 + 0.5·(-1+0.9·0-(-0.75))  = -0.875
step 2: max(Q[2,·])=max(0,1.5)=1.5
        Q(1,R) = -0.75 + 0.5·(-1+0.9·1.5-(-0.75)) = -0.200
step 3: max(Q[3,·])=max(0,7.5)=7.5
        Q(2,R) = 1.5 + 0.5·(-1+0.9·7.5-1.5)       =  3.625
step 4: terminal, r=+10
        Q(3,R) = 7.5 + 0.5·(10-7.5)                =  8.750

A short Python implementation of exactly this loop, checked line by line against the arithmetic above, confirms the same numbers:

states, actions = [0, 1, 2, 3, 4], ['L', 'R']
Q = {(s, a): 0.0 for s in states for a in actions}
alpha, gamma, goal = 0.5, 0.9, 4

def step(s, a):
    s_next = min(s + 1, 4) if a == 'R' else max(s - 1, 0)
    if s_next == goal:
        return s_next, 10.0, True
    return s_next, -1.0, False

for ep in range(3):
    s = 0
    for _ in range(4):
        s_next, r, done = step(s, 'R')
        max_next = 0.0 if done else max(Q[(s_next, 'L')], Q[(s_next, 'R')])
        Q[(s, 'R')] += alpha * (r + gamma * max_next - Q[(s, 'R')])
        s = s_next
        if done:
            break
    print(ep + 1, round(Q[(0,'R')],3), round(Q[(1,'R')],3),
          round(Q[(2,'R')],3), round(Q[(3,'R')],3))

Because the corridor's dynamics are known and deterministic, the true optimal Q* can also be solved exactly by hand, as a check on where the table above is heading. Working backward from the goal with the Bellman equation: Q*(3,R) = 10 (one step from the goal, nothing to discount). Q*(2,R) = −1 + 0.9 × 10 = 8.0. Q*(1,R) = −1 + 0.9 × 8.0 = 6.2. Q*(0,R) = −1 + 0.9 × 6.2 = 4.58. As a cross-check, the direct discounted sum of the whole four-step trajectory gives the same answer: −1 − 0.9 − 0.81 + 0.9³×10 = −1 − 0.9 − 0.81 + 7.29 = 4.58.

Every visited cell is climbing steadily toward its analytical optimum, one episode's worth of backward propagation at a time — Q(2,R) has already closed more than half the gap by episode 3, while Q(0,R), three steps further from the goal, is still climbing out of its initial negative dip. That gap is the direct, provable consequence of the update rule, not a coincidence: with enough episodes and continued visits to every state-action pair, the Q-learning update is a contraction mapping toward the Bellman fixed point, which is why Q-learning is guaranteed to converge to Q* under these conditions.

Exploration versus exploitation

The trace above fixed the action sequence to isolate the update arithmetic, but a real learning robot must also decide which actions to try. If it always greedily picks argmaxₐ Q(s, a), it can get permanently stuck exploiting an early, mediocre estimate — notice that at cell 1, Left currently has an untested (and misleadingly optimistic) value of 0, versus Right's true-but-still-converging 6.2. The standard fix is ε-greedy action selection: with probability ε take a uniformly random action (explore), otherwise take the current best action (exploit):

if random.random() < epsilon:
    action = random.choice(actions)
else:
    action = max(actions, key=lambda a: Q[(state, a)])

ε typically starts high (e.g. 1.0, pure exploration) and decays toward a small value as training progresses, so the robot spends early episodes mapping out the consequences of every action and later episodes mostly executing what it has learned. On a physical robot, unrestricted exploration is dangerous — a random action might drive an arm into a wall — which is one reason production robotics systems train the decision policy almost entirely in simulation (MuJoCo, NVIDIA Isaac Gym, PyBullet) at massive parallel scale, then transfer the learned policy to the real machine. Because a simulator is never a perfect physical model, engineers apply domain randomization during training — randomizing friction, mass, sensor noise, and latency across simulated episodes — so the policy that emerges is robust to the mismatch between simulation and reality rather than overfit to one exact (and wrong) physics model.

From tables to networks: continuous state and action spaces

The corridor example used a table with 5 states × 2 actions = 10 entries, small enough to store and update exactly. A real robot arm's state includes several continuous joint angles and velocities, and a mobile robot's state includes continuous position, orientation, and possibly a full lidar scan — there is no way to enumerate every state in a table. Modern robotics RL replaces the table Q(s, a) with a neural network Qθ(s, a) whose weights θ are trained by gradient descent to make the TD error small on average across sampled experience — this is the core idea behind Deep Q-Networks (DQN) for discrete actions, and behind policy-gradient methods like PPO for the continuous torque/velocity commands a real robot actually needs. The Bellman equation and the TD-error update rule from this chapter are unchanged; only the representation of Q changes, from a lookup table to a differentiable function approximator. This is exactly the bridge between classical reinforcement learning and the deep learning machinery covered elsewhere in this curriculum: the target r + γ·maxₐ′Q(s′,a′) becomes a regression label, and the network is trained to predict it.

Common misconception

A frequent misreading of episode 1 above is: "the robot reached the goal, so it has learned the correct policy." Look again at the actual numbers after episode 1: Q(3, R) jumped to 5.0, but Q(0, R), Q(1, R), and Q(2, R) each only moved to −0.5 — they still look worse than the untried, zero-valued Left action at their own state. A single successful trajectory only proves the environment is solvable and assigns a large value to the one state-action pair immediately adjacent to the reward. Every state further back has to wait for that value to arrive through repeated bootstrapped updates — the table shows it taking three full episodes for cell 2 to even turn positive, and cell 0 is still negative after three episodes despite Right being unambiguously optimal there. Reaching the goal once is necessary for learning to begin; it is nowhere close to sufficient for the policy to be correct at every state, which is precisely why RL training runs for thousands to millions of episodes rather than stopping at the first success.

Active recall

Attempt each question before reading its answer.

1. Using the episode-3 Q-table (Q(1,R) = −0.200, Q(2,R) = 3.625), compute Q(1,R) after one more update in episode 4, assuming the robot again takes R at cell 1 and Q(2,R) has not changed since episode 3.
2. Why does Q(0,R) remain negative for several episodes even though "always move right" is clearly the optimal policy in this corridor?
3. In the PID worked example, raising Kd from 0.02 to 0.5 dropped the command from 29.75 to 5.75. Explain physically what changes about the arm's motion, and name one risk of setting Kd too high.
4. Why can't a robot arm's low-level joint controller simply be replaced by a tabular Q-learning table running at the same 1 kHz rate as the PID loop?
5. Write the ε-greedy selection rule in words, and explain why ε is typically decayed over training rather than held constant.
6. A robot reaches the delivery point on its very first training episode. Has it learned the optimal policy? Justify your answer using the episode-1 Q-values above.

Answers.

1. max(Q[2,·]) = max(0, 3.625) = 3.625. Target = −1 + 0.9×3.625 = 2.2625. Q(1,R) = −0.200 + 0.5×(2.2625 − (−0.200)) = −0.200 + 0.5×2.4625 = −0.200 + 1.231 = 1.031. Note it has now crossed into positive territory.
2. Because the TD update only propagates value one bootstrapped step per episode along this fixed trajectory, and at each intermediate cell the max operator initially favors the untried, optimistically-zero Left action over the explored, currently-negative Right action. Q(0,R) only turns positive once Q(1,R) is large enough that 0.9×Q(1,R) exceeds the 1-unit step cost — which, per question 1, first happens during episode 5 (Q(1,R) reaches 2.3781, crossing the 1.111 threshold), so that value first feeds Q(0,R)'s own update in episode 6, where Q(0,R) turns positive (+0.3178).
3. A larger Kd more aggressively brakes the command as the error closes, damping the approach and reducing overshoot and oscillation around the setpoint — but it also makes the arm sluggish if pushed too far, and because a numerical derivative amplifies high-frequency noise, an overly large Kd can make the motor command jittery in response to ordinary sensor noise rather than to genuine motion.
4. The arm's state (joint angles and velocities) is continuous, so there are infinitely many states — a table cannot enumerate them. Low-level joint control also has a well-understood physical model (torque versus acceleration via the arm's mass and geometry) and needs a response every millisecond, which favors a fast, fixed-formula PID law over an approximate learned value; RL is reserved for the slower, higher-level decision layer where the environment (obstacles, paths, task goals) is not easily captured by a fixed formula, and continuous-state RL there uses a neural network function approximator (DQN/PPO-style), not a table.
5. With probability ε, pick an action uniformly at random (explore); otherwise, pick the action with the highest current Q-value (exploit). ε is decayed from a high value toward a low one because early in training the Q-estimates are unreliable and the agent needs broad experience to correct them, while late in training the estimates are more trustworthy and continued heavy exploration would waste episodes on actions already known to be worse.
6. No. After episode 1, only Q(3,R) — the state-action pair immediately before the goal — received a large update (5.0); Q(0,R), Q(1,R), and Q(2,R) each only reflect the local −1 step cost and are still below their same-state Left action's initial value of 0. One success confirms the task is solvable but has not yet propagated the goal's value back to the states furthest from it, so the policy at those states is not yet correct.

Think About It

Think about this: How would you explain robotics control and reinforcement learning to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Practice Exercises

Now it is time to practice! Complete these challenges to solidify your understanding:

• Exercise 1: Write a short program that demonstrates the core concept from this chapter. Test it with at least 3 different inputs.
• Exercise 2: Find a real-world example where robotics control and reinforcement learning is used in an Indian company (like TCS, Infosys, Flipkart, or ISRO). Write a paragraph explaining the connection.
• Exercise 3: Create a mind-map connecting robotics control and reinforcement learning to at least 3 other topics you have studied.