RL for Robotics: Sim-to-Real Transfer

In 2019, a team at OpenAI trained a Shadow Dexterous Hand — a five-fingered robot hand with 24 actuated degrees of freedom — to solve a Rubik's Cube one-handed. The policy that ran on the real hand was never trained on the real hand. Every gradient update came from a physics simulator. The real cube-solving session was the first time the policy had ever touched real silicone tendons, real motor backlash, or real cube friction. It worked anyway — not perfectly, but well enough to complete scrambles, recover from a researcher nudging the cube mid-solve, and keep going after a glove was thrown over two of its fingers. Nothing in training had looked exactly like a glove. The question this chapter answers is: how does a policy trained entirely inside a simulated world act competently in a physical one it has never seen? The answer is not "make the simulator perfect." It is a specific, learnable idea called sim-to-real transfer, and the workhorse technique behind it is domain randomization.

Why robot RL has to happen in simulation first

Recall the reinforcement learning loop: an agent observes state s, takes action a under policy πθ(a|s), the environment returns a reward r and next state s' according to transition dynamics P(s'|s,a), and training pushes θ to maximize expected discounted return. A modern policy-gradient algorithm such as PPO typically needs tens of millions of environment steps to converge on even a moderately complex continuous-control task. A simulator can produce those steps faster than real time and in parallel — thousands of copies running simultaneously on a GPU cluster, each one a few milliseconds of wall-clock time per episode.

A physical robot cannot do any of that. It runs at exactly the speed of physics: one second of robot time costs one second of wall-clock time, there is exactly one of it (or a handful, at enormous capital cost), and a bad exploratory action doesn't just produce a low reward — it can snap a $50,000 hydraulic actuator, strip a gearbox, or send a legged robot into a fall that damages sensors bolted to its frame. Training tens of millions of steps of raw exploratory RL directly on hardware is not merely slow; for most robots it is not survivable. Simulation is the only place where an RL agent can fail catastrophically for free, millions of times, until it stops failing.

That solves the sample-efficiency and safety problem — at the cost of introducing a new one. The simulator's transition function P̂(s'|s,a) is a model, and every model is wrong somewhere. The gap between what the simulator computes and what the real robot actually does is called the reality gap, and a policy optimized to exploit the specific dynamics of P̂ has no guarantee of working under the true P.

Where the reality gap actually comes from

It is tempting to think of the reality gap as one fuzzy source of "noise." It is not — it is a specific, enumerable list of places where physics simulators (MuJoCo, Isaac Gym, PyBullet, Bullet) systematically diverge from hardware:

Contact and friction. Rigid-body simulators approximate contact forces with simplified friction cones and restitution coefficients. Real contact between a gripper pad and an object involves micro-deformation, stick-slip transitions, and surface wear that no closed-form contact model captures exactly. Friction coefficients are also usually guessed, not measured, because measuring them precisely for every surface a robot might touch is impractical.

Actuation. A simulated motor applies the commanded torque instantly. A real motor has current-control latency, gear backlash, cable stretch, and torque ripple. Commanded 5 N·m and delivered 5 N·m are not the same thing on real hardware, and the discrepancy is worse near zero velocity (where static friction dominates) than at speed.

Sensing. Simulated state observations are exact and instantaneous. Real encoders have quantization noise, real IMUs drift, real depth cameras have flying pixels at object edges, and every sensor reading arrives after some non-zero, sometimes variable, latency.

Mass and inertia. CAD-derived mass and inertia estimates for a robot's links are close but not exact — cabling, connectors, and paint are usually left out.

Vision. If the policy consumes camera images, rendered simulation frames differ from real camera frames in lighting, texture, lens distortion, and sensor noise — a separate "visual domain gap" on top of the dynamics gap.

Every one of these is individually small. The mistake is assuming small individual errors stay small once they run through a physical process. The next section derives, with exact numbers, why they don't.

Worked example: a 50% friction error becomes a 50% distance error

Consider the simplest possible robotics task: a cart of mass m = 2 kg must slide across a floor and stop at a target d = 1.0 m away. The controller is a policy learned in simulation: apply a fixed push force F = 6 N for a duration t_push, then release. The cart decelerates under kinetic friction alone until it stops. This is an open-loop policy — it decides the entire action sequence in advance from the model, without watching the cart's actual position as it moves.

During the push, Newton's second law along the direction of motion gives acceleration a_push = F/m − μg (applied force minus friction resisting motion). After release, only friction acts, giving deceleration a_decel = μg. The simulator's best guess for the floor is μ_sim = 0.10. With g = 9.8 m/s²:

a_push = 6/2 − 0.10×9.8 = 3 − 0.98 = 2.02 m/s²
a_decel = 0.10×9.8 = 0.98 m/s²

Distance during the push phase is d₁ = ½·a_push·t², exit velocity is v = a_push·t, and the coasting distance after release is d₂ = v²/(2·a_decel). Setting d₁ + d₂ = 1.0 m and solving for t_push:

1.0 = a_push·t² · (0.5 + a_push/(2·a_decel))
1.0 = 2.02·t² · (0.5 + 2.02/1.96)
1.0 = 2.02·t² · 1.5306
t² = 1.0 / 3.0918 = 0.32343
t_push = 0.5687 s

The simulator-tuned policy is now fixed: push for exactly 0.5687 seconds, then stop pushing. This is the policy that gets deployed on the real robot. But real floors are rarely as slippery as the simulator's guess — measured surface friction usually comes in higher than a default simulator value, say μ_real = 0.15. Re-running the identical action sequence (same F, same t_push = 0.5687 s) under the real friction:

a_push_real = 3 − 0.15×9.8 = 1.53 m/s²
a_decel_real = 0.15×9.8 = 1.47 m/s²
v_real = 1.53 × 0.5687 = 0.8701 m/s
d1_real = 0.5 × 1.53 × 0.5687² = 0.2474 m
d2_real = 0.8701² / (2×1.47) = 0.2575 m
d_total_real = 0.2474 + 0.2575 = 0.5049 m

A 50% relative error in one physical parameter — friction going from 0.10 to 0.15 — did not produce a 50% error in the outcome by coincidence. It compounded through two multiplicative stages of the same trajectory (a slower push phase and a stronger deceleration phase both work against the cart), so the real robot lands at 0.505 m instead of 1.0 m: it stops barely halfway to the target, an almost total task failure, from one parameter that was off by less than most engineers would call "badly modeled." This is the reality gap in miniature, and it is exactly the kind of failure that made early sim-trained robot policies unreliable in the lab.

Domain randomization: stop guessing one number, train on a distribution

The fix is not to model friction more precisely. Chasing a perfectly accurate simulator is a losing game — contact dynamics, cable drag, and motor wear are expensive or impossible to characterize exactly, and even a very good simulator still leaves a nonzero, unknown gap. Domain randomization (DR) takes the opposite approach: since the exact real-world parameters are unknown, don't train on one guess — train on a wide distribution of guesses, and let the policy find behavior that works across all of them. Formally, standard RL optimizes

maximize_θ  E_π[ Σ γ^t r_t | M ]

for one fixed MDP M. Domain-randomized RL instead samples a randomization vector ξ — friction, mass, latency, motor strength, whatever is uncertain — fresh at the start of every episode from a distribution p(ξ), and optimizes

maximize_θ  E_ξ~p(ξ) E_π[ Σ γ^t r_t | M(ξ) ]

The real robot is then treated as just one more, unseen draw of ξ. If the training distribution p(ξ) is wide enough to contain the real robot's true parameters somewhere inside it, the policy has already had to succeed on dynamics close to that value during training — it never gets to overfit to one exact friction coefficient the way the open-loop cart policy above did.

Here is a minimal training-loop sketch showing what actually changes per episode. The parameters are sampled fresh each time and handed to the simulator's reset call before the episode runs:

import random
random.seed(42)

def randomize_dynamics():
    mu = random.uniform(0.05, 0.20)       # friction coefficient
    mass = random.uniform(1.8, 2.2)       # kg, cart mass
    latency = random.uniform(0.0, 0.02)   # sensor latency, seconds
    return mu, mass, latency

for episode in range(3):
    mu, mass, latency = randomize_dynamics()
    print(f"episode {episode}: mu={mu:.3f} mass={mass:.3f} latency={latency:.4f}")

episode 0: mu=0.146 mass=1.810 latency=0.0055
episode 1: mu=0.083 mass=2.095 latency=0.0135
episode 2: mu=0.184 mass=1.835 latency=0.0084

Notice that the friction range chosen, U(0.05, 0.20), spans both the simulator's original guess (0.10) and the real floor's true value (0.15) from the worked example. A policy trained across this range cannot rely on a single precomputed timing schedule, because the correct t_push is a different number for every episode. The only way to get consistently close to the target across the whole distribution is to stop being open-loop: the policy has to observe the cart's actual position and velocity as it moves and adjust force in real time — something closer to F(t) = k_p·(d_target − x) − k_d·v, a feedback controller shaped by RL rather than a fixed timer. Domain randomization doesn't just make one number more forgiving; it structurally forces the emergence of closed-loop behavior, which is what actually transfers.

In practice, robotics teams randomize far more than friction: link masses and inertias, motor strength and gear backlash, joint damping, sensor noise and latency, and — for vision-based policies — textures, lighting, and camera pose, closing the separate visual domain gap. OpenAI's Rubik's Cube result specifically used Automatic Domain Randomization (ADR), which starts training with a narrow range around a nominal simulator value and automatically widens it over training as the policy keeps succeeding, so the final policy has been exposed to a much broader range of physics than any hand-picked schedule would have set.

Other transfer techniques that complement domain randomization

System identification (SysID) runs the opposite direction: instead of randomizing broadly, measure the real robot's actual parameters (drop-test friction, load-test motor torque curves) and calibrate the simulator to match them as closely as possible, then randomize a narrower band around that calibrated point. SysID and DR are usually combined, not chosen between — calibrate what you can measure, randomize what you can't.

Teacher-student distillation is standard in legged-robot RL, including ETH Zürich's massively parallel ANYmal training pipeline, which trains thousands of randomized robot instances in parallel on a GPU physics engine. A "teacher" policy is trained in simulation with privileged information no real sensor can provide — exact terrain height maps, ground-truth contact forces, the true friction coefficient of the current episode. Once the teacher is competent, a "student" policy is trained by imitation to reproduce the teacher's behavior using only observations the real robot will actually have at deployment: joint encoders, an IMU, maybe a depth camera. The student, not the teacher, is what ships to hardware.

Careful real-world fine-tuning uses a small number of real rollouts — sometimes just dozens — to nudge a domain-randomized policy the last mile, using conservative, safety-constrained updates rather than raw exploratory RL from scratch.

The misconception this chapter needs to correct

The instinctive assumption is: if the simulator is accurate enough, the policy trained in it will transfer directly — the problem is simulator fidelity, and better physics engines eventually solve sim-to-real. This is wrong, and it is wrong in a specific, demonstrable way. The worked example above used a friction error of only 0.05 — a simulator most engineers would call "reasonably accurate" — and still lost half the task's precision through a single open-loop policy. No affordable increase in simulator fidelity closes every one of the gap sources listed earlier simultaneously (contact micro-mechanics, cable stretch, sensor latency jitter, paint mass) — and chasing that is not where the actual research advantage came from. OpenAI's own dexterous-hand results did not come from a more precise simulation of the Shadow Hand; they came from deliberately widening the range of physical parameters the policy had to survive during training. Robustness across a distribution of plausible dynamics is a different, and empirically more effective, target than point-accuracy of any single dynamics model. A student who understands sim-to-real as "make the simulator good enough" will keep hitting the same wall the field already hit and moved past.

Active recall

Attempt each question before reading its answer.

1. Why can't a legged robot simply be trained with raw RL directly on hardware, the way an agent is trained to play an Atari game on the actual console?

2. A cart policy is trained assuming friction μ = 0.12, then deployed on a floor with μ = 0.09 (smoother than the simulator assumed). Does the cart overshoot or undershoot the 1.0 m target, and roughly by how much?

3. Write the optimization objective for standard RL versus domain-randomized RL, and explain in one sentence what changes.

4. A grasping policy is domain-randomized over object mass and friction but never over camera lighting or exposure. On the real robot it grasps the wrong point on the object even though the physical grip itself is fine. What failure mode is this, and what's the fix?

5. OpenAI's cube-solving hand was trained without a highly precise model of the Shadow Hand's tendons and motors. Why did it still transfer, given the "accurate simulator" misconception this chapter corrects?

6. In teacher-student distillation for legged robots, what information does the teacher policy get in simulation that the student policy deployed on hardware does not, and why is that split necessary?

Worked answers

1. RL needs tens of millions of environment steps to converge. A simulator can generate those in parallel, faster than real time. A physical robot can only run at one second per second of wall-clock time, there are very few copies of it, and it is fragile — random exploratory actions during early training can damage actuators or cause falls that harm the hardware. Atari runs in an emulator at effectively unlimited speed and cannot be physically damaged; a robot cannot offer either property.

2. It overshoots. Trained on μ = 0.12: a_push = 3 − 9.8(0.12) = 1.824 m/s², a_decel = 1.176 m/s², solving 1.0 = a_push·t²(0.5 + a_push/(2a_decel)) gives t_push ≈ 0.6556 s. Deployed on μ = 0.09: a_push_real = 3 − 9.8(0.09) = 2.118 m/s², a_decel_real = 0.882 m/s², v_real = 2.118 × 0.6556 = 1.389 m/s, d1 = 0.455 m, d2 = 1.389²/(2×0.882) = 1.093 m, total ≈ 1.548 m — an overshoot of about 55%. Lower real friction means both a stronger push-phase acceleration and weaker deceleration than the model assumed, so the cart travels farther in both phases at once.

3. Standard RL: maximize_θ E_π[Σγ^t r_t | M] for one fixed MDP M. Domain-randomized RL: maximize_θ E_ξ~p(ξ) E_π[Σγ^t r_t | M(ξ)], an expectation over a distribution of MDPs. The change is that the policy is scored on its average performance across many plausible physical worlds, not its performance on one fixed model, so it can no longer overfit to that one model's exact numbers.

4. This is the visual domain gap, a separate failure mode from the dynamics gap — the perception module was never shown lighting or exposure variation, so its estimate of where the object is (not how the gripper should move once it's there) is wrong under real camera conditions. The fix is visual domain randomization (randomizing textures, lighting, and camera parameters during rendering) or a learned image-to-image domain adaptation step such as a sim-to-real GAN, alongside the dynamics randomization already in place.

5. Because transfer in that result came from Automatic Domain Randomization progressively widening the range of physical parameters — friction, gravity, hand geometry, action delay — that the policy had to remain competent across, not from precisely modeling the hand's tendons and motors. The real hand was simply one more sample from an already very wide training distribution, which is the domain randomization principle working exactly as intended, not evidence that the simulator happened to be unusually accurate.

6. The teacher gets privileged, ground-truth information that only exists inside the simulator and cannot be measured on real hardware in real time — exact terrain height maps, true ground-contact forces, the exact friction coefficient sampled for that episode. The student is trained afterward, by imitating the teacher's behavior, using only the observations the real robot will actually have available at deployment: joint encoder readings, IMU data, perhaps an onboard depth camera. The split is necessary because the real robot has no way to directly sense terrain height maps or true contact forces — only the student's realistic observation set is ever going to exist on the physical machine.

Think About It

Think about this: How would you explain rl for robotics: sim-to-real transfer to a friend who has never seen a computer? What real-world analogy would you use? Imagine you had to build a system using these concepts — what would be your first step? Try this: before moving on, write down three things you learned and one question you still have.

Key Takeaways — Summary and Recap

Let us recap what we covered: the core ideas behind rl for robotics: sim-to-real transfer, how they connect to real-world applications, and why they matter for your journey in computer science. Remember these key points as you move forward. For competitive exam preparation (CBSE, JEE, BITSAT), focus on understanding the WHY behind each concept, not just the WHAT.