Why Do Humanoid Robots Still Struggle With the Small Stuff?

That fluid gait, for example, comes from deep reinforcement learning. Roboticists once coordinated each movement with various hand-engineered algorithms, using equations to model the (simplified) physics of the robot. Now they train neural networks to act as “whole-body controllers” by running countless digital simulations of the humanoid. This process teaches the network a “policy” for how to translate feedback from its environment into actions.

“We use reinforcement learning to build a policy that’s handling the body coordination, collision avoidance, balance, all that stuff,” Kuindersma said. There’s no longer any need to model a robot’s leg as a linear inverted pendulum, for example. “That’s just gone by the wayside,” he said.

This strategy was aided by the proprioceptive actuators pioneered by Sangbae Kim of the Massachusetts Institute of Technology in his Cheetah series of robots. “Reinforcement learning has existed for a long time, you know. People tried it before,” Kim said. “But if you use conventional [motors], the robot just breaks” every time it fails to perfectly execute a policy in the real world — or encounters an obstacle or disturbance.

Kim’s actuators got around the problem with controllable “compliance,” or flexible springiness. Over the past decade, they’ve gotten cheaper and more widely accessible. “Reinforcement learning solved a lot of the [bipedal] locomotion problem, but the hardware was the enabler,” Kim said.

If reinforcement learning and compliant actuation were gifts to humanoid robotics, multimodal AI put a bow on it. In 2023, Google DeepMind introduced “vision-language-action” (VLA) models, which can take in video and natural language and produce movement commands as outputs.

“If you say ‘I’m thirsty,’ it knows you probably want to drink, and it can [generate] the steps that [the robot] needs to take: Go find a thing, and then pick it up in this way,” said Carolina Parada, head of robotics at Google DeepMind. “This is something that, before three years ago, you would have to go hard-code.” In a stroke, VLAs united previously disparate approaches to robotic perception, planning, and control into one general-purpose pipeline.

Robust embodiment, check. Generalizable intelligence, check. (A start, anyway.) So why don’t they add up to humanoids being scientifically “solved” — at least in principle?

May the Force Be With You

Pulkit Agrawal, who studies robot learning at the appropriately named Improbable AI Lab at MIT, had an answer when I reached him there last month. “To have robots which work like humans,” he said, “I think we have to master physics.”

He wasn’t referring to cosmic matters like general relativity or quantum gravity, nor to the virtual “world models” that currently excite leading AI researchers such as Yann LeCun. Instead, Agrawal is talking about mastering something a high school science student ought to be familiar with: force and inertia.