The rise of AI models is revolutionizing how humanoid robots are trained. Data collection, simulation, and new learning models are paving the way for machines capable of generalizing their knowledge to new situations.
Traditionally, robots were programmed to execute each movement, predetermined by code. This method worked well in highly controlled environments like factories or warehouses but proved too limited in unpredictable situations. Recent advancements in artificial intelligence and the emergence of foundational models have changed the game. Rather than solely following predefined rules, humanoid robots are now trained based on data. By observing human actions, they can replicate gestures, identify recurring patterns, and attempt to generalize their knowledge to new situations.
“We have shifted from a logic of programming behaviors to an approach where these behaviors are learned from data. This is the only way to scale,” summarizes Deepak Pathak, co-founder and CEO of Skild AI, an American start-up developing a model known as a “generalist brain for robots.”
Robots primarily learn from three types of data: robotic data (highly precise but challenging to collect on a large scale), video data (abundant but less rich in physical interaction information such as forces or object contacts), and data generated in simulated environments, which suffer from a gap with the real world (“Sim-to-real gap”).
Various methods exist for collecting this data. The simplest is learning through observation: the robot watches a human performing tasks, records movements and gestures using its cameras and sensors, and replicates them later. AI models can then identify recurring patterns. For instance, if hundreds of demonstrations show how to grip a cup at different locations, angles, and lighting, the robot can generalize to learn how to grab a cylindrical object.
However, the most common method is teleoperation. A human equipped with a remote control or VR headset controls the robot’s movements for it to memorize. The operator may use haptic gloves and motion sensors to gather more data, capturing detailed information like joint angles and applied forces. Major humanoid manufacturers employ this training method. 1X, launching the domestic robot NEO, will even offer a remote teleoperation service. An employee can take control of the humanoid to teach it household tasks at the client’s home.
Data collection via teleoperation has become a thriving industry, especially in China, where specialized centers employ operators to perform repetitive tasks to feed learning models for robots. Though more effective than programming, these approaches are time-consuming and demanding in human labor. To overcome these limitations, new methods have emerged, combining multiple data types, including video, and designed to help AI models understand physics laws.
Models like Video-Language-Action (VLA), such as GR00T N1 at NVIDIA, Gemini Robotics at DeepMind, or Helix at Figure AI, feed on images and text instructions to produce motor action sequences executable by robots. Skild AI implements a logic used in large language models: pre-training on vast data volumes followed by fine-tuning with specific real-world data.
Rhoda AI takes a different approach with its “Direct Video-Action” (DVA) model, allowing robots to learn directly from a high-performing video model to enhance their real-world action skills.
Another popular model is the World Models, enabling robots to grasp how the physical world operates and anticipate the consequences of their actions. Coupled with simulated environments, they enable robots to conduct millions of trials before real-world deployment.
By aggregating various data types for learning and training in simulation, these methods offer a solution for robots to comprehend their surroundings better. Humanoid robotics industry players envision this as a key step toward widespread deployment of their models.
About the Author
