Peter Chen and the education of a robot

At a Berkeley party, Peter Chen noticed someone sitting on a bouncy ball, working on a laptop. Rocky Duan had apparently found a perfectly serviceable study space in the middle of a social occasion. Chen went over to meet him. It was an encounter with a considerable afterlife: the two would become friends, researchers and company founders.

Their adviser, Pieter Abbeel, tells the story with another useful detail. Chen initially resisted artificial intelligence. In 2014, he thought it did not work. Duan wanted to pursue it anyway. Within months, Chen had reconsidered and volunteered in Abbeel’s laboratory. The future robotics entrepreneur began with an objection, then changed his mind. In a field fond of declarations about the future, that is an appealing qualification.

The volunteer and the learning machine

Chen’s old academic website preserves an earlier version of his ambition: “I’m building teachable robots.” The company named there is Embodied Intelligence, the original name of Covariant. He describes himself as a Berkeley doctoral student and a former OpenAI research scientist, interested in generative models and reinforcement learning. Those two interests would keep returning to the same practical question: how does a machine understand its surroundings well enough to do something useful?

The publication list supplies some clues. Chen worked with researchers including John Schulman, Ilya Sutskever and Diederik Kingma. He also co-organized a deep reinforcement learning workshop at NIPS in 2016. This was a career being assembled through collaborations and experiments, before the CEO title became its most convenient shorthand. The recurring concern was learning: extracting structure from observations, exploring possibilities and improving decisions.

Consider InfoGAN, a 2016 paper with Chen as first author. Its experiments explored whether a generative model could discover interpretable features without being explicitly taught each one. On images of handwritten digits, the system separated writing style from digit shape. Other experiments distinguished features such as pose and lighting. The attraction is easy to understand: a useful machine needs ways to organize a complicated world, rather than merely memorize its pictures.

PixelSNAIL, another paper led by Chen, approached image modeling through an architecture combining causal convolutions and self-attention. The technical vocabulary is dense, but the continuity matters. Before asking industrial customers to trust his software, Chen was working on how models learn relationships inside data. His later enthusiasm for foundation models had roots in this research. It arrived with a bibliography.

A warehouse is an exacting classroom

In 2017, Chen, Abbeel, Duan and Tianhao Zhang founded the business that became Covariant. Chen took the chief executive role. The setting they chose was a place where intelligence could be judged with unusual directness: a warehouse. An item either reaches the right destination or it does not. A successful presentation has limited influence over a parcel.

Covariant’s early work concentrated on pick-and-place tasks. Its software, the Covariant Brain, was intended to let robotic systems perceive objects and decide how to handle them. Over the following years, the company extended its offering into induction, depalletization and kitting. These are different operations with a shared nuisance: the world keeps presenting objects in arrangements that the engineer cannot conveniently script in advance.

The distinction helps explain Chen’s choice of workplace. A robot following a fixed sequence can be precise and useful. Change what it encounters, however, and the sequence may cease to be appropriate. An online order introduces variety: different shapes, packaging and positions. Teaching a system to cope with that variety is a research problem wearing a shipping label. The cardboard box makes a demanding examiner.

In February 2020, ABB announced a partnership with Covariant to develop integrated AI robotics, beginning with warehouse order fulfillment. ABB said Covariant had won its global competition to find an AI partner. An initial installation was being deployed at Active Ants in Utrecht, in the Netherlands. For a young software company, the relationship connected learning algorithms to an established industrial robotics supplier.

“If you want to advance artificial intelligence, we now need to take it out of the laboratory and apply it to the real world.”Peter Chen, February 2020

The partnership also made Chen’s argument unusually concrete. His company could bring its learning technology; ABB brought experience deploying industrial automation. Getting those strengths to work together required attention to the whole operation. The algorithm was part of a system that had to function where customers already worked. The laboratory door opened onto a loading dock.

The customer also supplies the lesson

Chen’s explanation for founding a company was more complicated than taking a finished invention to market. In his January 2024 conversation with Sarah Guo, he described a missing ingredient: enough physical interaction data to train broadly capable robot models. Commercial deployments offered a way to collect that experience while doing work customers would pay for. Revenue and research could share a task.

This made the sequence of progress important. Build something useful enough to deploy. Learn from its operation. Improve the model. Earn the opportunity to handle more work. Chen described an incremental route toward a much larger ambition. The warehouse was both the first market and a place to accumulate experience. Each expansion had to make commercial sense while creating the conditions for the next one.

A May 2023 partnership with Otto Group put that thesis on an operational map. The companies announced plans to begin deploying more than a hundred robots, starting at facilities in Haldensleben and Altenkunstadt in Germany. Their longer-term vision covered hundreds of robotic solutions across Otto Group’s fulfillment centers. These were deployment plans, with named places and a customer attached.

The announcement described robots learning together as a fleet, with improvements spreading through the network. It also reported Covariant customers in 15 countries and nearly 300 robots powered by its software at that time. Those figures offer a useful historical snapshot. Chen’s project had acquired a geography. A decision about grasping in one facility could contribute to a system serving other facilities.

There is an entrepreneurial temperament visible in that choice of arrangement. Chen tied an expansive research ambition to ordinary customer work. The approach made shipping, installation and reliability relevant to the science. It also made the company’s progress dependent on customers continuing to find the systems useful. A warehouse provides plenty of feedback, and very little courtesy applause.

Giving the arm a conversation

On March 11, 2024, Covariant announced RFM-1, its Robotics Foundation Model. The launch brought language and physical action into the same product story. The company described a model intended to help robots deal with unfamiliar situations, with demonstrations offered at its Emeryville headquarters and the MODEX trade event in Atlanta. Chen’s long-running interest in learning from varied data had become a public demonstration.

The underlying idea was to bring together information that robotics often treats separately. Text can describe a task. Images show the scene. Sensor readings and action records describe what a machine did and what happened. Learning across these forms of information offers a route toward connecting instructions with physical consequences. An instruction acquires a destination beyond the screen.

The attraction of such a system is also a reminder of the difficulty. Understanding a request and executing it reliably are separate accomplishments. A robot must contend with the exact object in front of it, its position and the consequences of contact. The charm of a conversational interface does not relieve the arm of those obligations. Gravity remains an uncooperative conversational partner.

The next workplace, with familiar colleagues

In August 2024, Amazon announced an agreement licensing Covariant’s robotic foundation models on a non-exclusive basis. Chen, Abbeel and Duan would join its Fulfillment Technologies & Robotics team, along with a group amounting to roughly a quarter of Covariant’s employees. Amazon said Covariant would continue serving its customers. The announcement described a technology license and a movement of people, rather than a purchase of the entire company.

For Chen, the move placed his work inside an organization already operating a large fleet of industrial robots. Amazon described its interest in generalizing how robotic systems learn and applying the technology within its operations. The destination suited the question that had animated Covariant: how to make learning systems work amid the variety of actual commerce. The scale of the workplace changed.

Chen now works in Amazon’s Frontier AI & Robotics organization. Its recruiting materials name him alongside Abbeel and Duan as leaders of the team. In a public recruiting post, Chen asked whether readers were tired of robot demonstrations that were variations on overfitting, and invited researchers to work on generalization in real operations. It is an unusually pointed job advertisement. A polished demonstration, in his telling, still has something to prove.

His public activity has also highlighted the team’s Holosoma open-source release for humanoid training and research on humanoid motion. The range has broadened beyond the warehouse arm. Yet a familiar pattern remains: Chen works with long-standing collaborators, connects fundamental learning questions to physical machines, and judges the distance between a promising experiment and useful behavior.

The student who once doubted AI ended up spending his career asking it harder questions. Can it recognize the unfamiliar? Can it learn from the attempt? Can it do the work again tomorrow? For Peter Chen, the future keeps arriving in objects that need to be picked up. There is something reassuring about an ambition that must occasionally answer to a box.