Multiple AI models help robots execute complex plans more transparently

Your each day to-do list might be pretty easy: washing dishes, grocery shopping, and other little things. It’s unlikely that you just wrote down “pick up the primary dirty dish” or “wash the plate with a sponge” because each of those small steps inside the task feels intuitive. While we are able to perform each step routinely and without much thought, a robot requires a posh plan with more detailed outlines.

MIT’s Improbable AI Lab, a bunch inside the Computer Science and Artificial Intelligence Laboratory (CSAIL), has helped these machines with a brand new multimodal framework: Compositional basic models for hierarchical planning (HiP), which develops detailed, actionable plans using the expertise of three different start-up models. Like OpenAI’s GPT-4, the bottom model on which ChatGPT and Bing Chat were built, these base models are trained on massive amounts of information for applications corresponding to generating images, translating text, and robotics.

Unlike RT2 and other multimodal models which are trained on paired vision, speech, and motion data, HiP uses three different base models, each trained on different data modalities. Each foundation model captures a special a part of the decision-making process after which works together when it’s time to make decisions. HiP eliminates the necessity for access to paired vision, speech and motion data that’s difficult to acquire. HiP also makes the argumentation process more transparent.

What is taken into account a each day task for a human could also be a robot’s “long-term goal” – a high-level goal that requires many smaller steps to be taken first – and that requires sufficient data to plan, understand and execute goals. While computer vision researchers have attempted to construct monolithic baseline models for this problem, pairing voice, image, and motion data is expensive. Instead, HiP represents a special, multimodal recipe: a trio that cost-effectively integrates linguistic, physical and environmental intelligence right into a robot.

“Basic models don’t need to be monolithic,” says NVIDIA AI researcher Jim Fan, who was not involved within the work. “This work breaks down the complex task of embodied agent planning into three constituent models: a language reasoner, a visible world model, and an motion planner. It makes a difficult decision-making problem more comprehensible and transparent.”

The team believes their system could help these machines do household chores, like putting a book away or putting a bowl within the dishwasher. Additionally, HiP could help with multi-step design and manufacturing tasks, corresponding to stacking and placing different materials in specific orders.

Review by HiP

The CSAIL team tested the sharpness of HiP on three manipulation tasks and outperformed comparable frameworks. The system is predicated on developing intelligent plans that adapt to recent information.

First, the researchers asked it to stack different coloured blocks on top of one another after which place others nearby. The catch: Because a number of the correct colours weren’t available, the robot had to position white blocks in a paint tray to color them. HiP has often adapted closely to those changes, especially when put next to cutting-edge task scheduling systems like Transformer BC and Action Diffuser, by adjusting its schedules to stack and place each square as needed.

Another test: Arrange items like candy and a hammer in a brown box while ignoring other items. Some of the items that needed to be moved were dirty, so HiP adjusted its plans and placed them in a cleansing box after which within the brown container. In a 3rd demonstration, the bot was in a position to ignore unnecessary objects to finish subgoals within the kitchen, corresponding to opening a microwave, putting away a kettle, and turning on a light-weight. Some of the prompted steps had already been accomplished, so the robot adapted by skipping these instructions.

A 3-tier hierarchy

HiP’s three-stage planning process works hierarchically and provides the power to pre-train each of its components on various data sets, including non-robotic information. At the top of this sequence is a big language model (LLM) that begins generating ideas by capturing all of the required symbolic information and developing an abstract task plan. Using common sense found on the Internet, the model breaks down its goal into subgoals. For example, “making a cup of tea” becomes “filling a pot with water,” “boiling the pot,” and the next actions required.

“All we would like to do is leverage existing pre-trained models and successfully connect them together,” says Anurag Ajay, a graduate student within the MIT Department of Electrical Engineering and Computer Science (EECS) and a CSAIL partner. “Instead of pushing for one model to do every thing, we mix multiple models that leverage different modalities of web data. When used together, they aid in robotic decision-making and might potentially assist with tasks in homes, factories and construction sites.”

These models also require some kind of “eyes” to grasp the environment during which they operate and accurately execute each sub-goal. The team used a big video diffusion model to enhance the initial planning of the LLM, which collects geometric and physical information concerning the world from footage on the Internet. The video model in turn generates an commentary history plan and refines the structure of the LLM to include recent physical knowledge.

This process, called iterative refinement, allows HiP to reflect on its ideas and acquire feedback at each stage to create a more practical design. The feedback flow is comparable to writing an article, where an writer can send their draft to an editor and the editor reviews and finalizes these revised versions for any final changes.

In this case, at the highest of the hierarchy is an egocentric motion model, or a sequence of first-person images that infer what actions should happen based on the environment. In this phase, the commentary plan from the video model is mapped over the space visible to the robot and helps the machine determine the right way to perform each task inside the long-term goal. When a robot uses HiP to make tea, it implies that it has determined exactly where the pot, sink, and other key visual elements are and begins completing each sub-goal.

Nevertheless, multimodal work is restricted by the shortage of high-quality video foundation models. Once available, they might be connected to HiP’s small video models to further improve visual sequence prediction and robotic motion generation. The next quality version would also reduce the present data requirements of the video models.

However, the CSAIL team’s approach used only a tiny amount of information overall. Additionally, HiP was inexpensive to coach and demonstrated the potential of using available baseline models to finish long-term tasks. “What Anurag has demonstrated is a proof of concept of how we are able to mix models trained on separate tasks and data modalities into models for robot planning. In the long run, HiP might be expanded with pre-trained models that may process touch and sound to create higher plans,” says lead writer Pulkit Agrawal, MIT assistant professor of EECS and director of the Improbable AI Lab. The group can also be considering using HiP to resolve real-world, long-term tasks in robotics.

Ajay and Agrawal are lead authors of a Paper describing the work. They are joined by MIT professors and CSAIL principal investigators Tommi Jaakkola, Joshua Tenenbaum and Leslie Pack Kaelbling; CSAIL research partner and MIT-IBM AI Lab research manager Akash Srivastava; graduate students Seungwook Han and Yilun Du ’19; former postdoctoral fellow Abhishek Gupta, now an assistant professor on the University of Washington; and former graduate student Shuang Li PhD ’23.

The team’s work was supported partially by the National Science Foundation, the US Defense Advanced Research Projects Agency, the US Army Research Office, the US Office of Naval Research Multidisciplinary University Research Initiatives and the MIT-IBM Watson AI Lab. Their results were presented on the 2023 Conference on Neural Information Processing Systems (NeurIPS).