Using language to grasp machines

Natural language conveys ideas, actions, information, and intentions through context and syntax; In addition, large amounts of it are contained in databases. This makes it a superb data source for training machine learning systems. Two master’s engineering students from the 6A MEng Thesis Program at MIT, Irene Terpstra ’23 and Rujul Gandhi ’22, are working with mentors within the MIT-IBM Watson AI Lab to harness this power of natural language to construct AI systems.

As computing becomes more advanced, researchers strive to enhance the hardware that runs it. This means innovations to supply recent computer chips. And since there may be already literature on modifications that could be made to realize certain parameters and performance, Terpstra and her mentors and advisors Anantha Chandrakasan, dean of the MIT School of Engineering and Vannevar Bush Professor of Electrical Engineering and Computer Science and researcher at IBM Xin Zhang develops an AI algorithm that helps with chip design.

“I’m making a workflow to systematically analyze how these language models can support the circuit design process. “What pondering skills have they got and the way can they be integrated into the chip design process?” says Terpstra. “And then again, if this proves useful enough, (we) will see in the event that they can mechanically design the chips themselves and connect them to a reinforcement learning algorithm.”

To this end, Terpstra’s team is creating an AI system that may iterate on different designs. It means experimenting with various pre-trained large-scale language models (like ChatGPT, Llama 2, and Bard), using an open-source circuit simulator language called NGspice that comprises the chip’s parameters in code form, and a reinforcement learning algorithm. Text prompts allow researchers to question how the physical chip ought to be modified to realize a particular goal within the language model and create instructions for adjustments. This is then transferred right into a reinforcement learning algorithm that updates the circuit design and outputs recent physical parameters of the chip.

“The end goal could be to mix the reasoning skills and knowledge base embedded in these large language models with the optimization power of reinforcement learning algorithms and allow them to design the chip itself,” says Terpstra.

Rujul Gandhi works with raw language itself. As a student at MIT, Gandhi studied linguistics and computer science and combined them in her MEng work. “I’m concerned about communication, each between people and between people and computers,” says Gandhi.

Robots or other interactive AI systems are an area where communication between each humans and machines should be understood. Researchers often write instructions for robots using formal logic. This helps ensure commands are followed safely and as intended. However, formal logic could be difficult for users to grasp, while natural language is straightforward to grasp. To ensure this smooth communication, Gandhi and her advisors Yang Zhang of IBM and MIT assistant professor Chuchu Fan are constructing a parser that converts natural language instructions right into a machine-friendly form. Gandhi’s system leverages the linguistic structure encoded by the pre-trained T5 encoder-decoder model and a dataset of annotated, basic English commands to perform specific tasks and identifies the smallest logical units or atomic sentences present in a given instruction .

“Once you give your instruction, the model identifies all of the smaller subtasks you would like it to perform,” says Gandhi. “Then, using a big language model, each subtask could be in comparison with the available actions and objects within the robot’s world, and if a subtask can’t be performed because a specific object or motion just isn’t recognized as impossible, the system can stop right there and ask the user for help.”

This approach of breaking down instructions into subtasks also allows their system to grasp logical dependencies expressed in English, reminiscent of “perform task reminiscent of navigation and manipulation, with an emphasis on household tasks. Using data written exactly the best way people would speak to one another has many advantages, she says, since it means a user could be more flexible in how they phrase their instructions.

Another of Gandhi’s projects concerns the event of language models. In the context of speech recognition, some languages ​​are considered “resource poor” because they could not have much transcribed speech available to them or may not have a written form in any respect. “One of the explanations I applied for this internship on the MIT-IBM Watson AI Lab was my interest in language processing for low-resource languages,” she says. “Many language models today are very data-driven, and if it isn’t that easy to capture all that data, it’s essential to make efficient use of the limited data.”

Language is only a stream of sound waves, but people conversing can easily determine where words and thoughts begin and end. In language processing, each humans and language models use their existing vocabulary to acknowledge word boundaries and understand meaning. In low-resource or low-resource languages, there could also be no written vocabulary in any respect, leaving researchers unable to supply one to the model. Instead, the model can determine which sound sequences appear together more often than others and infer that these could also be individual words or concepts. Gandhi’s research group then collects these derived words right into a pseudo-vocabulary that serves as a labeling method for the resource-poor language and creates labeled data for further applications.

The applications for voice technology are “just about all over the place,” says Gandhi. “You can imagine that individuals can interact with software and devices of their native language, their native dialect. You could imagine improving all of the voice assistants we use. You could imagine it getting used for translating or interpreting.”