Automated system teaches users when to collaborate with an AI assistant

Artificial intelligence models that recognize patterns in images can often do that higher than the human eye – but not at all times. If a radiologist uses an AI model to find out whether a patient’s X-rays show signs of pneumonia, when should they trust the model’s advice and when should they ignore it?

According to researchers at MIT and the MIT-IBM Watson AI Lab, a tailored onboarding process could help this radiologist answer that query. They have developed a system that teaches a user when to collaborate with an AI assistant.

In this case, the training method could find situations where the radiologist trusts the model’s advice – except that he shouldn’t since the model is mistaken. The system routinely learns rules for working with the AI ​​and describes them in natural language.

During onboarding, the radiologist practices working with the AI ​​through training exercises based on these rules and receives feedback on her performance and the AI’s performance.

The researchers found that this onboarding procedure resulted in a couple of 5 percent improvement in accuracy when humans and AI worked together on a picture prediction task. Their results also show that telling the user to trust the AI ​​without training resulted in poorer performance.

Importantly, the researchers’ system is fully automated, so it learns to create the onboarding process based on data from the human and AI performing a particular task. It can be adapted to different tasks, allowing it to be scaled and utilized in many situations where humans and AI models work together, equivalent to moderating, writing, and programming social media content.

“Often these AI tools are given to people with none training to determine once they will probably be helpful. We don’t do this with almost every other tool that folks use – there’s almost at all times some kind of instructional that comes with it. But for AI this appears to be missing. We are attempting to handle this problem from a methodological and behavioral perspective,” says Hussein Mozannar, a doctoral student within the Social and Engineering Systems doctoral program on the Institute for Data, Systems, and Society (IDSS) and lead creator of an article about this training process.

The researchers consider that such onboarding will probably be an important a part of the training of medical professionals.

“One could imagine, for instance, that doctors who use AI to make treatment decisions would first should undergo training much like what we propose. We may have to rethink every part from continuing medical education to clinical trial design,” he says Lead creator David Sontag, Professor of EECS, member of the MIT-IBM Watson AI Lab and the MIT Jameel Clinic, and leader of the Clinical Machine Learning Group of the Computer Science and Artificial Intelligence Laboratory (CSAIL).

Mozannar, who can also be a researcher on the Clinical Machine Learning Group, is assisted within the work by Jimin J. Lee, an electrical engineering and computer science student. Dennis Wei, senior research scientist at IBM Research; and Prasanna Sattigeri and Subhro Das, research associates on the MIT-IBM Watson AI Lab. The paper will probably be presented on the Conference on Neural Information Processing Systems.

Training that evolves

Existing human-AI collaboration onboarding methods often consist of coaching materials created by human experts for specific use cases, making them difficult to scale. Some related techniques depend on explanations, where the AI ​​tells the user how much they depend on each decision. But research shows that explanations are rarely helpful, says Mozannar.

“The capabilities of the AI ​​model are continually evolving, so the use cases where humans could potentially profit from it are increasing over time. At the identical time, the user’s perception of the model continues to vary. So we want a training process that also evolves over time,” he adds.

To achieve this, their onboarding method is routinely learned from data. It relies on an information set that comprises many instances of a task, equivalent to detecting the presence of a traffic light from a blurry image.

The first step of the system is to gather data concerning the human and AI performing this task. In this case, humans would try to make use of AI to predict whether traffic lights may be seen in blurry images.

The system embeds these data points right into a latent space, which is a representation of information by which similar data points are closer together. It uses an algorithm to find areas of this space where humans are incorrectly collaborating with AI. These regions capture cases where the human trusted the AI’s prediction, however the prediction was mistaken and vice versa.

People may mistakenly trust AI when images show a highway at night.

After discovering the regions, a second algorithm uses a big language model to explain each region, typically in natural language. The algorithm iteratively refines this rule by finding contrasting examples. One could describe this region as “AI ignore whether it is a highway at night.”

These rules are used to structure training exercises. The onboarding system shows the human an example, on this case a blurry highway scene at night, in addition to the AI’s prediction and asks the user whether the image shows traffic lights. The user can answer yes or no or use the AI’s prediction.

If the human is mistaken, they’re shown the proper answer and performance statistics for each the human and the AI ​​for those instances of the duty. The system does this for every region and at the top of the training process repeats the exercises that the human did incorrectly.

“Humans then learned something about these regions that we hope they’ll use in the longer term to make more accurate predictions,” says Mozannar.

Onboarding increases accuracy

The researchers tested this technique with users on two tasks – recognizing traffic lights in blurry images and answering multiple-choice questions from many fields (equivalent to biology, philosophy, computer science, etc.).

They first showed users a card with information concerning the AI ​​model, the way it was trained, and a breakdown of its performance into broad categories. Users were divided into five groups: some were just shown the map, some went through the researcher onboarding procedure, some went through a basic onboarding procedure, some went through the researcher onboarding procedure and got recommendations on when to accomplish that and when don’t trust the AI, others were only given recommendations.

Only the researchers’ onboarding procedure without recommendations significantly improved users’ accuracy, increasing their performance on the traffic light prediction task by about 5 percent without slowing them down. However, the onboarding for the query and answer task was not that effective. The researchers consider it’s because the ChatGPT AI model provided explanations for every answer, indicating whether it ought to be trusted.

But providing recommendations without onboarding had the other effect: Not only did users perform worse, additionally they needed more time to make predictions.

“If you only give someone recommendations, it looks like they’re confused and do not know what to do. It derails their process. People also don’t like being told what to do, in order that’s an element too,” says Mozannar.

Simply providing recommendations can harm the user if those recommendations are incorrect, he adds. When it involves onboarding, nevertheless, the most important limitation is the quantity of information available. If there’s not enough data, the onboarding phase won’t be as effective, he says.

In the longer term, he and his colleagues need to conduct larger studies to judge the short- and long-term effects of onboarding. You also need to leverage unlabeled data for the onboarding process and find methods to effectively reduce the variety of regions without leaving out vital examples.

“Humans are adopting AI systems willy-nilly, and AI actually has great potential, but these AI agents still sometimes make mistakes. “It is due to this fact critical for AI developers to develop methods that help people recognize once they can safely depend on AI’s suggestions,” says Dan Weld, professor emeritus on the Paul G. Allen School of Computer Science and Engineering on the University of Washington was not involved on this research. “Mozannar et al. have developed an progressive method to discover situations by which the AI ​​is trustworthy and (importantly) describe them to humans in a way that leads to raised human-AI team interactions.”

This work is funded partially by the MIT-IBM Watson AI Lab.