Image Recognition Accuracy: An Unprecedented Challenge Confounding Today’s AI

Imagine scrolling through the photos in your phone and coming across a picture which you could’t recognize at first. It looks like something fuzzy on the couch; Could it’s a pillow or a coat? After a couple of seconds it clicks – in fact! This ball of fluff is your friend’s cat, Mocha. While a few of your photos were immediately comprehensible, why was this cat photo rather more difficult?

Researchers on the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) were surprised to seek out that despite the critical importance of understanding visual data in key areas from healthcare to transportation to home appliances, the concept a picture is difficult for humans to understand recognize is nearly completely uncontrolled is ignored. One of the important thing drivers of progress in deep learning-based AI has been data sets, yet we all know little about how data drives progress in large-scale deep learning beyond “the larger the higher.”

In real-world applications that require understanding visual data, humans outperform object recognition models, although models perform well on current datasets, including those explicitly designed to challenge machines with distorted images or distribution shifts. This problem persists partially because now we have no indication of absolutely the difficulty of a picture or dataset. Without control over the issue level of the photographs used for evaluation, it’s difficult to objectively assess progress toward human performance, cover the range of human capabilities, and increase the challenge of a dataset.

To address this data gap, David Mayo, an MIT graduate student in electrical engineering and computer science and a CSAIL partner, delved into the deep world of image datasets and explored why certain images are harder for humans and machines to acknowledge than others. “Some images inherently take longer to acknowledge, and it will be important to know the brain’s activity during this process and its relationship to machine learning models. Perhaps our current models are missing complex neural circuits or unique mechanisms which might be only visible when tested with sophisticated visual tests.” “This exploration is critical to understanding and improving image processing models,” says Mayo, certainly one of the lead authors of 1 recent study Paper about work.

This led to the event of a brand new metric, the “minimal viewing time” (MVT), which quantifies the issue of recognizing a picture based on the time it takes an individual to take a look at it before making an accurate identification. Using a subset of ImageNet, a preferred data set in machine learning, and ObjectNet, a knowledge set for testing the robustness of object detection, the team showed participants images for various durations from as little as 17 milliseconds to so long as 10 seconds and quizzed them on 50 options select the appropriate object. After over 200,000 image presentation attempts, the team found that existing test sets, including ObjectNet, gave the impression to be biased towards simpler, shorter MVT images, with the overwhelming majority of benchmark performance derived from images which might be easy to humans.

The project identified interesting trends in model performance – particularly in relation to scaling. Larger models showed significant improvements on simpler images, but made less progress on more difficult images. The CLIP models, which integrate each speech and vision, stood out as moving toward more human-like recognition.

“Traditionally, object detection datasets have been biased towards less complex images, a practice that has led to inflation of model performance metrics and doesn’t truly reflect the robustness of a model or its ability to handle complex visual tasks. Our research shows that harder images are more difficult and cause a distribution shift that is commonly not accounted for in standard evaluations,” says Mayo. “We have released difficulty-labeled image sets in addition to tools to robotically calculate MVT in order that MVT might be added to existing benchmarks and prolonged to numerous applications. These include measuring test set difficulty before deploying real systems, discovering neural correlates of image difficulty, and further developing object recognition techniques to shut the gap between benchmark and real-world performance.”

“One of my biggest takeaways is that we now have a distinct dimension by which to guage models. We want models which might be in a position to recognize any image, even when – perhaps especially when – it’s difficult for a human to acknowledge. We are the primary to quantify what that will mean. “Our results show that not only is that this not the case with today’s state-of-the-art, but in addition that our current evaluation methods are unable to inform us when that is the case because standard datasets are so focused on easy images are,” says Jesse Cummings, an MIT graduate student in electrical engineering and computer science and, together with Mayo, first writer of the paper.

From ObjectNet to MVT

A number of years ago, the team behind this project recognized a major challenge in machine learning: models were combating images that were outside the distribution or not well represented within the training data. This is where ObjectNet comes into play, a dataset consisting of images collected from real-world environments. The dataset helped highlight the performance gap between machine learning models and human recognition capabilities by eliminating spurious correlations present in other benchmarks – for instance, between an object and its background. ObjectNet illuminated the gap between the performance of image processing models on datasets and in real-world applications and encouraged many researchers and developers to make use of it – which subsequently led to an improvement in model performance.

Let’s now rewind to the current: With MVT, the team has taken their research one step further. Unlike traditional methods that give attention to absolute performance, this recent approach evaluates the performance of models by comparing their responses to the simplest and most difficult images. The study also examined how image difficulty may very well be explained and tested for similarity to human visual processing. Using metrics similar to C-score, prediction depth, and adversarial robustness, the team found that harder images are processed otherwise by networks. “While there are observable trends, similar to simpler images being more prototypical, a comprehensive semantic explanation of image difficulty stays lacking within the scientific community,” says Mayo.

In healthcare, for instance, understanding visual complexity becomes much more essential. The ability of AI models to interpret medical images similar to X-rays will depend on the range and difficulty distribution of the photographs. The researchers argue for careful, professional-tailored evaluation of difficulty distribution to be certain that AI systems are evaluated based on expert standards quite than layman interpretations.

Mayo and Cummings are also currently studying the neurological basis of visual recognition and whether the brain exhibits different activity when processing easy and complex images. The study goals to seek out out whether complex images recruit additional brain areas not normally related to visual processing, and hopefully help demystify how our brains accurately and efficiently decode the visual world.

Towards human-level performance

Looking forward, researchers aren’t just focused on exploring ways to enhance AI’s predictive capabilities on image difficulty. The team is working to discover correlations with viewing time difficulty to generate harder or easier versions of images.

Despite the study’s major advances, researchers acknowledge limitations, particularly with regard to separating object recognition from visual search tasks. The current methodology focuses on object recognition and ignores the complexity created by cluttered images.

“This comprehensive approach addresses the long-standing challenge of objectively assessing progress toward human performance in object recognition and opens recent avenues for understanding and advancing the sphere,” says Mayo. “With the flexibility to regulate the Minimum Viewing Time difficulty metric for a wide range of visual tasks, this work paves the best way for more robust, human-like performance in object recognition, ensuring that models are truly put to the test and prepared for testing Complexities of Visual Comprehension within the Real World.”

“This is an interesting study of how human perception might be used to discover weaknesses in the best way AI vision models are typically evaluated, which overestimate AI performance by specializing in easy images says Alan L. Yuille, Bloomberg Distinguished Professor of Cognitive Science and Computer Science at Johns Hopkins University, who was not involved within the work. “This will help develop more realistic benchmarks that not only result in improvements in AI, but in addition enable fairer comparisons between AI and human perception.”

“It’s widely said that computer vision systems at the moment are outperforming humans, and that is true on some benchmark datasets,” says Simon Kornblith PhD ’17, a technical associate at Anthropic, who was also not involved on this work. “However, a big a part of the issue with these benchmarks lies in the dearth of clarity about what’s visible in the photographs. The average person simply doesn’t know enough to categorise different dog breeds. This work as a substitute focuses on images that individuals can only get right if given enough time. These images are generally rather more difficult for computer vision systems, but the perfect systems are only a bit worse than humans.”

Mayo, Cummings and Xinyu Lin MEng ’22 wrote this Paper alongside CSAIL Research Scientist Andrei Barbu, CSAIL Principal Research Scientist Boris Katz and MIT-IBM Watson AI Lab Principal Investigator Dan Gutfreund. The researchers are partners with the MIT Center for Brains, Minds, and Machines.

The team is presenting their work on the 2023 Conference on Neural Information Processing Systems (NeurIPS).