Bias has at all times been an issue in AI, but a brand new study shows that it’s covertly integrated into language models with potentially catastrophic consequences.

In what has already been heralded as a landmark study, a team of researchers, including Valentin Hofman, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King, documented how large language models (LLMs) discriminate against African American English (AAE).

In short, the study tests how different spelling and dialects affect LLMs’ behavior. It probes whether certain dialects and word usage influence an LLM’s behavior, specializing in bias and discrimination. 

We know that LLM outputs are highly sensitive to the input. Even small deviations in spelling and magnificence can influence outputs.

But does this mean certain inputs – e.g., those typed in AAE – produce biased outputs? If so, what are the possible consequences? 

To answer these questions, the researchers analyzed the prejudices held by a complete of 12 LLMs against AAE, revealing biases that match or exceed those typically held by humans. The study is obtainable on ArXiv.

The researchers then applied their findings to societal domains comparable to employment and criminal justice, where AI decision-making is becoming more common. 

Hofmann described the study methodology on X: “We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English (SAE) texts in prompts that ask for properties of the speakers who’ve uttered the texts, and compare the model predictions for the 2 sorts of input.” 

We analyze dialect prejudice in LLMs using Matched Guise Probing: we embed African American English and Standardized American English texts in prompts that ask for properties of the speakers who’ve uttered the texts, and compare the model predictions for the 2 sorts of input. pic.twitter.com/drTco67Ean

— Valentin Hofmann (@vjhofmann) March 4, 2024

This method allows the team to directly compare the responses of LLMs to AAE versus SAE inputs, unmasking the covert biases that will otherwise remain obscured.

The study’s findings are unsettling, to say the least.

Hofmann notes, “We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to those from before the civil rights movement.” 

We find that the covert, raciolinguistic stereotypes about speakers of African American English embodied by LLMs are more negative than any human stereotypes about African Americans ever experimentally recorded, although closest to those from before the civil rights movement. pic.twitter.com/07LgUY2bCj

— Valentin Hofmann (@vjhofmann) March 4, 2024

This suggests that the biases present in LLMs will not be merely reflections of latest stereotypes but are more aligned with prejudices that many believed society had moved beyond.

One of probably the most concerning elements of the study is the precise linguistic triggers of bias. 

Hofmann elaborates, “What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, comparable to using ‘finna’ as a future marker.”

This indicates that the bias isn’t just against using AAE usually but is tied to the distinct linguistic elements that characterize the dialect.

What is it specifically about African American English texts that evokes dialect prejudice in LLMs? We show that the covert stereotypes are directly linked to individual linguistic features of African American English, comparable to using “finna” as a future marker. pic.twitter.com/JhPhX7ZE5U

— Valentin Hofmann (@vjhofmann) March 4, 2024

The potential for harm

The potential for harm from such biases is immense. Previous studies have already demonstrated how AI systems are likely to fail women, darker-skinned individuals, and other marginalized groups. 

Before the previous couple of years, AI systems risked being trained on unrepresentative datasets. Some, like MIT’s Tiny Images, created in 2008, were later withdrawn on account of sexism and racism. 

One influential 2018 study, Gender Shades, analyzed tons of of ML algorithms and located that error rates for darker-skinned women were as much as 34% greater than for lighter-skinned males.

The impacts are stark, with healthcare models exhibiting high rates of skin cancer misdiagnosis amongst those with darker skin tones and prejudiced predictive policing models disproportionally targeting black people.  

We’ve already observed unequivocal proof of AI’s increasing use across the general public sector, from crime and policing to welfare and the economy. Addressing fundamental bias in sophisticated AI systems is totally critical if that is to proceed.

Building on this research, Hofman’s team investigated how LLM bias could impact several hypothetical scenarios.

Hofman shared, “Focusing on the areas of employment and criminality, we discover that the potential for harm is very large.” 

Specifically, LLMs were found to assign less prestigious jobs and suggest harsher criminal judgments against speakers of AAE.

First, our experiments show that LLMs assign significantly less prestigious jobs to speakers of African American English in comparison with speakers of Standardized American English, regardless that they will not be overtly told that the speakers are African American. pic.twitter.com/t5frzzzwJB

— Valentin Hofmann (@vjhofmann) March 4, 2024

Hofmann warns, “Our results point to 2 risks: that users mistake decreasing levels of overt prejudice for an indication that racism in LLMs has been solved when LLMs are the truth is reaching increasing levels of covert prejudice.” 

Second, when LLMs are asked to pass judgment on defendants who committed murder, they select the death penalty more often when the defendants speak African American English fairly than Standardized American English, again without being overtly told that they’re African American. pic.twitter.com/8VBaCXfNEi

— Valentin Hofmann (@vjhofmann) March 4, 2024

The study also determines that erasing these problems is technically difficult.

The authors write, “We show that existing methods for alleviating racial bias in language models comparable to human feedback training don’t mitigate the dialect prejudice, but can exacerbate the discrepancy between covert and overt stereotypes, by teaching language models to superficially conceal the racism that they maintain on a deeper level.”

It’s feasible to think these biases apply to other dialects or cultural-linguistic variations. More research is required to know how LLM performance varies with linguistic inputs, cultural use patterns, etc.

The study concludes with a call to motion for the AI research community and society at large. Addressing these biases is paramount as AI systems turn into increasingly embedded across society.

However, thus far, the inherent and systematically embedded bias of some AI systems stays an issue that developers are able to omit of their race for AI supremacy. 

The post African American English (AAE) influences LLMs towards discrimination appeared first on DailyAI.


This article was originally published at dailyai.com