AI programs can tell race from X-rays, but scientists don't know how. Here's why that's bad.

The MIT and Harvard study found that an artificial-intelligence program trained to read X-rays and CT scans could predict a person’s race with 90 percent accuracy.

Scientists at Harvard and MIT are part of an international team of researchers who found that artificial intelligence programs can determine someone’s race with over 90% accuracy from just their X-rays.

The problem is that no one knows how the AI programs do it.

“When my graduate students showed me some of the results that were in this paper, I actually thought it must be a mistake,” Marzyeh Ghassemi, an MIT assistant professor and coauthor of the paper analyzing the subject, told The Boston Globe. “I honestly thought my students were crazy when they told me.”

The researchers wrote in the study that many studies have shown that AI diagnostic systems seem to be using race in their considerations for diagnosis and treatment, to the detriment of patient health.

In the paper, they gave an example in which an AI program that examined chest X-rays was more likely to miss signs of illness in Black and female patients.

Thus, the aim of the study, which was published Wednesday in the medical journal The Lancet Digital Health, was to determine the degree to which AI systems can detect race from medical imaging, and to find out more about how these AI systems are detecting race.

To do this, the research team trained AI systems for the study using standard data sets of X-rays and CT scans of different parts of the body.

Each image was labeled with the person’s self-reported race, but contained no obvious racial markers, such as hair texture or skin color, or medical racial trends, such as BMI or bone density. The team then fed the AI systems images without race labelling.

The researchers found that the AI systems were somehow able to determine the race of the person who the images were taken from with over 90% accuracy. The AI systems were even able to detect race from medical images regardless of what part of the body the image was of.

Perhaps even more surprising is that the researchers found the AI programs could accurately predict race based on X-ray images and CT scans even when those images were heavily degraded.

The problem isn’t that AI systems can accurately detect race, the researchers wrote. The problem is that medical AI systems have been found to perform badly as a result of racial bias.

These AI seem to be making diagnoses or recommending treatments based on the person’s race, regardless of the individual’s specific health criteria, resulting in negative health outcomes.

Meanwhile, the actual physician could be oblivious to the AI’s racially biased results.

“In our study, we emphasise that the ability of AI to predict racial identity is itself not the issue of importance, but rather that this capability is readily learned and therefore is likely to be present in many medical image analysis models, providing a direct vector for the reproduction or exacerbation of the racial disparities that already exist in medical practice,” the authors of the study wrote.

Additionally, the fact that humans are unable to detect what features of the images are tipping off the AI systems to the patient’s race, combined with the fact that the AI systems were still accurately detecting the patient’s race regardless of what part of the body the image was taken from, as well as when the images were greatly degraded, means that it would be extremely hard to create an AI system using medical imaging that does not have a racial bias, the study authors wrote.

Ghassemi told the Globe that her guess as to how the AI is detecting race is that X-rays and CT scans are somehow recording the level of melanin in a patient’s skin in the images, and doing so in a way that humans have never noticed.

There’s also the possibility that the results of the study show some innate difference between races.

Alan Goodman, a professor of biological anthropology at Hampshire College and coauthor of the book “Racism Not Race,” told the Globe he is doubtful of this.

Goodman told the newspaper that he thinks other scientists will have trouble replicating the results of the study, but that even if they do, it is likely based on where a person’s ancestors evolved, rather than based on race.

While finding consistent racial differences in the human genome is difficult for geneticists, Goodman told the Globe, they do often find consistent genetic differences based on where peoples’ ancestors evolved.

Regardless, more research is needed on the study’s findings to make any firm conclusions, Ghassemi told the Globe.