An artificially intelligent (AI) software system that determines which patients get access to high-risk healthcare management programmes routinely lets healthier white people into the programmes ahead of less healthy black people, a new study has found.
The study was carried about by researchers at the University of California, Berkeley, the University of Chicago Booth School of Business and Boston-based non-profit Partners HealthCare.
Discover B2B Marketing That Performs
Combine business intelligence and editorial excellence to reach engaged professionals across 36 leading media platforms.
The researchers obtained the algorithm-predicted risk score for 43,539 white patients and 6,079 black patients enrolled at an academic hospital that uses an AI-based system to determine which patients will be given access to a high-risk care management programme. They then compared the risk score to more direct measures of a patient鈥檚 health, such as chronic illnesses. They found that, for a given risk score, black patients had significantly poorer health than white patients.
UC Berkeley associate professor of health policy and management Ziad Obermeyer said: 鈥淭he algorithms encode racial bias by using health care costs to determine patient 鈥榬isk鈥�, or who was mostly likely to benefit from care management programs.
鈥淏ecause of the structural inequalities in our health care system, blacks at a given level of health end up generating lower costs than whites. As a result, black patients were much sicker at a given level of the algorithm鈥檚 predicted risk.鈥�
Fixing the bias in the algorithm could more than double the number of black patients automatically admitted to these programs.
US Tariffs are shifting - will you react or anticipate?
Don鈥檛 let policy changes catch you off guard. Stay proactive with real-time data and expert analysis.
By GlobalDataRoman Family University professor of computation and behavioural science Sendhil Mullainathan said: 鈥淚nstead of being trained to find the sickest, in a physiological sense, these algorithms ended up being trained to find the sickest in the sense of those whom we spend the most money on. And there are systemic racial differences in health care in who we spend money on.鈥�
Patients whose risk scores landed in the top 97% were automatically selected for enrolment in a care management program. By correcting the algorithmic disparities between black and white patients, the researchers found the percentage of black people in the automatic enrolee group leapt from 18% to 47%.
By tweaking the software to use other variables to predict patient risk, such as avoidable cost or the number of chronic conditions that needed treatment in a year, the researchers were able to correct much of the bias initially built into the program.
Once the manufacturer of the software was alerted to the flaws in its algorithm, the company was 鈥渧ery motivated to address the issue鈥�, Obermeyer said.
Obermeyer said: 鈥淎lgorithms can do terrible things, or algorithms can do wonderful things. Which one of those things they do is basically up to us.鈥�
