The authors are far from the first to raise questions about COMPAS’s role in judicial decision making.
In one high-profile case, a Wisconsin man charged with evading the police said that his due process rights were violated when a judge used COMPAS in sentencing him to six years in prison. (He appealed the case to the United States Supreme Court, but it declined to weigh in last year.)
And, in 2016, the investigative journalism nonprofit ProPublica concluded that the software was unfair to black defendants.
While some academics disputed the certainty of that finding — arguing that it depends on how fairness is measured — the report inspired Ms. Dressel and Mr. Farid to conduct their own study.
In addition to their main finding, the authors also found evidence of racial bias from both COMPAS and the humans, even though neither considered race in the predictions. Compared with their white peers, a larger share of black defendants was falsely predicted to reoffend, while a smaller share was falsely predicted not to.
Equivant disputed the findings. In a statement on Wednesday, it said that the authors overstated the number of variables COMPAS uses to predict future crimes — it is six, the company said, not the 137 that the authors cited. Equivant also said that the conclusions were limited by the small data sample used, the same data behind the ProPublica report.
Ms. Dressel and Mr. Farid based their analysis on a database of more than 7,000 pretrial defendants in Broward County, Fla., from 2013 to 2014. For each, the database included demographic information, criminal history, COMPAS risk scores and arrest records for the two years following the scoring.
A subset of 1,000 defendants was then used to collect predictions from 400 people, recruited through Amazon’s Mechanical Turk, a marketplace for online labor. Those participants were presented with paragraphs about each of 50 defendants and asked to rate the likelihood that each would commit a crime in the following two years.
The descriptions the individuals were presented with included the defendants’ sex, age, crime they were charged with, the type of crime and their arrest history.