Criminal Justice Group Drops Support For Pretrial Risk Assessment Tools As Ohio Justices Seek To Block Their Use

An influential criminal justice organization has renounced its support of pretrial risk assessment tools as a growing chorus of voices question their effectiveness and criticize them as being inherently racially biased.

The change in position comes just weeks after the Ohio Supreme Court stripped language from proposed criminal justice reforms that would have required all judges in the state to be provided with pretrial risk assessment tools.

One of the first pretrial risk assessment tools was developed by the Vera Institute of Justice in the early 1960s, building on the work of parole risk assessment tools, some of which had been in place since the 1930s. Pretrial tools were intended to decrease pretrial mass incarceration and alleviate racial bias in the court system. The goal was to provide judges with a data-driven criteria they could use when deciding whether and under what conditions to release a person pending trial.

The Pretrial Justice Institute, which had continued to support pretrial risk assessment tools in the face of increasing opposition to them, reversed its long-held position on Friday, saying in a statement, “We now see that pretrial risk assessment tools, designed to predict an individual’s appearance in court without a new arrest, can no longer be a part of our solution for building equitable pretrial justice systems. Regardless of their science, brand, or age, these tools are derived from data reflecting structural racism and institutional inequity that impact our court and law enforcement policies and practices. Use of that data then deepens the inequity.”

Pretrial Justice Institute Executive Partner Meghan Guevara told The Appeal that the organization’s change of heart “has been in process for the past two years.” For years, the organization saw the emergence of pretrial risk assessments as “a step in the right direction” to decrease court systems’ reliance on money bonds. After working with civil rights advocates and others, however, “we’ve gained a greater understanding of the fact that the data that goes into developing those tools and scoring those tools is just inherently racially biased. And there is no way around that,” she said.

While an estimated 1,000 U.S. counties are using pretrial risk assessment tools, new research — most recently by Media Mobilizing Project and MediaJustice — suggests that the tools are having little impact on pretrial incarceration rates. That work mirrors the findings of other organizations in recent years. Insha Rahman, director of Strategy & New Initiatives for the Vera Institute of Justice, told The Appeal that her organization “really started to sound the alarm” about the tools after a 2016 ProPublica report that showed glaring racial disparities in the outcomes of more than 7,000 people arrested in Broward County, Florida, in 2013 and 2014. 

According to that report, ProPublica did a statistical test to isolate the effect of race from criminal history, recidivism, age, and gender. Even with those variables isolated, “Black defendants were still 77 percent more likely to be pegged as at higher risk of committing a future violent crime and 45 percent more likely to be predicted to commit a future crime of any kind.”

Academics and reform advocates say that disparate racial outcomes are only one of the issues with the assessment tools. In July, 27 researchers from MIT, Harvard, Princeton, NYU, UC Berkeley, and Columbia submitted a statement of concern about the tools to criminal justice policy makers in California, the city of Los Angeles, and Missouri. “Actuarial pretrial risk assessments suffer from serious technical flaws that undermine their accuracy, validity, and effectiveness,” the statement says. Among other problems, it says the tools use “inexact and overly broad” definitions of risks and that “no tool available today can adequately distinguish one person’s risk of violence from another.” In addition, the academics say, “people of color are treated more harshly than similarly situated white people at each stage of the legal system, which results in serious distortions in the data used to develop risk assessment tools.”

As a result of these flaws, the statement says, “These problems cannot be resolved with technical fixes. We strongly recommend turning to other reforms.”

Developments in Ohio suggest that at least some criminal justice officials are starting to hear those concerns. On January 15, the Ohio Supreme Court submitted criminal justice reform proposals to the state’s General Assembly that eliminated a task force’s recommendation that a “validated risk assessment tool” be provided to all the state’s judges. The justices voted 5-2 to strike that language. Under a 1968 reform to Ohio’s state Constitution, the state legislature can vote only to either accept or reject the entire package of proposed criminal justice reforms. If the General Assembly doesn’t take action by July 1, they will take effect automatically.

The court’s justices were persuaded to drop pretrial risk assessments by the American Civil Liberties Union of Ohio, according to a January 24 report on the decision by Cleveland.com. In an email to The Appeal, Policy Director Jocelyn Rosnick explained that the national ACLU took a public stance against the use of pretrial assessments in its March 2019 report, A New Vision for Pretrial Justice in the United States, which says that “Risk assessment instruments have not been shown to fix bias in pretrial decision-making, even as a supplement to decisions made by judges.”

“Risk assessment tools vary in the criteria they use. However, criminal history is largely included across the board,” Rosnick added. “This leads to racially biased outcomes.” According to 2017 figures released by the U.S. Department of Justice, the prison incarceration rate for Black men is more than five times that of white men, while Black men are three times more likely to be incarcerated in jails. 

Further, Rosnick said, there’s no proof that the risk assessments are actually effective at their key tasks: predicting the likelihood someone will return to court, and predicting how likely it is that an arrested person will commit violence if released pending trial. 

“There simply haven’t been robust studies showing that these tools actually work – meaning, if they are more accurate than judges, less biased than judges, or even improve on judicial decision-making,” she wrote.

In their July statement, the academics from MIT, Harvard, Columbia, Princeton, NYU, and UC Berkeley, reached the same conclusion, saying, “Within jurisdictions that have achieved positive outcomes, it is uncertain whether the risk assessment tools were responsible for that success or whether that success is due to other reforms or changes that happened at the same time.” 

Ohio Supreme Court Chief Justice Maureen O’Connor was one of two justices who voted to retain the language requiring the state’s judges to be provided with pretrial risk assessments. “The tools are just that — tools. They are no substitute for the discretion of the judge,” O’Connor told the Appeal. “Risk assessments are used every day and will continue to be used by judges who realize their value.”

In response to allegations that the tools perpetuate racial disparities, O’Connor said that “the claimed bias is due to the fact that a minority defendant may have more past convictions, and that is just a statistic that is real. It is imperative that a judge appreciate that and determine the value of that information.”

Insha Rahman, of the Vera Institute of Justice, said overly-broad judicial discretion is at the heart of the problem of mass pretrial incarceration whether or not a given court system uses a pretrial assessment tool.

“The jurisdiction can choose to not use them, and still have tremendous racial disparities and over-incarceration as a result of unfettered judicial discretion,” she said. “Or a jurisdiction can choose to use them and also still have those same problems, and the reason why is because risk assessment instruments are essentially a shortcut for, say, questions and the inquiry that judges should be making at the pretrial stage.”

Both Rosnick and Guevara also disagreed with O’Connor, saying that pretrial justice reform can be accomplished in other ways.

“There should be a broad presumption of release for most people; there should be a hearing within 48 hours of detention with access to counsel. Only the minimum necessary conditions should be set — things like employment or reporting requirements — and an ability to pay determination should occur prior to the use of money bond,” Rosnick said. “Additionally, risk assessment tools do not consider ways in which risk might be mitigated by support measures. For example, texting reminders by the court are a simple, cost-effective tool that studies show can increase court appearance rates.” One study in New York City found that texting reminders increased court appearance rates by 26 percent.

Guevara added that the choice isn’t as binary as the Chief Justice implied. “It is important to remember that systems don’t need to choose between being ‘money-based’ and ‘risk-based.’ Judges have the authority to weigh the information available to them and make a bail determination that doesn’t involve money without needing a risk assessment tool,” she said.