Part 2 of a five-part Project Milk Carton Research & Analysis

SERIES NOTE: Part I of The Number That Knocks was originally published in July 2026 and has since been updated for accuracy, sourcing, and clarity. [Read Part I here.] This new installment picks up where that information left off. The original long-form article is now being continued as a five-part series, with each installment examining a different piece of the record.

Part I examined what child-welfare predictive models actually measure and the administrative data they consume.

Now comes the harder test.

What happened when agencies put them to work?

THE RECORD

FOUR JURISDICTIONS, FOUR CAUTIONARY CASES

Illinois: 100 percent risk of death, and the children who actually died

In May 2016, the Illinois Department of Children and Family Services signed a $366,000 sole-source contract — no competitive bid — for a predictive tool called Rapid Safety Feedback, marketed by the Florida nonprofit Eckerd, with its for-profit partner MindShare Technology.

We will return to Eckerd in Part III of this series.

The system mined DCFS case files and assigned children scores from 1 to 100 representing their probability of death or serious injury.

What it produced, per reporting by the Chicago Tribune and the trade press that covers child welfare, belongs in a textbook — as a warning label.

The system assigned more than 4,100 Illinois children a 90-percent-or-greater probability of death or serious injury.

It rated 369 children under the age of nine at 100 percent — a literal certainty of death or serious harm.

The volume and extremity of the alerts made the system difficult to use as a meaningful prioritization tool, while highly publicized child deaths were not identified as high risk.

DCFS Director B.J. Walker shut the program down in December 2017, and her explanation deserves attention from every agency that filed an application before this July’s federal deadline: the predictive system was not reliably identifying the cases the agency most needed it to identify.

One more detail from Illinois.

The no-bid contract had been awarded under then-DCFS Director George Sheldon, who came to Illinois from Florida’s Department of Children and Families, the state where Eckerd held a major lead-agency contract.

Sheldon resigned in 2017 amid scrutiny of DCFS contracting practices. Illinois inspectors later reported mismanagement involving a no-bid contract, while contemporaneous reporting examined additional contracts involving people with whom Sheldon had prior professional or financial ties.

Los Angeles: 171 hits, 3,829 false alarms

Los Angeles County’s first attempt was Project AURA, developed with the software firm SAS.

The county later reported that the pilot, completed in October 2014, identified 171 cases at highest risk while producing 3,829 false positives — roughly 22 false alarms for every case identified at the highest level of concern.

The county’s own 2017 report said the volume of false positives threatened to overwhelm a system that could not effectively respond to that many alerts.

DCFS was no longer pursuing Project AURA.

Years later, Los Angeles launched a separate Risk Stratification Model pilot in three DCFS offices.

By late 2024, the county reported that the pilot had shown improvements in child safety and announced that the Risk Stratification Model would be permanently adopted and expanded.

It is a separate project from AURA, and the two should not be treated as the same failed model returning under a different name.

That distinction matters.

The record is complicated.

Some systems have failed badly. Others have been modified, replaced, or evaluated more favorably.

A rigorous examination has to report both.

Thanks for reading! This post is public so feel free to share it.

Share

Oregon: terminated weeks after the AP came calling

Oregon’s Department of Human Services built its Safety at Screening Tool explicitly on the Allegheny model.

It was terminated at the end of June 2022, weeks after an Associated Press investigation by reporters Sally Ho and Garance Burke examined racial and transparency concerns surrounding the Allegheny tool on which Oregon’s approach had been based.

Oregon reverted to Structured Decision Making, an older actuarial framework with a property the newer algorithms often lack: its factors can be placed on a form a human being can read, question, and challenge.

Pittsburgh: the tool that stayed

And in Allegheny County itself — the birthplace of one of the field’s most prominent implementations — the tool remains in daily use ten years on.

Civil-rights complaints alleging disability discrimination were filed in 2022, and Associated Press reporting in January 2023 said Justice Department civil-rights attorneys were scrutinizing the AFST.

We found no publicly announced DOJ findings, settlement, enforcement action, or disposition of those complaints as of publication.

We will return to the disability question directly in Part IV.

Hold the record in your head as you go:

One tool killed by its own director after severe performance problems.

One Los Angeles pilot abandoned after thousands of false positives, followed years later by a separate model the county says performed better and is now expanding.

One Oregon tool terminated under investigative scrutiny.

And one prominent Pennsylvania tool still in use after becoming the subject of unresolved civil-rights complaints.

That is the county-level record against which the federal government now proposes to support statewide predictive-analytics demonstrations.

But the models did not build themselves.

THE BUILDERS

WHEN THE INVENTORS GRADE THE INVENTION

Two academics, four jurisdictions

Much of the modern predictive-risk literature in American child welfare runs through two researchers.

Rhema Vaithianathan, a health economist at Auckland University of Technology and director of the Centre for Social Data Analytics, and Emily Putnam-Hornstein, formerly of the University of Southern California and now at the University of North Carolina at Chapel Hill, where her work is tied to the Children’s Data Network.

Together or through their research teams, they helped build the Allegheny Family Screening Tool.

Their work also contributed to Allegheny’s Hello Baby model, predictive-risk work in Douglas County, Colorado, and research connected to Los Angeles County’s later Risk Stratification Model.

Officials in these jurisdictions gave research and development teams access to integrated administrative data, and those teams worked with government partners to determine which features would enter the models.

This is not, by itself, an indictment.

Somebody has to build things.

The question is what happened next.

The conflict, stated plainly

A federally funded study co-authored by researchers involved in developing the Allegheny model later discussed predictive-risk modeling as a potential approach for other jurisdictions.

That overlap between development and evaluation is relevant and should be disclosed whenever the research is cited as evidence supporting broader adoption.

The same issue runs through parts of the evidence base that agencies cite in the tools’ defense.

One frequently cited finding favorable to the AFST — reporting reductions in unconditional and score-conditional racial disparities — comes from research co-authored by Putnam-Hornstein and Vaithianathan, both connected to development of the tool.

Allegheny County has also commissioned formal outside evaluations.

Hornby Zeller Associates examined the implementation process, and Stanford researcher Jeremy Goldhaber-Fiebert conducted an impact evaluation published in 2019.

A critical study from researchers at Carnegie Mellon University examined referrals in the county from August 2016 through May 2018.

Its findings raised a different concern: human call screeners frequently overrode the model’s recommendations, and those overrides affected racial disparities in screening outcomes.

Sit with that.

The humans were not merely carrying out the machine’s recommendation.

They were sometimes correcting it.

The Human Rights Data Analysis Group and ACLU have also argued that administrative data used in predictive models can encode existing patterns of surveillance and inequality.

We include the favorable findings, name their authors, and let the reader weigh them.

That is what a transparency organization does.

But provenance matters.

Several favorable studies involve researchers connected to development of the tools or evaluations commissioned by deploying agencies, while independent researchers have raised significant concerns.

Readers should weigh both the findings and the independence of the research behind them.

The next question is not academic.

It is financial.

Who sold these systems, and what infrastructure helped make them possible?

That is Part III.

YOUR MOVE: BUILD THE PAPER TRAIL

If Part I helped you identify a predictive tool, risk score, screening model, or decision-support system used where you live, do not stop at the tool’s name.

Now find its history.

Start with the exact name of the tool and search:

“[tool name]” child welfare
“[tool name]” evaluation
“[tool name]” contract
“[tool name]” audit
“[tool name]” bias
“[tool name]” vendor

Then answer five questions.

1. Who built it?

Look for:

a university
researcher
nonprofit
software company
consulting firm
government contractor

2. Who paid for it?

Search the tool name with:

contract
procurement
award
grant
RFP

Save anything that identifies the buyer, vendor, contract amount, or funding source.

3. Has anyone evaluated it?

Search:

“[tool name]” evaluation PDF

Then look at who conducted the evaluation.

Was it:

the tool’s developer?
the agency using it?
a researcher hired by the agency?
an outside independent evaluator?

4. Has the tool changed, expanded, or been shut down?

Search the tool name with:

pilot
terminated
expanded
replaced
suspended
2024
2025
2026

A ten-year-old article about a pilot does not tell you whether the tool still operates today.

5. Has anyone challenged it?

Search:

“[tool name]” lawsuit
“[tool name]” complaint
“[tool name]” civil rights
“[tool name]” discrimination
“[tool name]” investigation

Now create one simple note on your phone:

Tool:
Agency using it:
Developer/vendor:
What it predicts:
Evaluation found:
Current status:
Questions still unanswered:
Links saved:

Do not worry about solving the whole system.

You are building a paper trail.

Your second assignment:

Find out who built the score, who evaluated it, and whether it is still being used.

A NOTE ON SOURCES — AND WHAT WE COULD NOT CONFIRM

This installment is built on public records and identified reporting concerning the Illinois, Los Angeles, Oregon, and Allegheny County predictive-analytics programs; Associated Press reporting by Sally Ho and Garance Burke concerning the Allegheny Family Screening Tool; Allegheny County documentation and published evaluations of the AFST; research examining Allegheny referrals and worker overrides from 2016–2018; ACLU and Human Rights Data Analysis Group research concerning algorithmic child-welfare systems; Los Angeles County reports concerning Project AURA and the later Risk Stratification Model; and public records and reporting concerning Illinois DCFS and Rapid Safety Feedback.

Where the record requires qualification, we have done so:

Civil-rights complaints alleging disability discrimination involving the AFST were filed in 2022, and Associated Press reporting in January 2023 said Justice Department civil-rights attorneys were scrutinizing the tool. We found no publicly announced findings, settlement, enforcement action, or disposition. We therefore describe the public record as unresolved rather than characterizing the matter as a confirmed open federal investigation.
Some favorable AFST research involves researchers connected to development of the tool. Allegheny County has also commissioned outside process and impact evaluations. We identify that provenance rather than characterizing the AFST as never having undergone independent evaluation.
Project AURA and the later Los Angeles Risk Stratification Model are separate projects. The AURA pilot was completed in October 2014, and a 2017 county report said DCFS was no longer pursuing it. By late 2024, Los Angeles County reported favorable results from the separate Risk Stratification Model and announced its permanent adoption and expansion.
Los Angeles County reported 171 cases identified at the highest level of risk and 3,829 false positives in the AURA pilot. We do not characterize all 171 cases as confirmed true positives.
Statements concerning former Illinois DCFS Director George Sheldon and contracting practices are attributed to inspector findings and contemporaneous reporting rather than stated as independent allegations by Project Milk Carton.

Project Milk Carton is a 501(c)(3) transparency and public-data organization. This article is public-record research, not legal advice.

Corrections: if any figure or citation here is wrong, we will say so, prominently.

NEXT IN THE SERIES

The Number That Knocks — Part III: Follow the Money

The Vendors, Contracts, Federal Infrastructure, and Public Funding Behind Predictive Child Welfare.

Once it is published we will link it here.

If you enjoyed this work and feel encouraged by this article like this, please consider becoming a paid subscriber to 17sog.substack.com here on Substack, or donate at Project Milk Carton Inc.

Your gift keeps our site alive, spotlighting high-risk children! or at https://t.co/PvtnrODkNz

Your support helps us turn information into action for children and families by:

keeping high-risk and missing children visible
identifying gaps in the systems responsible for protecting them
giving families and communities clear, usable information they can act on
supporting education, outreach, and practical community-based solutions
maintaining the public-data and OSINT work that makes this accountability possible
building tools and resources designed to help communities protect children before they disappear into a system

Most importantly, please share this article.

Transparency survives when ordinary people help spread information.
Informed communities protect children.

ALL paid subscriptions are 100% tax deductible

Project Milk Carton | 501(c)(3) | EIN: 33-1323547

Project Milk Carton

The Number that Knocks - The Record