Robodebt

What They Said

In 2016, the Australian Department of Human Services announced an “Online Compliance Intervention” program. The pitch was efficiency: replace manual caseworker reviews of welfare recipients with an automated income-matching system that compared self-reported earnings against ATO tax records. Officials projected A$4.5B in savings over four years.

The system was sold as a modernization of welfare integrity. Then-Minister Alan Tudge told the public that “we’ll find you, we’ll track you down, and you will have to repay those debts.” The political message was clear: technology would catch fraudsters that humans had missed.

What Actually Happened

The system did not detect fraud. It generated debts where none existed. Robodebt took an annual ATO income figure and divided it by 26 fortnights, assuming uniform earnings across the year. For anyone with seasonal, casual, or interrupted work — exactly the population on welfare — the math produced phantom debts of thousands of dollars.

Between 2016 and 2019, the government issued roughly 470,000 incorrect debt notices. Recipients were given 21 days to disprove the algorithm’s calculation, often using payslips from employers who no longer existed. The burden of proof had been silently inverted. By 2019, a Federal Court ruled in Amato v. Commonwealth that the income-averaging method had no legal basis. In 2020, the government agreed to refund A$721M and write off A$1.76B in unlawful debts.

The 2023 Royal Commission, led by former Queensland Supreme Court Justice Catherine Holmes, found the scheme was “neither fair nor legal.” Senior public servants had been warned in 2017 that the legal foundation was shaky and proceeded anyway. The Commission referred individuals for civil and criminal prosecution. Coroners and parliamentary inquiries linked the program to multiple suicides, including that of 22-year-old Rhys Cauzzo and 28-year-old Jarrad Madgwick.

The Root Cause

The system measured the wrong thing and no one was empowered to say so. Income averaging was a statistical convenience, not a legal standard. Caseworkers had previously cross-checked employer records by hand because Australian welfare law required actual fortnightly income, not annualized estimates. When the automation replaced that judgment, it replaced the legal basis at the same time — and the people who knew that were either ignored or pushed out.

The second failure was governance. Internal legal advice in 2017 flagged the scheme as likely unlawful. That advice was suppressed. There was no independent audit, no external red team, no mechanism for a frontline officer to halt a wrongful debt notice. The political incentive to claim savings overrode every internal warning.

The Pattern to Watch For

Any time you automate a decision that was previously made by a human applying judgment to context, you are also automating the legal and ethical reasoning behind that decision. If you cannot articulate the rule the algorithm is enforcing in plain language a regulator would accept, you have not built efficiency. You have built liability that compounds with every transaction.

What You Should Steal

Before automating any compliance, eligibility, or enforcement decision, write the rule the system will apply in one sentence and have your general counsel sign their name to it. If counsel won’t sign, the system isn’t ready. Robodebt would have been killed in 2016 if anyone had been required to put their name on the income-averaging methodology. The signature isn’t bureaucracy. It’s the circuit breaker that forces the legal question to the surface before 470,000 wrongful notices go out.