Executive Accountability in the Age of AI

Governance is not what you claim. It's what you can prove. Reading version: chapters 1 and 2.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the author, except for brief quotations in reviews and certain other noncommercial uses permitted by copyright law. For permission requests, contact the author via apparens.nl.

Disclaimer. This book is intended for informational and educational purposes only. Nothing in this publication constitutes legal, regulatory, or compliance advice. The AI Control Index and the associated framework materials described in this book are governance design instruments. Their application to a specific organizational, legal, or regulatory context requires independent professional judgment. The author and publisher accept no liability for decisions made on the basis of this material.

Generative AI tools, including large language models, were used in the writing and editing of this book. All substantive claims, governance design choices, framework specifications, and editorial decisions are the author's. The AI tools were used as a writing assistant, not as a substantive authority. The author bears full responsibility for the published text.

About the framework. This book refers to the Apparens AI Control Index, a governance framework developed by the author and freely available at apparens.nl/ai-control-index. This book is not a manual for that framework. It builds the way of thinking from which the controls follow; the index remains the place where those controls are maintained. The author has a commercial interest in Apparens and in the adoption of the framework.

Part I. Introduction

1. $2,754

From Deanna Amato's tax refund, the Australian government withheld A$1,709.87. Centrelink, Australia's social-services agency, claimed she had received A$2,754 too much in income support during the years she was a student.

Amato disputed the debt. Only then did Centrelink request the data needed to establish what she had actually earned in the relevant periods. Of the original A$2,754, A$1.48 remained. That residual debt was waived.

So the government had first established a debt and set collection in motion. The evidence to determine whether the debt truly existed was gathered only after Amato objected.

The debt had not been worked out by hand. It came from a program later nicknamed Robodebt, and it began with a comparison of two records. Centrelink held the income people had reported for each fortnight while receiving income support. The Australian tax office held the total annual income their employers had reported. Where those figures diverged, it could suggest someone had received too much support.

Such a difference did not, in itself, prove a debt existed. It was a reason to look further.

Centrelink used to do exactly that. A caseworker requested data from the person or their employer and established what someone had actually earned in the weeks in question. Only then could it be calculated whether too much had been paid in a given period.

Under Robodebt, that work shifted to the individual. Through an online system they received a request to explain the difference and to confirm or correct income data from sometimes years earlier. If someone did not respond, or did not respond with enough data, the annual income from the tax records could be spread evenly across twenty-six fortnights. A debt was then based on that average.

The step in which Centrelink itself established what someone had actually earned in the relevant periods therefore no longer had to happen before the debt was imposed.

That step was the decisive one. Entitlement to income support did not depend on total annual income, but on income in each separate fortnight. Someone who worked full-time for a few months and then had no income could earn the same across the year as someone who received the same amount every two weeks. In the annual tax data, those two situations were indistinguishable. For the right to a benefit, they were fundamentally different.

A correct annual figure therefore did not tell you what someone had earned in a given period. And it was exactly that periodic income that determined whether a debt existed.

Robodebt thus used an administratively convenient figure as a substitute for the legally relevant fact. Centrelink did not first establish what someone had truly earned and then calculate the debt. The program calculated a debt first, and then asked the citizen to prove it wrong.

Anyone who could not rebut the calculation was stuck with the outcome. People had to dig up pay slips from years earlier, sometimes from employers that no longer existed. The absence of evidence was not treated as a failing of the organization imposing the debt. It became the problem of the person who received it.

Amato's A$2,754 was therefore not simply an arithmetic error. The amount had been calculated exactly as the system was designed to calculate it. The problem was that the calculation had been performed before anyone had gathered the data that could support it.

Amato was one case. Between 2015 and 2019, the Australian government used the same method to raise roughly 794,000 debts against some 526,000 people.

Behind those numbers lay not administrative corrections but years of letters, phone calls, and collection. People paid out of a benefit that was often barely enough to live on, watched their tax refunds vanish, or were called by private debt collectors while they were still disputing the debt. Some moved back in with their parents, put their studies or other plans on hold, or developed a lasting dread of every letter from Centrelink or another government agency.

Parliamentary inquiries documented depression, insomnia, stress-related physical illness, self-harm, and suicidal thoughts. In at least two of the cases examined, families connected the suicide of a young man to the pressure of Robodebt. A death cannot be reduced cleanly to a single cause; both men had also been dealing with other problems and vulnerabilities. But the debts, the repeated contacts, and the collection pressure were part of the circumstances they found themselves in.

Robodebt was not the Australian version of the Dutch childcare-benefits scandal. But it was the same kind of administrative failure: the state treated an uncertain calculation as an established fact, made the citizen prove the opposite, and in the meantime brought its full power to collect.

That the method worked this way was known early on. In December 2014, before the approach was automated at scale, internal legal advice held that a debt based on averaged annual income sat uneasily with the law. That law turned on the income a person had actually earned in the relevant periods.

The approach was rolled out anyway, first as a pilot and from 2016 as an automated online process. Almost immediately, complaints came in from people who received a debt they did not recognize. Welfare organizations warned about the reversed burden of proof and about the practical difficulty of retrieving wage data from years before.

The Commonwealth Ombudsman investigated the program and published a critical report in 2017. The method stayed in use.

Warnings came from the tribunals too. Between 2016 and 2022, the Administrative Appeals Tribunal ruled at first instance 431 times that a debt calculated wholly or partly on income averaging was insufficiently proven or could not be recovered. In roughly 114 other cases, averaging was accepted in the circumstances of that case.

Those first-instance decisions were not published. Nor did the government put the central legal question to a public, authoritative ruling on appeal. As a result, decisions rejecting the legal basis of the method could stay buried inside individual files while the same method continued elsewhere.

Only in November 2019, after advice from the Solicitor-General that averaged annual income was insufficient without further evidence, did the government abandon averaging as a standalone basis for a debt.

Shortly afterward, in Amato's case, the Federal Court recorded, with the state's consent, that the data used could not support her debt.

In hindsight, the test was simple. Take one debt and pull the evidence of the income actually earned in the periods it rested on. For Amato, A$2,754 came down to A$1.48.

For the hundreds of thousands of debts calculated the same way, that test had not been run before the letter went out. By the time the method was abandoned, people had paid, debt collectors had been engaged, and tax refunds had been withheld.

Robodebt used no language model and no independent agent. The automation was technically simple. The governance failure was not. A difference between two records was treated as proof of a debt. An average took the place of the factual finding the law required. The consequences were imposed before the organization could show that the facts they rested on were real.

A calculation, an approval, a document, or a dashboard can exist and can be produced. That does not make it proof of what the organization says it is. The governance test begins one layer lower: which fact has actually been established, from what source, by whom, and why may that fact carry these consequences?

Notes to chapter 1

1. Amato v Commonwealth, Federal Court of Australia, consent orders of 27 November 2019. The Australian government had withheld A$1,709.87 of Amato's tax refund over an alleged debt of A$2,754. On recalculation, an actual overpayment of A$1.48 remained; that residual debt was also cancelled. Victoria Legal Aid, "I hope everyone gets the opportunity for justice: a win for Deanna Amato in her robo-debt test case," 27 November 2019.

2. Royal Commission into the Robodebt Scheme, Report, 7 July 2023. Under the scheme, roughly 794,000 debts were raised against some 526,000 people. The roughly 470,000 debts announced in 2020 for refund or zeroing, and the averaging debts found unlawful by the Federal Court, concern smaller and partly overlapping groups.

3. Senate Community Affairs References Committee, Centrelink's Compliance Program: Second Interim Report, September 2020, chapter 2, especially paragraphs 2.7–2.14 and 2.38–2.45; Royal Commission into the Robodebt Scheme, Report, chapter 10. For the individual harms described: Luke Henriques-Gomes, "Robodebt-related trauma: the victims still paying for Australia's unlawful welfare crackdown," The Guardian, 21 November 2020; ABC News, "Roughly 443,000 Australians received false Robodebt notices. These are some of their stories," 8 July 2023. The two cases referred to in the text concern Jarrad Madgwick (22) and Rhys Cauzzo (28), from different families, whose mothers (Kath Madgwick and Jennifer Miller) connected their deaths to the collection pressure; see SBS News, "Mothers whose sons took their lives after robodebts detail anguish in heartbreaking letters," 18 August 2020. The Senate committee stressed that causal statements about suicide are complex.

4. Administrative Appeals Tribunal: between 2016 and 2022, first-instance decisions found 431 times that a debt calculated wholly or partly on income averaging was insufficiently proven or not recoverable. In roughly 114 other cases, averaging was accepted. These first-instance decisions were not published. Justice E. Kyrou, Federal Court of Australia, "Mechanisms in the ART Bill to thwart Robodebt-type maladministration," speech of 18 March 2024.

5. The internal legal advice of December 2014, the 2017 Commonwealth Ombudsman inquiry, and the 2019 Solicitor-General advice are described in the 2023 final report of the Royal Commission into the Robodebt Scheme and in Prygodicz v Commonwealth of Australia (No 2) [2021] FCA 634.

2. What Passed for Enough

In November 2022, Jake Moffatt wanted to book a flight for his grandmother's funeral. On Air Canada's website he asked the chatbot about the bereavement fare. The chatbot told him he could also apply for the reduced fare after the fact, within ninety days of the ticket being issued.

The answer turned out to be wrong. Air Canada had no retroactive bereavement fare. On another page of the same website, the correct policy was set out.

Moffatt brought the matter before the Civil Resolution Tribunal in British Columbia. Air Canada argued that it could not be held liable for what the chatbot had said. As the tribunal put it, that amounted to treating the chatbot as a separate party, responsible for its own words.

The tribunal rejected the argument. The chatbot was part of Air Canada's website. To a customer, it made no difference whether information came from a static page or from a conversation with a chatbot. Moffatt did not have to cross-check one part of the website against another to work out which part spoke for Air Canada. The company was ordered to pay him CAD 812.02 in total.¹

The case was small next to Robodebt. One traveler, one wrong answer, and damages of a few hundred dollars. The scale, the technology, and the consequences all differed. The likeness lay in what passed for enough inside the organization.

At Robodebt, the available figure was total annual income. That figure was used as though it also showed what someone had earned in each separate fortnight. At Air Canada, an answer appeared on the company's own customer channel. That answer took on the role of an accurate statement of the applicable policy, without anyone having established that it matched it.

In both cases, information was available. What was missing was an explicit judgment about what that information was allowed to prove.

That is also where the two cases differ. Robodebt is about evidentiary basis: which fact must be established before a consequence may be imposed? Air Canada is about evidentiary judgment and authority: who determined that this answer stated the policy correctly, and on whose behalf did the system speak when it gave that explanation?

An organization can make two mistakes before it acts. It can use the wrong fact. And it can fail to establish, explicitly, who decides that the available evidence is enough.

The second mistake is harder to see. Nothing conspicuous is missing. There is an answer, a score, an attestation, a logged ticket, or a prior decision. The file is not empty. That is exactly what lets the process move on without it becoming visible which judgment has not yet been made.

A model score shows this. An evaluation measures how a system performs on a particular dataset, for a particular population and threshold. In the decision that follows, that score is then made to answer a different question: is this system suitable for this use? That does not follow from the score alone. For it to follow, someone has to establish that the test matches the people, conditions, and consequences of real use closely enough, which errors are acceptable, and what happens when the system finds itself outside the tested conditions. Without that judgment, a measurement quietly turns into a decision to deploy.

That substitution rarely happens in a single visible moment. It happens in ordinary work. Someone selects a source. Someone else folds the result into an assessment. An attestation is added to the file. A committee sees a completed field or a green item on a dashboard. The process moves on without anyone having separately established what that source, score, or attestation is actually allowed to prove.

A vendor attestation can pass for sufficient assurance about a product. The attestation records what the vendor claims. It does not determine what evidence the receiving organization itself needs in order to accept that claim.

A prior decision can pass for current approval. That permission was once given does not mean the system, the use, and the circumstances have stayed the same since. Nor does it say who is now authorized to give that approval again.

A logged ticket can pass for a solved problem. That the ticket is closed shows only that the workflow ended. It does not yet say what was investigated, what was decided, or whether the cause was removed.

The system goes live. The vendor review is wrapped up. The ticket is closed. None of those outcomes decides, on its own, whether going on is responsible. Between what is available and what the organization does with it, a judgment belongs.

That judgment has to do two things. It has to establish what the available material actually proves. And it has to make visible who is authorized to allow the next step on that basis.

In a large organization, choosing the source, assessing it, and deciding rarely sit with the same people. One person determines which data is used. Another assesses the result. A committee grants approval and an operational team carries the decision out. At each handoff, a document can travel on as though the judgment about its meaning had already been made upstream.

Everyone involved sees that something is there. No one still establishes whether it is enough.

That is where the risk arises. Not because the organization acts without information, but because it moves on using information that was given more meaning along the way than anyone explicitly took responsibility for.

At Air Canada, the answer stood on the company's own customer channel. To Moffatt, that made it a statement from the company. Only afterward did Air Canada try to open a distance between the organization and the system that had spoken on its behalf.

For an organization, that distance may be technically imaginable. For the person who faces the consequences, it does not exist. A customer sees no model architecture, no content management, no internal division of authority. He sees one organization giving an answer.

The question, then, is not only what stands in the file, the evaluation, or the dashboard. It is who established what the available was allowed to prove, on what basis, and with what mandate.

Notes to chapter 2

1. Moffatt v. Air Canada, 2024 BCCRT 149, Civil Resolution Tribunal, British Columbia, 14 February 2024. The tribunal characterized Air Canada's defense as the implicit suggestion that the chatbot was "a separate legal entity that is responsible for its own actions." It rejected that reasoning: "It makes no difference whether the information comes from a static page or a chatbot." Moffatt was awarded CAD 650.88 in damages, CAD 36.14 in interest, and CAD 125 in fees, together CAD 812.02. The case returns in chapter 8, where the focus is not the evidentiary function of the answer but the authority the organization gave to a system that speaks on its behalf.

This is where Part I ends.

The full book is available as an e-book and in hardcover, in English and in Dutch.

Go to the book →