AI Audit

The Technical Definition

An AI audit is an independent assessment of an AI system’s behavior, performance, bias, safety, and compliance against a defined standard or set of expectations. Audits can be internal (run by a team that does not own the system) or external (run by a third party). They cover the model, the training data, the deployment context, and the controls around the system, and they produce a written report describing what was tested, what was found, and what was not in scope.

The result is supposed to be evidence — that the system does what it’s claimed to do, fails in the ways it’s claimed to fail, and doesn’t fail in ways nobody noticed.

What This Actually Means for Your Business

The market for AI audits is filling up with two kinds of work, and only one of them is worth paying for.

The first kind is the checklist audit. The auditor sends a questionnaire, your team fills it in, the auditor reviews the answers and a handful of policy documents, and a report appears in a few weeks confirming that you have an “AI governance program.” Nothing about the system was tested. No prompts were run. No data was probed. The deliverable looks rigorous because it’s long. It is not rigorous. It is the AI equivalent of a vendor security questionnaire — useful for procurement, useless for understanding whether the system is safe.

The second kind is the behavioral audit. The auditor demands access to the model or the deployed system and runs adversarial tests against it. They generate inputs the system has never seen. They probe for bias across protected classes by running matched-pair queries and measuring outcome differences. They try to extract training data. They test what the system does when its retrieval layer returns garbage, when its tools fail, when it’s asked to violate its own policy. They read the logs from real production traffic. The deliverable is shorter, harder to read, and far more useful — because it describes how the system actually behaves, not how the documentation says it should.

Don’t confuse an AI audit with SOC 2 or ISO 27001. Those audits assess controls — access management, change management, incident response. An AI audit assesses behavior. A system can pass every controls audit on the planet and still produce racially biased loan decisions, hallucinate citations into court filings, or quietly leak training data through completions. The controls were fine. The model was not.

Reality Check

What the vendor says: “Our AI is independently audited.”

What that means in practice: Ask for the report. Ask what tests were run, on what version of the system, by whom, against what standard. If the report is mostly procedural — policies, governance committees, risk registers — it’s a controls audit with “AI” stamped on the cover. If it includes adversarial test results, bias measurements, and a list of failure modes the auditor found, it’s the real thing.

What Operators Actually Do

The companies running AI seriously commission audits at two moments: before launch on any system that touches customers, money, or regulated decisions, and on a recurring cadence — annual at minimum — once it’s live. The audit scope is written before the auditor is hired, and the auditor is given enough access to break things. Locked-down audits where the auditor sees only outputs are nearly worthless.

They also separate the auditor from the integrator. If the same firm that built your AI system is the one auditing it, you don’t have an audit. You have a marketing exercise.

The pattern that works is read-the-report discipline. Operators read the full audit, not the executive summary. They interrogate every limitation the auditor flagged — what was out of scope, what couldn’t be tested, where the auditor lacked data — because that’s where the next failure will come from. The summary tells you what was checked. The limitations section tells you what’s still exposed.

The Questions to Ask

1. What did the auditor actually do? Specifically: how many prompts, what kinds, against what version, with what access. If the answer is documents and interviews only, this isn’t an audit of the system. It’s an audit of the paperwork around the system.

2. What’s in the limitations section? Every honest audit has one. If it’s a paragraph saying everything was tested and everything passed, the auditor isn’t being honest with you. Run.

3. Who is the auditor accountable to? A real auditor signs their name to findings and carries professional liability. A vendor’s “audit partner” hired to confirm the vendor’s pitch is not an auditor. They’re a reference.