How I check software that AI wrote
Coding agents write most of my code. Nothing ships until it passes checks I control.
AI made code fast to write. It did not make it correct. This is the method I use on my own work: a person decides what the software must do, tests come from that decision, and automated checks, not the agent, decide what ships.
The problem
AI-written code looks right when it is wrong
Code from a coding agent is clean, consistent and confident. That is what makes its mistakes hard to see. They are rarely typos. They are misreadings of what was asked, delivered in tidy code that passes a quick look.
So I don't review AI output the way I'd review a person's. I don't hunt for mistakes line by line. I ask for proof: run it, test it, and trace each requirement to what the software actually does.
The method
Four rules every build follows
Coding agents write most of the code. These four rules decide whether any of it ships.
I write what the software must do before any code exists.
Spec first.
Tests come from that spec and the API contract, not from the code the AI wrote.
Contract tests, typed schemas, generated OpenAPI.
Checks decide pass or fail by their result, not by what the agent reports.
Gate register, exit codes, blocking gates.
Every mistake I find becomes a permanent check, so it costs once.
Recorded frictions compiled into the next build's instructions.
Failure and fix
When my own checker lied
This one is about my own tooling, not the AI. One of my checks looked for a failure by reading four field names from the record of each run. The runtime sent none of those fields. So the check never found a failure, and every run showed PASS.
Over 240 runs it reported PASS and never once reported FAIL. 157 of those runs had a failure sitting in their own output.
The test for the checker passed for a month. It built its own made-up input, in the shape I expected rather than the shape the runtime actually sent, so it proved nothing.
I found it by reading the raw record and noticing that no row ever said FAIL. A check that never fails isn't checking anything.
The fix: the checker's tests now run against a recorded real input, and a separate test pins the shape of that input. If the runtime changes what it sends, a test fails instead of the checker going quiet.
The lesson for a client: count the verdicts before you trust them. If a system has never reported a failure, find out whether it can.
Caught before merge
Three defects that never shipped
On 14 August 2026 I finished one module of the AI SaaS backend. It had 157 tests and 96.24% coverage. The tests caught three defects in its responses, and all three were fixed before the code was merged.
That is the normal case, and the point of the method. The mistakes get found by a test before the code reaches anyone.
The numbers
What the checks cover
From the AI SaaS backend I am building, and the tooling I use to check it.
4,500
tests on an AI-built backend
91%
line coverage
81%
branch coverage
765
tests on its React client
~1,000
tests over my own checking tools
3
defects in one module, caught before merge
What a client gets
The software, plus the evidence
You get the software and its full record: the spec it was built against, the decision log, the tests with their results and coverage, notes on security, and the docs to deploy it and recover it if something goes wrong. It is written so anyone you choose, an engineer or your own customer, can check the claims without taking my word for them.
The record is yours, along with the code. Everything about your software belongs to you.
Have AI-built software you're not sure about?
Tell me what it does and I'll tell you where it's likely to break.
Start a conversationThe problem
Code written by AI agents is fluent and confident, and its mistakes are misreadings of the requirement rather than typos, so they pass a casual review. Software that looks finished in a demo can fail with real users, real data and real money.
Constraints
- One engineer, with coding agents writing most of the code
- An agent's report that something passed is not evidence that it did
- Checks must decide pass or fail by their result
- The tooling that runs the checks must be tested as well
What I built
The specification and the API contract are written before any code exists. Tests derive from the contract through typed schemas and a generated OpenAPI document, not from the code the agent wrote. A register of gates rules pass or fail by exit code, and a blocking gate stops the work. Frictions recorded during each build are compiled into the instructions the next build starts from, so a mistake is paid for once.
How it fits together
- Routed context and protocol files per task type for the coding agents
- A gate register that records lint, type check, test and audit results by exit code
- A knowledge compiler that turns recorded frictions into the next build's instructions
- Around 1,000 tests over the tooling itself
- Checker tests run against a recorded real input, with a separate test that pins the input's shape
Running in production
A working method, not a deployed product. The AI SaaS backend it is used on is not launched yet.
My role
- Designing the method and building the tooling that enforces it
- Writing the specification and architecture each build follows
- Owning the result of everything the agents write
Outcome
- AI SaaS backend: 4,500 tests across unit, integration, end-to-end, smoke and chaos suites, 91% line coverage and 81% branch coverage, over 1,030 commits in three months
- React client for the same product: 765 tests
- One module, 14 August 2026: 157 tests, 96.24% coverage, three response defects caught and fixed before merge
- A checker that reported PASS on 240 runs, 157 of which held a failure, was found and fixed. Its tests now run against a recorded real input
Last updated