Engineering method·2026

How I check software that AI wrote

Coding agents write most of my code. Nothing ships until it passes checks I control.

AI made code fast to write. It did not make it correct. This is the method I use on my own work: a person decides what the software must do, tests come from that decision, and automated checks, not the agent, decide what ships.

The problem

AI-written code looks right when it is wrong

Code from a coding agent is clean, consistent and confident. That is what makes its mistakes hard to see. They are rarely typos. They are misreadings of what was asked, delivered in tidy code that passes a quick look.

So I don't review AI output the way I'd review a person's. I don't hunt for mistakes line by line. I ask for proof: run it, test it, and trace each requirement to what the software actually does.

The method

Four rules every build follows

Coding agents write most of the code. These four rules decide whether any of it ships.

  1. I write what the software must do before any code exists.

    Spec first.

  2. Tests come from that spec and the API contract, not from the code the AI wrote.

    Contract tests, typed schemas, generated OpenAPI.

  3. Checks decide pass or fail by their result, not by what the agent reports.

    Gate register, exit codes, blocking gates.

  4. Every mistake I find becomes a permanent check, so it costs once.

    Recorded frictions compiled into the next build's instructions.

Failure and fix

When my own checker lied

This one is about my own tooling, not the AI. One of my checks looked for a failure by reading four field names from the record of each run. The runtime sent none of those fields. So the check never found a failure, and every run showed PASS.

Over 240 runs it reported PASS and never once reported FAIL. 157 of those runs had a failure sitting in their own output.

The test for the checker passed for a month. It built its own made-up input, in the shape I expected rather than the shape the runtime actually sent, so it proved nothing.

I found it by reading the raw record and noticing that no row ever said FAIL. A check that never fails isn't checking anything.

The fix: the checker's tests now run against a recorded real input, and a separate test pins the shape of that input. If the runtime changes what it sends, a test fails instead of the checker going quiet.

The lesson for a client: count the verdicts before you trust them. If a system has never reported a failure, find out whether it can.

Caught before merge

Three defects that never shipped

On 14 August 2026 I finished one module of the AI SaaS backend. It had 157 tests and 96.24% coverage. The tests caught three defects in its responses, and all three were fixed before the code was merged.

That is the normal case, and the point of the method. The mistakes get found by a test before the code reaches anyone.

The numbers

What the checks cover

From the AI SaaS backend I am building, and the tooling I use to check it.

4,500

tests on an AI-built backend

91%

line coverage

81%

branch coverage

765

tests on its React client

~1,000

tests over my own checking tools

3

defects in one module, caught before merge

What a client gets

The software, plus the evidence

You get the software and its full record: the spec it was built against, the decision log, the tests with their results and coverage, notes on security, and the docs to deploy it and recover it if something goes wrong. It is written so anyone you choose, an engineer or your own customer, can check the claims without taking my word for them.

The record is yours, along with the code. Everything about your software belongs to you.

Have AI-built software you're not sure about?

Tell me what it does and I'll tell you where it's likely to break.

Start a conversation

The problem

Code written by AI agents is fluent and confident, and its mistakes are misreadings of the requirement rather than typos, so they pass a casual review. Software that looks finished in a demo can fail with real users, real data and real money.

Constraints

  • One engineer, with coding agents writing most of the code
  • An agent's report that something passed is not evidence that it did
  • Checks must decide pass or fail by their result
  • The tooling that runs the checks must be tested as well

What I built

The specification and the API contract are written before any code exists. Tests derive from the contract through typed schemas and a generated OpenAPI document, not from the code the agent wrote. A register of gates rules pass or fail by exit code, and a blocking gate stops the work. Frictions recorded during each build are compiled into the instructions the next build starts from, so a mistake is paid for once.

How it fits together

  • Routed context and protocol files per task type for the coding agents
  • A gate register that records lint, type check, test and audit results by exit code
  • A knowledge compiler that turns recorded frictions into the next build's instructions
  • Around 1,000 tests over the tooling itself
  • Checker tests run against a recorded real input, with a separate test that pins the input's shape

Running in production

A working method, not a deployed product. The AI SaaS backend it is used on is not launched yet.

My role

  • Designing the method and building the tooling that enforces it
  • Writing the specification and architecture each build follows
  • Owning the result of everything the agents write

Outcome

  • AI SaaS backend: 4,500 tests across unit, integration, end-to-end, smoke and chaos suites, 91% line coverage and 81% branch coverage, over 1,030 commits in three months
  • React client for the same product: 765 tests
  • One module, 14 August 2026: 157 tests, 96.24% coverage, three response defects caught and fixed before merge
  • A checker that reported PASS on 240 runs, 157 of which held a failure, was found and fixed. Its tests now run against a recorded real input

Last updated

Claude CodeExpressGoPostgreSQLPrismaRedisBullMQPrometheusGrafanaReact