We shipped an order through the pipeline and it came out the other end with a clean sweep. Code review passed. Generated tests passed. The security scan found nothing. The application built, started, and answered its health check.

It also did not work. The feature it had been asked to build was not there.

The logic had been written into its own module, correctly, and then nothing ever mounted it. The application served two routes: the index, and the health check. Everything the specification had asked for existed as functions nobody called.

What makes this worth writing about is not that a code generator made a mistake. That is unremarkable, and it is why gates exist. What is interesting is that four independent quality gates looked at the result and all four reported success.

Why every gate was right

Each gate did exactly its job.

The code review gate read the diff. The code in the diff was reasonable: sensible names, no obvious defects, no unhandled errors. There was nothing to complain about, because the code that existed was fine.

The QA gate generated tests and ran them. The tests it wrote exercised the functions it could see, and those functions worked. Nothing failed.

The security gate examined the change for the things that should never reach a merge request. It found none, correctly, because the change did not contain any.

The health check asked the running container whether it was alive. It was.

Every one of those answers was true. The application was well-formed, secure, tested and running. It simply was not the application that had been asked for.

The shape of the blind spot

Conventional quality gates share an assumption so basic it is rarely stated: that the artefact in front of them is the thing being judged. They evaluate a presence. Is this code correct, is it safe, does it pass its tests.

None of them can evaluate an absence. There is no diff for the route that was never registered. There is no failing test for the endpoint nobody wrote, and generated tests are especially prone to this, because they are derived from the code that exists rather than from the requirement. A test suite written against an empty implementation will happily reach full coverage of nothing.

The gap only becomes visible when you compare the result against what was promised, and none of the four gates had access to the promise.

A pipeline that only measures what was built cannot tell you what was forgotten.

This matters more with generated code than with hand-written code, for a reason that is easy to miss. A human who implements four of five acceptance criteria usually knows they have done so. The knowledge sits with the person, outside the diff, and it surfaces in a stand-up or a pull request description. When an agent does the same thing, there is nobody holding that context. The omission is silent, and every downstream check inherits the silence.

What we changed

The pipeline already had something the gates were not using: an approved specification, agreed before any code was written, listing the acceptance criteria and the interfaces the change was expected to expose.

So we added a gate that reads it.

Before a merge request is opened, the spec completeness check extracts the routes and behaviour the specification promised, then compares them against what the built application actually serves. For a web service that means reading the live schema the running application publishes. For a front end it means the routes the build produced. Anything the specification promised that the application cannot demonstrate blocks the order.

Run against the broken build, it reported the missing endpoint by name, and nothing else. No false positives on the routes that genuinely existed.

The matching is duller than it sounds, and deliberately so. Parameter names are normalised away, because a specification that says /books/{id} and an implementation that says /books/{book_id} agree. A leading API prefix is stripped. Trailing sentence punctuation is removed after parameter substitution rather than before, because doing it in the wrong order mangles the very placeholders you are trying to match. Most of the work in a gate like this is not the idea, it is refusing to cry wolf.

What this suggests about AI code review generally

The industry conversation about AI-generated code is mostly about quality: is the code good, is it secure, is it idiomatic. Those are the right questions for a coding assistant, where a person reviews each suggestion and carries the requirement in their head.

They are not sufficient once an agent is producing whole units of work. At that point the failure mode shifts. The dangerous output is not bad code, which review catches. It is plausible, well-formed, incomplete code, which review waves through.

If you are evaluating any tool that writes code into your repository, the question worth asking is not what its quality gates check. It is whether anything in the pipeline compares the result against the original request, and what happens when they disagree.

We would rather tell you the gate exists because we needed it than pretend the pipeline was always right.

Where this sits in the pipeline

The spec completeness check is one of four gates that run before a merge request is opened, alongside code review, QA and security review. It is the only one that can fail a build for something that is not there.

You can read the full walkthrough of how an order moves from a work item to a merge request on the how it works page.