How I decide an AI-built change is ready to ship

In preparing a recent talk on AI engineering workflows, I kept coming back to the point where a developer accepts a change. An agent may have implemented it. Other agents may have reviewed it. Tests may have passed. Someone still needs to decide whether the evidence is enough to ship.

The developer needs to explain what changed, how it was checked and what could still go wrong. As agents take on more implementation, that judgement deserves more attention.

Start with an outcome someone can check

Specifications are rarely complete at the beginning. We learn more as we build. The implementation and the requirement need to be checked against each other as the work progresses.

For a concrete example, consider a hypothetical report-export feature. A customer selects some filters and downloads a report containing the work they are allowed to access.

“Export works” leaves several questions open. Does the file match the selected filters? Are its contents correct? Can a person download someone else’s report by changing the link? What happens when report generation fails?

Those questions turn a broad requirement into observable behaviour. They also expose decisions that need agreement. For example, should the report reflect the data when the export was requested or when it finished? The implementation will have some behaviour here, even if nobody has deliberately chosen it.

I want those choices made visible before they disappear inside the code.

Give each review a question to answer

An implementation review, a security review and a test run examine different parts of the change.

The implementation review can check whether the code follows the agreed behaviour and existing application patterns. A security review can examine access when the report is requested and downloaded. Tests can exercise the filtering, state changes and error handling.

Agents can help with each of those reviews. Clear responsibilities make the findings easier to assess. Several reviewers can still share the same wrong assumption, particularly when they all start from the same incomplete requirement.

Hence, the release decision needs evidence of what was checked, what happened and what remains uncertain. A conclusion such as “no issues found” needs enough detail for another engineer to understand its scope.

Follow the customer journey through to the result

For the hypothetical export, I would follow the journey from selecting the filters to opening the downloaded file. A successful response from the export endpoint only covers part of that journey.

The file needs inspecting. Do its records match the request? Are the expected fields present? Can the customer understand whether generation is still underway or has failed?

Then I would deliberately exercise the failures that matter for this change. An unavailable processing dependency and an attempt to download another customer’s report answer different questions. Both need an expected outcome before we can judge the observed behaviour.

I also want to know what happens after a failed attempt. Can the customer retry safely? Could that create duplicate work? Is an earlier download link still valid? These are decisions about the feature, with engineering consequences.

The review should state which paths were exercised. If access to the generated file has not been checked, that is an unresolved part of the feature. Passing the filtering tests does not answer it.

Make the acceptance decision easy to inspect

A release summary can be short and still useful. For this example, I would use a structure like this:

Outcome: The agreed report contents and access rules.

Evidence: The checks run, inputs used and observed results, with links to the relevant tests or review.

Limits: Paths that were not exercised and questions that remain open.

Operation: How a failed export is detected, investigated and recovered.

Decision: Who is accepting the change and any remaining risk.

Those fields need actual results before they support a release. Filling in a template is preparation for the decision.

Detection and recovery matter because a failure can occur after deployment. The developer accepting the change needs to know how to trace an export attempt without exposing customer data unnecessarily. They also need to understand what stopping or reverting the change would mean for work already accepted.

A code rollback may leave generated files or background work behind. The recovery approach has to account for the state the feature creates.

Keep ownership clear

A critical failure should stop progression until it is resolved or an appropriate owner explicitly accepts an exception. The exception needs to identify the risk, who owns it and what happens next. A deadline alone does not make the behaviour understood.

The developer accepting the change remains accountable for what ships. That includes understanding the implementation and the evidence, and recognising where specialist help is needed.

As an engineering leader, I want the team to get better at this decision. That means giving people room to question a proposed solution, investigate an incomplete customer journey and explain what the checks actually establish.

There will always be uncertainty. The useful response is to make it specific enough that someone can decide what to do about it.

If we say the change works, we need to be able to show what that means.