What I check before saying AI made us faster

I found an error in the way our engineering dashboard calculated cycle time and worked with a colleague to have it corrected.

It was a useful reminder to check how a dashboard arrives at a number before using that number to explain engineering performance. A chart can look clear while the calculation underneath it needs attention.

The correction was a piece of engineering work in its own right. It did not establish how much faster the team had become or what caused a change in delivery. Those questions need their own evidence.

That distinction matters when we talk about AI. Agents make implementation activity very visible. We can see code appear, tests run and changes prepared for review. It is tempting to turn that experience into a broad claim about team productivity.

I want to be more precise about what we are measuring and what we can reasonably conclude.

First, understand the number

Before I use cycle time in a discussion, I want a plain explanation of the start and finish points.

Does work start when someone creates a ticket, when it enters development or when an engineer first changes the code? Does it finish at merge, deployment or when the customer can use it?

The same change can have several finish points:

Those are different measurement boundaries. Comparing development-to-merge in one period with development-to-customer-availability in another would make the trend difficult to interpret.

The definition needs to stay visible alongside the result.

The population matters too. Are we counting features, fixes and maintenance together? Are we looking at one application or several teams? What happens to cancelled work, reopened tickets or tasks that are still in progress?

I would start by tracing a few representative items from the source records to the dashboard. That gives us something concrete to inspect. If the reported duration differs from the underlying events, I want that resolved before debating the trend.

I also want to know which period the chart covers and how it summarises the data. An average, a median and a percentile describe different parts of the same distribution. A useful report should make that choice visible.

Unfinished work deserves attention too. If we only look at completed items, we should also inspect what remains in progress. Otherwise, long-running work can disappear from the discussion while the finished items look faster.

Activity is one part of delivery

An agent producing a change quickly tells us something about that task. It leaves other questions open.

Was the requirement clear? How much review and rework followed? Did the change solve the customer’s problem? Could we detect and recover from a failure?

The effort can move between stages. A quick first implementation may still require careful investigation before release. Equally, an agent may help clarify the requirement or identify a missing check before anyone starts building. Counting generated code would miss that contribution.

For a useful comparison, I want to follow a change far enough to understand the result. A smaller implementation time is valuable when it helps useful work reach users with an acceptable level of quality and risk.

This is why I am cautious about using ticket counts or code volume as the headline. They can describe activity, but they need context before we use them to explain delivery.

Check what else changed

Our engineering workflow has been changing alongside our use of AI. For example, I asked the team to focus on one or two tasks and work from a single prioritised engineering board.

That is relevant context for any delivery comparison. If a period includes changes to work in progress, priorities or how the team organises work, I cannot assign the whole result to the coding tool.

The same applies to changes in team composition, task size, platform maturity or release practice. These are questions to investigate, rather than reasons to dismiss an improvement.

I would want the comparison to name the important changes we know about. We may still see a useful improvement without being able to separate every contributing factor. In that case, the wording needs to reflect the limits of the evidence.

“Delivery improved during this period” is a different claim from “AI caused this improvement”. I want us to be clear about which one we can support.

Collect enough evidence to make the next decision

I do not think every team needs a complicated measurement programme before using agents. I do want enough evidence to decide where they help and where the workflow needs attention.

For a bounded piece of work, I would start with a few practical checks:

1. Agree what starts and finishes the work, and keep those definitions consistent.
2. Record the task type and the relevant review, rework and release steps.
3. Verify the outcome against the requirement, including important failure paths.
4. Note other workflow changes that could affect the comparison.

I would keep those details beside the result in a small comparison note: period, scope, start and finish events, summary measure, handling of unfinished work, quality checks and other known changes.

That is an approach I want to build on. It would help us decide what to try next while keeping the remaining uncertainty visible. It would still leave causal questions that need stronger evidence.

If review keeps finding the same misunderstanding, the next improvement may be better task context. If work waits for an integration decision, generating the implementation faster may have limited effect. We need to inspect the whole flow to see where the help is useful.

That dashboard correction reinforced the habit for me. Check the calculation, understand the scope and then discuss the result.

I am comfortable saying where I have seen an agent help with a particular task. A claim about the whole team needs evidence that covers the whole claim.