Turning engineering mistakes into knowledge the team can use

In October, I shared a searchable set of engineering lessons with the team working on Arbitr. The material included lessons checked against the code, a domain glossary and a register of open findings. I asked contributors to update the lessons when they fixed the related problems.

The aim was practical. An engineer investigating a problem should be able to find what we already know. An agent working on the same area needs that context too.

Sharing those notes was a starting point. The next question is whether the information helps someone make a better decision during the work. A folder full of lessons gives us somewhere to look. The value comes when a lesson changes a requirement, a test or an implementation.

Start with the gap that let the problem through

When a defect escapes, fixing the immediate behaviour matters. I also want to understand what was missing from the way we built or checked it.

Was the requirement unclear? Did the test cover the individual component while missing the complete workflow? Did we assume another service would handle a failure? Was there already a rule, but the person doing the work could not find it?

Those questions lead to different actions. Adding a test helps when the expected behaviour is clear and repeatable. Clarifying a requirement helps when people have different interpretations of success. Recording an engineering rule helps when the same decision applies across several changes.

“Be more careful” gives the next engineer very little to work with. I want the finding to explain the missed assumption and the evidence that would have exposed it.

This also makes the discussion more useful. We can look at the conditions that allowed the mistake and decide what to change. The person investigating should not have to guess which part of the process we want them to improve.

Keep the lesson close to something verifiable

A useful lesson needs enough detail to act on. It also needs a boundary. A decision that makes sense for one workflow can cause problems when copied into another.

Language-code normalisation gives me a useful way to explain this. I worked on that in Tiri’s data pipeline, where equivalent legacy labels needed consistent handling. The example below is a hypothetical finding based on that kind of work. It is not a report of a particular defect:

• Observation: a language filter skips records carrying an equivalent legacy label.
• Missed assumption: every incoming record already uses the current language-code convention.
• Rule to confirm: apply the agreed mapping at the appropriate boundary before selection.
• Verification: include equivalent legacy and current labels in a test, and check that already-normalised values remain unchanged.
• Scope: identify the source, target and language-pair fields affected. Keep unknown-label handling explicit.
• Status: proposed until the implementation and regression check have been inspected.
• Source and owner: link the requirement, mapping, implementation and test, with someone responsible for keeping the record current.

The next person has something concrete to investigate. They can locate the mapping and check the behaviour. They can also see that the proposed rule does not answer every question about an unfamiliar label.

The distinction between an open finding and a verified fix matters. I would keep a proposed lesson marked as such until someone can point to the relevant change and check. Otherwise an engineer or agent may reuse an idea that has never been established.

Once verified, the record needs the supporting links and the conditions under which it applies. If the mapping changes later, those links give the maintainer somewhere to start.

Make the information available during the work

Our domain glossary has a different job from the lessons. It helps people understand what terms mean in our product. That matters when similar words refer to different states, responsibilities or business rules.

A lesson then adds the decision and its consequences. The open-findings register keeps track of questions that still need an answer. I want these pieces to connect, so someone reading a task can follow the relevant terms, constraints and unresolved concerns.

With agents, access alone does not establish that they understood the material. An agent can retrieve a note and still apply it too broadly. I want the review to show which rule mattered, how it affected the change and what evidence supports the result.

Customer information needs care as well. A reusable lesson usually needs the behaviour and the constraint. It rarely needs a customer’s actual files or identifying details. The knowledge should be available within the same access boundaries as the work it describes.

Give the lesson somewhere to go

In my notes for a recent talk on AI engineering workflows, I set out three responses when a problem gets through: add the missing test, clarify the requirement or document the rule for next time.

I want each useful finding to lead to one of those actions, or to a clear reason for leaving it open. Some findings will need more investigation. Some will only apply to an old implementation. Keeping those distinctions visible helps the team avoid treating every note as permanent guidance.

As the code changes, the lessons need attention too. A linked test can be removed. A service can take on a different responsibility. An old workaround can become unnecessary. Asking contributors to update the material with their fixes is part of keeping it useful.

I would judge this work by whether an engineer can find the relevant decision, understand its limits and verify the behaviour. The number of notes tells me very little on its own.

That is the feedback loop I want to strengthen: a problem gives us evidence, and we use that evidence to improve the next piece of work. We still need to check whether it is helping.