What machine translation taught me about evaluating AI

Before I worked with LLMs and AI agents, I was preparing translation data and training statistical machine translation models with Moses. That included cleaning datasets, generating word alignments with GIZA++, and experimenting with different training approaches.

Later came neural machine translation, then small language models and agent workflows.

The tools have changed considerably. Some questions keep coming back. What data are we using? What does a useful result look like? What evidence would convince us that something improved?

A lot of the work needed to answer those questions happens before the model runs.

Getting the data into a usable shape

A translation memory contains previous translations. That gives you useful material to work with, but it still needs selecting and preparing. The language, domain and age of the content affect what you can reasonably use it for.

During my contribution to Tiri, I implemented exports from Elasticsearch into reusable datasets, using Dask to organise the export work. The pipeline could select data by language, date and domain, then map existing records into a consistent format.

This was shared infrastructure. My contribution was specific parts of the data preparation and evaluation workflow, alongside the wider team’s work on training and fine-tuning.

Some changes were quite ordinary. I expanded tests for whitespace and non-breaking-space cleaning, corrected output-directory handling, and normalised legacy language codes.

Language codes are a useful example. An older system may label the same language in different ways. If those labels reach the next stage unchanged, equivalent records can be selected differently. A filter that looks reasonable can give you an incomplete view of the available data.

Normalising those codes makes the selection more consistent. It does not tell you whether every translation is good training material. That still needs a separate decision.

I think this is where evaluation starts: understanding what went into the dataset and what the preparation changed. Otherwise, a later difference in results can be difficult to explain.

Making evaluation usable

I added BERTScore evaluation and integrated existing TER results into the evaluation reports. These gave us additional ways to compare model output with reference translations.

Around that work were evaluation configurations, missing reference-data fields, inference-client behaviour and metric failures. Those details affect whether an evaluation run produces something the team can inspect and use.

A completed report can look convincing. It may contain several metrics, aggregate results and detailed comparisons. We still need to understand the conditions behind it.

Adding a metric gives us another view of the output. It does not, by itself, demonstrate that the translation model improved.

When I look at an evaluation result, there are four things I want to understand:

The dataset: what work does it represent, and what is missing?
The configuration: which settings could affect the result?
The comparison: what are we comparing, under which conditions?
The limitations: which conclusions would go beyond the evidence?

These questions help separate an interesting score from a decision we can defend. They also give someone else a way to question the result without having been involved in the original work.

An aggregate result can be a useful starting point. I would still inspect individual outputs before deciding whether it reflects the behaviour that matters.

Suppose, as an illustrative comparison, we evaluate two models on different exports. One export contains older language labels that were excluded by the selection filter. The other uses normalised labels and includes those records. Any difference in scores now needs investigation. We changed the data as well as the model, so we cannot attribute the result to the model alone.

A short comparison note would help make that visible:

Question: What behaviour are we trying to compare?

Inputs: Which dataset version, selection rules and preparation steps were used?

Run: Which model, inference settings and metric configuration produced the result?

Inspection: Which individual outputs support or challenge the aggregate result?

Conclusion: What can we say from this comparison, and what needs another run?

This is a proposed review aid, rather than a record of a particular experiment. The point is to give the next person enough information to reproduce the comparison and question the conclusion.

Carrying that thinking into agent workflows

With an agent, the output might be a code change, a document or a completed workflow. There are more steps to inspect, and more opportunities to misunderstand the goal.

The task and its context need the same care as the evaluation inputs. If the requirement changes during implementation, an earlier passing result may no longer answer the current question.

An agent can implement its interpretation of a requirement correctly while missing what the person needed. Another agent can review the change and share the same assumption. I still check that agreement against the system’s behaviour.

That does not make agent reviews unhelpful. They can identify questions and problems quickly. I want to understand what they examined and what evidence supports their conclusions, just as I would with an evaluation report.

The underlying habit is the same: keep the conclusion close to the conditions that produced it. If the input, configuration or scope changes, check whether the earlier result still answers the current question.

This has a leadership implication too. The team needs enough understanding to recognise weak evidence and investigate an unexpected result. That deserves investment alongside access to better models and tools.

My machine translation work gave me an early introduction to that discipline. Working with agents keeps bringing me back to it.

Before I ask whether the model is good enough, I want to understand how we would know.