Tiri: data and evaluation

The problem

Useful translation models depend on the work around the training run. Operational translation data needs to be extracted, prepared and made consistent. Evaluation needs to work across different models and imperfect datasets, and give the team a useful view of translation behaviour.

I contributed to that work for Tiri, our small translation language models. My contribution included model training and fine-tuning, with specific hands-on work in shared data and evaluation repositories.

Making operational data usable

I implemented translation-memory exports from Elasticsearch, using Dask to coordinate the work. The exports supported selection by language, date and domain, and produced consistent JSONL datasets for downstream tools.

This meant working with the shape of existing data: mapping legacy source and translation fields, controlling concurrent exports and allowing already-produced domain files to be skipped.

I also contributed cleaning tests, output-directory fixes and language-code normalisation. These are small details individually, but they affect whether data can be selected and compared consistently later.

Building evaluation capabilities

I added BERTScore evaluation using hypothesis and reference translations, with precision, recall and F1 results at segment and aggregate levels. That included multilingual baseline handling and configurable rescaling.

I integrated TER into evaluation reporting alongside other metrics. I also worked on dataset and model configurations, inference-client integration and handling failures such as missing reference data or tokenisation errors.

The distinction matters: implementing a metric gives the team another way to inspect behaviour. It does not, by itself, prove that a model has improved.

Working as part of the team

The training, data and evaluation infrastructure was shared work. I contributed specific capabilities and participated in training and fine-tuning, rather than owning the whole stack or every model result.

That combination kept the engineering questions connected: where the data came from, how it was prepared, what the evaluation measured, and how the model would fit into an operational workflow.

What I took from it

My earlier machine-translation work used Moses, bilingual datasets and word-alignment experiments. The model technology changed, but the need for useful data and honest evaluation remained.

I still find those questions valuable when working with AI agents: what evidence supports the result, what are we measuring, and where do we need human judgement?