---
title: 1,260 benchmarks in 12 days
subtitle: Scaling an ML-agent benchmark from
hand-ported competitions to about
105 a day, for a frontier AI lab
slug: benchmark-scale-up
hook: Four pods, 32 trainers, a QA pipeline
with a dry run before every write,
and a PII review before anything shipped.
role: ML Engineer, then Workstream Lead, then
Project Lead (4 pod leads
reporting to me)
employer: Turing (contract, remote)
client: A frontier AI lab
period: Sep 2025 – May 2026
tracks:
- evaluation
- full-stack-fde
- mlops
tags:
- benchmark design
- MLE-bench format
- agent trajectories
- QA automation
- program leadership
- PII review
metrics:
- value: 1,260
label: real-world ML competitions turned
into AI-agent benchmarks, in 12 days
(about 105 a day)
when: 3–14 Feb 2026
- value: 75 → 254
label: Competitions in the agent
environment I grew as Workstream Lead
when: Nov 2025 – Jan 2026
- value: 56.8 min
label: Mean fix time per competition,
across 32 trainers in 4 pods
when: Feb – May 2026
- value: 1,180
label: Competitions scanned for PII before
delivery (LLM-assisted)
when: Feb – May 2026
---
## Context
A frontier AI lab trains and tests
ML-engineering agents on Kaggle-style
competitions, packaged in the public
MLE-bench format. Each competition is a
small software project: a data split, a
grader, a task description, a leaderboard
and checksums.
I started on this work in September 2025,
porting competitions by hand. In February
2026 I led the push that delivered 1,260
of them in 12 days.
## The problem
A benchmark is only useful if it's both big
and right. Big is a staffing problem.
Right is an engineering problem. At an
average of 56.8 minutes of fix time per
competition, doing them one at a time
doesn't scale. Doing them fast and sloppy
produces a benchmark that quietly lies to
the people training on it.
## Constraints
- **Volume, fast.** The lab wanted far more
competitions than a hand-porting team
could produce.
- **Graders must be provably right.** A
perfect submission scores 1.0, every time.
- **Compliance before delivery.** Personal
data and sensitive patterns had to be
found and cleared, not discovered later.
- **People at different levels.** 32 trainers,
organized into 4 pods, each with a pod
lead reporting to me.
- **One source of truth.** The tracking sheet
was the system of record for every
competition's status.
## What I built
**1. Porting by hand first (Sep – Oct 2025, ML
Engineer).** I learned the format by doing
it: config, data-split script, grader,
task description, leaderboard and
checksums. I implemented and tested
competition metrics, including Spearman
correlation and a character-span micro-F1
with its own grading tests, and drafted a
Recall@20 setup for a recommender
competition. I proved each grader by
scoring a perfect submission at 1.0. Along
the way I built two helpers: an MD5
checksum generator for dataset integrity,
and a dispersion calculator for the
benchmark's inclusion rule. I also wrote
the case for and against including one
borderline competition.
**2. Growing the environment (Nov 2025 – Jan
2026, Workstream Lead).** I grew the
team's agent environment from 75 to 254
competitions, adding 179 with a 384-line
installer script. The team delivered 179
agent trajectory files in four December
batches, on two tracks. I coordinated a
40-person team across 82 tasks, reviewed
and merged pull requests, wrote the
documentation index and setup guide, and
ran trajectory-review calibration with a
peer reviewer.
**3. The scale-up (Feb – May 2026, Project
Lead).**
- **Pods.** 32 trainers in 4 pods, each pod
lead reporting to me.
- **Sourcing at volume.** 1,971 candidate
competitions sourced and tracked. 359 were
discarded on quality grounds.
- **Automated QA.** A processing notebook
checks each competition and writes its QA
status back to the tracking sheet, with a
dry run before every write. Nobody
overwrites a row by accident.
- **A review app.** I prototyped a review
application in FastAPI, SQLAlchemy, React
and TypeScript, about 3,500 lines, so
reviewers could work from a queue instead
of a spreadsheet.
- **PII review.** Before delivery, 1,180
competitions were scanned with
LLM-assisted detection, and 1,491 flagged
patterns went to legal review.
## How it fits together
A pipeline with people in it. Competitions
enter at sourcing and pass an inclusion
check. Pods port and fix them. An
automated QA notebook checks the result
and records its verdict in the tracker,
dry run first. Pod leads review.
Everything that passes goes through the
PII scan; anything flagged waits for
legal. Clean competitions are packaged
into batches with checksums and delivered.
Rejection paths matter as much as the happy
path. Competitions can leave at triage or
at QA, and the reason is recorded. 359
left that way.
## Stack
- **Code:** Python, pandas, NumPy,
scikit-learn, PyYAML, pyarrow, hashlib,
Jupyter.
- **Benchmark tooling:** the MLE-bench format,
the Kaggle API and CLI, metric
implementations (Spearman, span micro-F1,
Recall@k), grader tests.
- **Operations:** the Google Sheets and Drive
APIs, gspread, matplotlib, Git and GitHub.
- **Review app:** FastAPI, SQLAlchemy, React,
TypeScript.
- **Compliance:** LLM-assisted PII detection
with human legal review.
## Results
- **1,260 competitions delivered in 10 batches
between 3 and 14 February 2026**: about
105 a day on average.
- **1,971 sourced, 359 discarded** on quality
grounds rather than shipped.
- **32 trainers in 4 pods**, at a mean fix
time of 56.8 minutes per competition.
- **PII review:** 1,180 competitions scanned,
1,491 flagged patterns sent to legal.
- **Earlier on the same program:** the agent
environment grown from 75 to 254
competitions, and 179 trajectory files
delivered.
## What I'd do differently
- **Start with the review app.** It arrived as
a prototype partway through. It should
have been the pods' cockpit from day one.
- **Scan for PII at intake, not just before
delivery.** Every hour spent fixing a
competition that legal later blocks is an
hour wasted.
- **Measure fix time by failure type.** 56.8
minutes is an average of very different
jobs. Knowing which failure types eat the
hours is how you decide what to automate
next.
- **Publish the rejection reasons.** 359
discards are a dataset. Turned into a
checklist, they make the next sourcing
round faster.
## What I left out
This is client work under NDA. Not included,
on purpose:
- the lab's name and the program's and tracks'
internal names;
- which competitions were included or
discarded;
- graders, data splits and task descriptions;
- the inclusion rule's thresholds;
- details of the PII findings;
- the people: trainers, pod leads and their
performance.
## How to verify
- **LinkedIn, Turing experience entry** (Mar
2024 – present): the 1,260 competitions
turned into an ML-agent benchmark in 12
days, as Project Lead.
- **LinkedIn About:** the 1,260 ML benchmarks
in 12 days and the 377-person program.
- **The format is public.** MLE-bench is an
open benchmark format, so the structure of
each competition (data split, grader,
description, checksums) can be checked
against it.
- **Employment check:** Turing, official title
Consultant, contract, Mar 2024 – present.
- **In an interview:** a walk-through of the
method, within NDA limits.
- **Code:** walkthroughs available on request.
Case study 03 · Turing (contract, remote) · Sep 2025 – May 2026
1,260 benchmarks in 12 days
Scaling an ML-agent benchmark from hand-ported competitions to about 105 a day, for a frontier AI lab
Four pods, 32 trainers, a QA pipeline with a dry run before every write, and a PII review before anything shipped.
✓ Compiled from this page's own markdown.
Context
A frontier AI lab trains and tests ML-engineering agents on Kaggle-style competitions, packaged in the public MLE-bench format. Each competition is a small software project: a data split, a grader, a task description, a leaderboard and checksums.
I started on this work in September 2025, porting competitions by hand. In February 2026 I led the push that delivered 1,260 of them in 12 days.
The problem
A benchmark is only useful if it's both big and right. Big is a staffing problem. Right is an engineering problem. At an average of 56.8 minutes of fix time per competition, doing them one at a time doesn't scale. Doing them fast and sloppy produces a benchmark that quietly lies to the people training on it.
Constraints
- Volume, fast. The lab wanted far more competitions than a hand-porting team could produce.
- Graders must be provably right. A perfect submission scores 1.0, every time.
- Compliance before delivery. Personal data and sensitive patterns had to be found and cleared, not discovered later.
- People at different levels. 32 trainers, organized into 4 pods, each with a pod lead reporting to me.
- One source of truth. The tracking sheet was the system of record for every competition's status.
What I built
1. Porting by hand first (Sep – Oct 2025, ML Engineer). I learned the format by doing it: config, data-split script, grader, task description, leaderboard and checksums. I implemented and tested competition metrics, including Spearman correlation and a character-span micro-F1 with its own grading tests, and drafted a Recall@20 setup for a recommender competition. I proved each grader by scoring a perfect submission at 1.0. Along the way I built two helpers: an MD5 checksum generator for dataset integrity, and a dispersion calculator for the benchmark's inclusion rule. I also wrote the case for and against including one borderline competition.
2. Growing the environment (Nov 2025 – Jan 2026, Workstream Lead). I grew the team's agent environment from 75 to 254 competitions, adding 179 with a 384-line installer script. The team delivered 179 agent trajectory files in four December batches, on two tracks. I coordinated a 40-person team across 82 tasks, reviewed and merged pull requests, wrote the documentation index and setup guide, and ran trajectory-review calibration with a peer reviewer.
3. The scale-up (Feb – May 2026, Project Lead).
- Pods. 32 trainers in 4 pods, each pod lead reporting to me.
- Sourcing at volume. 1,971 candidate competitions sourced and tracked. 359 were discarded on quality grounds.
- Automated QA. A processing notebook checks each competition and writes its QA status back to the tracking sheet, with a dry run before every write. Nobody overwrites a row by accident.
- A review app. I prototyped a review application in FastAPI, SQLAlchemy, React and TypeScript, about 3,500 lines, so reviewers could work from a queue instead of a spreadsheet.
- PII review. Before delivery, 1,180 competitions were scanned with LLM-assisted detection, and 1,491 flagged patterns went to legal review.
How it fits together
A pipeline with people in it. Competitions enter at sourcing and pass an inclusion check. Pods port and fix them. An automated QA notebook checks the result and records its verdict in the tracker, dry run first. Pod leads review. Everything that passes goes through the PII scan; anything flagged waits for legal. Clean competitions are packaged into batches with checksums and delivered.
Rejection paths matter as much as the happy path. Competitions can leave at triage or at QA, and the reason is recorded. 359 left that way.
Stack
- Code: Python, pandas, NumPy, scikit-learn, PyYAML, pyarrow, hashlib, Jupyter.
- Benchmark tooling: the MLE-bench format, the Kaggle API and CLI, metric implementations (Spearman, span micro-F1, Recall@k), grader tests.
- Operations: the Google Sheets and Drive APIs, gspread, matplotlib, Git and GitHub.
- Review app: FastAPI, SQLAlchemy, React, TypeScript.
- Compliance: LLM-assisted PII detection with human legal review.
Results
- 1,260 competitions delivered in 10 batches between 3 and 14 February 2026: about 105 a day on average.
- 1,971 sourced, 359 discarded on quality grounds rather than shipped.
- 32 trainers in 4 pods, at a mean fix time of 56.8 minutes per competition.
- PII review: 1,180 competitions scanned, 1,491 flagged patterns sent to legal.
- Earlier on the same program: the agent environment grown from 75 to 254 competitions, and 179 trajectory files delivered.
What I'd do differently
- Start with the review app. It arrived as a prototype partway through. It should have been the pods' cockpit from day one.
- Scan for PII at intake, not just before delivery. Every hour spent fixing a competition that legal later blocks is an hour wasted.
- Measure fix time by failure type. 56.8 minutes is an average of very different jobs. Knowing which failure types eat the hours is how you decide what to automate next.
- Publish the rejection reasons. 359 discards are a dataset. Turned into a checklist, they make the next sourcing round faster.
What I left out
This is client work under NDA. Not included, on purpose:
- the lab's name and the program's and tracks' internal names;
- which competitions were included or discarded;
- graders, data splits and task descriptions;
- the inclusion rule's thresholds;
- details of the PII findings;
- the people: trainers, pod leads and their performance.
How to verify
- LinkedIn, Turing experience entry (Mar 2024 – present): the 1,260 competitions turned into an ML-agent benchmark in 12 days, as Project Lead.
- LinkedIn About: the 1,260 ML benchmarks in 12 days and the 377-person program.
- The format is public. MLE-bench is an open benchmark format, so the structure of each competition (data split, grader, description, checksums) can be checked against it.
- Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
- In an interview: a walk-through of the method, within NDA limits.
- Code: walkthroughs available on request.