Syed Saqib AI-first. Full-stack. Forward-deployed. Data scientist.

Case study 03 · Turing (contract, remote) · Sep 2025 – May 2026

1,260 benchmarks in 12 days

Scaling an ML-agent benchmark from hand-ported competitions to about 105 a day, for a frontier AI lab

Four pods, 32 trainers, a QA pipeline with a dry run before every write, and a PII review before anything shipped.

✓ Compiled from this page's own markdown.

Role
ML Engineer, then Workstream Lead, then Project Lead (4 pod leads reporting to me)
Client
A frontier AI lab
When
Sep 2025 – May 2026
Tracks
Evaluation · Full-stack & FDE · MLOps

Clients anonymized, no task content. The numbers are mine; the secrets aren't.

// recorded 3–14 Feb 2026

1,260

real-world ML competitions turned into AI-agent benchmarks, in 12 days (about 105 a day)

3–14 Feb 2026

// recorded Nov 2025 – Jan 2026

75 → 254

Competitions in the agent environment I grew as Workstream Lead

Nov 2025 – Jan 2026

// recorded Feb – May 2026

56.8 min

Mean fix time per competition, across 32 trainers in 4 pods

Feb – May 2026

// recorded Feb – May 2026

1,180

Competitions scanned for PII before delivery (LLM-assisted)

Feb – May 2026

How it fits together

A pipeline with people in it

Hover or tap a box to see what it talks to.

Context

A frontier AI lab trains and tests ML-engineering agents on Kaggle-style competitions, packaged in the public MLE-bench format. Each competition is a small software project: a data split, a grader, a task description, a leaderboard and checksums.

I started on this work in September 2025, porting competitions by hand. In February 2026 I led the push that delivered 1,260 of them in 12 days.

The problem

A benchmark is only useful if it's both big and right. Big is a staffing problem. Right is an engineering problem. At an average of 56.8 minutes of fix time per competition, doing them one at a time doesn't scale. Doing them fast and sloppy produces a benchmark that quietly lies to the people training on it.

Constraints

  • Volume, fast. The lab wanted far more competitions than a hand-porting team could produce.
  • Graders must be provably right. A perfect submission scores 1.0, every time.
  • Compliance before delivery. Personal data and sensitive patterns had to be found and cleared, not discovered later.
  • People at different levels. 32 trainers, organized into 4 pods, each with a pod lead reporting to me.
  • One source of truth. The tracking sheet was the system of record for every competition's status.

What I built

1. Porting by hand first (Sep – Oct 2025, ML Engineer). I learned the format by doing it: config, data-split script, grader, task description, leaderboard and checksums. I implemented and tested competition metrics, including Spearman correlation and a character-span micro-F1 with its own grading tests, and drafted a Recall@20 setup for a recommender competition. I proved each grader by scoring a perfect submission at 1.0. Along the way I built two helpers: an MD5 checksum generator for dataset integrity, and a dispersion calculator for the benchmark's inclusion rule. I also wrote the case for and against including one borderline competition.

2. Growing the environment (Nov 2025 – Jan 2026, Workstream Lead). I grew the team's agent environment from 75 to 254 competitions, adding 179 with a 384-line installer script. The team delivered 179 agent trajectory files in four December batches, on two tracks. I coordinated a 40-person team across 82 tasks, reviewed and merged pull requests, wrote the documentation index and setup guide, and ran trajectory-review calibration with a peer reviewer.

3. The scale-up (Feb – May 2026, Project Lead).

  • Pods. 32 trainers in 4 pods, each pod lead reporting to me.
  • Sourcing at volume. 1,971 candidate competitions sourced and tracked. 359 were discarded on quality grounds.
  • Automated QA. A processing notebook checks each competition and writes its QA status back to the tracking sheet, with a dry run before every write. Nobody overwrites a row by accident.
  • A review app. I prototyped a review application in FastAPI, SQLAlchemy, React and TypeScript, about 3,500 lines, so reviewers could work from a queue instead of a spreadsheet.
  • PII review. Before delivery, 1,180 competitions were scanned with LLM-assisted detection, and 1,491 flagged patterns went to legal review.

How it fits together

A pipeline with people in it. Competitions enter at sourcing and pass an inclusion check. Pods port and fix them. An automated QA notebook checks the result and records its verdict in the tracker, dry run first. Pod leads review. Everything that passes goes through the PII scan; anything flagged waits for legal. Clean competitions are packaged into batches with checksums and delivered.

Rejection paths matter as much as the happy path. Competitions can leave at triage or at QA, and the reason is recorded. 359 left that way.

Stack

  • Code: Python, pandas, NumPy, scikit-learn, PyYAML, pyarrow, hashlib, Jupyter.
  • Benchmark tooling: the MLE-bench format, the Kaggle API and CLI, metric implementations (Spearman, span micro-F1, Recall@k), grader tests.
  • Operations: the Google Sheets and Drive APIs, gspread, matplotlib, Git and GitHub.
  • Review app: FastAPI, SQLAlchemy, React, TypeScript.
  • Compliance: LLM-assisted PII detection with human legal review.

Results

  • 1,260 competitions delivered in 10 batches between 3 and 14 February 2026: about 105 a day on average.
  • 1,971 sourced, 359 discarded on quality grounds rather than shipped.
  • 32 trainers in 4 pods, at a mean fix time of 56.8 minutes per competition.
  • PII review: 1,180 competitions scanned, 1,491 flagged patterns sent to legal.
  • Earlier on the same program: the agent environment grown from 75 to 254 competitions, and 179 trajectory files delivered.

What I'd do differently

  • Start with the review app. It arrived as a prototype partway through. It should have been the pods' cockpit from day one.
  • Scan for PII at intake, not just before delivery. Every hour spent fixing a competition that legal later blocks is an hour wasted.
  • Measure fix time by failure type. 56.8 minutes is an average of very different jobs. Knowing which failure types eat the hours is how you decide what to automate next.
  • Publish the rejection reasons. 359 discards are a dataset. Turned into a checklist, they make the next sourcing round faster.

What I left out

This is client work under NDA. Not included, on purpose:

  • the lab's name and the program's and tracks' internal names;
  • which competitions were included or discarded;
  • graders, data splits and task descriptions;
  • the inclusion rule's thresholds;
  • details of the PII findings;
  • the people: trainers, pod leads and their performance.

How to verify

  • LinkedIn, Turing experience entry (Mar 2024 – present): the 1,260 competitions turned into an ML-agent benchmark in 12 days, as Project Lead.
  • LinkedIn About: the 1,260 ML benchmarks in 12 days and the 377-person program.
  • The format is public. MLE-bench is an open benchmark format, so the structure of each competition (data split, grader, description, checksums) can be checked against it.
  • Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
  • In an interview: a walk-through of the method, within NDA limits.
  • Code: walkthroughs available on request.

Want the walkthrough?

Code walkthroughs are available on request, within NDA limits. Email is fastest.

Email me +92-333-4141218 LinkedIn Upwork Résumé (PDF)