Syed Saqib AI-first. Full-stack. Forward-deployed. Data scientist.

Case study 09 · Turing (contract, remote) · Sep – Oct 2025

13 ML problems to test a data agent

An auto-graded data-science benchmark for an agentic data assistant

Multi-table data up to 33.8M rows, hidden ground truth, and graders proven perfect before shipping. In about 9 days.

✓ Compiled from this page's own markdown.

Role
Data Scientist and Benchmark Engineer, one of about 10 developers
Client
An enterprise data & AI platform company
When
Sep – Oct 2025
Tracks
Data science · Evaluation

Clients anonymized, no task content. The numbers are mine; the secrets aren't.

// recorded Oct 2025

13

End-to-end, auto-graded ML problems built in about 9 days

Oct 2025

// recorded Sep – Oct 2025

33.8M

Rows in the largest multi-table dataset (3–10 tables per dataset)

Sep – Oct 2025

// recorded Sep – Oct 2025

1.67M

Blockchain transactions engineered into 55,350 address-level rows for a fraud-detection pilot

Sep – Oct 2025

How it fits together

What the agent sees, and what only the grader sees

Hover or tap a box to see what it talks to.

Context

An enterprise data & AI platform company wanted to measure how well its agentic data assistant handles real data-science work, not toy CSVs. The plan was a benchmark of end-to-end ML problems in the spirit of MLE-bench. In September and October 2025 I was one of about ten developers building it, as a data scientist and benchmark engineer.

The problem

Agents are getting good at single-file, Kaggle-style problems. Real business data doesn't come in one tidy file. It lives in many tables, and most of the work is joining them sensibly and engineering features across them. A benchmark that hands over one flat CSV measures the wrong thing.

Constraints

  • Hidden ground truth. The agent must never see the answers the grader uses.
  • Graders must be correct before they ship. A wrong grader quietly corrupts every score.
  • Reproducible data. Splits and ingestion must rebuild identically from scratch.
  • No duplicates across developers. Ten people sourcing datasets in parallel will collide unless someone checks.
  • The format changed mid-stream. Reviewers asked for a new split structure in October.

What I built

Dataset sourcing. I proposed 32 high-quality open, non-Kaggle datasets, each described against a structured schema. 11 of them went into the team's shared source list of 464 datasets.

Two pilot problems.

  • Fraud detection on 1.67M blockchain transactions, engineered into 55,350 address-level feature rows with extreme class imbalance, and modeled with Random Forest, XGBoost and logistic regression with oversampling, scored on F1.
  • A retail site-selection recommender, built from public US census business patterns and economic time series.

13 production problems (October 2025). Domains: grocery e-commerce, music streaming, grocery demand, recipe ratings, digital-media retail, corporate bankruptcy and bank lending. The data came from 7 public multi-table datasets, each with 3–10 tables and up to 33.8M rows. Every problem shipped with:

  • Training, Test and hidden Test_groundtruth splits;
  • an instructions.md task brief;
  • an eval.py grader against hidden ground truth (accuracy, RMSE, MAE or macro-F1).

Each problem was designed around multi-table relational joins, so an agent has to engineer features across tables rather than read one flat file.

Proof before shipping. A grader test harness (test_all_evals.py) proves that every grader scores the ground truth perfectly before a problem ships. Split scripts per problem and dataset ingestion through the Kaggle API make the data pipeline reproducible.

Review-driven iteration. I moved every October problem to the new split format the reviewers asked for, and dropped one dataset another developer was already using, to keep the benchmark free of duplicates.

How it fits together

What the agent sees and what the grader sees are deliberately different.

A public multi-table dataset passes a schema review: is this really multi-table work? A split script produces training and test tables, plus a hidden ground-truth file. The agent gets the task brief and the training tables. The grader gets the hidden answers. Before anything ships, the test harness proves the grader gives the ground truth a perfect score, and peer review can send a problem back for rework.

Stack

  • Python: pandas, NumPy, scikit-learn, XGBoost, imbalanced-learn, joblib, openpyxl.
  • Data: the Kaggle API for reproducible ingestion, public multi-table datasets, SQL-style relational joins.
  • Evaluation: custom graders (accuracy, RMSE, MAE, macro-F1, F1), a grader test harness.
  • Delivery: Git.

Results

  • 13 end-to-end, auto-graded ML problems in about 9 days, covering classification, regression, demand forecasting and recommender systems.
  • Multi-table data of up to 33.8M rows, from 7 public datasets with 3–10 tables each.
  • Every grader proven against ground truth before shipping.
  • 32 dataset proposals, 11 adopted into the team's 464-dataset source list.

What I'd do differently

  • Ship a naive baseline score with every problem. An agent's score means little until you know what predicting the mean, or the majority class, would have scored on the same split.
  • Test graders against plausible wrong answers, not just the perfect one. Proving a perfect submission scores perfectly is necessary, not sufficient. In my later RL work (case study 01), every grader also had to fail known wrong answers. I'd bring that in here.
  • Check for leakage across tables automatically. Multi-table data makes it easy to leak the label through a sibling table. A scripted check per problem beats eyeballing joins.
  • Freeze dataset versions with checksums from the first commit.

What I left out

This is client work under NDA. Not included, on purpose:

  • the client's name and product, and the benchmark's internal name and size targets;
  • which public datasets went into which problem;
  • task briefs, graders and hidden ground truth;
  • reviewers' comments and the client's internal results.

How to verify

  • LinkedIn project "ML benchmark suite for an agentic data assistant" (Turing, Sep – Oct 2025): 13 problems, multi-table data up to 33.8M rows, hidden ground truth, self-tested graders, built in ~9 days.
  • LinkedIn, Turing experience entry: the fraud-detection pilot on 1.67M blockchain transactions.
  • Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
  • In an interview: a walk-through of the method, within NDA limits.
  • Code: walkthroughs available on request.

Want the walkthrough?

Code walkthroughs are available on request, within NDA limits. Email is fastest.

Email me +92-333-4141218 LinkedIn Upwork Résumé (PDF)