Syed Saqib AI-first. Full-stack. Forward-deployed. Data scientist.

Case study 01 · Turing (contract, remote) · May – Nov 2026

RL environments an agent can't sweet-talk

Reinforcement-learning environments, self-tested graders and a 15+ gate quality pipeline for an autonomous data agent

≈39% of the whole team's tasks. Each one with a frozen gold answer, a self-tested grader and 20+ graded runs.

✓ Compiled from this page's own markdown.

Role
AI Engineer, RL environments and evaluation; later founder of the team's task-authoring kit and operator of an agent fleet
Client
An enterprise data & AI platform company
When
May – Nov 2026
Tracks
Evaluation · Agentic & GenAI · MLOps

Clients anonymized, no task content. The numbers are mine; the secrets aren't.

// recorded May – Nov 2026

≈39%

of the whole team's RL tasks, delivered by me

May – Nov 2026

// recorded Final batch, 2026

54%

of the final batch: the one the client scrutinized most closely

Final batch, 2026

// recorded Jun – Nov 2026

20+

Graded runs per task, split across two frontier models

Jun – Nov 2026

// recorded Jun – Nov 2026

15+

Automated quality gates every task passes

Jun – Nov 2026

// recorded Oct 2026

0

delivered tasks broken after a 99-table workspace expansion (389 re-checked)

Oct 2026

How it fits together

One task, from design memo to delivery

Hover or tap a box to see what it talks to.

Context

An enterprise data & AI platform company was training its autonomous data agent with reinforcement learning. The agent works inside a data platform: it searches the workspace, writes SQL, runs notebooks and answers business questions. To learn, it needs environments: realistic tasks with answers that can be checked.

I joined the program through Turing in May 2026 as an AI engineer building those environments end to end. By the client hand-off in November 2026, I had delivered ≈39% of the team's tasks.

The problem

An agent learns from its environment. A sloppy environment teaches sloppy habits, or worse, teaches the agent to game the grader. The client needed tasks that were:

  • Realistic. They read like a real company's production data, not a tutorial.
  • Verifiable. One frozen, reproducible gold answer.
  • Hard for the right reason. The agent fails because of its reasoning, not luck or wordplay.
  • Fair. Every failure can be explained, and every pass is earned.

Constraints

  • The bar moved every few weeks. Prompts stopped naming tables, so the agent had to discover data across the whole workspace. Starter notebooks went empty. Gold answers needed multi-table SQL. Calibration moved to two models. Discovery depth became a gated requirement.
  • Difficulty was contractual. The primary model had to fail some runs and pass some. A stronger control model was not allowed to ace the task.
  • Two graders had to agree. The client re-graded every run with its own LLM judge, and our grades had to match.
  • The workspace was shared. Hundreds of delivered tasks depended on its exact data, so nothing that already existed could be edited.

What I built

A task is a package, not a prompt. Each one I shipped contained:

  1. A design memo first. A business question an analytics team would really ask, and the specific wrong turn an agent is likely to take. The wrong turn has to be explainable from the data itself, at a specific stage of the pipeline.
  2. A data estate. Realistic raw feeds, silver and gold layers, notebooks and rival sources, registered through an asset registry. Nothing pre-existing was ever edited.
  3. A gold answer. Frozen from live SQL (multi-table joins, CTEs, window functions), with an executed reference notebook and the expected wrong answers written down.
  4. A grader with its own tests. A Python checker, self-tested to pass the gold answer and two or three genuinely different correct answers, and to fail every known wrong answer. Plus an LLM-judge rubric of five checks or fewer, graded over the agent's whole trajectory.
  5. Calibration. A small probe round with a GO/STOP readout. Then 10 graded runs on the primary model and 10 on a frontier control model.
  6. Analysis. Every pass marked genuine or fluke. Every failure marked fair or unfair. Turn and tool-call statistics, each model's error patterns, and a contamination audit.
  7. Delivery. A pull request carrying a machine-generated gate report, a tracker row generated from the package, a workspace hand-over and evidenced cleanup.

A quality pipeline of 15+ automated gates. Contamination checks on both model arms. A scan for tell-tale signs of synthetic data. Trace integrity. Rubric alignment. A starter-notebook check. The per-run reasoning review. Taxonomy validation. Discovery scope and reachability. A readiness judge. The client's judge reproducing every grade. And a local CI audit that runs all of it, plus the delivery validators, before anything leaves my machine.

The reference task. I wrote the model task that the client's project lead turned into the design blueprint for every later task on the discovery track.

A bigger workspace that broke nothing. In October the client wanted a larger, more realistic workspace in which every delivered gold answer still held. I co-led the expansion with one other engineer: 26 waves added 99 tables and 99 notebooks in 83 new schemas, and my 14 waves added 51 of each. Then we regression-checked all 389 delivered tasks over 990 graded runs. I verified 48 of those tasks myself. One of my waves surfaced a new kind of risk, a rival source created by later task work. Checking for it became a permanent gate.

How it fits together

Think of an assembly line with a quality lab at every station.

The author side runs left to right: design memo, data estate, gold answer. The verification side proves the grader before the grader judges anyone. Calibration runs the task against two models and reads every run's reasoning, not just its score. Only then do the gates run, and only a fully green gate report becomes a pull request.

Two loops keep it honest. Any failed gate sends the task back to its design memo, not to a quick patch. And any change to the shared estate triggers a regression across every delivered task.

Stack

  • Data platform: SQL, governed schemas, raw / silver / gold layers, notebooks, the platform's CLI and Python SDK.
  • Languages: Python; SQL (multi-table joins, CTEs, window functions); PowerShell; Bash.
  • Agents and judges: Claude Code (subagents, skills, hooks), Cursor, LLM-as-a-judge rubrics, two frontier models for calibration.
  • Automation: a Playwright run harness, GitHub (gh CLI, PR workflow, branch strategy), the Google Sheets and Drive APIs for the tracker and asset registry, JSON Schema validation.
  • Statistics: pass-rate calibration bands, Fisher's exact test, run distributions and medians.

Results

  • ≈39% of the team's tasks delivered. 135 of them were merged at the hand-off.
  • 54% of the final batch, the one the client scrutinized most closely, and 13 tasks in the final week alone.
  • Every task shipped with evidence, not claims: code-graded results on 20+ runs, a machine-generated gate report and, from September, the client's own judge agreeing run by run.
  • 0 of 389 delivered tasks broke after the workspace expansion, across 990 graded runs.
  • The kit, skills and gates I started became the shared way of working for every operator on the project. That story is case study 02.

What I'd do differently

  • Build the regression harness on day one. We reached zero broken tasks because we checked all 389. I'd rather have had that check running from the first delivery than switched on for the expansion.
  • Test graders against tricky correct answers earlier. Most grader bugs I found were about language, not math. Adversarial-but-correct answers, written before the gold answer is frozen, flush those out sooner.
  • Track difficulty over time. Agents improve. A task calibrated in July can be easy by August (case study 04). A scheduled re-baseline beats a surprised one.
  • Treat rival sources as a design input. We found that risk class during the expansion. It belongs in the design memo template from the start.

What I left out

This is client work under NDA. Not included, on purpose:

  • the client's name, product and internal program names;
  • the names of the models under training and of the control model;
  • task text, prompts, rubrics, grader code, trap designs and the trap taxonomy;
  • the client's exact calibration thresholds and contract terms;
  • the business domain of the synthetic data, beyond "enterprise";
  • screenshots, traces, and anything from the delivery repositories.

How to verify

  • LinkedIn, Turing experience entry (Mar 2024 – present): the RL environments for an autonomous data agent, LLM-as-a-judge grading over 20+ runs on two frontier models, and zero breakages across 389 regression-tested tasks.
  • LinkedIn project "Multi-agent coding-agent factory for RL environments" (Turing, Jun – Nov 2026): ~39% of the program's tasks and the 15+ CI quality gates.
  • Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
  • In an interview: a walk-through of the method, within NDA limits.
  • Code: walkthroughs available on request.

Want the walkthrough?

Code walkthroughs are available on request, within NDA limits. Email is fastest.

Email me +92-333-4141218 LinkedIn Upwork Résumé (PDF)