Syed Saqib AI-first. Full-stack. Forward-deployed. Data scientist.

Case study 04 · Turing (contract, remote) · May – Aug 2026

The judge, and the statistics that judge the judge

An LLM-as-a-judge QA tool, grader calibration, and a statistical re-baseline for an RL environment program

One task went from 3/20 to 20/20 after a platform redeploy. Better agent, or broken task? I ran the numbers before anyone argued about it.

✓ Compiled from this page's own markdown.

Role
AI Engineer, RL environments and evaluation
Client
An enterprise data & AI platform company
When
May – Aug 2026
Tracks
Evaluation · Data science · Agentic & GenAI

Clients anonymized, no task content. The numbers are mine; the secrets aren't.

// recorded May – Jun 2026

≈89%

of the first delivery's tasks carry a verdict from the QA judge I built (92 of 103)

May – Jun 2026

// recorded Jul – Aug 2026

p = 2.6 × 10⁻⁸

One task's jump from 3/20 to 20/20 passes, by Fisher's exact test

Jul – Aug 2026

// recorded Jul – Aug 2026

3 of 4

Re-baselined tasks with a significant shift; the fourth (3/10 → 8/10, p = 0.07) was not

Jul – Aug 2026

How it fits together

Two loops: QA on the way in, trust when a number moves

Hover or tap a box to see what it talks to.

Context

Same RL-environment program as case studies 01 and 02, earlier in its life. Every eval team eventually faces two questions:

  1. Is the grader right?
  2. When a number moves, did the thing we measure change, or did the ruler?

Between May and August 2026 I built tooling for the first question and ran the statistics for the second.

The problem

Graders fail in two directions. One that fails a correct answer teaches the model the wrong lesson. One that passes a wrong answer hides the failure. Both look fine on a dashboard.

Pass rates fail too. With 10 or 20 runs per task, a "big jump" can be noise, and real improvements can hide inside noise. When a platform redeploy sent some delivered tasks' pass rates soaring, the team needed to know which tasks still worked as intended before anyone retired or rewrote them.

Constraints

  • Runs are expensive, so samples are small: 10 or 20 per task per model.
  • The client re-graded everything with its own judge. Our judge and theirs had to agree.
  • Delivered tasks are load-bearing. You don't rewrite one because of a hunch.

What I built

1. An LLM-as-a-judge QA tool (May – Jun 2026). An internal QA judge built on a frontier LLM's API: about 1,800 lines of Python, packaged so other authors could run it. For each new task package it runs:

  • duplicate-task detection against everything already delivered;
  • a live catalog collision check, so a new task's tables can't collide with existing ones;
  • format, taxonomy and review checks;
  • an auto-generated delivery row and verdict file.

2. Graders that are tested before they judge. Every grader I shipped was self-tested against the gold answer, two or three genuinely different correct answers, and every known wrong answer. Judge rubrics were written asymmetrically: sharp about the defect, loose about the route, five checks or fewer. From the third phase of the program, I reproduced every run's grade with the client's own judge.

One example. A probe run reached the correct answer but was graded a non-pass. The grader had read the agent's description of its own verification step as an admission of a shortfall. We fixed the rule (a verification sentence is a check, not a confession), re-ran the grader's self-tests and re-graded all 13 graded runs of that task. Only the misgraded run moved.

3. A statistical re-baseline (Jul – Aug 2026). After the client redeployed its agent, pass rates on some delivered tasks jumped. One went from 3/20 to 20/20. The project lead asked for a re-baseline. I re-ran the baselines of 4 tasks (50 runs), checked the workspace for drift, and tested each task's shift with Fisher's exact test.

  • Three of the four shifts were significant, with p-values from 2.6 × 10⁻⁸ (3/20 → 20/20) to 0.011. One of the three, 2/10 → 10/10, came in at p = 7.1 × 10⁻⁴.
  • The fourth, 3/10 → 8/10, looked like a win and wasn't one: p = 0.07.
  • The workspace had not drifted. So the gains were real: the agent had improved, not the tasks broken.
  • I flagged which traps needed recalibration, and the lead decided from evidence rather than from a hunch.

How it fits together

Two loops.

The QA loop runs on the way in. A new task package goes to the judge, which checks it against every delivered task, the live catalog, and the format and taxonomy rules, then writes a verdict file and the delivery row.

The trust loop runs when a number moves. Re-run the baselines, test each shift, and check the environment for drift, in parallel. Then decide: a real gain means the trap needs recalibrating; a broken task means fixing the task.

Stack

  • Judge: a frontier LLM through its Python SDK, structured verdict files, LLM-as-a-judge rubrics.
  • Statistics: Fisher's exact test (SciPy), pass-rate calibration bands, run distributions.
  • Data: live catalog queries against the data platform, SQL, Python, pandas.
  • Delivery: Git and GitHub, packaged CLI tooling for other authors.

Results

  • The judge's verdicts appear on 92 of the 103 tasks in the first delivery repo.
  • A misgrade caught and fixed without disturbing any correct grade: 13 runs re-graded, one moved.
  • The client's judge reproduced every shipped run on both model arms, so delivery had no regrade surprises.
  • A re-baseline that separated signal from noise: 3 of 4 shifts significant, one not, and no workspace drift.

What I'd do differently

  • Write the protocol before rerunning anything. The test, its sidedness, the threshold and the run counts belong in a short standing document. Choosing them after seeing the data is the easiest way to fool yourself, and the 3/10 → 8/10 case is exactly where it happens.
  • Re-baseline on a schedule. We re-baselined because of a surprise. A small canary set re-run every week would have caught the shift the day it happened.
  • Calibrate the judge against human labels first. A small human-labeled agreement set, measured before launch, finds misgrades earlier than probe runs do.
  • Show intervals next to pass rates. With 10 runs, the uncertainty is wide. Putting the interval on the dashboard ends a lot of arguments before they start.

I wrote up the method in the essay "Is that eval improvement real?".

What I left out

This is client work under NDA. Not included, on purpose:

  • the client's name and product, and which tasks moved;
  • the judge's prompts and the client's judge prompt;
  • trap designs, the taxonomy and task text;
  • model names and the details of the platform redeploy;
  • the delivery repository and its contents.

How to verify

  • Recompute it yourself. Fisher's exact test on the before/after counts reproduces every p-value here. In Python, scipy.stats.fisher_exact([[3, 17], [20, 0]]) gives p ≈ 2.6 × 10⁻⁸, and [[3, 7], [8, 2]] gives p ≈ 0.07.
  • LinkedIn, Turing experience entry (Mar 2024 – present): LLM-as-a-judge grading over 20+ runs on two frontier models.
  • Employment check: Turing, official title Consultant, contract, Mar 2024 – present.
  • In an interview: a walk-through of the method, within NDA limits.
  • Code: walkthroughs available on request.

Want the walkthrough?

Code walkthroughs are available on request, within NDA limits. Email is fastest.

Email me +92-333-4141218 LinkedIn Upwork Résumé (PDF)