Managed human evaluation · code & robotics
Eval data from people who can read the diff.
Named teams of vetted engineers grading coding agents, and trained reviewers grading robot episodes. Every delivery ships with inter-rater agreement scores. No anonymous crowds.
{
"client": "[redacted]",
"task_type": "pairwise_comparison",
"domain": "agentic_coding",
"items_delivered": 1840,
"evaluation_team": ["ENG-04", "ENG-11", "ENG-17"], // same engineers since batch_001
"inter_rater_agreement": {
"metric": "cohens_kappa",
"score": 0.81, // measured, not assumed
"double_rated_share": 0.20,
"disagreements_adjudicated": 112
},
"status": "shipped"
}what a code delivery actually looks like
// what we grade
01 / the-problem
Generic labeling platforms break on hard domains.
Not for lack of QA. The failure is structural: evaluating code and evaluating robot behavior are both expert work, staffed and priced like they aren't.
failure_mode[0]
Generalist raters miss what matters
A generalist can tell you which paragraph reads better. They can't tell you which of two diffs introduces a race condition, or whether a robot's grasp succeeded by policy or by luck. Judging what an AI system did is domain work.
failure_mode[1]
Unmeasured labels are just noise
Most platforms hand you labels with no reliability metrics attached. If you don't know whether two raters would agree on the same item, you don't know whether you're training on signal or on coin flips.
failure_mode[2]
Anonymous raters, rotating weekly
Recruiting thousands of raters in a week is a staffing achievement, not a data one. When the people grading your outputs change constantly, calibration never compounds, rubric interpretation drifts batch to batch, and there's no one to ask when a label looks wrong.
failure_mode[3]
Piecework pricing rewards speed
A careful agent-trajectory review, or a frame-accurate failure segmentation on a long-horizon episode, takes thirty to forty minutes. Per-task marketplaces price it like a captcha, so a captcha's worth of attention is exactly what it gets.
02 / domains
Two domains. One measurement stack.
Evaluating a coding agent and evaluating a manipulation policy are different jobs requiring different people. What doesn't change is the machinery around them: gold-set calibration, double-rating, adjudication, and an agreement score attached to every batch.
domain/code
Code & agents
Eval data from people who can read the diff.
Software engineers who read the diff, run the code, and follow an agent through a multi-step trajectory before scoring it. Vetted with the same coding screen you'd run for a hire.
who grades
Working software engineers
vetting = coding_screen + paid_calibration
task types
domain/robotics
Robotics & embodied AI
Eval data from people who can see the failure.
Reviewers trained on manipulation and navigation footage: was the task actually completed, where exactly did the policy break, and is this demonstration clean enough to train on. Calibrated against a gold set before they touch production clips.
who grades
Calibrated video reviewers
vetting = perception_screen + paid_calibration
task types
// scaling a robotics data program and need measured capacity fast? we staff and calibrate managed teams as a partner to labs and data companies both.
03 / how-it-works
From eval spec to measured delivery.
Scope the rubric
We start from your eval spec, or help you write one. Task types, grading rubric, edge cases, and a gold set we calibrate against before anything scales.
Assemble your team
A named team matched to the domain — engineers for code, trained reviewers for robot footage — vetted through screens and paid calibration rounds on the gold set. You see who passed, and who's grading.
Run the evaluations
Evaluators read the diff or scrub the episode, then grade against the rubric. A fixed share of items is double-rated; disagreements are adjudicated by a senior reviewer.
Deliver with receipts
Every batch ships with an agreement score per task type, disagreement logs, and per-evaluator stats. If agreement drops, we recalibrate before the next batch, not after.
04 / why-us
The parts that are hard to fake.
vetting = domain_screen + paid_calibration
Domain-qualified evaluators
Every evaluator passes a real screen for their domain — a coding screen you'd run for a hire, or a perception and failure-analysis screen for robotics — then paid calibration rounds before touching production tasks.
delivery.kappa = 0.81 // every batch
Agreement scores on every delivery
An inter-rater agreement score ships with every batch, computed on double-rated overlap. You see the reliability of the labels before you train on them. We're accountable to a number, not a vibe.
team = ["ENG-04", "ROB-09", "ENG-17"]
Named, consistent teams
The same evaluators grade your work week over week, identified in every delivery. Calibration compounds, drift gets caught early, and when you question a label, there's a person behind it.
conflicts_of_interest = []
Two domains, deep. Lab-independent.
We don't train models, we aren't owned by a lab, and we don't do general-purpose labeling. We go deep on code and robotics and decline everything else. Your rubrics, prompts, and episode data stay yours, and we have no stake in any benchmark.
05 / the-numbers
Measured, batch over batch.
κ 0.00
avg Cohen's Kappa across recent batches
0%
of items double-rated, every single batch
0%
of deliveries shipped with an IRA report
0 days
to stand up and calibrate a new domain team
06 / the-platform
Bring your own evaluators. Keep the measurement.
The platform is the stack our own evaluation teams work in every day, opening up as a product. If you run your own evaluators, an in-house team or a labeling company scaling a robotics or coding program, you can run them on it and get the same measured output.
Try it yourself
you're the evaluator. pick a domain — two tasks are assigned to you ↓
queue / task_2481
1 of 2 assigned
// Bug report: retry storm under load. Grade the model's patch.
src/retry.ts
@@ -13,5 +13,6 @@
for (let attempt = 1; attempt <= max; attempt++) {
const base = opts.baseMs;
- const delay = base * attempt;
+ const delay = base * 2 ** attempt;
+ const jitter = rand() * base;
await sleep(delay + jitter);
if (await tryOnce()) return;
}
// your label
correctness
code_quality
root_cause_fix
tests_adequate
regression_risk
flags
justification0/20
domain_native_review
Review surfaces built for the artifact
Side-by-side diffs with repo context and agent-trajectory playback for code. Frame-scrubbed episode playback with timeline segmentation for robotics. Not a text box with a viewer bolted on.
rubrics_as_versioned_config
Rubrics with a version history
Rubrics live as versioned config behind calibration gates. Every label records exactly which rubric version graded it, so drift is visible instead of silent.
agreement_math_built_in
IRA computed, not compiled
Double-rating, Cohen's Kappa, Krippendorff's Alpha, and disagreement queues run automatically on every batch. The reliability report your clients ask for writes itself.
// early partners onboard directly with our team
07 / next-step
Skeptical is the right starting point.
Book twenty minutes. We'll walk through a redacted real delivery in whichever domain you care about: agreement scores, disagreement logs, evaluator profiles. You can judge the signal yourself. No deck, no drip campaign.