E20 · Trust an automated judge?
Synthetic practice, not company performance.
One decision
Should automated review replace bilingual checks?
Run the exact query
SELECT candidate,COUNT(*) cases,SUM(task_pass=0 AND judge_pass=0) true_failure_detected,SUM(task_pass=0 AND judge_pass=1) missed_failure,SUM(task_pass=1 AND judge_pass=0) false_alarm,SUM(task_pass=1 AND judge_pass=1) true_pass FROM sim_eval_results JOIN sim_eval_cases USING(case_id) WHERE split='heldout' GROUP BY candidate;
Download SQL · Result CSV · Metabase saved question
Chart and reading
Compute failure-detection recall and false alarms; inspect critical misses.
Challenge your recommendation
How many important failures did the judge miss?
Guardrails: Synthetic labels; compute failure recall, inspect slices and critical misses before using a live blocking judge.
Next evidence: Real adjudicated labels, agreement slices and operational false-positive costs.
PRD sections: 12, 13. Open proposed PRD
Your spoken decision
State one takeaway, one caveat and one action in 30 seconds. Then stop.
Connect this finding
What this chart helps decide · Follow the connected decision · Canonical metric definitions
Optional model answer — try your own first
E20 · Trust an automated judge?
Decision: Should automated review replace bilingual checks?
Readout: Compute failure-detection recall and false alarms; inspect critical misses.
Actual query output (synthetic), first rows:
| candidate | cases | true_failure_detected | missed_failure | false_alarm | true_pass |
|---|---|---|---|---|---|
| baseline | 108 | 18 | 4 | 7 | 79 |
| candidate | 108 | 13 | 3 | 5 | 87 |
Model spoken answer: I would calibrate the judge against domain reviewers and retain human review where missed errors matter.
Caveat/counterargument: How many important failures did the judge miss? Synthetic labels; compute failure recall, inspect slices and critical misses before using a live blocking judge.
Next evidence: Real adjudicated labels, agreement slices and operational false-positive costs.
Deliberately left out: A single judge pass score as sufficient release evidence.