E13 · Reject a bad experiment readout
Synthetic practice, not company performance.
One decision
Can the apparent variant difference support rollout?
Run the exact query
WITH b AS (SELECT * FROM v_user_cohort WHERE ('all'='all' OR persona='all') AND ('all'='all' OR primary_workflow='all') AND ('all'='all' OR substr(signup_at,1,7)='all') AND date(signup_at)>='2026-05-01' AND date(signup_at)<'2026-10-01') , acc AS(SELECT account_id,COUNT(*) eligible_seats,COUNT(DISTINCT variant) variants FROM b WHERE mature7=1 GROUP BY account_id) SELECT COUNT(*) mature_accounts,SUM(variants>1) mixed_variant_accounts,1.0*SUM(variants>1)/NULLIF(COUNT(*),0) interference_rate,SUM(eligible_seats) eligible_seats FROM acc;
Download SQL · Result CSV · Metabase saved question
Chart and reading
Q13 counts mature accounts with mixed seat assignments; inspect this interference before inference.
Challenge your recommendation
Are team members exposed to both variants?
Guardrails: This lab assigns users, so team accounts can have mixed variants. This audit is not a causal comparison; fix assignment before inference.
Next evidence: Account-level assignment and actual exposure, preregistered success and harm criteria.
PRD sections: 12, 14. Open proposed PRD
Your spoken decision
State one takeaway, one caveat and one action in 30 seconds. Then stop.
Connect this finding
What this chart helps decide · Follow the connected decision · Canonical metric definitions
Optional model answer — try your own first
E13 · Reject a bad experiment readout
Decision: Can the apparent variant difference support rollout?
Readout: Q13 counts mature accounts with mixed seat assignments; inspect this interference before inference.
Actual query output (synthetic), first rows:
| mature_accounts | mixed_variant_accounts | interference_rate | eligible_seats |
|---|---|---|---|
| 167 | 35 | 0.20958083832335328 | 280 |
Model spoken answer: I would reject a causal conclusion here because seat assignment can mix variants inside a team account.
Caveat/counterargument: Are team members exposed to both variants? This lab assigns users, so team accounts can have mixed variants. This audit is not a causal comparison; fix assignment before inference.
Next evidence: Account-level assignment and actual exposure, preregistered success and harm criteria.
Deliberately left out: A ship recommendation based only on a favorable point estimate.