Probability Lab // Cabinet 09

False Positive Arcade

Run the same A/B test again and again. Watch chance turn identical truths into conflicting headlines.

01 — Load the experiment

Baseline conversion10.0%

2%50%

True relative lift for B+0%

−50%+100%

Visitors per arm1,000

50010k

Confidence threshold95%

80%99%

Simulation seed

02 — Race one test

True rates loaded: A 10.00% · B 10.00%

—converts

Insert a probability token.

Choose settings, then run a race. The same seed and settings replay the same result.

03 — The wall of outcomes

A wins

—

B wins

—

Inconclusive

—

No batch loaded yet.

Batch runs reveal how often one fixed experiment design points in each direction.

Significant for ASignificant for BNot significant

Educational simplification: independent binomial samples, a pooled two-proportion z test, and a two-sided confidence threshold. Controls keep expected conversions at five or more per arm, but this remains an approximation. Real experiment decisions also need power planning, data-quality checks, guardrails, and correction for repeated peeking or multiple comparisons.