Halcyon Bank: The Test Says Ship. Should You?
Halcyon Bank ran an experiment on its retail banking product in UK.
The new checkout flow was tested against control with 8,710 users per arm over 10 days. Control converted at 3.5%; the variant converted 4.4% relatively higher. The team reports the result as significant at p < 0.05.
Two details sit further down the document. The team tested 3 variants against the same control, and checked results daily, calling the test when it crossed the threshold. A guardrail metric — refund rate — rose by 1.5 percentage points, which was reported as "not significant".
The PM wants to ship on Monday.
design
- run days
- 10
- stopping rule
- checked daily, stopped when p < 0.05
- users per arm
- 8710
- variants against one control
- 3
results
- absolute lift pts
- 0.15
- relative lift pct
- 4.4
- control conversion pct
- 3.5
guardrails
- reported as
- not significant
- refund rate change pts
- 1.5
Advise the PM. Your answer should provide:
- Analysis — whether this result supports the conclusion, given the sample and how the test was run.
- Risks — the specific ways this readout could be wrong.
- Recommendation — ship, iterate, or re-run, with what you would require.
State any assumptions you make.
80 points, 60% to pass.
- recommendation15
- market analysis15
- risk assessment25
- financial analysis25
Reveal suggested structure
Check power, then the stopping rule, then multiple comparisons, then guardrails. A p-value from a peeked test is not a p-value.