Pinecrest Grocers: The Test Says Ship. Should You?
Pinecrest Grocers ran an experiment on its grocery product in US.
The new checkout flow was tested against control with 1,640 users per arm over 15 days. Control converted at 7.3%; the variant converted 13.4% relatively higher. The team reports the result as significant at p < 0.05.
Two details sit further down the document. The team tested 3 variants against the same control, and checked results daily, calling the test when it crossed the threshold. A guardrail metric — refund rate — rose by 2.2 percentage points, which was reported as "not significant".
The PM wants to ship on Monday.
design
- run days
- 15
- stopping rule
- checked daily, stopped when p < 0.05
- users per arm
- 1640
- variants against one control
- 3
results
- absolute lift pts
- 0.98
- relative lift pct
- 13.4
- control conversion pct
- 7.3
guardrails
- reported as
- not significant
- refund rate change pts
- 2.2
Advise the PM. Your answer should provide:
- Analysis — whether this result supports the conclusion, given the sample and how the test was run.
- Risks — the specific ways this readout could be wrong.
- Recommendation — ship, iterate, or re-run, with what you would require.
State any assumptions you make.
80 points, 60% to pass.
- recommendation15
- market analysis15
- risk assessment25
- financial analysis25
Reveal suggested structure
Check power, then the stopping rule, then multiple comparisons, then guardrails. A p-value from a peeked test is not a p-value.