Running A/B tests but not sure if I'm doing it right. Need guidance from experienced practitioners!
My Current Setup:
- Testing new onboarding flow
- Sample: 50% control, 50% variant
- Running for 7 days
- Tracking D1, D3, D7 retention
My Questions:
1. Sample Size Calculation:
- How many users needed for reliable results?
- Does it vary by metric (retention vs monetization)?
- Tools to calculate minimum sample size?
2. Test Duration:
- Is 7 days enough for retention tests?
- Should I wait for full cohort completion (D7, D30)?
- How to handle weekly seasonality?
3. Statistical Significance:
- What p-value threshold do you use? (0.05? 0.01?)
- Confidence intervals - how important?
- Dealing with multiple comparisons?
4. Common Mistakes:
- Stopping tests too early?
- Sample ratio mismatch issues?
- Novelty effects?
5. When to Trust the Results:
- Small improvements (2-3%) worth implementing?
- Conflicting metrics (better retention, worse monetization)?
What's your A/B testing checklist? Share your frameworks!