Write the hypothesis, control policy, target population, primary outcome, minimum detectable effect, power and sample-size assumptions, assignment unit, duration and stopping rule. Use pre-treatment variance estimates and allow for repeat observations, clustering, delayed labels and attrition. Count independent assignment units, because rows are not automatically independent. A retention test needs its full outcome horizon, and a pricing test needs mature returns and costs.
A power sanity check. For two equal independent arms, a continuous outcome and a fixed two-sided 5% test with 80% power (the chance of detecting the minimum effect), a normal approximation gives:
n_per_arm ≈ 2 × (1.96 + 0.84)^2 × outcome_variance / minimum_effect^2With an invented contribution standard deviation of ₹200 per eligible account and a minimum useful mean difference of ₹5, this is about 25,088 accounts per arm. Halving the effect needs roughly four times as many accounts. This is a planning illustration, not a campaign-size rule. Heavy tails, unequal arms, multiple comparisons, covariate adjustment, attrition and clustered assignment change the calculation. Use design-based simulation or a suitable power method with representative pre-treatment data. Pricing market-days and repeat contacts are not independent accounts. Allow time for outcomes to mature after the last assignment.
Keep assignments stable where repeat exposure matters. If inventory, household sharing or market competition cause interference, consider cluster randomisation or a switchback design (switching units between arms over time), with its own carryover assumptions. Prespecify important segment checks and multiplicity handling. For ongoing peeking, use a valid sequential method. A daily dashboard does not turn fixed-horizon intervals into valid stopping rules.
Where feasible, run an A/A test (both groups treated alike) or a replay to check instrumentation. Reconcile assignment with delivery and outcome joins, and diagnose sample-ratio mismatch before reading lift. Microsoft documents the pre-experiment design practices and SRM checks. Use intention-to-treat as the main analysis, report actual delivery separately and explain any extra estimand.
Analysis by assignment still needs defensible outcomes. Report follow-up completeness by arm and recover missing outcomes where possible. Keep unreachable accounts in, and leave unknown margins missing instead of zero. Predeclare justified missing-data, censoring and sensitivity methods. If treatment affects whose outcome you observe, randomisation alone cannot prevent bias (Lee's sample-selection analysis). Separate a verified zero-purchase outcome from an unobserved one, and postpone the impact claim when evidence is thin.
For high-variance outcomes, a predeclared covariate adjustment such as CUPED can improve precision. Use only covariates available before assignment, not purchases or engagement that treatment affects. June 2026 CUPED research examines adjustment and valid variance estimation. Cross-fit learned adjustments where required, and keep the assignment and clustering structure in inference. Check coverage in an A/A simulation, and update the power calculation with justified variance estimates instead of assuming a fixed reduction.
Compare against the current business policy, not only a deliberately weak model. Look for a meaningful contribution result with acceptable guardrails, since a significant click-rate gain is a different claim. Investigate missing telemetry and immature labels before interpreting any result.
Decide how results generalise before expanding the pilot. An experiment among eligible existing customers in one region shows an effect for that population and period. In a new population, compare eligibility, action support, baseline outcomes, costs and delivery capacity. If those change much, replicate in stages: a larger model cannot make a transportability claim valid. Keep a control or run a new valid experiment when expanding the action policy.