There are two separate experiments. The first collects randomised arm data to learn which offers work. The second tests the complete proposed policy, with its eligibility, delivery, budgets and no-offer choices, against the current one. A policy chosen offline stays a proposal until the second experiment supports its value.
Before running either, specify the population, assignment unit, outcome horizon, primary metric, smallest useful effect, sample-size/power assumptions, guardrails, duration and stopping rule. For several arm comparisons, state the multiple-comparison correction. For sequential monitoring, use a valid sequential method, not repeated stops on an ordinary p-value. Large profit outliers and cluster assignment affect power, so plan with historical outcome distributions and the real assignment design, not an arbitrary “run it for two weeks” rule.
For the primary intention-to-treat analysis, compare groups by randomised assignment. Comparing only delivered, opened, exposed or redeemed offers selects on post-assignment behaviour and can break randomisation. Keep delivery failure as a diagnostic. Before accepting the lift estimate, check assignment counts, missing outcomes and sample-ratio mismatch. Microsoft's SRM guide shows how biased inclusion can reverse an experiment's conclusion.
Shared household coupons, limited stock, contact-centre queues and marketplaces can create interference, where one customer's treatment affects another. Decide whether account, household, store, geographic cluster or a carefully designed time-block experiment fits the mechanism, then adjust inference and power for that unit. A time-block design may need washout for carryover. Microsoft's pre-experiment guidance discusses stable identifiers and network effects. Cluster assignment also changes precision and the effect being estimated.