When decision dates are available, split train, validation and final test by time. Leave a gap, or filter training rows, so their outcome and return windows close before the training cutoff. Tune models, calibration, policy thresholds and budget rules on validation. Use the final test once, on the frozen policy.
For response probabilities, keep class weighting and any sampling inside training folds. Calibrate and evaluate on deployment-representative arm data or documented design weights. A balanced conversion sample asks a different probability question. Calibration folds must respect the same time and account boundaries as model selection, as described in Chapter 3. Report positives and observation coverage by arm: a pooled reliability plot can hide an unsupported offer arm.
If production serves existing customers repeatedly, they can appear in several periods, provided features are available at the right time and labels do not overlap. You need not force every customer to be unique across a time-based test. When new customers matter, also report a separate account holdout for cold start.
| Layer | Useful metrics | What they establish |
|---|---|---|
| Retrieval | Recall@K, eligible candidate coverage, new-item coverage | Whether relevant items reach the ranker |
| Ranking | NDCG@K, precision/recall with a stated candidate protocol | Ordering quality under the chosen relevance benchmark |
| Response | Log loss, PR-AUC, Brier score, reliability by arm | Prediction and probability quality |
| Causal targeting | Uplift curve, AUUC/Qini with the estimator and normalisation stated | Group-level treatment targeting on suitable held-out treatment/control data |
| Business policy | Incremental contribution per eligible account, total value, cost, uncertainty | Whether the chosen actions improve the decision objective |
| Operations and customer impact | Latency, fallback rate, delivery failure, budget violations, complaints, unsubscribes | Whether the system works within its constraints |
AUUC or Qini is a diagnostic, not the final campaign objective. State whether it evaluates a binary treatment, one arm against control, or a multi-action policy, because these differ. Tune any uplift curve before the final test, so its chosen contact depth stays an independent result. For public data without margins, report conversion/visit policy estimates with clearly labelled cost sensitivity, not verified incremental profit.
Why ranking alone can select a losing campaign. In this made-up example, two accounts have true incremental contribution of ₹12 and ₹6 before contact costs. Contacting either costs ₹10. Capacity allows at most two contacts. One model estimates effects of ₹12 and ₹6 and contacts only the first account: actual net value ₹2. Another estimates ₹24 and ₹12 and contacts both: actual net value ₹2 − ₹4 = −₹2. Both rank the accounts the same way. The error is in effect size and the cost-based threshold, which a ranking curve cannot see.
The August 2026 UpliftBench preprint studies this gap and metric sensitivity across outcome regimes. Its results are dataset-specific. Some threshold comparisons reuse the evaluation data, and its intervals measure split variability. Treat it as a reason to choose thresholds independently, not as a universal model ranking. On a validation experiment, inspect arm differences or DR estimates within effect-score bins fixed in advance, with counts and uncertainty. This checks effect sizes for groups, not any one person's counterfactual. Choose the threshold there, then evaluate the frozen policy on a separate experiment or test sample. Picking the best of many policies on one test set makes the winner look better than it is, even when each value estimator is valid.
For a fixed target policy pi and net reward r from the accounting contract,
inverse-propensity scoring estimates policy value with a weight pi(a|x) / p_logged(a|x). Here
a_i is the recorded assignment, including assignments whose delivery failed, and r_i is its
eventual observed outcome. The target policy must stay within the logged feasible set and
action/context support. In a held-out randomised sample, a doubly robust estimator combines
reweighting with an outcome model:
V_DR(pi) = mean_i [sum_a pi(a|x_i) * Q_hat(x_i, a)
+ pi(a_i|x_i) / p_logged(a_i|x_i)
* (r_i - Q_hat(x_i, a_i))]
incremental value = V_DR(pi) - V_DR(no_offer)Take nuisance predictions from a separate fit or cross-fitting. Inspect weights, action/context
support and effective sample size. For nonnegative weights w, the diagnostic effective sample size
is (sum w)^2 / sum(w^2). Large weights can make an estimate too unstable to use. Clipping weights
trades variance for bias, so report the threshold and sensitivity. Without extra assumptions, this
estimate cannot identify a target action with zero support.
Dudík, Langford and Li's policy-evaluation paper explains the
direct-model and importance-weighting trade-off behind doubly robust evaluation.
This estimates one decision at a time, over the logged opportunities. Include pre-decision history, feasible actions and resource state when they affect assignment. A new policy may change future contact eligibility, budget use, stock or customer behaviour, so future states can differ from the log. This formula cannot give the value of the whole adaptive campaign. Evaluate the complete policy in a controlled experiment, or justify a sequential design separately. Coordinate simultaneous price, offer and retention actions through the shared delivery plan. Separate per-action probabilities are not, in general, the probability of their joint assignment.
Compute confidence intervals at the assignment unit, such as the account, and keep cluster or stratum structure. If the logging policy changed through online learning, use inference suited to adaptive data. Intervals that treat rows as independent can mislead with repeated customers, campaign-wide shocks or adaptive assignment. For policy minus baseline, use paired uncertainty instead of subtracting two unrelated interval endpoints.
Evaluate profit and contact rate by action, tenure, sparse history, channel, region where relevant, and other pre-decision business segments. Show support counts as well as scores. If the apparent winner changes with a small accounting assumption, a slightly later time period, or reasonable weight clipping, the result needs more proof.