Report predictive metrics separately from causal and policy results. An offline comparison of people who did and did not get a discount can mostly reflect how the old team chose them. If randomised logged actions support the target policy, evaluate that frozen policy on held-out assignments. For a frozen deterministic target policy on a discrete action set, an inverse-propensity estimate is:
estimated_value(policy) = mean(
I[logged_action == policy(context)] * net_outcome / logging_probability
)For a stochastic target policy, replace the indicator with the target probability of the logged action, given the same context and feasible set.
The logging probability is the chance of that action under the actual assignment policy, given context and candidate eligibility. It is not the model's confidence. Inspect overlap, extreme weights and effective sample size, since a deterministic old policy cannot evaluate actions it never tried. State the uncertainty and any clipping bias. Doubly robust estimation can reduce some errors, but it cannot repair missing action support or arbitrary unobserved confounding (Dudík, Langford and Li, 2011).
Here net_outcome is the observed, mature net contribution after actual costs for every eligible
assignment unit, including no purchase and no offer. A contacted nonbuyer can have negative
contribution. It is measured, not a predicted score or a baseline-subtracted uplift. The
formula evaluates an assignment policy when logged_action records assignment. Keep that original
assignment for intention-to-treat after human overrides or delivery failures. Executed actions need
their own justified propensities, so do not reuse the original probability for an override chosen
in an unknown way. Constraints applied before the random draw belong in the known assignment
mechanism. A delivery failure after the draw is a separate record, not a reason to invent a new
randomised probability.
For combined actions, log the joint assignment mechanism. An action can be a tuple of offer, price and retention choice, each with its no-change option. Its propensity is the probability of that tuple under the coordinator, given eligible candidates and pre-decision inventory, budget, contact history and other assignment state. Multiplying three marginal probabilities is valid only when the design makes the choices conditionally independent, which mutually exclusive or budget-sharing choices usually are not. For staged random draws, use the recorded conditional probability at each stage, given earlier choices. Their product is the tuple probability. Keep the coordinator version and assignment history to reproduce it.
This estimate covers one contextual decision over the stated outcome horizon. It does not identify a sequential campaign policy, where contacts change later eligibility or prices use up shared inventory. That needs a design that handles sequences and interference, or a prospective experiment of the whole policy. Conditioning on recorded stock and budget cannot show a new policy's effect on their future distribution, and a chain of probabilities for one compound action cannot evaluate a policy over time. Sequential off-policy methods need supported decision histories, correct assignment probabilities and a suitable dependence model. Jiang and Li's ICML 2016 paper develops a sequential doubly robust estimator under its stated trajectory assumptions. When allocations interact across customers, start with a controlled test of the whole coordinator.
If assignment probabilities adapt as results come in, keep their value at each draw and use inference valid for that adaptive design. Ordinary independent-row intervals can fail even with known probabilities. Hadad et al., 2021 (author preprint) studies policy-evaluation inference for adaptive experiments. Check its conditions, because logging propensities alone is not enough.
For continuous prices, this discrete formula applies only to a deliberately logged, finite price grid. A new continuous-price policy needs density-based or structural methods with more assumptions. Prefer a bounded randomised experiment to unsupported extrapolation. Uplift ranking scores alone cannot show net business value.