Prediction validation. Use forward rolling windows on the SKU-store panel. Use a later final period once; tune only on earlier windows. Add held-out stores or products when deployment must serve unseen groups. A group split alone cannot prevent temporal leakage, and a time split alone cannot test cold-start products. Fit encoders and transformations within each training window. Use dependence-aware folds for causal nuisance models, since library defaults may assume independent rows.
Report seasonal-baseline, GLM and challenger results on the same rows. Use count deviance and MAE, with bias and observed-versus-predicted totals by SKU, store, season, stock availability and price band. MAPE breaks at zero demand. WAPE is undefined when total actual demand is zero and can hide poor low-volume performance. Keep uncensored demand checks apart from realised sales checks, and state what is being forecast.
Response validation. On randomised or otherwise identified held-out data, compare demand and profit by assigned price, show uncertainty, and inspect monotonicity and heterogeneous effects. Report how much supported price variation remains after controls. Good MAE under the current price policy does not validate a new price curve.
Uncertainty. Keep three things apart: variation in future customer demand, uncertainty about the estimated causal mean response, and uncertainty about the policy's profit improvement. Quantile models can estimate demand prediction intervals; check their coverage, width, pinball loss and calibration on later data and operational slices. Check distributional count models for tail behaviour and stockout probability too. A refit/cluster bootstrap or estimator-specific inference can measure response-estimation uncertainty under the relevant assumptions.
Ordinary conformal coverage relies on assumptions that temporal drift may break, so rolling coverage checks and adaptation are needed. Gibbs and Candès study adaptive conformal inference under shift. Neither conformal prediction nor quantile calibration gives causal validity or guaranteed coverage for an unsupported candidate price.
Coverage after choosing a price. Per-price average coverage may not hold once a policy picks different prices in different contexts. The July 2026 Zheng and Jin preprint studies this with policy-coupled coverage under i.i.d. logging, supported identified actions and known assignment probabilities, separating model fitting, policy learning and calibration. It is optional research. Validate the frozen policy's selected-action coverage on independent identified data with a justified weighting or design strategy. Pooled coverage under the current policy is not enough, and a multiplicity adjustment for mean improvement does not solve this predictive-risk problem.
A made-up counterexample makes this concrete. Two prices each miss five of the same 100 contexts, on separate sets, so each has 95% coverage. A policy that picks each price in exactly the contexts it misses has only 90% selected-price coverage. These are constructed outcomes, not observed customer counterfactuals, but an optimiser can unintentionally favour poorly covered contexts.
Policy validation. Compare against the current pricing rule, with matured contribution profit per randomised unit and horizon as the primary outcome. Also track sell-through, revenue, stockout rate, returns, complaints, repeat purchase and constraint violations. Estimate uncertainty at the assignment unit: thousands of customer events in one treated market are not thousands of independent treatment assignments.
Offline policy evaluation needs the candidate policy's actions to have support under the logged assignment mechanism, and valid reward/assignment assumptions. On a finite price grid, IPS and doubly robust estimates can help. Inspect propensity errors, extreme weights, effective sample size and interval width. Dudík, Langford and Li describe the doubly robust approach, which cannot repair missing support or unmeasured confounding. Generic IPS on logs from a deterministic current policy cannot evaluate a different deterministic price. Continuous-price evaluation needs suitable density or smoothing assumptions, and binning prices afterwards does not create randomised data. A model-only profit simulation is not an observed lift.
Evaluating a fixed learned policy differs from evaluating an adaptive learner that updates as feedback arrives. The learner's actions depend on its history, so a one-shot IPS/DR score for its final policy misses cold-start learning, pending rewards and inventory depletion. For those claims, use a justified sequential evaluation design or a bounded live trial.