| Data | Good exercise | What it cannot establish |
|---|---|---|
| Your own appropriately governed randomised campaign | Multi-arm incremental-profit policy with realistic constraints | Effects outside the supported population, actions and horizon |
| Criteo's corrected uplift dataset | Binary uplift benchmarking; policy evaluation under explicit sampling assumptions | Coupon amount effects, real contribution margin, temporal production validation, or original advertiser incrementality |
| MovieLens or another interaction dataset | Popularity, similarity, retrieval and ranking | The causal effect or profitability of financial incentives |
| A clearly labelled simulator with known potential outcomes and logged probabilities | Check policy logic, budget allocation, overlap failures, and estimator behaviour | Evidence that the learned policy improves a real business |
| An ordinary CRM/campaign snapshot | Response baselines, feature audits, and a design for collecting better evidence | Causal uplift without additional identification assumptions and evidence |
Criteo's official page describes an advertiser leakage problem in the original release and links the
corrected one. Use the corrected version, keep assignment proportions when sampling, record the file
checksum, and keep enough examples in both arms. Treat treatment as the assignment. exposure
happens after assignment, so do not use it to filter the causal sample or as a baseline feature. The
public features are anonymised, and non-uniform sampling hides the original incrementality.
Benchmark estimates do not translate into the original advertiser's business lift. The original
assignment and sampling logs are unavailable, and the released arm proportions are not the original
logging propensities. For any benchmark policy estimate, state the assignment and sampling
assumptions, test sensitivity, and validate the estimator on known-propensity simulations. The
release lacks the dated campaign history, multiple coupon actions and profit accounting that the
full retail decision contract needs.
For an undated benchmark, use a suitable held-out split and say that it cannot test future-period performance. You can add a dated simulator to exercise feature availability and deployment logic, but keep its results separate. Check each dataset's terms before use. Publish code, a manifest and download instructions instead of casually bundling source data.
Practise with the reference lab before a benchmark. The reference lab's evaluation report compares risk ranking with known-effect ranking at a fixed contact budget. Its simulated randomised experiment keeps failed deliveries in their assigned arm. Use it to check incremental-value arithmetic, budget allocation and the intention-to-treat analysis. Its effects are simulator assumptions, not evidence for a real offer.