4. Choose a model according to the evidence

Model-selection decision tree

Choose the question before the model. The incremental-effect branch asks whether an action changes profit against no offer. The relevance branch asks which product or offer is likely to interest the customer. “Identified” means randomised assignment with action support, or an observational design you can defend. Redemption or targeted-campaign history alone does not pass that gate.

Check data readiness first. If you lack the pre-decision features, exposure logs or mature labels, start with eligibility rules, popularity where supported, and the no-offer baseline. Collect decision, exposure and outcome logs using the data contract, then follow the tree.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Offer model-selection decision tree
    accDescr: Choose retrieval and ranking for a large relevance catalogue, logistic response prediction without identified effects, or uplift modelling when treatment effects are identified.
    Q{"Incremental<br/>effect?"}
    Q -->|No| C{"Large<br/>catalogue?"}
    Q -->|Yes| E{"Identified?"}
    C -->|Yes| O1["O1<br/>Retrieve<br/>+ rank"]
    C -->|No| O2["O2<br/>Logistic"]
    E -->|No| O2
    E -->|Yes| O3["O3<br/>S/T uplift"]

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class Q,C,E decision;
    class O1,O2,O3 model;

If a retrieved product carries an incentive, use O1 for relevance and follow the incremental-effect branch separately for the incentive. A better product rank does not show discount lift.

O1 — Large catalogue: retrieve, then rank

• Model and why: Start with eligible popularity/content candidates. When scoring the whole catalogue costs too much, compare retrieval plus LightGBM LambdaRank. Use two-tower retrieval only when interaction volume and measured serving needs justify learned embeddings.
• Minimum data: For learned models: customer–item interactions, item metadata, candidate and exposure/position logs, and meaningful relevance labels with query/session groups.
• Promotion check: Improve retrieval recall, ranking quality and latency at the same eligible candidate budget. Check cold-start coverage. Continue with retrieval and ranking.

O2 — Small table or unidentified effects: predict response

• Model and why: Fit logistic regression, then compare one CatBoost or LightGBM response model. That gives an inspectable baseline and a nonlinear challenger. Comparing predicted response scores cannot identify what an intervention caused.
• Minimum data: Pre-decision customer/offer/context features, observed assignments or exposures, and mature response labels including non-responses, with a deployment-matched holdout.
• Promotion check: Compare log loss, rare-event PR metrics, calibration and serving cost on the same split. Without effect identification, keep claims predictive and collect randomised offer/no-offer data. Redemption prediction is not uplift. Follow the predictive workflow and collection design.

O3 — Identified effects: estimate uplift and choose a profit policy

• Model and why: For randomised data, compare arm profit with a regularised S-learner or T-learner, respecting assignment probabilities and strata. For observational data, they need a defensible confounder-adjustment set, overlap (each action used for similar customers), and effect or value estimates that this design justifies. Raw historical arm averages do not show uplift. The target is each action's difference from control, over a small supported action set including no offer.
• Minimum evidence: Mature net outcomes and supported action/control assignments with known randomised probabilities, or a defended observational design that handles confounding, overlap, treatment definition and interference.
• Promotion check: Require held-out incremental contribution, support and uncertainty checks, then a policy experiment. Compare a cross-fitted DR-learner or causal forest only when arm sizes and heterogeneity justify it. Continue with uplift and profit policy and policy evaluation.

A bandit is a separate step. Consider one only once these work: assignment probabilities logged after constraints, action support, joins to mature rewards, safe exploration and a randomised holdout. A bandit cannot supply missing causal identification. Use the bandit prerequisites after the fixed-policy baseline.

Model comparison ladder

The table below is my suggested order of work. You still choose the model on your own held-out data, costs and latency budget.

Predictive baseline: logistic regression, then one boosted tree

Model response from pre-decision context and offer attributes. Compare it with rules and offer-level historical rates. CatBoost suits tables with categorical customer, product and offer attributes, because it handles them directly. LightGBM is another good candidate with explicit categorical handling. Choose on validation quality, memory, training cost, serving latency and team familiarity, since neither wins everywhere. CatBoost's categorical-feature documentation and LightGBM's advanced topics describe how each one works.

Measure log loss, PR-AUC for rare conversions, and reliability curves by action and customer segment. Calibrate on a separate, later validation sample, or with a split that respects time and account identity. After any negative sampling or class weighting, evaluate at the original deployment prevalence. Calibration matters because profit uses the size of a probability, not just its rank. The scikit-learn calibration guide explains why calibration data must be separate from model fitting.

For a small table, an optional tabular foundation-model comparison can be worthwhile. Use the version-specific guidance in Chapter 3. Compare on the same future opportunities, per-action calibration and serving budget. Good outcome prediction does not make it a causal learner.

This model is a good predictive baseline. Past offers were targeted at different customers for a reason, and comparing its predictions for two such offers does not remove that reason. Label a profit simulation built on these predictions as model-based and unvalidated causally. Feature importance or SHAP helps inspect a prediction. It cannot show which features cause customers to respond.

Incentive selection: randomized uplift and profit policy

Run a multi-arm campaign with no offer and the candidate actions. Apply the common eligibility rules, then randomise at a stable account or customer level. At each assignment, record the known probability of every feasible action. Stratifying by a few pre-decision segments can improve balance, but the logged probabilities must reflect that design. Every action the new policy might take needs enough support in its context.

Start with average profit per arm and its uncertainty. When the treatment-effect signal is weak, a segment or no-offer policy can beat a personalised model. Then compare:

• S-learner: one outcome model with the assigned action as an input. It shares information across arms, but if the main outcome signal dominates, it may learn little treatment variation.
• T-learner: one outcome model per arm, using regularised CatBoost/LightGBM or another suitable learner. Subtract the no-offer estimate. It is easy to inspect but can be noisy in small arms.
• X-learner: a useful comparison for a binary treatment/control problem with unequal arm sizes. Whether it helps depends on how the data was generated. Künzel and colleagues' meta-learner paper finds that no meta-learner is best in every case.
• DR-learner: combines outcome and assignment models with cross-fitting. It can handle several categorical treatments. Choose and validate the nuisance (helper) and final models separately. With a properly randomised design, give the known assignment probabilities to estimators that accept them. EconML's DR guide covers multi-treatment estimation and its assumptions.
• Causal forest: an optional comparison when you have enough data to study nonlinear effect differences and their uncertainty. Use an estimator and inference method suited to the treatment representation. Wager and Athey's causal-forest paper develops inference under stated assumptions. Those guarantees do not cover any forest trained on a CRM export.

Cross-fitting means predicting helper quantities for a row with models fitted on other rows. Keep related account records together, and leave the final future test period untouched. For observational data, a defensible causal analysis also needs no unmeasured confounding, overlap, a well-defined treatment, and a suitable interference assumption. A rich CRM table does not prove these. An agent's unlogged judgement about which customers need a discount can confound the data even when the response model has excellent AUC. Double robustness cannot repair an unobserved confounder, or an action never assigned in a customer context.

Optional research challengers: learn the effect or the policy?

A treatment-effect model is one part of a decision. You can also learn a restricted policy, such as a shallow decision tree, from cross-fitted estimates of each action's value. This can make a campaign easier to review while enforcing a contact budget. Athey and Wager's policy-learning paper, published in Econometrica in 2021, develops binary decisions under explicit identification and estimation assumptions, including binary treatment allocation and small continuous-treatment nudges. It is not a ready-made multi-arm budget allocator. For several offers, estimate each supported action's net value and cost, then use an allocation method that enforces your real constraints. A simple policy with reliable value can beat a more accurate effect model whose decision threshold is unstable.

This extension is outside the 25-hour starting project. Add it only after the randomised baseline and held-out evaluation work. A recent paper is a reason to test a candidate, not to replace them.

Large catalogue: retrieve, rank, then apply the policy

If scoring every product costs too much, first retrieve a manageable candidate set. A two-tower model embeds the customer/context and the item separately. Item vectors computed ahead of time make nearest-neighbour retrieval fast. A richer ranker with customer–item interactions comes next. The TensorFlow Recommenders retrieval tutorial shows this structure, and Google's YouTube paper describes candidate generation followed by ranking at large scale.

Train retrieval on observed customer–item interactions, with features available for those examples. State how you sample negatives. When item IDs repeat, another row's positive can become an accidental negative. The retrieval task API removes these duplicate-positive hits with remove_accidental_hits=True and candidate_ids, and corrects logits using negative-candidate sampling probabilities. Those probabilities describe the training sampler, not campaign assignment or causal exposure. Report retrieval over the eligible catalogue, or say how candidates were sampled. An in-batch metric is not full-catalogue Recall@K.

LightGBM LambdaRank is a reasonable candidate when you have grouped recommendation opportunities and meaningful relevance labels. Keep each query/session group whole during training and evaluation. Log what appeared, where, and which candidates were available: a click near the top of the screen differs from a product never shown. Correcting position bias needs a suitable exposure model or experiment. Adding position as an ordinary feature does not by itself give a causal relevance estimate.

For a new customer, use stated preferences and context with recent eligible popularity. For a new product or offer, use metadata and carefully budgeted exploration, not an ID-only embedding. Metadata must go into a tower trained to use it. Adding descriptions to an ID-only model at serving does not create content-based retrieval. The side-feature tutorial shows train/serve feature representations. Test on users and items first seen after the training cutoff, using only metadata available when they appeared. Version the model and its matching candidate index together, and measure index refresh lag. Check retrieval recall before ranking: a ranker can only pick products that retrieval returned. Apply eligibility during retrieval where possible and again before the final decision, especially when stock changes.

If incentives are attached to retrieved products, there are two decisions: product relevance, and whether its incentive adds profit. The incentive still needs its own randomised test.