3. Use a model ladder that matches the available evidence

Model-selection decision tree

Use the prediction tree to choose a risk formulation, then the action tree if the workflow will recommend a retention intervention. These stages compose: an R1 classifier or R2 survival model can supply risk for triage alongside an A1 action-effect model. Risk alone does not show who an offer will save, and causal modelling does not need a separately trained risk classifier.

1. Choose the prediction formulation

“Usable” means you can reconstruct eligibility, feature availability and outcome observation at a real decision time. Incomplete follow-up is usable for survival only when event or censoring times and the observation limit are recorded. A missing date is not a censored duration.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Choose a churn prediction formulation
    accDescr: Repair missing dates or outcomes; use mature fixed-horizon classification, or survival when recorded censoring or event timing matters.
    D{"Usable dates<br/>and outcomes?"}
    D -->|No| R0["R0<br/>Repair data"]
    D -->|Yes| Q{"Censoring or<br/>event timing?"}
    Q -->|No| R1["R1<br/>Classifier"]
    Q -->|Yes| R2["R2<br/>Survival"]

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    class D,Q decision;
    class R0 data;
    class R1,R2 model;

2. Add the retention-action stage

Use this second tree to decide whether to contact a customer, or which offer to deliver. Action effects need a randomised comparison, or a defensible identified design with its assumptions and overlap checked. A contact-response history chosen by the old policy is not enough on its own.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Add evidence for a retention action
    accDescr: Without identified action effects, use risk for triage and collect a randomised pilot; with valid evidence, evaluate uplift or doubly robust action models.
    E{"Action effects<br/>identified?"}
    E -->|No| A0["A0<br/>Triage + pilot"]
    E -->|Yes| A1["A1<br/>Uplift / DR"]

    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    class E decision;
    class A0 execution;
    class A1 model;

Read each leaf as a starting recommendation with an evidence gate, not an automatic release.

Model ladder

Fit and calibrate the classifier

For the first classifier, start with unweighted log loss and a small tuning budget. Keep class weighting only if it improves the operational tradeoff. Tune tree depth or leaves, minimum leaf size, learning rate, regularisation and the number of trees on earlier validation cohorts. Run early stopping there too, not on the final test. Apply the same feature availability rules to every challenger. Sequence models are an extra experiment, for when event histories hold information your aggregates miss and you have enough data to validate the added complexity.

For smaller tables, you can compare a released tabular foundation model using the Chapter 3 protocol. Report calibration and contact-capacity performance on the same future cohort, including sparse-history customers. Use a bounded tuning and inference budget. A leaderboard average does not decide this queue's value.

Handle imbalance inside the training process. Avoid SMOTE or undersampling before splitting. If you test resampling, apply it only inside each training fold. Keep the natural class balance in validation, calibration and test data. This avoids both leakage and evaluation on an invented prevalence. imbalanced-learn's resampling pitfalls.

Class weighting and sampling can change probability estimates even when ranking improves. LightGBM warns that is_unbalance and scale_pos_weight can give poor individual class-probability estimates. Use one or the other, not both together. LightGBM objective parameters.

After choosing and freezing the classifier, fit a sigmoid calibrator on a later matured cohort that the classifier did not train on. Isotonic calibration is a challenger when you have enough examples for a flexible fit. Assess reliability plots, log loss and Brier score on the untouched final cohort. A Brier score mixes calibration and discrimination, so inspect the reliability plot separately. A decimal from the API is not a probability until you check its calibration. Calibrators need out-of-sample predictions, and refitting the classifier afterwards can break the fitted mapping. scikit-learn calibration guide.