4. Validate the next campaign, with its real training cutoff

Use expanding or rolling chronological backtests. At each simulated retraining time, use only features and labels available by that cutoff. Then score a later cohort with that cohort's historical feature state. A chronological row split alone is not enough if training labels reach into the validation period and were unknown when that model would have been trained.

For a transparent first project, reserve successive time blocks for these purposes:

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Chronological churn validation
    accDescr: Separate successive cohorts for model selection, frozen-model calibration, policy selection and final untouched testing.
    subgraph Selection["Earlier cohorts"]
        direction LR
        T["Fit model"] --> V["Select<br/>+ freeze"]
    end
    subgraph Preparation["Later mature cohorts"]
        direction LR
        C["Calibrate"] --> P["Freeze<br/>policy"]
    end
    subgraph Holdout["Untouched future cohort"]
        E["Final test"]
    end
    Selection --> Preparation
    Preparation --> Holdout

    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class T,V,C model;
    class P decision;
    class E neutral;

Validation sequence. Calibration and capacity or policy selection use separate later cohorts. The classifier, calibrator and operating policy are all fitted without the final future cohort.

Leave enough time for outcomes to mature before each fitting or policy-selection cutoff. Small datasets may instead use carefully designed forward out-of-fold predictions for calibration and policy selection. Document exactly which earlier data each fitted component saw. Keep a final future period untouched. If you have only enough events for two blocks, report the resulting uncertainty and simplify the model instead of quietly tuning on the final test.

Match customer overlap to deployment. Weekly scoring usually includes existing customers seen in earlier training periods. Their earlier history is legitimate if it was available at the simulated training time. Putting every customer into exactly one lifetime fold answers a different question: performance on unseen customers. When onboarding or new-account generalisation matters, also report a customer-disjoint, time-respecting test. A random row split across monthly snapshots is unsuitable for the main deployment backtest. Group-only cross-validation does not guarantee chronological validity either.

Overlapping historical feature windows can stay: the past usage they share can be legitimate. The danger is future labels, future information, near-duplicate decisions and uncertainty estimates that treat repeated customers as independent. Use an embargo or purging (a gap between training and test periods) where label intervals and the retraining protocol require it. Justify its length from horizon, ingestion delay and data dependencies.

Report by month, tenure, plan, acquisition channel and relevant operating segment. Check both the repeat-customer and new-customer cohorts. For repeated snapshots, estimate uncertainty by resampling customers. Where shared calendar shocks matter, use time blocks or another dependence-aware method. State the resampling unit. A narrow interval from bootstrapping single rows means little when one customer appears twelve times.

Say whether each result weights scoring opportunities or unique accounts. One churn event can label several earlier weekly snapshots. That suits a recurring scoring task, but it does not create several independent churn events or saved accounts. Evaluate capacity on the actual decision cadence after cooldowns and exclusions, and count realised contribution once in the economic follow-up.