Accuracy rarely helps when churn is uncommon. Average precision (AP) is a useful ranking summary. If you report "PR-AUC", say whether it means AP or trapezoidal area under the precision-recall curve, because they are computed differently. The scikit-learn AP definition weights precision by changes in recall. Average precision documentation.
ROC-AUC remains a valid discrimination measure, but it does not tell you whether precision is usable at the team's capacity. Precision depends on prevalence, so compare AP and precision-recall curves on comparable populations and show each cohort's churn rate.
| Report | Question it answers |
|---|---|
| Churn prevalence and eligible count | What population and difficulty did this score describe? |
| AP and ROC-AUC | How well does the model order customers across thresholds? |
precision@k | Of the k customers the team can investigate, how many later churn? |
capture@k / recall at capacity | What fraction of all observed churners is in that queue? |
lift@k | How much more concentrated is churn in the queue than in the eligible population? |
| Reliability plot, log loss, Brier score | Are the probabilities useful on future cohorts? |
| Cost or value by capacity, with intervals | Does expanding the queue improve the stated objective? |
An arithmetic example, not a benchmark result. In a matured cohort of 10,000 accounts, 500
churn. A queue of 1,000 contains 200 of them. Precision is 200 / 1,000 = 20%, capture is
200 / 500 = 40% and lift is 20% / 5% = 4. This is good risk concentration. It says nothing about
how many of those customers a retention offer would save.
For a purely diagnostic alert with stated false-positive and false-negative losses, you can derive a
probability threshold from those losses. For a retention campaign, use expected incremental
value, action eligibility and capacity. Choose its operating point on the policy-selection cohort,
freeze it and evaluate it on the final cohort. Choose at most k feasible actions with positive
expected benefit, even if that leaves a quota unfilled. When budgets or action costs differ, use a
constrained allocation instead of a single risk threshold.