You already write software. This book helps you use that experience to build, evaluate and run AI systems. Start with a working baseline and keep asking what your results really support.

Check your starting point

Try these without an assistant writing the first attempt. Documentation is fine. This is a self-check, not a timed exam.

1. Read a CSV with Python, count missing values, and write a function with one meaningful test.
2. Make a Git commit and explain what it includes and excludes.
3. Call an HTTP endpoint, inspect a failed response, and keep credentials out of commits.
4. Explain the difference between a percentage and a percentage-point change: from 20% to 25% is 5 percentage points and a 25% relative increase.
5. Explain why (100, 1) + (100,) can create an unexpected array. If this is new, Chapter 2 teaches it.

If the first three are new, start with the programming resources in Chapter 1. If only the number work is new, begin the normal path. You can write your first baseline before finishing any mathematics textbook.

Your next useful hour

Create a repository and a learning log. Write one sentence naming a person, a decision and the data that might help: “A support lead wants to spot tickets that need escalating, using information available when the ticket arrives.” Then follow orientation week.

Choose one role to explore while keeping up the shared foundations. The role branches in Chapter 7 explain what to build, what to explain and what to recognise. You do not have to finish every branch.

The book has three reading layers

Core. Follow the chapter plan, work through its example, and try the exercise before opening the answer. Each chapter ends with something you can show.

Practice. Diagnose a failure, compare alternatives, and note the limits of a result. Build on the same project where the domain and data make sense.

Research. Optional notes and annotated resources go deeper. Treat a recent paper as something to understand and test, not as a default production choice.

Website reading times cover reading only. Practice-hour figures are planning estimates for someone who already develops software, not measured completion times.

Two ways to work on the same path

The reference lab gives you synthetic data, reproducible commands, filled-in reports and deliberate failures. Run it locally with Python. Its results illustrate its stated assumptions, not real commercial impact.

Your own project uses data you found and are allowed to use. Start with the dataset checklist in Chapter 2. As soon as you know what you want to predict, set aside a test set, before you look at the outcomes or fit any preprocessing, and leave it alone until the end. If your data cannot support the next claim, make the claim smaller or find better data. The reference lab is always there for practice.

A useful completion standard

Work is finished when you can run it from a clean checkout, explain one decision that mattered, diagnose a plausible failure, and show a result with its limits. Documentation can stay open. An honest, reproducible negative or inconclusive comparison is a finished analysis too.

Read offline

The website includes a complete print edition and an EPUB download. Interactive figures also have static explanations, and the Python lab runs without a model API. Reading progress is saved only in this browser on this device, and you can clear it from the reader controls.

A working developer's path into AI and machine learning.

The short version. Eight core chapters take you from shipping software to shipping AI systems in about 450 hours. A ninth applied chapter covers offer recommendation, price recommendation and churn, with model choices and a full business decision workflow. You build on one thread — one dataset becomes one model becomes one AI product. Each chapter has a practical plan and a resources folder for deeper reading.

Contents

The chapters

1. Getting oriented  ·  resources
2. Data foundations  ·  resources
3. Core machine learning  ·  resources
4. Deep learning  ·  resources
5. Language models and AI engineering  ·  resources
6. Running it in production  ·  resources
7. Choosing a specialisation  ·  resources
8. Finding the work  ·  resources
9. Business machine learning: offers, prices and churn  ·  resources

Applied guides

• Model-selection decision trees
• Offer recommendation
• Price recommendation
• Churn prediction and retention
• Shared delivery plan and completion gates

Before you start

• Who this book is for
• How to read this book
• Start here: your first week
• The path at a glance
• One thread through the whole book
• How much time it takes
• How each chapter works
• Using AI assistants while you learn

Reference

• Terms in plain English
• Projects, in order of difficulty
• Common pitfalls
• Where thoughtful people disagree
• Keeping this book current
• A note from the author

Who this book is for

This is written for someone who already writes software. You use Git, you call APIs, you have deployed something, and you can read a stack trace without panic. What you want is a clear route into machine learning and AI engineering, without spending six months on material that turns out to be out of date.

If you are not yet programming, this book will be hard going. From a standing start, add a substantial programming stage before these chapter budgets. A planning allowance of 1,000 to 1,500 total hours is an estimate, not a measured completion time or a job guarantee. CS50P is an excellent place to begin, and this book will still be here afterwards. You will move through it much faster.

A word on what this can and cannot promise. It is a well-researched, carefully checked path, and I think it is a good one. It is not the only good one, and it will not fit everyone. Where the evidence is genuinely mixed, I have tried to say so rather than sound more certain than I am.

How to read this book

Follow the core chapters in order. Chapter 9 is an applied track after Chapters 3 and 6. Use one guide as your Chapter 7 capstone, or work through all three afterwards. Everything optional is marked as optional. When you are unsure what to do next, do the next thing on the list. Most roadmaps offer a branching menu, and that turns out to be part of the problem: a long list of choices is easy to browse and hard to finish.

Build something in every stage. Dunlosky and colleagues' 2013 review rated practice testing and distributed practice highly across the learning conditions it examined. Re-reading and highlighting were less useful overall. Roediger and Karpicke's 2006 experiments found that recalling studied text, rather than studying it again, helped people remember it later. Neither study tests a software curriculum directly. This book applies their ideas: explain from memory, space out sessions, and build the key exercise with the tutorial closed. Then use your references to check and correct the result.

Keep a short log. One file, one line per session: the date, the hours, what you built, and what broke. It gives you a way to revise, and it is a genuinely good answer when someone asks how you learned all this.

Give yourself room. The plan below assumes ten to twelve hours a week, not twenty. A slower plan you finish is worth more than a faster one you abandon, and there is no prize for rushing.

Find one place to ask questions. You will get stuck, and a stuck hour with no one to ask is how most self-study ends. Pick one community early and use it: the fast.ai forums, the DataTalks.Club Slack, or the Hugging Face forums are all free, active and kind to beginners. Ask with what you tried and what you saw. That habit is itself a skill people hire for.

When you feel lost, do this: open the chapter you are on, find the week plan near the top, and do only that week's row. Use the resource list to look things up, not to hunt for a better next hour.

Start here: your first week

If eight chapters and forty-four weeks feel like a lot, start smaller. Week 0 is eight hours. You do not need to understand machine learning yet. You need a target role, a public repo, a log file, and a notebook that runs. The day-by-day plan for that week is at the top of Chapter 1.

End of week 0: you have a public repo, a log, a role, and Colab working. Open Chapter 2 and follow Week 1 of its plan.

The path at a glance

The diagram shows the order. The table gives each chapter’s time budget and outcome. Start Chapter 8 alongside Chapter 6 in week 33. Chapter 9 is an applied extension after Chapters 3 and 6, and can be your Chapter 7 capstone.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: The core learning path
    accDescr: Complete foundations, machine learning and AI engineering in order, then choose one specialisation. Job search begins alongside production in week 33.
    F["Foundations<br/>Ch 1–2 · weeks 0–5"]
    M["Machine learning<br/>Ch 3–4 · weeks 6–22"]
    E["AI engineering<br/>Ch 5–6 · weeks 23–37"]
    S{"Choose one role<br/>Chapter 7"}
    A["AI<br/>Engineer"]
    B["ML<br/>Engineer"]
    C["Data<br/>Scientist"]
    F --> M --> E --> S
    S --> A
    S --> B
    S --> C
    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef action fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class F data
    class M model
    class E action
    class S decision
    class A,B,C neutral

One thread through the whole book

You are building one connected project. One dataset becomes one model becomes one AI product. That is an end-to-end story an interviewer can follow in five minutes.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart LR
    accTitle: One project, developed across the book
    accDescr: Your Chapter 2 data becomes a model in Chapters 3 and 4, then an AI product in Chapters 5 and 6.
    D["Data<br/>Ch 2"] --> M["Model<br/>Ch 3–4"] --> P["AI product<br/>Ch 5–6"]
    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef action fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class D data
    class M model
    class P action

The same project grows. The table shows what each chapter adds.

For the business cases, a connected retail or subscription thread can start with transaction and customer snapshots, move to demand or churn prediction, add a document feature over product or support policies, and end with an offer, price or retention policy and measured feedback. Choose that domain early if Chapter 9 is your goal. For intervention exercises, public transaction data usually needs clearly labelled synthetic exposure and assignment data. Do not invent evidence that real discounts or prices caused the observed purchases.

Pick a domain you care about in Chapter 2 and stay with it until Chapter 6 at least. If you change datasets every chapter, you will finish tired and with nothing coherent to show.

How much time it takes

The week ranges in the roadmap are planning windows. The job search overlaps Chapters 6 and 7, because it takes several months and starting early tells you a lot. It adds three to four hours a week on top of the chapter hours, so weeks 33 to 43 are the fullest in the book. If that is too much, slow the chapters down rather than skipping the search.

About the hour figures. These are your hours on each chapter, assuming you use each resource as described rather than finishing it end to end. Several listed courses are much longer than the chapter that mentions them. Where that matters, the chapter says which parts to do. The resource tables give each item's full length, so those numbers are often larger. That is expected, not a contradiction.

The applied extension has its own budget. Chapter 9 adds about 40 hours for one case or 90 for all three, including shared data and delivery work. The original core estimates total 453 hours, not counting the parallel job search. The core schedule covers 44 calendar weeks: orientation week 0 plus weeks 1–43. Doing all of Chapter 9 afterwards makes it 543 hours, or roughly eight to nine extra weeks at 10–12 hours per week. If one case doubles as Chapter 7's 30-hour capstone, count those hours once. The practice budgets leave out waiting for real churn labels or for a properly powered live experiment to finish.

Allow extra calendar time for the job search. Four to eight months is a planning buffer, not a median estimate or a forecast for you. Location, experience, work authorisation, target level and hiring conditions can change it greatly. Chapter 8 shows how to adjust the plan using current postings and your own application funnel. Bootcamp cohort reports do not show how long AI/ML searches take in general.

How each chapter works

Every chapter runs on the same loop.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: The learning loop used in each chapter
    accDescr: Learn and practise with a reference, wait a day, build with the tutorial closed, compare and correct, then explain from memory. Revisit the exercise if you cannot explain it.
    L["Learn + practise<br/>One guided exercise"]
    W["Wait a day"]
    B["Build from memory<br/>Tutorial closed"]
    R["Compare + correct<br/>Reference open"]
    C{"Explain it<br/>without notes?"}
    N(["Next chapter"])
    L --> W --> B --> R --> C
    C -->|Yes| N
    C -.->|Revisit| L
    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef action fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class L,W,R neutral
    class B model
    class C decision
    class N action

Solid arrows show the normal order. The dashed arrow is the revision loop.

The step people skip is build it yourself, and it is the one that does most of the work. If you can finish a chapter by watching, something has gone wrong with the chapter.

Chapters 1 to 6 share one layout: week plan (what to do each week), what you will be able to do, what to learn, resources, what to build and before you move on, usually closing with a few things worth knowing. Chapters 1 and 5 add day-by-day plans where the work needs finer scheduling, and Chapters 7 and 8 are organised around the three branches and the job search instead.

Chapter 9 has a shared decision and delivery plan and three case guides. Each guide covers the target, data, baseline, candidate models, evaluation, constraints, experiments, operations and completion evidence. It is an applied extension, not a prerequisite for the job search.

Using AI assistants while you learn

How assistant use affects learning depends on the task and how you use it.

The 2026 Stack Overflow survey, released on 6 October, shows wide use: 73% of respondents who use AI coding assistants or agents say they use them daily. That share is of assistant users, not all developers. This self-selected survey describes reported practice and feelings. It does not show that assistants improve learning or independently measured productivity.

There is now a small controlled experiment too. In a randomised trial published by Anthropic in January 2026, 52 developers, mostly junior, learned an unfamiliar Python library with or without an AI assistant. The assistant group was not reliably faster, and scored 50% versus 67% on an immediate comprehension quiz. That is a reason to protect first attempts, not proof that every kind of AI help harms every learner.

Other designs give different results. A preprint revised on 4 October 2026 randomised AI access for 211 undergraduates studying unfamiliar topics and writing essays. It found higher scores on an unaided knowledge test, and a smaller gain a week later. This was not a coding task, and its tutoring-versus-delegation comparisons were descriptive, not separately randomised. A 2026 programming-classroom workshop paper links feedback with short-term completion, but does not show lasting mastery without help. Before applying a study's result to your learning, compare its intervention, population and unaided outcome.

An assistant can take away the effort of recall, so keep some practice unaided. The book's practical rule has two modes: do first attempts yourself, and when you ask for help, ask for an explanation rather than the fix.

This is a reasonable compromise, not a proven one. The coding trial measured immediate understanding. Other studies use different tasks and follow-up periods. Rebuild a small exercise without help at the next session and again a week later, then compare mistakes. Finishing with help and feeling confident do not show what you retained. Adjust how you ask for help based on what you find.

One thing worth taking seriously either way: only put code in your portfolio that you could talk through line by line, and be ready to explain its data and evaluation too.

Terms in plain English

Every term below is also explained in plain words where it first matters in a chapter. Come back here whenever a word feels fuzzy.

Projects, in order of difficulty

What each one proves, and separately what makes it stand out. Those are different questions. The Where you build it column names the chapter whose build exercise covers each project, or notes when it is optional or spans several chapters.

Three punch above their weight for the effort involved: the deliberate leakage write-up (project 3), the published failure taxonomy (project 7), and the honest fine-tuning comparison whose answer may be "not worth it" (project 8). That last kind of negative result shows a tested hypothesis and a fair decision. Explain the comparison, uncertainty and operating costs so a reviewer can judge it.

Common pitfalls

Collected from practitioner writing and from resources that turned out to be dated when checked. None of these are character flaws. Most are design problems with the material available.

About sequencing

Resources that are commonly recommended and now dated

Technical traps

About money and portfolios

• Certificates mostly signal enrolment. Audit courses free and spend the money on compute.
• Expensive cohort courses are often excellent, and their authors frequently publish most of the same material free. Read the free version first.
• Buying a GPU before exhausting free tiers is usually procrastination in physical form.
• Ten shallow projects read worse than three deep ones.
• Code you cannot explain is a real risk now that reviewers ask about repositories in detail.
• Building in public works better in the quiet version: ship small things and write up what broke. The high-profile examples usually had a large audience already.

Where thoughtful people disagree

This book takes positions throughout. These are the places where competent, experienced people genuinely disagree, and where stating both sides is more impressive than picking one.

Keeping this book current

Three kinds of information in this book age fastest: prices and free tiers, model names and capabilities, and salary figures. Treat any specific number as a snapshot, and check the source before you rely on it.

The illustrated edition — 30 October 2026

The web edition adds shorter lessons, eight original interactive field notes, local search, optional on-device reading progress, focus reading, a complete print edition and an EPUB. Chapter Markdown is still the source for all of them.

Chapters 2–6 now include original worked examples, independent exercises, hints and explained answers. Chapter 2 sets aside a holdout before you look at outcomes, and a shared causal primer prepares the business cases. Completion gates ask to see your reasoning and debugging.

The reference lab supplies deterministic synthetic data, training-only model fitting, separate calibration and test cohorts, a paired comparison, business decision examples and filled reports. Its results illustrate a stated simulator and implementation, not real commercial performance. The source inventory records cited URLs, without claiming that every source was independently rechecked on the edition date.

Two studies shaped how the book asks you to work: Dunlosky and colleagues (2013), on which study techniques actually hold up, and Roediger and Karpicke (2006), on why self-testing feels worse in the moment and works better in the end.

A few places deserve extra scepticism, and I would rather point them out myself. Causal and experimental methods rest on assumptions about assignment, confounding and interference. Chapter 9 now states those assumptions for the business cases and adds primary sources. The two-mode rule for AI assistants is a reasonable compromise, not a proven method. And the four-to-eight-month job-search allowance is a planning judgement, not an outcome estimate.

If something here has drifted out of date, that is the field doing what it does. Check it against the source, adjust, and keep going. If you find a broken link or a claim that no longer holds, an issue or a pull request on this repository is very welcome. Corrections from readers are how a book like this stays useful.

What changed in the illustrated edition review

• Edition 2026.10.1. Every diagram in the PDF now prints in its own place, and the EPUB passes the EpubCheck validator that ebook stores use.
• One thread, end to end. Chapters 4 and 5 now start with a same-thread build, and Chapter 6 runs the Chapter 3 model alongside the Chapter 5 application. Chapter 7 shows how each capstone stays connected, and Chapter 8 adds operations evidence to the portfolio template.
• New practice. A broadcasting exercise in Chapter 2, a training-debugging checklist in Chapter 4, document field extraction and output guardrails in Chapter 5, and explained exercises in Chapters 6, 8 and the offer, price and churn guides.
• Production. Safe artifact loading, provider outages, retention and erasure, and incident severity with blameless write-ups.
• Laboratory. Stricter manifest and sample-ratio checks, an experiment result reported with its uncertainty, and a test that the checked-in reference matches a fresh run.
• Corrections. Consistent fine-tuning placement, hour figures and reading selections. Missing links and resource notes added. A citable edition with EPUB and PDF downloads.
• Web edition. Illustrated plates for the start page, reader's guide, applied guides and lab reports. Full-width field notes in the lab. Dollar amounts no longer render as mathematics.
• Licence. The text, illustrations and lab reports are now CC BY 4.0, so you may share, translate and adapt them with credit. The website and code remain under the MIT licence.

What changed in the October 2026 research revision

• Business cases. Added Chapter 9 with detailed offer, pricing and churn guides, conditional model recommendations, worked economics, practice data and production delivery gates.
• End-to-end coverage. Added rules for feature availability, label maturity and outcome observation, plus calibration and test discipline, regression and forecasting, optional segmentation and anomaly methods, experiment protocols, model lifecycle and delayed-outcome feedback. The business guides separate competing events, the uncertainty of picking the best-looking policy, and joint or adaptive assignment.
• Current evidence. Reviewed 2026 tabular, causal, survival, recommendation, pricing and agent research. Recent preprints appear as conditional challengers, with criteria for reproduction and adoption, not as universal winners.
• Corrections. Updated library compatibility, course editions, contribution policies and hiring data. Qualified benchmark claims by model variant and sampling protocol. Strengthened guidance on leakage, uncertainty, permissions, release evaluation and rollback.
• Sources. Primary papers and official documentation are linked where the chapters use them, with assumptions and practical limits alongside model recommendations.

What changed in the September 2026 revision

For readers already partway through, these are the changes worth knowing:

• Setup. Python 3.14 in the setup commands, and a note on uv's new default project layout (Chapter 1).
• AI assistants. A 2026 randomised trial on learning with an assistant, and a two-mode rule adjusted to fit it (above).
• Short new sections. Reasoning-effort costs and agent skills (Chapter 5), regulation as part of production (Chapter 6), and a security reference for the AI Engineer branch (Chapter 7).
• Tabular models. A more careful picture of tabular foundation models against boosted trees (Chapter 3).
• The job search. Newer hiring and pay data, and interviews that now include an AI assistant (Chapter 8).
• Resources. Chapter 5 now points to Anthropic's free Claude Academy course, one course website no longer loads, and many notes were refreshed for new versions, prices and owners.

A note from the author

I have spent my career as a developer, moving between domains, and every move brought a new set of challenges. Through all of them, one habit has helped me more than any tool: starting from a roadmap. A good roadmap does two quiet things. It builds the foundation in the right order, and it teaches you to think in systems — to see how the pieces connect before you start collecting them.

This book is that habit written down, for the move I think matters most right now: from software engineering into AI and machine learning.

I want to be open about how it was made. I used AI models, including Fable 5, to help me research and draft it. The plan, the judgements, and the final words are mine, but it felt right that a book about working with these systems should be honest about being written with their help.

I am not only the author here. I follow these chapters myself, to refresh my own skills, because in this industry the learning never stays done. If you find something that has aged, treat it the way this book will teach you to treat every claim: check it, correct it, and keep moving.

— Santosh Shinde

Week 0  ·  about 8 hours.

Book index  ·  Chapter resources

The short version. Choose a target role, a weekly time budget and a working Python environment. Start on your own CPU or a free notebook service. You finish with a repository another person can run, and no hardware to buy.

In this chapter: Plan  ·  Learn  ·  Build  ·  Completion  ·  Resources

Your week 0 plan

About eight hours. Do the days in order, and finish every step that ends in a commit or log line.

What you will be able to do

Name the specific role you are working towards, and open a project without fighting your tools.

A quick entry check

Try these with documentation but without a tutorial. They show what to prepare, and are not a timed admission test.

1. Read a small CSV, check that an amount is numeric, and return a useful error for a bad row.
2. Write a function and a test, run them from a terminal, and explain the stack trace of a deliberate failure.
3. In a throwaway repo, make a Git commit, look at its diff, and undo one accidental change.
4. Explain why 12 events among 80 eligible cases is 15%, then say what would make that denominator wrong.

If the first three feel unfamiliar, use the programming resources in Chapter 2's resource list before counting the chapter budgets. Take one Python course, such as CS50P or Python for Everybody, then Learn Git Branching and The Missing Semester for Git and the shell. If fractions or function notation feel rusty, list the maths topics to revisit and work through the examples alongside Chapters 2 and 3. You can start without a full mathematics course. The reference lab is a small project with no API calls for checking your local workflow.

What to learn

Pick one target role. The five common titles look similar from outside and lead to quite different curricula. Trying to prepare for all of them at once is one of the more common reasons people stall. Treat the table as a guide to the work. Employers use these titles inconsistently.

Data Engineer is worth considering as well: ingestion, data quality, modelling, orchestration and reliable analytical systems. Your SQL, backend and infrastructure experience can transfer.

If you cannot decide, AI Engineer is a reasonable first choice for someone who already ships applications. Revisit it after checking local vacancies and after your first project. A title alone does not show hiring demand or an employer's degree requirements.

Know what interviews can assess. Eugene Yan has published his own hiring rubric. He assesses software engineering fundamentals, data literacy, comfort with uncertainty, evaluation frameworks, and breadth of knowledge. It is one practitioner's rubric, so expect other interviews to differ. The projects in this book are built to show the two skills in bold.

Set up your tools. Use one documented workflow, and check version-sensitive examples against current documentation.

• Environments: use uv. It manages dependencies, virtual environments and Python installations. First follow its installation instructions for your operating system, reopen your terminal if needed, and check uv --version. Then create the project:

  uv python install 3.14
  uv init --python 3.14 my-learning-repo
  cd my-learning-repo
  uv add pandas numpy jupyterlab

  Check that the installed packages import:

  uv run --locked python -c "import numpy as np; import pandas as pd; print(np.__version__, pd.__version__)"

  Both version numbers should print without an error. Open notebooks with uv run jupyter lab. Commit pyproject.toml, uv.lock, .python-version and the generated src/ package. Note the Python patch version, uv version and operating system. Use uv sync --locked and uv run --locked ... for a reproduction check. The flag fails if the lock disagrees with project metadata, rather than quietly changing it. See locking and syncing. A lock does not pin your data, GPU driver or every hardware-dependent result.

  Two details trip people up. Since uv 0.12, uv init creates a small package under src/, which suits the tested cleaning functions you write in Chapter 2. Add --no-package if you want the older single-file layout. For the Python version, pick the newest one that PyTorch, the deep-learning library you meet in Chapter 4, has ready-built packages for. In September 2026 that is 3.14, as the official PyTorch installation guide shows. If a needed library lacks a compatible wheel, choose a supported Python version and regenerate the lock on purpose.

• pandas 3.0 changed some defaults. Copy-on-Write is now the only mode, so chained assignment cannot update the original dataframe. Use a single .loc[...] = ... assignment. Text columns also default to a string dtype. Some operations still support inplace=True, so seeing it does not date a tutorial. Check the pandas migration guide. NumPy's old aliases such as np.int, np.float and np.object expired in 1.24. NumPy 2.0 reintroduced np.bool with boolean dtype semantics. Use explicit dtypes when precision matters.

• Editor. VS Code with the Python and Jupyter extensions is a practical default. Select the project's .venv interpreter and notebook kernel. Its data science tutorial also covers conda and TensorFlow. Those are valid, but use one documented environment here.

A note on package sources. At work, follow your organisation's package and licensing policy. Anaconda's commercial terms depend on the service and organisation, so read its current terms if you use it. Miniforge with conda-forge is another option if you need conda.

Where your compute comes from.

Nothing on the main path of this book needs a GPU you have to buy. "I cannot learn deep learning without a GPU" is a common worry and, happily, not true.

Start locally or in a free notebook. Choose a GPU service later, to fit the exercise's memory, runtime and storage needs. A provider's allowance can change, so keep a CPU fallback and checkpoint training so it can resume. Official compute links are in this chapter's resources.

Resources

All 13 resources for this chapter, with notes on each, are in resources/.

The ones to begin with:

• How to Interview and Hire ML/AI Engineers — Eugene Yan. A free article, 1 hour. The hiring rubric this book is built around.
• The Rise of the AI Engineer — Shawn "swyx" Wang. A free article, 1 hour. The essay that first framed the role. Compare it with current vacancies.
• AI and Job Postings: From Destruction to Creation? — Guillermo Gallacher. A free article, under an hour. Realistic expectations about the junior market.
• Teach Yourself Programming in Ten Years — Peter Norvig. A free article, under an hour. The antidote to "AI engineer in 30 days".

What to build

Create your learning repository, add a LOG.md, make your first commit, and push it. Public is better, even this early. One line per session is enough:

2026-10-09 · 2h · Created repo with uv; CPU notebook runs · Broke: selected wrong kernel

From a fresh clone, run uv sync --locked, then repeat the import check above. Write those exact commands in the project README, with your role and weekly study sessions. Keep credentials and private datasets out of the repository. A public project helps when its data and code can be shared.

Before you move on

• You can state the role you are training for, and one reason for it that is not salary.
• Your weekly hours and study sessions are written down. If ten hours a week is unrealistic, stretch the schedule.
• A fresh clone installs from the committed lock and passes the documented import check.
• You can correct a pandas chained-assignment example using the migration guide.
• You have a working CPU environment and know how to check an optional GPU service's current limits.

A few things worth knowing

• Course collecting. CS50P, Python for Everybody and Automate the Boring Stuff overlap heavily. Doing all three is a lot of hours to learn one thing once. Pick whichever suits you and finish it.
• Certificates. A certificate shows you finished a course, not that you can build or evaluate things on your own. Historical MOOC completion rates count everyone who registered, including people who never meant to finish, so they are a poor guide to your own success. Alongside any certificate, build something a reviewer can inspect.
• Perfecting the plan. It is possible to spend a happy weekend on a study schedule. The schedule is not the work.

Back to the index  ·  Next: Data foundations

Weeks 1–5  ·  about 50 hours.

Book index  ·  Chapter resources

The short version. You already program. This chapter teaches the data dialect — arrays and shapes, dataframes and cleaning judgement, and enough SQL to be trusted with real data. You finish by publishing a small, reproducible analysis of a dataset you found yourself, which becomes the seed of everything you build later.

This dataset carries forward. Chapter 3 models it. Chapter 5 can use documents in the same domain. Keep the domain where you can. Document a needed change, rather than forcing an unsuitable dataset to support a claim it cannot answer.

Choose a dataset you can finish

Before investing in cleaning, check five things:

• Permission: you may download, analyse and publish the intended examples.
• Grain: you can say what one row represents and spot duplicates or repeated entities.
• Question: one useful descriptive question and a feasible binary or numeric prediction target.
• Clocks: you know when inputs would be available and when outcomes can be seen. A current export with no history supports a limited exercise, not a verified historical backtest.
• Size and continuity: a small sample runs on your machine, and related public documents could support the later application.

If choosing stalls, use the reference lab while looking for your own portfolio dataset. It uses synthetic data, so its results show behaviour under stated assumptions. A city open-data export, a permitted export of your own data, or a small transaction dataset can work if it passes these checks. An ordinary, modest dataset is enough.

Set aside the test set before you explore outcomes. As soon as you know what you want to predict, write down the prediction time and time or group split, and set aside the final holdout. All-row checks of source, schema and integrity are useful. Learned transformations, charts that use the target, and feature choices belong on training and development data only. If you have already explored every outcome, label that work exploratory and find fresh data to confirm it. Splitting afterwards does not undo what you have seen. Chapter 3 reviews this design.

In this chapter: Plan  ·  Learn  ·  Build  ·  Completion  ·  Resources

Your five-week plan

About ten hours a week. Each week ends with something on GitHub.

New words in this chapter, such as array, broadcasting, dataframe and point-in-time, are explained where they first appear below and collected in Terms in plain English.

Work in the Chapter 1 project. Add the SQL, plotting and test dependencies, then commit the changed project files:

uv add duckdb matplotlib
uv add --dev pytest

After writing the Week 2 tests in tests/test_*.py, run uv run --locked pytest from the project root. Use pytest's getting-started guide for discovery and assertions. Start the linear-algebra videos with chapters 1–4, ending with "Matrix multiplication as composition". Later lessons are optional, inside the same 50-hour budget.

What you will be able to do

Take a messy dataset you found yourself and turn it into a published, reproducible analysis that a stranger can run with one command.

What to learn

Worked lesson: the event happened, but you did not know it

Suppose you predict at the start of 3 January, using the previous seven days. These invented transactions separate when an event happened from when the service received it:

The historical total is 10. Filtering on event time alone gives 30 and leaks a late arrival. Using today's complete customer total gives 60 and also includes a future event. Values correct today can be wrong inputs for a past decision.

from datetime import datetime, timedelta, timezone

def day(number):
    return datetime(2026, 1, number, tzinfo=timezone.utc)

events = [(10, day(1), day(1)), (20, day(2), day(4)), (30, day(4), day(4))]
as_of = day(3)
total = sum(amount for amount, event_at, available_at in events
            if as_of - timedelta(days=7) <= event_at < as_of
            and available_at <= as_of)
assert total == 10

Try it yourself. Move the decision to 5 January. Which records now qualify, and what is the total? Then add an event exactly at the prediction time. Decide whether it belongs before running.

Use this tiny fixture to test a larger SQL or pandas implementation. The reference lab builds the same habit on a reproducible synthetic project.

Core concepts

Thinking in arrays. You are fluent in loops. Machine learning is written in shapes and batches, and the second dialect takes a little while. What you need is small but you need it cold: shape and dtype, indexing and slicing, views against copies, reshape and transpose, axis semantics, vectorising a loop into an array expression, and above all broadcasting.

Broadcasting deserves the extra time. It is the set of rules by which NumPy stretches a smaller array to match a larger one's shape. A large share of the debugging you will do over the next six months is shape errors, silent broadcasts and transposes in the wrong place.

import numpy as np

predictions = np.arange(100.0).reshape(100, 1)  # one column, shape (100, 1)
targets = np.arange(100.0)                      # shape (100,)
errors = predictions - targets

Try it yourself. Predict errors.shape before running it. Every prediction equals its target, so the mean squared error should be zero. What does (errors ** 2).mean() return, and why?

Alongside this, watch 3Blue1Brown's Essence of Linear Algebra. Three hours of video, and it gives you one idea that pays off immediately: a matrix is a transformation of space, not a grid of numbers to row-reduce. After that, "a layer is a linear map followed by a nonlinearity" stops being jargon. It helps to pair each chapter with code, though. It is quite possible to finish the series feeling enlightened and still be unable to compute a matrix product.

Thinking in tables. pandas, and the judgement that goes with it. The library is the easy half.

Alongside your notebook, record every cleaning decision you made, what you rejected, and how the choice would change a model downstream. Here is an example decision log, with invented percentages:

## delivery_time — 8.2% missing
Pattern: missingness correlates with carrier == "regional" (34% missing) against 0.4%
for national. This is not missing completely at random: it depends on observed carrier.
This pattern alone cannot establish MAR versus MNAR; missing delivery times are unobserved.
Rejected: mean imputation. It would smear the national distribution across regional rows
          and hide the fact that we simply do not observe this segment.
Rejected: dropping the rows. It removes 8% of the data, and non-randomly.
Chose:    keep the nulls and a delivery_time_observed flag for descriptive coverage checks.
          Report observed timing by carrier, alongside missingness and sample counts.
Impact:   observed rows alone cannot establish timing for all regional-carrier deliveries.

That paragraph is the thing worth showing people. Anyone can call dropna(). The judgement is the scarce part, and it maps directly onto "data literacy" in a hiring rubric.

The author-hosted missing-data chapter explains why dependence on an observed field can fit MAR (missing at random), and why MNAR (missing not at random) needs sensitivity analysis.

Separate missing features from missing targets. If delivery time is the Chapter 3 target, its eventual value and observed flag cannot be predictors at dispatch. Avoid filling an unfinished target with zero or a mean. A package still being tracked has a right-censored duration: you know how long it has gone undelivered, not when it will arrive. Evaluating only completed deliveries can leave out the slowest ones. The scikit-survival introduction explains event indicators and censoring. A survival model is optional depth, not an extra prerequisite. For a first project, a fixed-horizon target such as “delivered within seven days” is often easier to define. After seven days, a tracked non-delivery is negative. A lost tracking record is still unknown. State which population and follow-up your evaluation covers.

For a feature available before prediction, compare an imputer and missing indicator fitted inside the training pipeline with a model that handles that feature's missing values. Either way, keep the raw missingness information in your analysis.

SQL. Grain, joins and aggregation carry across tools, though syntax and NULL behaviour vary by engine. Get comfortable with CTEs, window functions (ROW_NUMBER, RANK, LAG, running totals), join cardinality, and point-in-time correctness.

That last one matters more than it sounds. Most serious data-leakage bugs in machine learning are, underneath, a query that accidentally used the future.

If an assistant writes SQL, check it against a schema you understand and a tiny table with duplicates, NULLs and boundary dates. Plausible SQL can give the wrong denominator with no error.

A schema is more than column names. Define dtypes, units, allowed categories and what a missing value means. Keep leading zeros in identifiers. Choose a timezone policy. Store money with a documented currency and rounding rule. Tell apart a zero, an unknown value and a value that does not apply. Recheck pandas string and nullable dtype behaviour after upgrades. In pandas 3.0, default string storage uses PyArrow when installed and falls back otherwise, as the string migration guide explains.

Write the expected join cardinality before joining. One customer with three orders and four support events produces twelve rows if both event tables are joined directly. Summing the joined orders inflates revenue fourfold. Aggregate each event table to the intended grain first. In pandas, use validate="many_to_one" for a unique customer lookup and indicator=True to check unmatched rows. Its merge documentation also warns that NULL keys on both sides match each other, unlike usual SQL joins. Test this explicitly.

Write a data contract before training. Name the unit of a row, its unique key, the prediction time, the feature lookback, the outcome window, and when that outcome becomes complete. Keep both event_time (when something happened) and available_at (when your service could first use it). A transaction from Monday that arrived on Friday was unavailable to Tuesday's model. Historical features must pass both clocks, and later corrections need version history. Feast's point-in-time joins show historical retrieval. Check how your own query and tool handle availability time.

Some public exports hold only a current snapshot, with no availability history. Record that limit, and avoid inventing timestamps or claiming a verified backtest. Test your join logic with small synthetic late-arrival cases, and choose a Chapter 3 split your data can support.

For a customer snapshot at time t, build features from [t - lookback, t), using records available by t. Build labels from the outcome window that follows. An unresolved outcome is unknown, not a negative. Record label_available_at and outcome_observed separately from the planned maturity date, because a window can pass while its source ledger is still incomplete. At a simulated training cutoff, use only labels that the observation window, reporting delay and actual availability allow. Report missing outcomes by cohort and exclusion reason. Elapsed time alone does not prove follow-up is complete.

Test unique keys, join row counts, timezone conversion, missingness, late arrivals and duplicate events. Store the extraction cutoff and source version with the data. Record permission to use the data, keep customer identifiers out of public examples, and publish only data whose licence and permissions allow it. When the original data is private, a reproducible synthetic example is enough.

Reproduction needs a data manifest. Record the source URL or export/query version, licence, retrieval date, extraction cutoff, raw-file checksum, schema version, code commit and environment. Where permitted, keep raw inputs unchanged. Document any sampling, seeds and exclusions, and use the same population denominator in your text and charts. Run the cleaning and analysis in a fresh process with the locked environment, not from the notebook's execution history. The questions in Datasheets for Datasets help explain who is represented and which future uses the dataset cannot support.

If raw data is not in the repository, include its exact download or export steps and expected checksum. If the documented input is missing or differs, the analysis should fail clearly, not quietly analyse a newer export.

Resources

All 31 resources for this chapter, with notes on each, are in resources/.

The ones to begin with:

• Python for Data Analysis, 3rd Edition — Wes McKinney. A free book, 40–60 hours. Chapters 4 to 8 and 10 are the ones that matter.
• NumPy: the absolute basics for beginners — NumPy documentation contributors. Free documentation, 3–6 hours.
• Mode / ThoughtSpot SQL Tutorial (Basic, Intermediate, Advanced) — ThoughtSpot. Free lessons. Run selected exercises in DuckDB. Budget 10–15 hours for the full tutorial, fewer for this chapter's selections.
• uv — official documentation — Astral maintainers. Free documentation, 2–4 hours. Getting Started and Projects are enough.

These are whole-resource estimates, and some are alternatives, so do not add them up. Fit selected chapters and lessons inside the 50-hour project budget, and finish the analysis before optional depth.

If Python is not yet one of your languages, do CS50P first. Its 90 to 120 hours sit on top of this chapter's 50, not inside them, and the chapter will make much more sense afterwards.

What to build

A published data product. A public repository containing:

• A README stating the question in one sentence, the answer in one sentence, and the limitation in one sentence
• pyproject.toml, uv.lock, Python/platform details and a command tested after uv sync --locked
• A src/ package with your cleaning functions, not just a notebook
• At least eight meaningful checks spanning cleaning functions, unique keys, joins, timestamps and missingness
• Your cleaning decision log
• Only charts that answer named questions, each with a denominator, units, source and limitation. Cover the relevant distributions, missingness/coverage and time or group variation. Show uncertainty when making an estimate. Use readable labels and a palette that works without colour.
• A data contract covering row keys, both timestamps, feature windows, label maturity, actual outcome availability/coverage and permitted use
• A source manifest with raw-input checksums, extraction cutoff and documented exclusions

Give the package an analysis entry point and document its exact uv run --locked ... command. It must read the documented inputs and rebuild the cleaned output, summary results and chosen charts without notebook state. Run uv run --locked pytest separately. Passing tests alone do not reproduce the analysis.

Choose a dataset that you found yourself: a city open-data portal, a government export, a niche marketplace, or your own data exported from a service you use. Before Week 4, pick an outcome you could later predict, such as a delay, a price or a category, and the inputs available at prediction time. Chapter 3 will ask you to model it. In the handoff, state the prediction time, row/group key, reserved holdout and a workable split. If the dataset cannot support a supervised question, change the dataset and write down why. Avoid inventing labels or using the outcome as an input.

Before you move on

• A stranger can install the locked environment, run the tests and regenerate your analysis with the documented command.
• You can defend two cleaning decisions against "why not just drop those rows?"
• You can write and explain a top-N-per-group query, checking ties and duplicate keys on a fixture. Documentation is allowed.
• You can predict the output shape of a broadcast before running it.
• You can show that a late-arriving record and an unfinished label cannot enter a training row.
• You can tell apart a mature observed negative, a censored duration and a missing outcome, and explain how each enters your denominator.
• You can show that a join preserves the intended grain and explain the denominator of each chart.
• The Chapter 3 handoff names a target, prediction time, available inputs and defensible split, and states any historical-data limits.

A few things worth knowing

• The free Python Data Science Handbook at jakevdp.github.io is the 2016 first edition, written against Python 3.5, though still widely recommended. The NumPy and matplotlib chapters are useful for the concepts. The pandas and scikit-learn chapters teach idioms that may now raise errors or behave differently, so check its examples against current documentation. McKinney's book plus the pandas migration guides is the main route here.
• Polars is optional. This chapter uses pandas to match its selected tutorials. Polars is a useful alternative when your team's stack or measured memory and runtime needs call for it. scikit-learn supports Polars transformer output. Rewrite one cleaning function only after the analysis works, and check correctness before speed.
• Titanic, Iris and MNIST are fine for a ninety-minute warm-up. As portfolio pieces they are so common that they tell a reviewer very little about you, which is a shame given the effort. A dataset you sourced yourself does much more work on your behalf.

Previous: Getting oriented  ·  Next: Core machine learning

Weeks 6–13  ·  about 80 hours.

Book index  ·  Chapter resources

The short version. This chapter teaches you to train a model carefully. Split the data so the score means something, pick a metric tied to a real cost, compare against a simple baseline, and explain the result. The same habits carry straight into evaluating language models later. You finish with a model served behind an API.

Use your Chapter 2 dataset. Same repo or a fork — the story should connect. Review the holdout you set aside when choosing what to predict, and keep it fixed after seeing outcomes.

In this chapter: Plan  ·  Learn  ·  Build  ·  Completion  ·  Resources

Your eight-week plan

About ten hours a week. Follow the rows in order, including the baseline and the README.

The table uses binary classification as its first-project example. For a numeric target, use DummyRegressor, regularised linear regression, RandomForestRegressor and a boosted regressor. Swap the threshold exercise for error/interval analysis and a cost-based decision rule. The same split, holdout and API requirements apply. A well-explained negative result completes the comparison. Avoid searching for a misleading split just to beat the baseline.

For a first multiclass project, compare a class-prior baseline. Report a confusion matrix and per-class precision/recall alongside the aggregate you chose. A binary cost threshold does not carry over to several classes. Define the cost of each action/outcome pair and choose the lowest expected cost using suitable probabilities. Multilabel tasks also need their own label-level decision and evaluation design. Leave them for later if the basic workflow is new.

In your uv project, start with uv add scikit-learn. In Week 9, add the boosted-tree and SHAP packages you chose, following their official install guides, and commit the updated lock. Keep one boosted candidate. Installing three libraries is not a model comparison.

Is this still worth eight weeks in 2026?

Yes. The skills here support both tabular business decisions and generative systems. A more capable pretrained model still needs a deployment-matched test, a baseline, a suitable metric and a cost-aware decision. The split and leakage habits carry into Chapter 5, though agent evaluation also needs its own tests for tools, safety and interaction.

Eight weeks, then, rather than six months.

What you will be able to do

Take an unfamiliar dataset, choose a validation scheme and defend it, choose a metric and tie it to a real cost, compare to a simple baseline, and explain in writing why each model helped or did not.

What to learn

Worked lesson: a probability is not an action

Four made-up development cases have predicted event probabilities [0.1, 0.4, 0.7, 0.9] and true labels [0, 1, 0, 1]. A false positive costs 2 units and a false negative costs 8. Assume correct decisions have zero cost and there is no capacity limit.

At threshold 0.5, the last two cases are positive: one false positive and one false negative cost 2 + 8 = 10. At threshold 0.2, the last three are positive: one false positive costs 2. The model has not changed. The action policy has.

With calibrated probabilities and these costs, compare the cost of acting, 2 × (1 - p), with the cost of not acting, 8 × p. Acting is cheaper when p > 2 / (2 + 8) = 0.2. Set a rule for ties. This assumes the error costs describe the decision. It cannot show a treatment benefit from a churn-risk score.

probabilities = [0.1, 0.4, 0.7, 0.9]
labels = [0, 1, 0, 1]

def observed_cost(threshold, false_positive_cost=2, false_negative_cost=8):
    return sum(false_positive_cost if p >= threshold and y == 0 else
               false_negative_cost if p < threshold and y == 1 else 0
               for p, y in zip(probabilities, labels))

assert observed_cost(0.5) == 10
assert observed_cost(0.2) == 2

Try it yourself. Swap the costs: false positives cost 8 and false negatives cost 2. Work out the threshold and its observed cost. Is a four-case table enough to show calibration?

The reference lab has a reproducible baseline comparison. Write down your expected result before running it, and explain any difference from what you see.

Core concepts and practical failure modes

On a first pass, focus on the split, baseline, pipeline, metric and decision. The dated benchmark, advanced uncertainty and causal sections below are optional depth after that baseline works.

The learning problem. A model maps inputs to outputs, and a loss measures its errors. Many models optimise differentiable parameters with gradient-based methods. Trees learn partitions, and linear regression can also use a direct solver. Learn regularisation, bias and variance, and how each candidate is optimised. Not every model trains like a transformer.

Learning curves are the most practically useful idea here and the most often skipped. Plotting training and validation score against training-set size helps you judge whether more representative data or a different model might help. Treat it as a clue to test, not a firm diagnosis. Label noise, distribution shift and a changing population mix can produce similar patterns. Keep time or group structure while building the curves.

Validation, which is where the real care goes. Your validation score estimates performance under its sampling and deployment assumptions. Information not available at the simulated training or prediction time can quietly invalidate it. Leakage often makes scores look too good, and even a clean holdout cannot promise performance after an untested change.

Three families to know:

Two questions, asked before you choose a splitter, prevent most of this: does a row's identity repeat, and does time matter?

Match the split to deployment. Predicting next month's activity for existing customers can fairly reuse their earlier histories. If the launch includes new customers, also evaluate on unseen ones. A time split and a group split answer different questions, and sometimes you need both. Leave out immature training labels. If a label needs 30 days plus seven days of reporting delay, those 37 days must have passed by the simulated training cutoff. A fixed row-count gap in TimeSeriesSplit does not by itself enforce a calendar-based label window.

Keep separate training, model-selection, calibration/decision-selection and final test data, or use carefully designed out-of-fold estimates. Refitting a model can change its calibration, so evaluate the exact model and calibrator you will serve. Early stopping, feature selection, resampling and hyperparameter tuning all belong inside the training procedure. Open the final test only after the choices are frozen.

Label maturity and observation are separate checks. A 30-day window can end with the source record missing. Keep that out of the negatives, and avoid quietly scoring only the matched outcomes. Report coverage by time and cohort, and say which population the score represents. A duration target with incomplete follow-up needs a censoring-aware design, as Chapter 2 explains.

A pipeline is necessary, but it does not choose the right time or group handling for every component. For example, scikit-learn's TargetEncoder uses cross-fitting in fit_transform. Calling fit(...).transform(...) on the same training rows does something different. Default internal folds are random. Since 1.9 you can pass grouped splitters through metadata routing, but every row must appear in a validation fold exactly once. An ordinary forward-only TimeSeriesSplit breaks that rule. Time-ordered encodings need an explicit past-only design or a different encoder. Keep a training row's own and future labels out of its feature encoding.

Repeatedly picking the best score across many models and settings also overfits validation. Use a fixed tuning budget and an untouched test. To estimate the whole selection procedure, use deployment-matched nested validation. Compare models on the same rows and horizon, and record version, seed, search budget and failed runs. The nested-CV example explains the selection bias, but its random folds need adapting for time or groups.

Metrics, which are decisions in disguise. On a dataset with 1% positives, a model that always predicts "no" scores 99% accuracy. A metric is a claim about what it costs to be wrong, and choosing one is a business decision you are making on someone's behalf, usually by leaving it at the default.

Use DummyClassifier(strategy="prior") as the probability baseline. It estimates class frequencies from training labels and predicts the majority class. Unlike strategy="most_frequent", its probabilities are not forced to zero and one, so it is a useful log-loss/Brier comparator. Also report the simplest real action policy under your stated costs, such as always taking no action. A probability baseline and an action baseline answer different questions.

Learn precision and recall well, and when each matters more. ROC-AUC measures ranking, but can coexist with poor precision when positives are rare. Report average precision or a clearly defined PR-AUC, positive prevalence, and precision/recall at your team's real capacity. PR metrics also depend on prevalence, so they are not directly comparable across datasets.

Probability and action are separate. A score of 0.8 should mean about 80% positives among similar scored cases. Check a reliability diagram and log loss or Brier score. Those losses also reflect discrimination, so neither alone proves calibration. Fit sigmoid or isotonic calibration on independent, representative data and recheck it by cohort. Small calibration samples make flexible isotonic fits unstable. Calibration on last month's sample does not guarantee calibration after prevalence, eligibility or data collection changes. scikit-learn's calibration guide explains the methods. Choose thresholds on validation data using costs and capacity, then evaluate once on untouched test data. See the threshold tuning guide.

Class weighting, oversampling and undersampling can change a classifier's probability scale. Fit any sampler inside training folds only. Keep calibration, decision selection and final evaluation at deployment prevalence, or document justified sampling weights. A balanced test set answers a different question, as imbalanced-learn's leakage examples show. Cost-weighted scores are not automatically raw event probabilities. Check their calibration before applying a cost-derived probability threshold, even if you used costs in training.

Calibration needs the same care with splits. CalibratedClassifierCV defaults to stratified folds for classification, which ignore your time or entity boundary. A simple route: wrap a fitted pipeline in FrozenEstimator and calibrate it on a separate, representative set that follows the same deployment split. Keep threshold-selection and final test data apart. For cross-fitted calibration, check the API's fold requirements and class support, and confirm your time splitter works with your ensemble setting.

A tiny threshold example. Suppose false negatives cost ten times false positives — missing a fraud case is ten times worse than flagging an honest transaction. With calibrated probabilities, zero cost for correct classifications and no capacity limit, expected loss is minimised at C_FP / (C_FP + C_FN) = 1 / 11, about 0.091. Measure precision and recall at that threshold on validation data, since the cost ratio does not determine them. Capacity limits or case-varying costs need a richer decision rule. On 1% positives, "always no" still scores 99% accuracy, which shows why accuracy alone is not enough.

Tree models. Gradient-boosted trees are a strong practical candidate for many tabular problems. Compare them against a simple model on your actual split. Learn what bagging and boosting each buy you, and what max_depth, min_child_weight, subsample, colsample_bytree and early stopping actually do to bias and variance. XGBoost, LightGBM and CatBoost are alternatives for this week's single boosted candidate. CatBoost is convenient for many categorical fields, and LightGBM can be efficient on large tables. Measure those benefits on your data and hardware.

Also learn interpretability: permutation importance and why it misleads under correlated features, partial dependence, and SHAP, which attributes a single prediction to the features that drove it. Feature attributions describe a model's prediction, not the effect of changing a customer's behaviour. A SHAP plot alone cannot show causality, fairness or fitness for a high-stakes decision. When explanation is a requirement, compare simpler interpretable models. scikit-learn's causal interpretation example shows why predictive explanations can mislead about interventions.

Regression and forecasting. For a numeric target, compare a constant or seasonal baseline, regularised linear regression and a boosted regressor. Report MAE in business units, look at errors by segment, and use quantile/prediction intervals when uncertainty changes the decision. MAPE is undefined at zero and unstable near zero, so choose scaled or absolute errors on purpose. Forecasting: Principles and Practice explains the tradeoffs. For demand over time, compare seasonal-naive, exponential-smoothing or ARIMA baselines with a lag-feature tree. Backtest at several rolling origins and at the real forecast horizon. Compute lags and rolling statistics from past data only. Start with the scikit-learn lagged-features example.

Update features the way serving will. Rolling one-step forecasts may use observations that arrive between predictions. A 24-hour forecast issued now cannot use tomorrow's realised demand in later lags. For fixed-origin multi-step forecasts, recursively feed back predicted values, or train direct per-horizon models on origin-available inputs only. Report error by horizon, as in the forecast cross-validation example. A forecast at today's price cannot tell you demand at an untested price.

For a forecasting project, an optional challenger can be Chronos-2 or TimesFM 3. Their current interfaces support multivariate/covariate-informed forecasts. Check the chosen checkpoint's frequency, context and horizon limits. Future covariates must really be known at the forecast origin. A scheduled holiday is, but tomorrow's realised weather or sales are not. Compare business error and interval coverage at the same rolling origins, including holidays, stockouts and regime changes. The model's own benchmark wins do not replace this check. TimesFM 3's downloadable weights have noncommercial/nonproduction terms separate from its code and hosted commercial offering. Keep seasonal/statistical baselines even when a foundation model wins.

Clustering and anomaly detection, when the question calls for them. Use scaled K-means as a segmentation baseline, PCA for exploring correlated features, and Isolation Forest as an anomaly candidate. Check segment stability and business usefulness. A silhouette score alone does not validate a customer strategy. Review anomalies against labelled examples and measure precision at the review budget. These are optional extensions. Your supervised project comes first.

Intervals that match your data. Compare models in pairs on the same test cases. When customers repeat, bootstrap customers rather than rows. For serial dependence, use suitable time blocks. Fold standard deviation is not a confidence interval for future deployment. Freeze the comparison plan before the final test, and report the sample size and dependence assumptions.

Prediction intervals and confidence intervals answer different questions. The first covers one future outcome, the second a measured model comparison. As optional depth, split conformal prediction wraps a model using separate calibration residuals. Its usual guarantee is marginal coverage under exchangeability: an average over cases, not a promise for each customer or under arbitrary temporal drift. Report interval width and observed coverage by horizon/cohort. Neither quantile forecasts nor conformal intervals give causal evidence.

From prediction to intervention. Offer recommendation, pricing and retention ask what changes when you take an action. That claim needs randomised treatment data or a causal identification design you can defend. Chapter 9 has the data and model recipes. A churn classifier, sales regressor or click ranker can be a useful baseline without showing that a campaign increases profit.

The maths, paid for as you go. This is the most argued-about part of any AI roadmap, so here is my reading of it, plainly.

The people who teach the hands-on courses agree to an unusual degree. fast.ai's own prerequisites page says high-school maths is enough. Karpathy's course asks for "intro-level math (e.g. derivative, gaussian)". Parr and Howard tell readers to come to their matrix-calculus paper only after they are already training networks. The failure I see most often is over-preparing, not under-preparing: committing to a long linear algebra course before writing any model, and stalling around month three with nothing built.

The minimum is small enough to name:

• Linear algebra: vectors, matrices and tensors. Shapes and axes. Matrix multiplication and why shapes must line up. Transpose, dot product as similarity, broadcasting. A matrix as a transformation, and matrix products as composition. Eigenvectors, SVD, rank, norms and orthogonality can wait a few weeks.
• Calculus: the derivative as sensitivity, partial derivatives and the gradient. The chain rule, which carries more weight than anything else here. Recognising a Jacobian, and gradient descent. Integrals, Hessians and hand-derived matrix calculus can wait, or may never come up.
• Probability and statistics: random variables, expectation and variance, independence, conditional probability, Bayes' rule, the Gaussian, Bernoulli and categorical distributions, likelihood and maximum likelihood, log-probabilities, and cross-entropy as a loss.

That is perhaps twenty to forty hours of focused study rather than a semester. Classical hypothesis testing gets less weight here on purpose. It matters a great deal for experimentation work, which the Data Scientist branch of Chapter 7 covers.

The counter-argument deserves respect too. People who take the code-first route and never come back to the maths tend to plateau at the point where they need to read a paper, debug an unusual loss, or design an architecture. The bill arrives either way. Paying it here, once you have something concrete to attach it to, tends to go better.

Resources

All 43 resources for this chapter, with notes on each, are in resources/.

Choose one main textbook or course, plus selected reference sections. Full-resource reading estimates are alternatives, not a total to fit inside 80 hours. Pick up the maths as you need it. Once the required project is complete, use the buffer week for one optional research comparison.

The ones to begin with:

• An Introduction to Statistical Learning (ISLP — Python edition) — Gareth James, Daniela Witten, Trevor Hastie & Robert Tibshirani, with Jonathan Taylor. A free book. Pick sections that fit this chapter's 80 hours, rather than the full 60–90-hour reading estimate.
• scikit-learn: Common Pitfalls and Recommended Practices — scikit-learn core developers. Free documentation, 2–3 hours. Read it in week 7, before the leakage exercise.
• Approaching (Almost) Any Machine Learning Problem — Abhishek Thakur. A free book, 20–30 hours.
• Interpretable Machine Learning — Christoph Molnar. A free book, 12–18 hours. Start with the SHAP and permutation-importance chapters.

What to build

A model behind an API. Take your Chapter 2 dataset and complete the loop:

1. A split you chose deliberately and justified in writing
2. All preprocessing inside a Pipeline
3. A simple baseline, then a linear/logistic model, a random forest and a boosted tree for the target type
4. A metric tied to a stated cost, reported with confidence intervals
5. A classification threshold or regression decision rule taken from the stated costs and constraints
6. SHAP explanations, including a misclassified case or a large-error regression case
7. A FastAPI endpoint with schema validation, and tests for both code and data
8. A reliability plot or regression error analysis, an untouched test comparison, and documented label maturity, outcome coverage and calibration/decision-selection splits
9. A versioned model/preprocessor/calibrator bundle with dependency and data provenance, plus a fresh-process API check

The README is the deliverable. Include a table of every model and what it bought you, a section on what did not work, and a section on known failures. Those last two are unusual in a junior portfolio and they do a lot of quiet good.

Return a model version with predictions, validate feature names/types/units, and define behaviour for missing or unknown categories. Serve the preprocessing/model/calibration bundle you tested, and keep retraining and scaler fitting out of the request. For Week 11, add the API and testing dependencies:

uv add "fastapi[standard]"
uv add --dev pytest httpx
uv run --locked pytest

Follow the official request-body and testing guides. In a fresh process, test a valid request against the saved bundle's prediction, including the probability or numeric prediction, model version and chosen decision rule. Also test a malformed request and your chosen handling of missing/unknown inputs. Load a Python model artifact with the training environment's dependency versions. Loading across scikit-learn versions is unsupported, as the persistence guide explains. Document measured latency, memory and CPU fallback. Chapter 6 covers production operations.

The Week 7 leakage exercise: fit preprocessing on the full dataset on purpose and record the score. Then move the fitting inside a Pipeline and repeat on the same split. Report the real difference, even if it is small, since this leak does not always change the score much. Availability-time and group errors need their own data or split fixes. A pipeline fixes this kind of leakage, not every kind.

Handoff to Chapter 4. Commit the split definition, feature pipeline and every model's results. Chapter 4 can train a small network on these same features and compare it with these numbers on the same rows. Keep the split fixed and the test untouched.

Before you move on

• Given a dataset description, you can choose a split and a metric and defend both in a minute.
• You can explain the costs and derive your classification threshold or numeric action rule out loud.
• You can explain what SHAP computes, and one thing it does not tell you.
• You can call your endpoint, match a valid prediction to the saved bundle and show it rejecting a malformed request.
• You can explain which decisions your predictions support and which need an experiment.
• Your comparison states the actual split, model/checkpoint versions and compute limits. The served bundle reproduces the tested predictions.
• You can explain how missing outcomes or a changed class balance could change the reported score, probability scale and action threshold.

A few things worth knowing

• Start with ISL rather than ESL. They share authors and the titles are similar, but Elements of Statistical Learning is a graduate statistics text. People often start there, stop at chapter three, and conclude they are not a maths person. They usually are. It was the wrong book.
• If you want Strang's linear algebra, MIT 18.065 is the machine-learning-facing course. 18.06 is excellent and spends its first month on material you will not use here.
• Kaggle is a good gym and a weak portfolio. Competitions give fast, honest feedback on validation design, which is genuinely valuable. They also remove problem framing, data sourcing and metric choice, which are three of the harder parts of the job.

Previous: Data foundations  ·  Next: Deep learning

Weeks 14–22  ·  about 90 hours.

Book index  ·  Chapter resources

The short version. Build a tiny autograd engine, learn an explicit PyTorch training loop, debug a model that fails to learn, and implement attention. Finish with a trained model, a fair evaluation and a measured quality, memory and runtime trade-off. Modern tools speed this up, but you still need to understand the data, gradients and evaluation.

Same thread, new skill. You already know tabular ML from Chapter 3. Here you learn how neural nets learn — the machinery behind LLMs. The first build option below keeps your Chapter 3 data, split and baseline.

In this chapter: Plan  ·  Learn  ·  Build  ·  Completion  ·  Resources

Your nine-week plan

About ten hours a week. Start with Karpathy, or with fast.ai lessons 1–3 if you need an early win (see below).

What you will be able to do

Write backpropagation from an empty file, train a network in PyTorch without a template, and take a model that is quietly not learning and work out why.

On the order to do this in

You will be told three incompatible things about how to start deep learning, and all three come from people worth listening to.

• Top-down (fast.ai): train a working model in lesson one, then come back to the mechanics. This can keep you engaged, though it is a teaching choice, not a proven cure for giving up.
• Bottom-up (Karpathy, d2l.ai): start from a single derivative and build up to a transformer. The argument is that nothing else produces real understanding of what is happening.
• Maths first: foundations before code. This is the one ordering I would gently steer you away from, for the reasons in Chapter 3.

I suggest bottom-up first, top-down alongside: build the autograd engine, then learn PyTorch, then use fast.ai for transfer learning and momentum. If you know that you lose interest without an early win, invert it and do fast.ai's first three lessons before the autograd exercise. Both work.

On frameworks. Use PyTorch here so the exercises share one implementation. Choose another framework later when a project or team needs one. Keras supports custom PyTorch training steps (Keras custom-training-step guide), so it still lets you learn the loop. JAX is useful beyond very large training runs too. The point is a consistent first project.

The official PyTorch 2.14 release, published 2 September 2026, adds compiler, hardware and distributed-runtime work. Pin a supported release and use its docs. A release-note speedup on one kernel or device does not predict your model's end-to-end speed. Several new features are marked experimental.

What to learn

Worked lesson: gradients must accumulate

Let x = 3, a = x × x and y = a + a. The forward pass gives a = 9 and y = 18. a is used twice, and each use contributes 1 to dy/da, so its accumulated gradient is 2. The two inputs to x × x then each contribute x × 2 = 6 to dy/dx. The final gradient is 12. Assigning a gradient with = instead of accumulating with += silently loses a path.

def objective(x):
    a = x * x
    return a + a

x, epsilon = 3.0, 1e-5
numeric_gradient = (objective(x + epsilon) - objective(x - epsilon)) / (2 * epsilon)
assert abs(numeric_gradient - 12.0) < 1e-6

This numerical gradient is a small, independent check on your backward pass, for debugging rather than training. It needs many forward evaluations and can be unstable near non-differentiable points or with a badly chosen step size.

Try it yourself. Change the graph to y = a × a, keeping a = x × x. At x = 3, predict the value and derivative before running the numerical check. Which reused nodes need accumulation?

Core concepts

Backpropagation, by building it. Write a scalar autograd engine: a Value class with the arithmetic operations, a topological sort for the backward pass, and correct gradient accumulation when a node is used more than once. That last detail is where most people's bug lives, and finding it is the moment the chain rule stops being a formula.

PyTorch as a language. Tensors, Dataset and DataLoader, and the explicit training loop you should be able to implement and explain. Know why model.train() and model.eval() differ, why zero_grad() exists, what torch.no_grad() saves you, and why both the model and the data need to be on the same device.

Start with this explicit loop and inspect each state change:

for epoch in range(num_epochs):
    model.train()
    for x, y in train_loader:
        x, y = x.to(device), y.to(device)
        optimizer.zero_grad()
        loss = criterion(model(x), y)
        loss.backward()
        optimizer.step()

For validation, call model.eval() and run the forward pass inside torch.no_grad() or torch.inference_mode(). Evaluation mode changes dropout and batch-normalisation behaviour. Turning off autograd avoids building a backward graph. Neither does the other's job. When debugging, check mode and gradient state separately rather than guessing. After validation, switch back to training mode at the start of each epoch. PyTorch's optimisation tutorial shows separate training and evaluation loops.

Check the loss contract too. For ordinary multiclass classification, pass unnormalised logits to CrossEntropyLoss, with class-index targets of the expected shape and dtype. Applying softmax first changes the objective. Use valid probability targets only when the task calls for them. The loss documentation also explains ignored targets and reduction. For epoch loss, sum the loss numerator and its intended denominator across batches, then divide. A plain average of batch means can overweight a short last batch or a sequence with few supervised tokens.

Training dynamics, and how to break them. This is the part that separates people who can debug training from people who change the architecture and hope. Instrument your runs with loss curves for both splits, gradient norms per layer, and activation histograms.

Five habits fix most problems:

1. Try to overfit ten clean samples with regularisation and augmentation turned off. An unexpected failure is a clue, though noisy labels, a constrained model or the loss can limit the fit. Check shapes, target encoding and gradient flow before adding complexity.
2. Compare against a trivial baseline.
3. Sweep the learning rate across orders of magnitude and plot it.
4. Turn off every extra (augmentation, scheduler, dropout), get a working baseline, then add one thing at a time.
5. Look at the data. Print ten samples as the model sees them, after transforms. A surprising share of "it will not learn" turns out to be inverted labels or a normalisation applied twice.

When it still will not learn, work down this checklist. Change one thing per run and log it.

• Overfit one small batch. If a few examples cannot reach near-zero training loss, suspect the loss, targets or update step before the architecture.
• Data, labels and shapes. Assert shapes and dtypes where the loss is computed, and check label values against the output layer. (B, 1) outputs against (B,) targets broadcast to (B, B) in a regression loss, as in the Chapter 2 exercise.
• Loss at initialisation. With near-uniform initial predictions, multiclass cross-entropy starts near ln(C), about 2.30 for ten classes. Much higher suggests overconfident initialisation. An output bias matching the class prior or target mean is a cheap fix.
• Learning rate, then warmup and schedule. Find a working constant rate by sweeping orders of magnitude. Only then add a short warmup for early instability, and a decay schedule.
• AdamW and weight decay. Adam with weight_decay adds an L2 term to the gradient by default. AdamW decouples the decay from the adaptive update. Check which one you use, and whether biases and normalisation weights are excluded from decay.
• Gradient norms and clipping. Log the global norm each step (clip_grad_norm_ returns it). Clip to cap occasional spikes, calling scaler.unscale_(optimizer) first if you use a gradient scaler. Clipping on most steps acts as a smaller learning rate.
• Vanishing or exploding gradients. Early-layer gradient norms orders of magnitude below later ones, with a flat loss, suggest vanishing. Growing norms, loss spikes or NaNs suggest exploding. Check initialisation for the activation, normalisation, residual paths and the learning rate.

Karpathy's recipe for training neural networks and Google's Deep Learning Tuning Playbook expand these steps. Both have notes in resources/.

Transfer learning. When suitable pretrained weights exist, adapt them and compare against a small from-scratch baseline. For tiny datasets, freeze the backbone and train a new head. Unfreeze more as you get more data, and try lower learning rates for earlier layers. CNNs make weight sharing and local inductive bias concrete. Vision transformers and self-supervised encoders are useful challengers when their data, memory and latency needs fit. Avoid claiming one family wins every small-data or edge task. Augment training data only, deduplicate before splitting, and hold out whole subjects or capture sessions when deployment needs that generalisation.

Freezing parameters with requires_grad_(False) does not freeze batch-normalisation buffers. Decide whether to keep the pretrained running statistics or adapt them on training data. To keep them, return the relevant backbone modules to evaluation mode after each outer model.train() call, leaving the new head in training mode. Record the choice, and keep validation data out of statistics updates. PyTorch's BatchNorm documentation describes running statistics. Evaluation mode and gradient freezing are separate controls.

Attention, built by hand. Follow Karpathy's build-a-GPT lecture, typing rather than watching, then build a byte-pair tokeniser. Inspect how it splits whitespace, numbers and several languages. Tokenisation affects the input and token cost, but is only one possible cause of reasoning failures. Add tests for causal masking, padding, position handling and encode/decode round trips. Split a language corpus by source document before making token windows, so neighbouring or duplicated text cannot leak into validation. Learn BPE merges and any fitted preprocessing from training text only, then keep them fixed for validation and the final test.

The scale has a useful derivation. Suppose the d query and key components are independent, with zero mean and unit variance. Then each product has variance 1 and the dot product, their sum, has variance d. Dividing by sqrt(d) restores variance 1 under those assumptions. This helps stop dot products growing with head dimension and over-concentrating the softmax. Learned components need not meet those assumptions exactly, so this explains the scale rather than proving exact normalisation for every trained head. Apply a causal mask before softmax, so a masked future token gets no attention weight, however large its unmasked score.

To test causal masking, change a future token and check earlier logits stay the same within numerical tolerance, in evaluation mode. Shift targets correctly for next-token prediction, checking whether the model already shifts them internally. Exclude padding from the loss too. Masking it in attention alone leaves its training targets in place.

Read Attention Is All You Need after this, not before. It is a 2017 machine-translation paper describing an encoder-decoder model. Compare its attention mechanism with your decoder-only model. There is no useful percentage for how much of its design a current model shares. Read cold as a first resource, it tends to discourage people. Read after you have built the thing, it is easier to connect to the code. Its notes are in resources/.

Efficient training, after correctness. Profile an eager baseline, then change one thing at a time. Automatic mixed precision can cut memory and runtime on supported hardware. Choose the dtype for the device, check loss and gradients stay finite, and use a gradient scaler when your precision and setup need one (PyTorch AMP recipe).

Gradient accumulation changes the effective batch size. Scale the loss correctly and step the optimiser at the intended interval. With variable-length sequences, divide the summed token loss by all non-ignored prediction targets in the accumulation window, including a short final window. Averaging each microbatch's mean equally changes token weighting. Check your pinned trainer's handling so you do not divide twice. The Transformers accumulation guide documents this denominator. Accumulation does not reproduce a large batch's batch-normalisation statistics. Activation checkpointing trades recomputation for activation memory. These are different controls, not interchangeable fixes.

Try torch.compile once the model is correct. Record first-run compilation separately from steady-state time, because changing shapes and graph breaks can erase the benefit. Report peak memory, examples or tokens per second, and the same validation metric as the eager model. A slower or less stable result is still a useful experiment (official compile tutorial).

Where recent research fits. The material below was checked on 9 October 2026. It is orientation and optional depth within the 90 hours. Reproducing a frontier model is extra work.

For adaptation, tell full fine-tuning apart from LoRA, which trains low-rank weight updates, and QLoRA, which trains adapters on a quantised base model. Neither guarantees lower serving latency or unchanged task quality. Supervised fine-tuning learns from demonstrations and preference optimisation from comparisons. RL with verifiable rewards needs a real verifier. Learn these distinctions here. Run one bounded adaptation experiment only when it addresses a measured failure in your project: optionally in Chapter 5's week 31 fine-tuning decision, or as Chapter 7 depth work. The primary papers and current implementation guides are in the resource list.

Resources

All 30 resources for this chapter, with notes on each, are in resources/. The main path uses selected sections, so the full-course hour estimates are not extra work.

The ones to begin with:

• Neural Networks: Zero to Hero — Andrej Karpathy. A free course, 40–70 hours; lecture 1 (micrograd) is week 14.
• PyTorch official tutorials — "Learn the Basics" — PyTorch documentation team. Free documentation, 8–12 hours.
• Practical Deep Learning for Coders — Jeremy Howard. A free course, 40–80 hours; lessons 1 to 4 are what this chapter uses.
• Understanding Deep Learning — Simon J. D. Prince. A free book, 60–80 hours, used for lookup, not read cover to cover.

What to build

A model you trained yourself, with an honest ablation — retraining with one component removed or swapped at a time, to measure what each one contributes. Pick one:

• Same thread: a small network on your Chapter 3 features, in raw PyTorch. Tune on the same training and validation data, then compare once on the same test with your Chapter 3 baseline and boosted tree. Losing to the trees is worth reporting. The reference lab sets up the same experiment
• A classifier on 200-plus photos you took, in raw PyTorch, deployed as a small demo
• A character-level language model on a corpus that means something to you, with the modern architecture pieces swapped in one at a time
• A reproduction of one bounded paper component, implemented from the paper and compared against a reference

Whichever you pick, save the split and data versions, configuration, environment, seed and a full training checkpoint: model, optimiser, scheduler and scaler if used, random-generator states and progress. Save preprocessing and tokeniser versions too. Resuming mid-epoch can also need the sampler or data-loader position. Practise resuming from it. To judge whether an ablation helped, record how results vary across repeated runs. A seed does not guarantee identical results across releases or devices (PyTorch reproducibility notes).

Keep a final test untouched until model selection is done. Include a real ablation, failed runs, one diagnosed failure, and a demo with its inference settings recorded. If hosting is unavailable, a reproducible local demo is fine, but do not present a screenshot as a live service.

The ablation is where most of the value sits. A table showing that one change mattered and three were within noise, with wall-clock times alongside, says more about you than a good final number.

Handoff to Chapter 5. Carry these habits forward: an untouched test, replayable runs and a fair baseline comparison. Chapter 5 applies them to documents from your Chapter 2 domain, starting from a pretrained model rather than one you trained.

Before you move on

• Your autograd engine passes a numerical gradient check on a graph with a reused node, and you can explain and fix a deliberately broken accumulation rule.
• You can write a training and validation loop, find a wrong mode or gradient setting, and show that validation does not update the model. Documentation is allowed.
• Given an unhealthy loss curve, you can name a likely cause and the instrument that confirms it.
• You can label a causal attention matrix with its shapes and mask, and explain the variance reason for dividing by the square root of the head dimension.
• You can reload a checkpoint and explain what would be lost by saving only model weights.
• You can compare a precision or compile change on quality, memory and measured runtime.

A few things worth knowing

• The Goodfellow, Bengio and Courville book is free, famous, and where a lot of people stall. Published in 2016 with only minor corrections since, it has no transformer chapter and its sequence and generative sections are well out of date. Chapters 2 to 9 are still a clean rigorous treatment of the mathematics. Simon Prince's Understanding Deep Learning, also free, is the more useful modern choice.
• Check the date on course materials. The older CS231n recordings teach basic training mechanics. The Spring 2026 schedule adds current vision topics. Its slides and assignments are public, but the current lecture videos may not be.
• Rewrite one fast.ai exercise in an explicit PyTorch loop. This checks your grasp of transforms, optimiser updates and validation behaviour under the convenience API.
• Use materials that contain working exercises. Read microgpt, a compact pure-Python teaching implementation, and inspect nanochat, an author-maintained training pipeline. Pin its commit and compute assumptions before running it. Neither a tiny names model nor a repository headline shows production capability.

Previous: Core machine learning  ·  Next: Language models and AI engineering

Weeks 23–32  ·  about 100 hours.

Book index  ·  Chapter resources

The short version. Treat a language model as a dependency: control its context, retrieve the right facts, constrain its actions, and measure whether it works. Finish with a small application, an independent release evaluation, measured operating costs, and a user who is not you.

Same thread. Build on documents from your Chapter 2 domain: if you modelled parking-ticket disputes, extract fields from the appeals. If the domain has no suitable documents, use API docs from work or your exported notes, and record that change. The eval discipline is the same as in Chapter 3, applied to text.

In this chapter: Plan  ·  Learn  ·  Product sprint  ·  Completion  ·  Resources

Your ten-week overview

Weeks 23–32, about ten hours a week. Read selected lessons rather than every resource. Weeks 27–28 set aside about 20 hours for the two-week product sprint. Its days are work sessions, not fourteen full working days. Reuse the cases, pipeline and deployment from weeks 23-26.

New words in this chapter (context engineering, retrieval, gold set, pass rate, trace, judge) are explained where they first appear below, and collected in Terms in plain English.

What you will be able to do

Ship a language-model application with an evaluation suite you built before you started optimising, an error taxonomy taken from real traces, and a cost per thousand requests you measured.

The one idea to take from this chapter

A plausible answer is an observation, not proof that the system works. You need examples with criteria you can check, traces that show how answers were produced, and a release decision that was not tuned to the test answers. This holds for document extraction and for agents that change files or call services.

An application with a good interface and no labelled examples is hard to improve, because you cannot tell whether a change helped. Add a hundred hand-labelled traces, a written list of failure modes and a judge checked against your own labels, and it becomes a different thing. A trace is the saved record of one request: input, intermediate steps and output. A judge is a model prompted to grade another model's answers against criteria you wrote. These records help you explain failures and compare changes, but they do not replace a domain expert or application-enforced permissions.

What to learn

Worked lesson: evaluate the change on the same cases

Take 50 invented task families, sampled independently, with one evaluated case each. Freeze the rubric, then score the same cases with a baseline and a candidate. This development example uses human pass/fail labels and no repeated attempts:

The baseline passes 39/50 (78%) and the candidate 41/50 (82%). The paired improvement is (6 - 4) / 50 = 4 percentage points. Reporting only “82%” hides four regressions. The ten discordant pairs, where the versions disagree, hold the evidence about this change.

Assuming no change, an exact paired test treats the ten discordant outcomes as fair coin flips. Six wins against four losses is not surprising under that null:

from math import comb

wins, losses, total = 6, 4, 50
discordant = wins + losses
paired_difference = (wins - losses) / total
two_sided_p = min(1.0, 2 * sum(comb(discordant, k)
                             for k in range(min(wins, losses) + 1)) / 2 ** discordant)
assert paired_difference == 0.04
assert abs(two_sided_p - 0.75390625) < 1e-12

The p-value is not the probability that the models are equal, nor does it show whether an improvement would be useful. If your declared rule requires at least five percentage points of supported improvement and no severe regression, this table does not justify promotion. Record the decision, inspect development failures and collect enough independent cases for the next release. For an interval, resample the paired task-family differences, preserving the sampling unit. A tiny or dependent sample can make an ordinary interval misleading.

This is the exact binomial form of McNemar's paired test. The statsmodels documentation describes the exact and large-sample options. A repeated-trial or clustered design needs its own sampling assumptions, instead of treating every attempt as an independent pair.

Now audit a judge on these candidate outputs, treating failure as positive. Suppose it catches 7 of the 9 human-labelled failures, misses 2, and wrongly flags 4 of the 41 human passes. Failure sensitivity is 7/9 ≈ 78%, false-alarm rate is 4/41 ≈ 10%, and agreement is 44/50 = 88%. That reassuring agreement still hides two missed failures, so check severity and slices before trusting the judge. If you use these development labels to tune the judge, they can no longer give independent release evidence for it.

Try it yourself. A new candidate fixes all five cases both versions failed, but breaks three cases both passed. What is its pass rate and paired improvement over the original baseline?

Core concepts and optional depth

The first product needs a bounded task, a baseline, a trace, a checked output and independent evaluation. MCP, agent harness research and post-training are optional extensions once that path works. The reference lab is a no-API starting point for reproducible evaluation. Its synthetic outcomes do not measure a hosted language model.

The model as a dependency. Treat it like a database: a stochastic function with a latency budget, a token bill and a schema contract. Learn structured output, tool calls, streaming, cancellation and error handling. Where an endpoint enforces a schema, it controls the shape, not the truth. Validate values, references and tool arguments in application code. Record the model identifier, sampling/effort settings, prompt version and API/SDK version for reproducible comparisons.

As the baseline, use a current general-purpose model that supports your interface. Compare a smaller model and a stronger or reasoning model on your own tasks. Effort controls, reasoning visibility and billing differ by endpoint, so raise the budget only when evaluations justify it. Self-reported model confidence is not a calibrated routing score. Validate a router against labelled outcomes, including escalation calls in the comparison.

The 2026 linguistic-confidence study, whose authors report acceptance to EMNLP Findings, separates useful score ordering from calibrated probabilities. For your router, measure error among accepted answers against the fraction accepted, then freeze the threshold before the release test. Recheck calibration after changing model, prompt or domain. Token probabilities and agreement between sampled answers are also signals that need validating.

Economics you can measure. Record input, cached input, output and separately reported reasoning usage, plus retrieval, tool, storage and hosting charges, retries and failed requests. Compare cold and warm caches. Cache minimums, lifetimes, write charges and batch discounts depend on provider and model. Check dated pricing and cache documentation instead of assuming a fixed saving applies to the whole bill. Match usage fields to the provider's billing schema instead of summing every breakdown. Claude's thinking-cost guide explains that thinking usage is already included in billed output tokens, so adding it again double-counts cost.

If 1,000 attempted requests cost $8 and 800 pass task checks, the observed cost is $0.01 per successful request. Report cost per attempt and per success, p50/p95 latency and timeout rate. A cheaper model needing frequent escalation can cost more for the task. Set a spending limit and a fallback before public deployment. Enforce per-request and whole-application limits, counting retries and concurrent calls. Check whether a provider budget setting stops requests or only alerts.

Context engineering. The context window is a budget you allocate, not a bucket you fill. Order it so the stable parts (system prompt, tool definitions, examples) come first and can be cached, and the dynamic parts (retrieved documents, history) come after. The harder problems are all about time: compacting history as it grows, deciding what persists across sessions, compressing errors before they enter the context, and noticing that a long context window does not mean the model attends usefully to all of it.

Give sources identifiers and dates. When compacting history, keep unresolved tasks, decisions and source references. Test whether compaction drops a constraint or turns an unverified claim into a fact. Stored memory needs an owner, access permissions, update rules and deletion. NoLiMa (2025) tests long-context retrieval without easy literal matches, showing why a simple needle test can miss failures. Its results cover only the tested models and tasks. Include paraphrases, distant evidence and conflicting versions in your own tests.

Choose the information path. Retrieval-augmented generation (RAG) fetches selected sources. Long context supplies a larger body directly. Search fetches outside information when freshness or coverage requires it. Compare them before assuming you need a vector store.

The EMNLP 2024 RAG/long-context study compares both, plus hybrid routing. It supports a task-specific comparison, not a ranking of 2026 models. Hold the permitted corpus, source versions, query set and answer criteria constant.

Retrieval that you can diagnose. Start with keyword and dense baselines. Choose embeddings by language, domain, length, deployment cost and licensing, and follow their query-prefix and similarity conventions. Changing the embedding model usually means re-embedding the index. MMTEB (2025) broadens multilingual evaluation, but your own domain queries should decide. Chunk around document structure, keep source/version/parent IDs and deduplicate chunks. Apply user access before retrieval and check it again before returning sources, including caches. A cross-encoder reranker rescores retrieved query-document pairs at higher cost and cannot recover a missing candidate. See the official retrieve-then-rerank example.

Deploy the query encoder, document embeddings, chunk/source manifest and retrieval configuration as one compatible version. A new query encoder against an old index can silently change results. Test source updates, deletions and permission revocations as well as ingestion. A cached answer must never restore access to a source the user can no longer read.

The most important habit: measure retrieval separately from generation. If you only measure end-to-end, you cannot tell whether to fix the retriever or the prompt, and you can lose a week rewording a prompt when the document was never fetched. Label relevant source spans and measure recall at k, ordering and latency. Count source families, not duplicate chunks, as successes. Then score answer correctness, whether cited passages support claims, and sensible abstention on missing or conflicting evidence. A citation's presence alone is not support. Twenty query families are a debugging exercise, not a precise production quality estimate.

Extracting fields from documents. Appeals, invoices and forms often arrive as PDFs or scans. Extracted text (a PDF's text layer, or OCR for scans) is cheap to send and easy to inspect, but can lose tables, checkboxes, layout and handwriting. A vision-capable model reading page images keeps layout, has its own cost and can still misread. Compare both on the same labelled pages. Return each field with its evidence (page number and source span or region), or an explicit "not present" instead of a guess, so a reviewer can check it and code can reject unsupported values. Split the labelled set by document. Normalise dates, amounts and IDs, then report per-field accuracy, whole-record correctness and invalid outputs, sliced by scanned, rotated and multi-page documents. Check formats and cross-field rules in application code.

Evaluation as a development loop. Write a small task specification first, then improve it using real development traces. A practical order is:

1. Write representative cases and acceptance criteria, and reserve independent release cases.
2. Build the smallest complete version and collect development traces.
3. Read failures, write free-text notes, and group recurring failure modes.
4. Add deterministic assertions where outputs or final state can be checked directly.
5. Add a judge only for criteria needing interpretation. Ask it for evidence and a rubric decision.
6. Validate the judge on separately labelled examples before relying on its measurements.
7. Put regression cases in CI. Fix one failure mode and compare on development cases.
8. Freeze the candidate and decision thresholds, then run the independent release evaluation.

Keep test answers out of development. Split by source document, customer/session, time or question family as appropriate, then check duplicates and paraphrases across splits. Split before generating synthetic variants. A RAG question may rightly refer to a readable source, but its expected answer and grading notes must stay hidden from the application and its tools. Prompt-tuned cases are development cases. A CI regression set catches known failures, but repeated optimisation against it rules out an independent release claim. Periodically collect a fresh, independently labelled holdout.

Web agents can find leaked benchmark answers during evaluation, even ones absent from training. Anthropic's March 2026 BrowseComp investigation documents leaked answers and answer-key searches by an evaluation-aware model in its tested setup. Audit provenance, isolate grading assets and use fresh private cases. Public benchmark scores are supporting evidence, not your release test.

Use binary criteria for binary requirements, and a defined graded rubric when quality has degrees. Check human-rater agreement. Choose a viewer that makes inputs, sources, calls and outputs easy to inspect. Read the task, trial and grader sections of Demystifying evals for AI agents (January 2026). Its examples describe Anthropic's experience, not universal guarantees.

Where practical, blind reviewers to candidate identity. Test answer-order and verbosity sensitivity. Overall agreement can hide a judge that passes nearly everything. If failure is the positive label, failure sensitivity is TP/(TP+FN) and false-alarm rate is FP/(FP+TN). Missing labels and grader errors are explicit outcomes, not passes. Reset files, databases and tools between independent agent trials, and disclose retry/attempt budgets. Report sample size, uncertainty and meaningful slices. Respect task/session grouping instead of treating correlated trials as new users. Towards a Science of AI Agent Reliability (ICML 2026) organises reliability around consistency, robustness, predictability and safety. One average success score is insufficient for an action-capable system.

Agents and workflows. When the steps are known, start with a fixed workflow. Give an agent discretion where its next step depends on what it observes and you can verify progress and constrain effects. Weigh that flexibility against extra calls and failure modes.

Build one small loop or state machine before choosing a framework. No line count makes it production-ready. Make control flow explicit: allowed tools, argument validation, deadline, token/cost/step limits, bounded retries, cancellation, checkpoints and stop/handoff conditions. Use idempotency keys and check the real service state before retrying writes. A timeout does not tell you whether a remote action happened. Multi-agent designs also need bounded fan-out, shared-state ownership and a way to reconcile findings. Compare with a single-worker baseline.

For a write-capable extension, save the validated action and its status before dispatch, and reuse the same business idempotency key on retries. Tie any user confirmation to the exact action, arguments, recipient and time window. If those change, ask again where the product requires confirmation. Recheck authorisation at execution, because a saved approval is not a permanent access grant. Chapter 6 covers durable delivery and recovery.

Test recovery from tool failure and context compaction. The September 2026 Mid-Harness preprint studies sampling and verifying candidate actions before running one, in terminal-agent benchmarks. After a bounded baseline, optionally compare action verification with extra generation at the same total budget. Its results do not prove a verifier reliable for your actions, and application authorisation still decides whether an action may run.

Access and tool boundaries belong in the first product. Authenticate users when documents or actions are private, and apply their document permissions before retrieval. Treat retrieved text and tool output as untrusted input. Validate tool arguments and enforce permissions in application code. The model does not grant its own access. Add timeouts, bounded retries, idempotency for writes and a confirmation step for consequential actions. Redact secrets and personal data from traces. Test cross-user retrieval and documents that try to override instructions. The OWASP prompt-injection reference covers the threat and layered mitigations, and the AgentDojo benchmark gives tool-agent attack and defence examples. Use harmless canary data for exfiltration tests. Where relevant, use narrowly scoped service credentials, network/file allowlists and isolated execution. Keep grading secrets and other users' data outside the tool environment, recheck access at execution, and set a trace-retention period. A public, read-only demo can keep controls small because its data and permitted effects are small.

Check outputs as well as inputs. Injection defences guard what enters the model. Output guardrails check what leaves it, before a user or tool receives it. Validate schemas and citations. Block or reroute answers exposing secrets, personal data or another user's documents. Add a moderation or policy classifier when the product has a content policy. A guardrail is another classifier, with misses and false blocks. Measure both on labelled examples, including the injection cases above, and keep application permissions as the enforcement layer.

Check the evaluation boundary itself: use fake credentials and harmless services, test outbound access restrictions, and keep production secrets outside the harness. Anthropic's August 2026 incident account reports misconfigured evaluation environments that allowed unauthorised access. This vendor incident report concerns deliberately less-protected cyber evaluations and does not estimate your application's failure rate. Test safe stopping when a task or tool cannot be completed.

MCP, the Model Context Protocol, is an open standard connecting clients to tools and data. Learn the specification revision your client and server implement. The 2026-07-28 changelog introduces a stateless core without the earlier initialisation handshake or protocol sessions, and adds multi-round-trip requests. Stateful business operations use explicit handles whose access the server must enforce. Roots, sampling and logging are deprecated but still work during their deprecation window. Earlier-version tutorials can still help, but test compatibility rather than assuming it.

Learn request/version metadata, discovery, schemas, results/errors and cancellation. Transport retries are not exactly-once business transactions. For remote servers, validate token audience, avoid token passthrough, protect OAuth redirects and restrict outbound fetches against SSRF. Follow the official MCP security practices. Installing a server does not grant access to every connected user.

Agent skills are a smaller idea worth knowing next to MCP. A skill is a folder holding a SKILL.md file of instructions, plus optional scripts and reference files. The agent reads only each skill's name and description at first, and loads the rest when a task calls for it. That is context engineering in a reusable form. The format began at Anthropic, is now an open standard, and coding agents from several vendors support it. Writing one skill for a repeated task lets you compare loading on demand with inline instructions. Read its scripts and references, and enforce normal tool permissions. Loading a skill does not raise its trust.

Fine-tuning and RL when the error warrants them. You do not have to add RAG before training. Match the fix to the failure and available evidence.

The DeepSeek-R1 paper studies reasoning post-training and distillation. It does not prescribe RL for a document assistant. The official TRL GRPO documentation covers reward and training controls for an optional experiment. Validate reward against human or task outcomes. A rising reward alone does not show better quality.

Split by document/customer/task family and time where appropriate, and deduplicate before training. Keep teacher-generated examples independent of evaluation labels. Check synthetic labels, record provenance and permitted data use, and compare at the same deployment budget. Quantised adapter tuning reduces some memory costs, but sequence length, batches, optimiser state and serving context also matter. If training adds no useful benefit, keep the baseline and report that result.

Resources

All 49 resources, with notes and a targeted research section, are in resources/. The chapter's 100 hours do not include study time for optional resources. Start with selected lessons and sections:

• Deep Dive into LLMs like ChatGPT — Andrej Karpathy. A free video, 4 hours. Watch it in week 23 for the map.
• Building with the Claude API (Claude Academy) — Anthropic education team. A free course, 9 hours.
• Your AI Product Needs Evals — Hamel Husain. A free article, 1 hour. Then the Evals FAQ by Hamel Husain and Shreya Shankar.
• Building Effective Agents — Erik Schluntz & Barry Zhang. A free article, 1 hour.
• Demystifying evals for AI agents — Anthropic. Task, trial and grader sections for week 26.

What to build

Warm up in about three hours. Put a pretrained model or read-only API function behind a URL, using non-sensitive data. Measure one request and one failure, and reuse this deployment path.

Next, the evaluation-first application. Pick a narrow task you care about and can judge, ideally on documents from your Chapter 2 domain, such as extracting fields from its appeals or forms. If the domain has no suitable documents, triage your own issue tracker or draft first-pass replies. Follow the evaluation loop above. Skip retrieval when the input already holds everything needed.

The result is a repository whose evaluation suite predates optimisation, with evaluation history, failure examples, the development/release split and a measured operating-cost table.

Then re-scope it into a small product with a public URL, following the two-week plan below.

Your first AI product: a two-week plan

One user, one job, one screen, one main quality metric, plus cost, latency and access checks. Spread about 20 hours across these sessions, reusing weeks 23–26 work. Cut down any task needing complex integration or a custom authentication system before this sprint.

Write the non-goals on day one. No payments, autonomous writes, custom vector store, mobile app or model training. Use public/non-sensitive data, or platform authentication with verified per-user access for a private application. Optional agents and tuning experiments come later, in weeks 29–31.

Use deterministic checks for objective tasks. A model judge is optional. Run some repeated trials for stochastic or tool-heavy cases within budget, and disclose the count. If a release run fails, investigate on development data rather than rerunning it until it happens to pass. If your changes were tuned to the old holdout, collect fresh release cases.

The Chapter 5 minimum is explicit: a read-only, narrowly scoped interface, a repeatable local regression command, independent quality evidence, known failures and limits on any paid requests. Use synthetic/public inputs, or verified platform authentication for private ones. A static demo with recorded or deterministic responses is a valid first deployment if it is labelled accurately and does not claim to call a live model. If you add a paid API, keep its credential on the server, validate requests, cap input and output, enforce request and spending limits, and fail safely. These controls apply now, before the later hardening chapter.

Chapter 6 takes this exact application as its starting point. Its container and CI work is not needed to complete Chapter 5. Check current hosting eligibility and charges before choosing a route. Hugging Face Spaces still offers free Static Spaces. Creating Gradio or Docker Spaces has account-plan requirements, with some ZeroGPU exceptions. Free CPU hardware and permission to create a Space are separate questions. Modal pricing separates compute credits from Shared Endpoints, whose per-token charges those credits do not cover. Neither route guarantees free model API calls or a free running demo.

Completion means a usable URL or documented local-demo fallback, a candid independent quality report, a repeatable local regression command, the access and spending controls that deployment needs, known failures, a cost table and feedback from at least one other user. A measured lack of improvement is a valid result. Release claims need independent evidence.

After release, model/SDK upgrades, prompt changes and corpus refreshes are separate versioned changes. Recheck retrieval coverage, access filtering, task quality and cost before rollout. Keep the previous compatible configuration and index for rollback, subject to current permissions and deletions. A rollback must never restore revoked access or withdrawn sources. Review sampled real failures and user corrections under your trace-retention policy. Chapter 6 develops this operating loop. If the sprint's release results shape later changes, collect fresh release cases for week 32 and keep the earlier ones for regression testing.

Before you move on

• You can explain why error analysis comes before evaluation tooling.
• You can validate deterministic graders. If you use a model judge, you can show its confusion matrix, failure sensitivity and false-alarm rate.
• You can tell CI regression cases apart from an independent release evaluation.
• You can show a denied tool action, timeout and tested fallback.
• You can state cost per attempt/success and the limits of the quality estimate.

A few things worth knowing

• Tutorial code ages. Check repository status, dependencies, available models and endpoint versions before reproducing it. Keep the lesson and update the code details.
• Frameworks are a choice. They can help with persistence, tracing and integration. A small explicit workflow can be easier to inspect. Compare what your application actually needs.
• Model names age quickly. Everything specific in this chapter will be superseded. Tokenisation, attention, cache economics, retrieval, evaluation and context management will not, so that is where the study time is best spent.

Previous: Deep learning  ·  Next: Running it in production

Weeks 33–37  ·  about 55 hours.

Book index  ·  Chapter resources

The short version. Your Chapter 5 product becomes a system: containerised, tested on every push, deployed, watched, and cheap enough to defend in a budget conversation. Little here is glamorous, but it separates a demo from something a team would trust.

From this week, Chapter 8: Finding the work runs alongside this chapter and the next. Open it now. The search it describes takes longer to pay off than anything left to build.

In this chapter: Plan  ·  Learn  ·  Build  ·  Completion  ·  Resources

Your five-week plan

About eleven hours a week. You are finishing the Chapter 5 app, not starting a new one. Bring the Chapter 3 model along. Its saved bundle and API go through the same container build, CI, release manifest, monitoring and rollback in these weeks. The retraining, decision-tracing and stale-feature checks at the end of the chapter use that classical model. If nothing needs a live decision, a scheduled batch-scoring job can replace its API.

What you will be able to do

Containerise, deploy, monitor, roll back, and say what your system costs per thousand requests.

Most of this will not set you apart. People notice its absence and assume its presence. That is why it gets five weeks, not twelve. The exception is cost engineering, which very few people can discuss with numbers from a system they built themselves.

What to learn

Worked lesson: a timeout is an unknown outcome

Your service asks a destination to issue a voucher with operation ID retention-42. The destination issues it, but the acknowledgement is lost. Retrying with a new ID can issue another voucher. A local timeout is a fact about the response, not proof that the remote action failed.

This tiny fake destination shows a business idempotency contract:

issued = {}

def issue_voucher(operation_id, amount):
    if operation_id in issued:
        if issued[operation_id]["amount"] != amount:
            raise ValueError("same operation ID with different arguments")
        return issued[operation_id]
    result = {"voucher_id": f"voucher-{len(issued) + 1}", "amount": amount}
    issued[operation_id] = result
    return result

first = issue_voucher("retention-42", 10)
retry = issue_voucher("retention-42", 10)
assert first == retry and len(issued) == 1

The dictionary is for teaching only. Real delivery needs durable state, atomic handling of concurrent requests, retained deduplication keys, and the destination's documented idempotency and reconciliation behaviour. A transactional outbox closes the local gap between decision and dispatch. On its own, it cannot deduplicate a remote service's effects.

Try it yourself. The worker crashes after the voucher is issued but before its local delivery row is marked complete. What should it save before dispatch? Which key should it reuse? What should happen if the destination cannot report or deduplicate the operation?

Core operations and optional systems depth

Finish one deploy, regression command, monitored failure and rollback before trying distributed serving or framework optimisations. The reference lab gives a deterministic starting point. Production claims still need measurements from your own system.

Containers, and about forty per cent of Docker. Images and layers, why your build is slow, multi-stage builds, .dockerignore, sensible Dockerfile hygiene for Python, volumes, environment variables and secrets, and Compose for local work. Keep a small CPU/API application lean, and check large layers for unused dependencies. GPU runtimes and model weights can genuinely need larger images. Judge size alongside build time, cold start and deployment limits.

Choose one deployment environment you can run within your budget. A container service or serverless API is enough for this chapter. If hosting is unavailable, a local container with a recorded load test is a fair fallback. Learn Kubernetes when a target role or real requirement calls for it. A first release does not need a cluster.

Continuous integration that means something. On every push, run your tests, your data tests (schema assertions, range checks, point-in-time and label-maturity checks, group separation when the evaluation needs unseen entities) and evaluation assertions. Put deterministic checks such as output schemas, permissions and forbidden tool actions in the fast suite. Run a bounded quality suite on pull requests and a larger one before release. Fix the dataset and scorer versions, and compare against the current release on the same cases. Repeat stochastic runs, and report uncertainty and important slices. Set the smallest useful quality difference and the release rule before seeing the result. A noisy one-point judge wobble should not pass for a real regression. MLflow's regression-testing workflow shows how to record tests and traces. The scoring design is still your job.

Keep workflow permissions minimal. Checks get read-only repository access, and only the deployment job gets deployment access. Run untrusted pull-request code without production secrets. Do not use a privileged workflow trigger to check out and run a contributor's branch. Pin third-party actions to reviewed commit SHAs, lock dependencies, and load model artifacts only from trusted, recorded sources. A passing test suite cannot make an untrusted build artifact safe.

Loading an artifact can execute it. Pickle-based files, including joblib bundles and full PyTorch checkpoints, can run arbitrary code on load. Check the recorded hash first, and never unpickle an untrusted file. For weights, prefer safetensors, or torch.load with weights_only=True, the default since PyTorch 2.6 (serialisation notes). That narrows the risk but does not remove it. scikit-learn's persistence guide compares safer formats.

Version prompts, retrieval settings, tool schemas, decoding settings and model identifiers with the application. Keep a held-out evaluation set for release decisions. Repeated tuning on the same public benchmark or failure queue eventually fits that evaluation. Check automated judges against a blinded human-labelled sample, track disagreement by failure type, and recalibrate when the judge or task changes. Upgrading the judge's model is an evaluation change too. Once release cases shape fixes or thresholds, treat them as development data and collect fresh, independent release cases. A private holdout can be overfit just like a public benchmark.

A rollback runbook. Write it, then rehearse it. Define alert owners, the compatible previous release, a fallback, and the signal that shows recovery worked. Fill the placeholders below from a load test and the product's error tolerance, including minimum sample sizes for quality signals. An immediate security or action-execution failure can need a different response from a slowly arriving statistical quality signal.

# Rollback runbook — [service name]

## When to use this

- Alert: [quality measure] crosses [declared bound] with [minimum sample/window]
- Alert: p95 latency above [service objective] for [window] at [minimum traffic]
- Immediate stop: [unauthorized action / invalid price / duplicate delivery condition]
- Manual: user reports systematic wrong answers

## Roll back application

1. Redeploy previous immutable image: `[registry]/app:[last-good-tag-or-digest]`
2. Confirm readiness and replay [recorded smoke cases]; verify feature and output schemas

## Roll back decision release

1. Select [previous release manifest]: compatible model, features, calibrator and policy
2. Load and validate the complete release; route requests after its readiness check passes
3. Replay [recorded decisions]; preserve current access/revocations, constraints and freshness checks
4. Confirm fallback and idempotency before resuming actions

## Kill switch

- Disable new model-driven actions with [delivery feature flag]; pause action workers/outbox consumers
- Reconcile in-flight actions before retrying or resuming; a timeout does not prove nondelivery
- For read-only LLM responses, disable `LLM_ENABLED` or route to the approved static fallback

## Who to notify

- [Your name / on-call channel]

## After rollback

- Open incident doc; attach traces from first failing request

Serving. FastAPI is a reasonable API starting point. Daily business decisions can use a batch job. For self-hosted language models, compare a supported engine such as vLLM or SGLang on your workload and hardware. Learn how KV-cache allocation, continuous batching and queueing affect memory, throughput and latency. Record arrival rate, concurrency, input/output lengths, time to first token, time per output token, p95 end-to-end latency, failures and cost per successful task. Report the load at which the service still meets its objectives, not peak tokens per second at unusable latency. vLLM's serving benchmark gives a starting harness. Include the warmup and runtime configuration in your report.

Separate process liveness from readiness to serve a compatible, loaded release. Load artifacts and required resources before accepting traffic. FastAPI's lifespan guide shows model startup and cleanup. Bound queues and request deadlines under overload, and test that the fallback answers before the caller's deadline. Retries on an overloaded service can add to its load, so count them in both capacity and cost measurements.

A hosted model is a dependency that can fail. Set a client timeout inside the caller's deadline. Retry only safe calls, with bounded backoff, and respect rate-limit responses. Decide what users see during a provider outage: a cached or static answer, a queued job, or a clear "unavailable" message. Rehearse it by blocking the provider endpoint, and record what the user actually saw.

Optimisation is conditional. Prefix caching needs reusable prefixes. Quantisation needs supported kernels and a quality check. Speculative decoding adds draft work and depends on the workload. The current vLLM speculative-decoding guide targets memory-bound, medium-to-low request-rate workloads. Test one change under your load before adopting it. Use distributed serving only when the simpler deployment cannot meet the declared requirement.

Monitoring. For classical models, watch input distributions, prediction distributions and drift, knowing that labels often arrive weeks late. For language-model applications, watch traces, tool-call failure rates, evaluation pass rate on a live sample, cost and latency. Check upstream freshness, schema changes and join coverage early, because a broken pipeline can look like model drift. An input distribution alert shows a change, not its effect on prediction quality. Tie quality alerts to outcomes once those outcomes are available.

An agent also needs repeated-task consistency, sensitivity to small input changes, correct escalation, and checks for unauthorised actions. The peer-reviewed ICML 2026 paper Towards a Science of AI Agent Reliability separates reliability from average task accuracy. Use that idea to build tests for your own permissions and failure costs. A benchmark score is not a production guarantee.

For business ML, join outcomes to the original decision_id and report only mature cohorts. Track feature freshness, missingness, calibration, precision at contact capacity, demand errors, margin, budget use, opt-outs and action delivery. Compare the policy against its control group. A falling input-drift score does not show rising business value. See Chapter 9's delivery plan for the full feedback path.

Alert on things that mean act now, and keep the list short. An alert nobody acts on trains everyone to ignore the channel.

Reproducibility. Record code, immutable data/split versions, configuration, dependencies, hardware/runtime, random state and checkpoints. A commit hash cannot rebuild a changing customers table, a deleted API response or an unversioned prompt. Include query and snapshot cutoffs, source licences and the commands needed to recover artifacts. For an API model, save the provider model identifier, settings and recorded evaluation outputs. A changing hosted model may not allow exact retraining or replay.

Be precise about which level you claim. Rerunnable means the command works today. Reproducible within a declared tolerance means reruns land in a stated metric range on the same evaluation. Report the runs that set that range. Bit-identical needs tighter control, and even after seeding it is not guaranteed across platforms or releases. PyTorch reproducibility notes.

The classical ML lifecycle. Schedule extraction, snapshot construction, training, evaluation and batch scoring as explicit jobs with dependency checks and retries. Save preprocessing with the model and compare batch and online features on recorded requests. Each release needs data/query versions, training cutoff, feature schema, metrics and an immutable artifact identifier. A simple manifest works at first. A registry adds managed versions and aliases when several models need them (MLflow's registry workflow).

Pin one complete release manifest to each request or batch, covering preprocessing, model, calibrator, decision policy and any retrieval index. Prepare and validate the next release before routing traffic to it. Workers should never mix separately refreshed components within one decision. After retraining or changing score meaning, recheck calibration and decision thresholds on data separate from model fitting before promotion. Rolling back artifacts still respects current permissions, revocations, costs and operational constraints.

Serve risk scores in batch if a team acts once a day. Use an API where a session really needs a decision. Either way, check authentication, request limits, fresh features and graceful fallback. Shadow a new policy, check its outputs, then canary it with explicit stop conditions. On an alert, inspect data and label freshness before retraining. Evaluate a challenger on recent mature cohorts and the preserved benchmark. Promote only if the declared quality, business and latency gates pass. Recalibration, data repair or rollback may be the right fix instead of a new model.

Try it yourself. Release v8 of your Chapter 3 model went live at 09:00. At 11:00 an alert fires: the share of requests above the decision threshold has doubled. The freshness check shows the activity table behind its recency features last loaded at 02:00 yesterday. Outcome labels will not mature for 30 days. Do you roll back to v7, retrain, or do something else?

Feedback is a data product. Log eligibility, the chosen action, policy version, execution status and, when randomised, the assignment probability. Then reconcile outcomes and label availability. Keep training examples and evaluation cohorts separate as they mature. User ratings and model judgements are useful diagnostic signals. But selected feedback is not an unbiased estimate of product performance. Sending a retention offer or changing a price also changes later training data. Keep a suitable control or exploration design to measure incremental effects. For LLM applications, sample successful as well as failed traces, redact sensitive content, have a person check proposed labels, and route approved cases into a versioned development set.

Keep access, retention and deletion rules with the trace pipeline. They need to reach every copy: traces, evaluation cases drawn from them, training snapshots and derived artifacts. Record where personal data flows and how long each store keeps it. Then rehearse one erasure request end to end, including whether a model trained on the erased records needs retraining.

Durable actions, when the application can write. Commit the validated decision and a pending delivery record together before dispatch. A transactional outbox is one way to close the gap between database write and message send. The AWS design guide also requires handling duplicate delivery. Reuse the business operation ID, check the destination's idempotency contract, and reconcile unknown outcomes after a timeout. If a destination cannot deduplicate or report its state, stop and reconcile instead of promising exactly-once effects. Test crashes before dispatch and after the remote effect but before its acknowledgement. This extension is for action-capable systems. The Chapter 5 read-only sprint does not need an action worker.

Cost. Take one working feature and build a real table. Start with baseline cost per thousand requests, then add the effect of restructuring the prompt so the static prefix caches, routing easy requests to a cheaper model, and moving latency-tolerant work to a batch endpoint. Report the evaluation pass rate beside each variant, because the classic trap is a cost cut that quietly lowers quality.

Fill in your own numbers from provider logs. Treat any figure in a tutorial, including this table, as a placeholder until you measure.

Keep provider details in the experiment configuration. OpenAI's prompt-caching guide describes exact-prefix reuse. Minimums, retention and read/write pricing depend on model generation and settings, so do not budget with a universal discount. Its Batch API currently offers a 50% discount for supported work with a 24-hour completion window. A batch can expire with partial results. Before retrying unfinished work, reconcile successes and request-level errors by custom_id, not output order. Measure turnaround instead of promising interactive latency. When comparing alternatives, count retries, tool calls, judge usage, storage and idle compute.

For self-hosting, the answer is arithmetic, not preference. Compare monthly API cost with GPU hours (including idle time) plus the engineering hours to run it. Then find the request volume where the lines cross. Knowing how to do that sum matters more than which side you land on.

Resources

Annotated resources are in resources/. For the 55-hour plan, use the selected sections. Full courses and deeper distributed-systems material are optional extras.

The ones to begin with:

• Docker documentation — Get started, and the Python guide — Docker documentation team. Free documentation, 4–6 hours. Week 33.
• FastAPI official documentation and tutorial — Sebastián Ramírez. Free documentation, 2–4 hours of selected API/testing work. Use your existing endpoint.
• GitHub Actions documentation — GitHub documentation team. Free documentation, 3–5 hours of selected Python CI work. Week 34.
• Prompt caching guide — OpenAI developer documentation. Free, 1–2 hours. Read your own provider's equivalent too. Week 36.

What to build

Take the application from Chapter 5 and finish it properly: a multi-stage Dockerfile, CI running tests and evaluations, deployment within a declared budget, redacted structured request logs, a small dashboard, two alerts you would actually act on, a rollback runbook, a cost table, and a model card. A model card is a one-page note on what the system is for, what it was evaluated on, and what it should not be used for.

Deploy the Chapter 3 model beside it, as an API or a batch-scoring job. Give it a release manifest (data/query versions, training cutoff, feature schema, metrics and artifact hash) and a decision log that joins each decision_id to its features, release and, once mature, its outcome. Add a freshness check that switches to the documented fallback when inputs are stale. The reference lab's release manifest is a filled-in, simulated example.

Then break it on purpose and write a short incident report: what you injected, how long detection took, what fired, how you rolled back, and what you would change. Set severity by user impact, such as wrong decisions served versus a late dashboard. Write it blamelessly. Name the conditions and missing safeguards that let the failure through, not a person to blame. The lab's incident rehearsal is a filled-in example. Your own report is worth more than another model in a portfolio, because very few people have one.

Before you move on

• You can explain each Docker build stage and fix a build with a missing runtime dependency (documentation allowed). Show that the image starts and its readiness check catches a missing or incompatible model bundle.
• You can rerun training from the release manifest within a declared metric tolerance, or explain the limits of replaying a hosted model.
• You can show your dashboard and justify each alert.
• You can read your incident report and your cost table out loud.
• You can trace a decision to its features, model and eventual outcome, and rehearse a fallback on stale data.

A few things worth knowing

• Deploy on day one of the week, not day five. The first deploy nearly always turns up something: a missing system library, a secret that was only in your shell, a port that is not exposed. Finding that on Monday leaves the week for fixing it.
• Inspect image size. docker history shows large layers. Separate required GPU runtimes and weights from unused build tools, caches or training data. Measure cold start after changing the image, since a smaller image does not always fix latency.
• Secrets never go in the image. Pass them in at run time as environment variables or from the platform's secret store. Keep a .env.example with the names but no values, and check that .dockerignore and .gitignore both exclude the real file.
• Document applicable obligations with an owner. Scope depends on the intended use, where the system is used, and your organisation's provider or deployer role. For an EU-facing release, check the Commission's current AI Act guidance and the applicable enacted text when assessing classification, transparency and dates. Record the assessment and open questions. A model card alone does not show compliance. This book does not treat every application as high risk, or a proposed timeline change as enacted law. Data-protection law can apply as well, for example the EU's GDPR or India's Digital Personal Data Protection Act, 2023, depending on whose data you process and where. Record that assessment the same way. This book is not legal advice.

Previous: Language models and AI engineering  ·  Next: Choosing a specialisation

Weeks 38–43  ·  about 70 hours.

Book index  ·  Chapter resources

The short version. The shared path ends here. Pick one of three roles: AI Engineer, ML Engineer or Data Scientist. Spend about forty hours going deeper and thirty on a capstone matched to that role's interviews. One branch done well beats three sampled.

Extend the same thread. Build your capstone on the dataset, model or LLM app from Chapters 2–6, not a new tutorial project. One connected story beats three unrelated repos.

In this chapter: AI Engineer  ·  ML Engineer  ·  Data Scientist  ·  Completion  ·  Resources

Plan about 40 hours of depth and 30 on the capstone, and budget the capstone first. It is the point of this chapter, so fit the depth work around it.

Do one branch. The shared path you have finished keeps the other two open for later.

You do not need to master every topic below. Think in three levels of depth:

These hours assume the project and environment from earlier chapters. Record your actual time, and shrink the capstone if integration or getting data eats the budget. The goal is one finished system, not familiarity with every research direction listed.

Chapter 8: Finding the work should already be running alongside this chapter. If it is not, start it today.

For offer, pricing or churn work, use Chapter 9 as your applied track. Its shared delivery work plus one guide is about 40 hours. Use the 30 capstone hours here for a small prototype, then about ten more for the remaining shared checks. ML Engineers should stress data, serving and feedback. Data Scientists should stress identification, experiment design and the decision memo. Both need the basic experiment checks before recommending an intervention.

What you will be able to do

Go deep on one role, finish a capstone that matches its interviews, and explain in writing why you did not sample the other two.

Branch A — AI Engineer

Go deeper on four things.

• Agent harnesses. State, tool contracts, durable execution, limited retries and verification. Add sub-agents or memory when a measured task needs them. Make external actions idempotent (safe to repeat), and put human review at the decisions that need it.
• Evaluation as a speciality. Judge the whole trajectory as well as the final answer. Check tool-call correctness and multi-turn behaviour, and keep a judge calibrated over time.
• Reliability and safety. Prompt injection (hostile instructions hidden in content the model reads), privilege boundaries, escalation, rate limits and per-user cost caps. Test allowed and forbidden actions together. A system that refuses every task avoids execution mistakes but fails the product goal.
• The product layer. Design for a component that is sometimes wrong, and capture user feedback that feeds your evaluation set.

For depth, apply the strongest Chapter 5 and Chapter 6 resources to your capstone, within the depth budget. Read the Anthropic engineering blog's agent sequence end to end, then 12-Factor Agents. Apply Husain and Shankar's evals material to your own traces. Add the dated research and official security guidance from this chapter's resource list. Recommending these sources is not a claim that any agent framework guarantees reliability.

The ICML 2026 paper Towards a Science of AI Agent Reliability organises reliability around consistency, robustness, predictability and safety. Its arXiv version was submitted 18 February and revised 2 June 2026. For your capstone, repeat typical tasks, perturb inputs, test escalation and permissions, and measure recovery after tool failures. Report completion, action errors, latency and cost together. Record harness, model and evaluator versions, because the same final answer can hide a very different sequence of tool actions. Use the OWASP LLM Top 10 2026, published 3 August 2026, to choose relevant attack cases. A risk checklist alone does not show that your defence works.

Capstone, pick one:

• A reproducible evaluation of one important failure mode.
• An agentic system with trajectory evaluation and a small user pilot.
• An open-source tool or contribution, with a reproduced problem and documented validation. Start from a bug or gap you hit in a library your Chapter 5–6 app uses, and show the change working in that app.

Keep the work to 30 hours. Record real pilot feedback when available. Adoption, novelty and a maintainer's merge decision are separate outcomes.

Interview preparation: practise an AI systems design discussion, then confirm each employer's rounds and tool rules using Chapter 8. Good practice prompts: a triage system for ten thousand tickets a day, showing it works, halving its cost, recovering from a high error rate, and whether the task needs an agent. Use your artifacts to support the tradeoffs. These prompts are practice. Real formats vary.

A simple six-week plan (about 12 hours a week):

Branch B — ML Engineer

Go deeper on four things.

• The data layer. Orchestration, warehouse modelling, data quality tests, and point-in-time correctness at scale.
• Training at scale. Data and model parallelism, mixed precision, gradient accumulation and checkpointing, profiling, and scaling laws.
• Serving and inference optimisation. Quantisation stores numbers at lower precision and can save memory. Any speed-up depends on the supported kernels and the workload, so compare quality and latency before adopting it. Also learn distillation, compilation, batching strategies and management of the KV cache, the memory a model keeps of tokens it has already processed.
• Post-training. Supervised fine-tuning, preference optimisation, and reinforcement learning from verifiable rewards.

Before spending the 70 hours, choose a production path or a language-model implementation path. The default production path below spends about 40 hours on selected systems readings and 30 on one capstone, with no from-scratch language model.

For the alternative, CS336 Spring 2026 has public materials and five assignments covering basics, systems, scaling, data and alignment/reasoning. It assumes strong programming, deep learning and maths. Plan about ten hours of selected lectures, fifty for Assignment 1, and ten for tests, profiling and documentation. That assignment replaces the production capstone and its depth work. It also steps off the project thread, which is fine when model implementation is your target role. Your Chapter 2–6 project stays your portfolio's production story. These are planning estimates, so allow more time if correctness needs it. The full graduate course is beyond this chapter's budget. See Branch B — ML Engineer depth in resources/.

For efficient training, use the official FSDP2, AMP and compiler guides for your pinned framework version. Explain the memory and communication tradeoff of sharding before renting several GPUs. Read the MLSys 2026 FlashAttention-4 paper on hardware-aware attention kernels, alongside its March arXiv version. Its results are hardware-specific, so test them on your own workload. For adaptation, compare a no-training baseline with one well-chosen SFT or adapter run. Keep demonstrations, preference data and verifiable rewards separate, hold out a task evaluation, and include memory and serving cost. A good fine-tuning loss does not show that the product task improved.

Capstone, pick one:

• An end-to-end training and serving system with a documented retraining trigger. Keep the model simple on purpose, since the system is what is assessed.
• A paper reproduction with a frank account of where your numbers differ. Pick a method that applies to your Chapter 3–4 model, and finish by running it on your own dataset.
• One from-scratch language-model stage, done to course standard.

Interview preparation: rehearse a 45-minute machine learning systems design discussion covering data, training, evaluation, serving and monitoring. The length is for practice. Formats vary, and some employers also test coding or statistics. Explain a simple starting design and what results would justify more complexity, then adapt to the real round.

A six-week production plan (about 12 hours a week). If the implementation path is your capstone, use the CS336 hours above instead:

Branch C — Data Scientist

Go deeper on four things.

• Experimentation. Estimands, randomisation units, power and minimum detectable effect before you run, sample-ratio mismatch, multiple comparisons, novelty and carryover, interference in marketplaces, variance reduction and guardrail metrics. Repeated observations from one user are not independent experimental units.
• Causal inference. Potential outcomes, confounders and colliders, difference-in-differences, instrumental variables, regression discontinuity, matching, and sensitivity analysis.
• The classical statistics the book deliberately left for later.
• Communication. Turn a metric, its uncertainty and its limitations into a clear decision memo.

In applied work, separate prediction from treatment effects before choosing a causal method. Chapter 9 adds primary research, identification assumptions and a shared experiment protocol for the three business cases. The annotated list is in resources/ under Branch C — Data Scientist depth: Kohavi, Tang and Xu on trustworthy online experiments (companion site at experimentguide.com), and two books free online, Cunningham's Causal Inference: The Remix and Huntington-Klein's The Effect.

For each experiment, write down in advance who is eligible, the unit randomised, treatment and control, assignment logging, the outcome window, the intended estimand and the stopping rule. Analyse by assignment for an intent-to-treat effect. Dropping failed deliveries or non-responders after assignment can break the comparison. Reconcile assignment, exposure and outcomes, and look into any sample-ratio mismatch before reading a lift. Match the uncertainty to clustered or repeated units. Use a fixed analysis horizon or a well-founded sequential method. Checking ordinary p-values repeatedly and stopping when one crosses 0.05 changes the error rate.

Include one variance-reduction experiment in the depth work. Compare unadjusted estimates with CUPED, using only covariates the treatment cannot affect. Then run simulated A/A tests to check interval coverage. Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED, a preprint submitted 17 June 2026, examines choices of adjustment and variance estimation, including multi-arm and two-stage sampling. It is recent method research, not a promise that any adjustment makes every business experiment more precise.

For observational work, state the identification argument before choosing an estimator. Difference-in-differences needs a credible comparison and trend assumptions. Instruments need relevance and exclusion. Regression discontinuity needs a cutoff design you can defend. An uplift model, a doubly robust learner or a flexible predictor cannot supply missing randomisation, overlap or control of unmeasured confounding. Use sensitivity checks that suit the design, and write down what remains unidentifiable. Chapter 9 connects these choices to offer, price and retention actions.

Capstone, pick one:

• An experiment analysis with a well-supported causal claim and sensitivity checks.
• A metric system for a real product, with a decision memo.
• A data engineering project on your Chapter 2 sources and snapshots, showing validated SQL, lineage and reliable metric delivery.

Choose the one closest to your target role's actual work.

If you have no randomised data of your own, run the reference lab. It writes a simulated experiment to data/assignments.csv and data/experiment_outcomes.csv in its output directory, with assignment probabilities and failed deliveries kept in their assigned arm. Practise the intent-to-treat analysis, sample-ratio check and uncertainty there, and label the result as simulated.

Interview preparation: practise live SQL, statistics, product reasoning and experiment design, putting the target employer's actual rounds first. Pair a dashboard with a memo that links its metrics, uncertainty and limitations to a decision.

A simple six-week plan (about 12 hours a week):

Resources

The annotated resources are in resources/, grouped by branch. Hours for full books and courses are alternatives, not additions to the 70-hour plan.

The ones to begin with depend on the branch you chose:

• Branch A. The relevant sections of the "Start here" lists in Chapter 5 and Chapter 6, then 12-Factor Agents by Dex Horthy (free, 3–5 hours). For the security week, the OWASP Top 10 for LLM Applications from the OWASP GenAI Security Project (free, 3–4 hours).
• Branch B. For the default production path, the training, data and serving guides in the branch resource list. For the alternative implementation capstone, Stanford CS336.
• Branch C. Trustworthy Online Controlled Experiments (companion site) by Ron Kohavi, Diane Tang & Ya Xu: free companion materials to a paid book. Plan 6–10 hours of selected reading here, or 20–30 for the full book. Then The Effect by Nicholas Huntington-Klein, a free book: 6–10 hours of selected reading here, or 25–40 for the whole book.
• Any branch. The Kaggle Playground Series from the Kaggle competitions team is free validation-design practice between capstone sessions, 10–25 hours per entry.

Before you move on

• You finished one branch's capstone, not three half-started ones.
• Your README states the metric, the result with a number, and what the system should not be used for.
• You can defend your capstone choice in one minute without mentioning salary.
• For Branch A: you can show repeated-task results, a permission-boundary test and recovery from a failed tool call, with the cost of successful completion.
• For Branch B's production path: your release links data, model and serving configuration. Load and checkpoint/rollback checks support your claims. For the CS336 alternative, the completion evidence is correctness tests, checkpoint resume, profiling and the documented implementation.
• For Branch C: the decision memo states the estimand, assignment or design checks, uncertainty and the limits of the causal claim. It separates the hypothesis set in advance from the observed result, includes a relevant sensitivity or robustness check, and labels synthetic data or a simulation as such.

Previous: Running it in production  ·  Next: Finding the work

Week 33 onward  ·  about three to four hours a week, alongside chapters 6 and 7.

Book index  ·  Chapter resources

The short version. The search is a process you run, not a verdict on you. Present three deep projects well, practise your interview stories out loud, track your numbers, and use the doors that are actually open. This chapter runs alongside the last two, because a search can take months and starting early shows you what the market wants.

In this chapter: First month  ·  Portfolio  ·  Interviews  ·  Search  ·  Completion

Your first month (alongside Chapters 6–7)

Spend about three to four hours a week on the search while you keep building. Full-time job hunting comes later.

After month one, log applications weekly and refresh your README when metrics move. Start now, before Chapter 7 finishes.

What you will be able to do

Present three projects so that a reviewer understands them in two minutes, tell three interview stories out loud without notes, and run a search you can measure and adjust rather than endure.

Three projects is a planning target. The case studies can come from one connected project: data and validation, model or experiment results, and production delivery. Give each a clear question and its own evidence, and put work relevant to your target roles first.

Presenting your work

Make a reviewer's first two minutes count. These are presentation suggestions, not measured hiring effects:

• Archive the tutorial forks. A weak repository sits next to your strong one and gets opened too. Make your strongest projects easy to find.
• Make the README useful without running the code. State the question, the result with a number, the architecture, the key decisions, what did not work, and the known failures. Those last two sections show a reviewer your judgement.
• Show something working. Offer a public demo, recorded walkthrough or reproducible local command that suits the data and cost.
• Write things up. One short post per major topic, explaining it plainly. It is good revision, and it is how people find you.

A README outline that works. Copy this into each project and fill in the blanks:

# [Project name]

## Question

One sentence: what did you try to answer?

## Result

State the baseline, result, held-out sample size and uncertainty.
Say whether the evidence is synthetic, retrospective or from a live experiment.

## How it works

Three to five sentences, or a simple diagram. Link a live URL if you have one.

## Key decisions

Two or three bullets: what you chose and what you rejected.

## What did not work

One or two failures, with evidence and what you changed.

## Operations evidence

If it is deployed: release manifest, the alerts you act on, cost per 1k requests, and a link to
the incident report from a deliberate failure.

## Known limitations

What the system should not be used for.

For the operations section, link your Chapter 6 artifacts directly: the incident report, rollback runbook, cost table and model card. The reference lab has filled, simulated examples of a model card and an incident rehearsal. Use them as formats, not as your evidence.

A business project can use the Chapter 9 evidence levels. "Improved policy value in a simulator" and "increased live contribution in a randomised test" are different results. Include the assignment unit, horizon, no-action baseline and costs. If you only have observational transactions, show predictive performance and say what intervention evidence is missing. A reviewer should be able to reproduce the claim you actually make.

Try it yourself. A README's Result section says: "Churn model achieves 94% accuracy and will save the business $200k a year." What would a reviewer object to, and how would you rewrite it?

For open source, start with a bug you hit while building your own project. The entries in Chapter 7's resources cover the mechanics. Read each project's current contribution policy. scikit-learn requires contextual understanding, careful human review and disclosure of AI tool use. Transformers welcomes coordinated, verified AI assistance and requires disclosure and test evidence. You stay responsible for every changed line and for the maintainers' review burden.

The interview loop

Use Eugene Yan and Jason Liu's hiring account to connect your projects to coding, data and evaluation questions. Confirm the actual stages, time limits and allowed tools with the recruiter. Practise coding on your own, and explaining assistant-written code where the employer allows it.

Have three stories ready and practise them out loud, at ninety seconds each:

1. A time your numbers were wrong and you found out. The Chapter 3 leakage exercise is built for this.
2. A time you chose the simpler option deliberately, with the numbers behind it.
3. A time you shipped something someone else used.

Story structures, not results to borrow. Swap each scenario for work you actually did and numbers you can defend. Keep the task, mistake or tradeoff, evidence, change and outcome.

1. Wrong numbers. "My delivery model used a final-status field that only arrived after the decision. I removed it, rebuilt decision-time features, and evaluated on a later period with mature labels. Here are the original and corrected scores, sample sizes and release test."
2. Simpler on purpose. "I compared retrieval with fine-tuning on the same held-out questions. The paired comparison did not show a meaningful improvement at a cost we could support. I chose retrieval and documented the remaining errors and the evidence that would change my mind."
3. Someone else used it. "A team used my triage tool for a defined pilot. I recorded eligible cases, usage and failures. I fixed one failure class and tested the change on a separate release set. Here is the measured scope of the pilot and the reproducible evaluation."

A result on 50 examples is a useful first signal, not a precise population guarantee. One extra pass moves the rate by two percentage points. Show paired disagreement counts or a suitable interval. Keep pilot adoption separate from evidence that your change caused a business improvement.

The search itself

Run it as a process. Track applications and three conversion rates: applications to screens, screens to onsites, onsites to offers. Calculate each rate from resolved stage transitions, show the denominators, and keep pending decisions separate. Compare cohorts with a similar time to resolve. Different response delays can change resolved-only rates without any real improvement. Use each stage to find a question to investigate before choosing more preparation.

A tracker you can actually use. One spreadsheet or markdown file, updated weekly:

If few applications become screens, review role fit, eligibility, CV and portfolio signals. If few screens or onsites move forward, ask for feedback on the actual rounds, including coding, SQL, design and communication where assessed. Review level or pay expectations when those were a stated obstacle. A funnel rate alone cannot identify the cause.

Before weeks of preparation, review a few current postings for your target city and level. Record the date, actual responsibilities, required skills, interview format if known, and which project supplies evidence. Revisit this monthly. Titles alone can mislead. An "AI Engineer" role can mean product integration, retrieval, modelling, infrastructure or research.

Track referrals, community contacts and targeted applications separately, and use your own funnel to decide where to spend time. Small counts jump around, and a low response rate can reflect fit, location, timing or application quality. Change one thing, then review a new batch. One rejection is no reason to redesign your whole portfolio.

Consider nearby roles that match your experience: data engineering, backend work on an AI product, contract projects or an internal transfer. Check the actual mentorship, work and transfer policy, because a backend offer does not promise a move into ML within a year. Employers outside technology may also have relevant tabular and document projects.

On pay, compare your level, location and pay components. Indeed's September 2026 US analysis finds seniority affects advertised-pay comparisons, though its seniority-specific evidence is only suggestive. It studies US postings with an advertised salary, not individual offers or Indian salaries. Self-reported pay trackers and aggregate AI wage premiums cannot predict your offer. Check current local ranges, and separate base pay, equity, bonuses and contract terms.

Staying current afterwards

Split your attention. The fast layer turns over in a year or two: model names, frameworks, pricing, free tiers and specification versions. The slow layer will still be true in a decade: attention, backpropagation, evaluation design, generalisation, retrieval, cost and latency, statistics and SQL. Spend your study time on the slow layer and simply monitor the fast one. A couple of hours a week on a few named, human-written sources is plenty.

Once a quarter, spend ninety minutes checking whether anything you depend on has changed price, been deprecated or moved. Re-run your evaluations before adopting a model version change. Rolling model aliases can change without a code change. Record the resolved version, and investigate unexpected regressions with your evaluation suite.

Resources

All 12 resources for this chapter, with notes on each, are in resources/.

The ones to begin with:

• AI Engineer Compensation Trends (Q3 2025) — Alina Kolesnikova, levels.fyi. A free article, under an hour, for setting pay expectations by level.
• How to Interview and Hire ML/AI Engineers — Eugene Yan and Jason Liu. A hiring-side account to help turn your projects into interview evidence.
• normcore-llm-reads — Vicki Boykis. A free curated reading list for staying current without the noise.
• State of AI Report 2026 — Nathan Benaich and Air Street Capital. An annual industry overview. Follow its primary sources, and check real openings before choosing a sector.

Before you move on

• Your project case studies follow the outline above, including "what did not work."
• You have said each of the three interview stories out loud at least twice.
• Your tracker separates pending from resolved outcomes and records dates, role, location and channel. With small counts, describe the uncertainty and ask for specific feedback, rather than declaring one funnel stage definitely weakest.
• You have checked pay expectations at your level and city, not the headline average.

Previous: Choosing a specialisation  ·  Next: Business machine learning  ·  Back to the index

Applied track after Chapters 3 and 6  ·  about 40 hours for one case, 90 for all three.

Book index  ·  Chapter resources

The short version. Build a system that chooses a useful business action and measures its effect. Offer recommendation, price recommendation and churn share data and operations, but ask different statistical questions. Start with a transparent baseline, test a stronger model on a realistic split, and ask for proof before turning predictions into interventions.

The examples use retail and subscriptions. Adapt the outcome, allowed actions, customer policies and economics to your domain. This is an engineering recommendation based on the research below, not a promise that one model wins in every company.

In this chapter: Plan  ·  Decision trees  ·  Model choices  ·  Architecture  ·  Build  ·  Completion

Your applied plan

Read the short causal primer and the shared delivery plan, then choose a guide. Use one case for the Chapter 7 capstone, or finish all three as one connected business project. The hours are for a small learning prototype on public or synthetic data. Collecting real data over time and running a powered business experiment take extra calendar time.

These are scaffolded practice budgets: you reuse the earlier environment, a small prepared dataset and the service boundary. Starting from a blank repository or collecting your own historical data needs extra setup time. The reference lab gives a synthetic starting point, so it cannot show a real business effect. On the first pass, finish the baseline, constraints and evidence report before opening the optional research sections.

One guide plus the two shared blocks takes about 40 hours. Every row takes 90. Finish one baseline before adding a second model family. The starting models do not need a GPU. The shared delivery plan lists the artifacts and the completion gates.

The questions are different

Prediction estimates outcomes in observed data. Intervention estimates what would change under an action. A model can predict well from patterns that break once you act, as scikit-learn shows in its causal interpretation example. Randomisation is the preferred start for actionable effects. Observational designs need stated identification assumptions and sensitivity analysis, and a more complex learner cannot make up for missing identification.

Choose your ML approach

Start with the business question and the evidence you have. Follow the decision tree in the right guide, then read the matching recommendation: which model to start with, why it fits, what data and evidence it needs, and what to validate before adding complexity.

You can combine branches. A recommender can retrieve candidates and estimate offer effects. A churn system can estimate both risk and retention effects. The final business policy still applies costs, eligibility, budget, capacity and the approved fallback.

Each model branch starts with a baseline and has an evidence gate for promotion. If timestamps, outcomes, costs or action support are missing, do the data-collection step first. Recent research challengers stay in the detailed model sections, with their reproduction and evaluation conditions.

Suggested models and approach

These are starting candidates. The final choice comes from your business metric, fair evaluation over time, inference cost, calibration and evidence about treatment effects. Only a model that improves the decision enters the production path.

CatBoost's categorical-feature support makes it convenient for many customer tables. LightGBM is another candidate when its training, memory and serving behaviour suit the data. Neither wins by default, so compare on the same splits and cost budget. Current tabular results depend on model version and evaluation protocol. Compare BeyondArena’s June 2026 benchmark with the September TabPFN 3.5 report. The second reports aggregate gains but does not lead everywhere across full grouped, temporal and large-data slices, and its group-feature protocol differs. Chapter 3 explains how to make the comparison match deployment.

The guides explain the specific conditions behind each row. Read them before choosing a library.

How to use the newest research

The research review includes sources available through 9 October 2026. The guides include optional challengers: causal, generative recommendation, inventory-learning and survival foundation models. Being new does not make them the best starting models. Some are recent preprints or workshop results. Industrial engagement results also answer a different question from incremental business contribution.

Promote a candidate only after it clears these comparisons:

1. Task and evidence: match the prediction, treatment, censoring and deployment setting to yours. Write the identifying assumptions before fitting.
2. Reproduction: check the release, licence, preprocessing, tuning budget and evaluation split. Compare on the same data, action support, hardware and allowed candidate count.
3. Decision value: measure calibrated risks or effect sizes, supported policy value and uncertainty, as well as ranking. Include no action and a simple nonpersonalised policy.
4. Operation: test cold start, missing inputs, drift, inference cost, latency, constraints and fallback. A research benchmark says little about reliability in production.

The practical starting point remains a transparent baseline and a calibrated boosted-tree challenger for tabular prediction, identified causal estimation for interventions, and an explicit constrained decision policy. Optional research comparisons sit outside the prototype time budget.

One architecture, three decision paths

Use the same lifecycle and execution contract for all three cases. Only the model output changes: offer response or uplift, price-dependent demand, or churn risk, survival and retention effects.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Shared business ML architecture
    accDescr: Prepare point-in-time snapshots, validate the case model, choose a permitted action, persist the assignment before execution, and join mature outcomes for monitoring and the next validated release.
    S["Point-in-time snapshots<br/>Verified training targets"]
    M["Validate case model<br/>Offers · prices · churn"]
    D["Constrained policy<br/>Include approved default"]
    A["Record before delivery<br/>Reserve + execute once"]
    F["Join mature outcomes<br/>Costs + contribution"]
    S --> M --> D --> A --> F
    F -.->|Evaluate + refresh| S
    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef action fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class S,F data
    class M model
    class D decision
    class A action

Blue: data and outcomes. Indigo: model validation. Amber: policy decisions. Teal: delivery. Labels identify each role as well as colour.

Training targets need mature fixed-horizon outcomes or properly recorded event and censoring follow-up, as set out in the data contract. Save the original decision, assignment and relevant logging probabilities before delivery. Record delivery events separately, including nondelivery and overrides. The delivery lifecycle shows the order in detail.

The coordinator is application logic or an optimisation step. An LLM can help explain approved results or extract inputs from documents. Price, eligibility, causal effects and budget limits come from your data and rules, not from the LLM.

Start with one pipeline, a SQL snapshot job, one fitted model artifact and one batch job or API. Add a feature store, registry platform, embedding service or bandit when a concrete need justifies its running cost. The feedback loop is the architecture that matters.

Make the three systems cooperate

Offers change the price the customer actually pays. Retention offers use the same customer attention and often the same subsidy budget. If each system maximises its own score alone, a customer can get competing discounts, a campaign can contaminate a pricing test, and the same revenue can be counted twice.

Log list price, recommended price, final paid price, subsidy source and every concurrent campaign assignment. Start with mutually exclusive experiments or one shared assignment plan. Give one coordinator ownership of the final action, including no change/no contact. Cap total spend, contact frequency and allowed price movement. Treat customer-level policy effects and inventory spillovers as part of experiment design.

Optimise combined actions only after you have data supporting them. Separate offer and price models cannot identify the effect of an untested discount-plus-price change. Use a factorial design (testing the combinations) where it fits, or test the combined policy directly. Report total net contribution once, with component metrics to explain it.

For logged policy evaluation, record the coordinator's joint assignment probability, or its stage-by-stage rule, with pre-decision budget, stock and contact state. Separate marginal probabilities do not usually describe a combined action. A new policy can also change future inventory and eligibility, so a one-decision evaluation cannot show the whole system's long-term value. The shared policy evaluation protocol sets out these limits and the controlled-test path.

What to build

A business decision system with an evidence report. Choose one guide and follow the delivery plan. Publish the decision brief, data contract, model comparison, policy comparison, experiment protocol, batch/API contract, monitoring and rollback artifacts. For private business data, publish code and synthetic examples instead of customer records.

Keep three evidence levels separate in the report:

A finished prototype can have an experiment proposal and a rehearsed rollout. A measured live profit lift needs executed assignments and observed outcomes. Say this plainly.

Before you move on

• You can state the decision, outcome horizon, baseline and unit of assignment in one minute.
• The feature query excludes events not yet available. Fixed-horizon labels are mature, and survival evaluation uses recorded censoring and supported follow-up.
• You can defend your model choice against a simple baseline using the same data and budget.
• You know which effects are identified, and where the policy must abstain or fall back.
• You can trace one chosen action through execution, cost, outcome and experiment assignment.
• You can disable the policy and reproduce its last release from its recorded inputs.

Resources

The annotated source list separates official implementation documentation, research papers and practice datasets. The three guides cite their supporting sources directly.

Previous: Finding the work  ·  Delivery plan  ·  Back to the index

Chapter 9  ·  Shared delivery plan

Read this before the offer, price or retention guide. It helps you state the causal question and its assumptions before choosing an estimator. The arithmetic below is made-up teaching data.

Name the comparison

A prediction asks what is likely in the observed setting. A causal question asks what would change if the same eligible people received a different action. Write the estimand: the quantity you want to estimate, with its population, action, comparison and outcome horizon.

For example: “Among currently eligible subscribers, how does assigning this retention offer rather than no contact change 30-day retained contribution per assigned account?” This policy comparison is more useful than asking whether some model “reduces churn.”

For each account, imagine two potential outcomes: retention if assigned the offer, Y(1), and retention if assigned no contact, Y(0). You see only one, so ordinary data never labels the difference for one person. An average treatment effect compares the averages, and a conditional effect compares them within a stated context. Neither tells you for certain which person would have changed their decision.

Worked lesson: risk and benefit answer different questions

Suppose a randomised, fully observed experiment gives these point estimates:

With equal segment weights, the estimated average retention effect is (0.10 + 0) / 2 = 0.05. Assume an extra retained account contributes 100 units and an offer costs 5 per assigned account, with no other costs or interference between customers. The estimated extra contribution is:

• Higher risk: 0.10 × 100 - 5 = +5 per assigned account.
• Lower risk: 0 × 100 - 5 = -5 per assigned account.

These are point estimates, not promised gains. Report intervals at the assignment unit, check guardrails, and confirm a policy's value on data you did not use to choose it. Real economics may need contribution as the direct outcome, not retention times a constant. The churn guide explains when high risk and benefit from the action disagree.

Try it yourself. An offer costs 12 instead of 5. What do the point estimates recommend?

What randomization supplies—and what it still needs

Random assignment makes groups alike on average. Still check assignment logs, eligibility, sample-ratio mismatch (unplanned group sizes), outcome coverage and the planned analysis horizon. Analyse outcomes by original assignment for an intention-to-treat effect. Dropping failed deliveries after assignment can leave different kinds of customers in each arm.

The comparison also needs a clear action and outcome, a real chance of receiving each compared action, and a design for any interference or carryover (lasting effects of earlier actions). Randomising one customer's price may change another's demand or use up shared stock. Choose the assignment unit and interpret results accordingly.

When assignment was not random

A confounder affects both who gets the action and the outcome. For example, an account manager may target unhappy customers, who were already more likely to leave. A treated-versus-untreated comparison then mixes that selection with the offer's effect.

A mediator is changed by the action and sits on a path to the outcome, such as post-offer engagement. Adjusting for it usually changes the causal question and can bias a total-effect analysis. A collider is a shared result of other variables, and conditioning on it can create a false association. A diagram helps state these choices, but cannot prove the assumptions.

Observational adjustment usually needs a believable case that measured pre-action variables block the relevant confounding. It also needs overlap: each compared action has a nonzero chance in the contexts where the policy would choose it. If an action was never tried in a segment, a flexible model can extrapolate a number without identifying that effect. Restrict the policy or collect data.

The propensity is the probability that the assignment policy selects an action, given recorded pre-action context. It differs from the predicted chance of a purchase. Estimated propensities and outcome models are often called nuisance models, because the main target is the effect or policy value, not their predictions.

Cross-fitting trains a nuisance model on other folds before scoring a fold. It can reduce some overfitting, but cannot repair unmeasured confounding, unsupported actions or a wrong time or group split. “Doubly robust” is a property under stated assumptions, so the design still matters. The linked guides give each estimator's requirements.

Before using an effect estimate

Write the estimand, assignment unit, clocks and outcome coverage. Say which identification assumptions come from the design and which are your judgement. Keep simulation, past-data prediction and measured intervention results apart. Then go to the delivery plan and one case: offers, prices or churn and retention.

To go deeper, try The Effect and the metalearners paper, both annotated in the chapter resources.

About 25 hours for the learning project.  ·  Chapter 9  ·  Index  ·  Resources

The short version. Start with eligibility rules and a measurable baseline. For a small offer catalogue and customer data in tables, compare logistic regression with CatBoost or LightGBM. When the offer costs money or changes how customers behave, learn incremental contribution profit against a no-offer control from randomised assignments: the extra profit compared with sending nothing. For a large catalogue, use retrieval and ranking. Consider contextual bandits only once you can log action probabilities and measure outcomes reliably. These solve different problems, and no model is best for all.

You finish with a decision service, a reproducible evaluation and an experiment plan, which a learning project can prepare in about 25 hours. A properly powered experiment and its full outcome window take extra calendar time.

In this guide: Decision contract · Data · Model decision tree · Evaluation · Serving · Completion gates

Your learning plan

Build this after the validation work in Chapter 3 and the serving work in Chapter 6. Deep learning is optional here.

Start with the causal primer. The 25-hour schedule assumes small, prepared data, your earlier environment and a reused service boundary. Finding data, a new delivery integration or approval for a production campaign needs extra time. First finish one supported baseline for a single binary action. Pretrained causal models, generative ranking and bandits are optional depth.

1. Decide what “offer” means

Two business requests often arrive under the same name.

A product can be relevant without needing a discount. A customer may be likely to redeem a coupon and just as likely to buy at full price. A model trained to predict redemption often favours such customers, because redemption is easier to predict than behaviour changed by the offer. Attribution reports and conversion probabilities cannot show incrementality, the sales the offer caused. Microsoft's incrementality documentation draws the same line between attributed conversions and causal impact.

For this guide, picture a retailer or subscription service with a small, explicit action set: no_offer, reminder, free_shipping, and coupon_10. A reminder is an active treatment if it sends a message. The no-offer control gets your defined usual experience. Note whether that includes other campaigns. For retention, connect this policy to the churn guide. For an offer that changes the selling price, use the demand and margin method in the pricing guide.

Write the contract before you choose a model:

• Decision: one action for an eligible customer or account, at a stated time, through a stated channel. If the decision is a whole ranked slate, say so, because its logging and evaluation differ.
• Outcome: contribution profit over, for example, the next 30 days, with time for returns to arrive. Count every transaction, whatever the coupon redemption or click attribution.
• Comparison: a no-offer policy and the current business policy. A new policy must beat both, on the same eligible population and accounting rules.
• Constraints: consent, eligibility, stock, minimum margin, contact limits, total spend, concurrent campaigns, and any limits the business sets.
• Guardrails: complaints, unsubscribes, returns, service load, and longer-term buying or renewal behaviour. Set the thresholds that matter before you see the result.

2. Turn the target into a decision

Let x hold what is known at decision time, a be an eligible action, and 0 be no offer. Define:

Q(x, a) = E[contribution profit over the window if assigned action a | x]
delta(x, a) = Q(x, a) - Q(x, 0)
chosen action = argmax delta(x, a), among feasible actions including 0
delta(x, 0) = 0

This is a conditional average effect: the expected difference for customers with similar observed context. You see only one of the two outcomes for each person, so you cannot directly label anyone's true uplift. Take the logged reward from the same accounting contract: contribution over the window minus the specified incentive, fulfilment and contact costs, including failed delivery attempts. If net contribution already subtracts a cost, do not subtract it again. Use that net reward for both Q and policy evaluation, no-offer arm included.

A worked example. Suppose an order contributes ₹500 before the incentive. The incentive applies to every completed order in its assigned arm, and sending the offer costs ₹2. Assume no returns or other effects. Suppose randomised data and a validated model gave these made-up purchase probabilities for one customer context:

The largest benefit wins on conversion but loses to the ₹100 benefit on incremental profit. If the ₹100 benefit were infeasible, the policy would pick from the remaining feasible actions. If every active action lost money against no offer, choose no offer. In a real campaign, discounts may apply only on redemption, baskets can change, and returns add later costs. Measure those parts, or model net profit directly, instead of forcing this simple formula onto the business.

Try it yourself. Sending the offer now costs ₹10 instead of ₹2, with the same purchase probabilities. Recompute the increments. Which action does the policy choose, and which active action now loses money against no offer?

A discount paid to customers who would have bought anyway is a real cost. Also check whether the offer only pulls a future purchase into the window, moves revenue from a higher-margin product, or pushes out an existing offer. That is how a short-term redemption metric can produce an expensive policy.

3. Build the data around decision opportunities

Each training row is an eligible opportunity at a decision time, including those assigned no offer. A table of redeemed offers alone cannot estimate the comparison above. For each opportunity, join all relevant purchases, returns and costs over the fixed outcome window. Keep delivered, seen, clicked and redeemed as separate events.

Use at least these tables. The schema is my suggestion, not a library requirement.

For a feature window ending at decision_at, use only events whose event time and availability time are both at or before the decision. A purchase dated Monday but first loaded on Wednesday was unavailable to Tuesday's service. Joins to today's customer profile, today's stock or a backfilled lifetime-spend field can all leak future information. Fit encoders, imputers and other learned transforms on training data only.

Useful inputs include purchase recency and frequency, past contribution, category affinity, tenure, earlier eligible contacts, time since the last offer, pre-decision complaints, channel preference, offer/product attributes, calendar context and available stock. Add missingness and history length so thin histories are visible. Leave out post-assignment opens, coupon redemption and future spend.

Do not invent labels for unchosen actions. When a customer gets free shipping, you see their outcome under that assignment only, not under a coupon or no offer. Products never shown are unobserved, not confirmed dislikes. Negative sampling can keep retrieval training manageable, but it changes the training distribution. It does not turn missing interactions into causal negatives.

Make overlapping decisions explicit. One purchase cannot quietly become an independent positive label for five competing offers. Start with one decision per account per non-overlapping measurement window, or use a documented design that models sequences and concurrent campaigns.

Mature does not mean observed. A closed window with a reconciled transaction and return ledger can give a zero-purchase outcome. A closed window with lost identifiers, an incomplete channel feed or an unresolved refund cannot. Store observation status and label availability separately, and report missing outcomes by assigned arm and pre-decision cohort. Keep every assigned account in the experiment accounting. Dropping missing outcomes can select customers after assignment, and neither randomisation nor correct propensities repair that. Reconcile coverage, and for any remaining exclusions state the assumptions and sensitivity analysis they need, as the experiment inclusion guidance shows.

4. Choose a model according to the evidence

Model-selection decision tree

Choose the question before the model. The incremental-effect branch asks whether an action changes profit against no offer. The relevance branch asks which product or offer is likely to interest the customer. “Identified” means randomised assignment with action support, or an observational design you can defend. Redemption or targeted-campaign history alone does not pass that gate.

Check data readiness first. If you lack the pre-decision features, exposure logs or mature labels, start with eligibility rules, popularity where supported, and the no-offer baseline. Collect decision, exposure and outcome logs using the data contract, then follow the tree.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Offer model-selection decision tree
    accDescr: Choose retrieval and ranking for a large relevance catalogue, logistic response prediction without identified effects, or uplift modelling when treatment effects are identified.
    Q{"Incremental<br/>effect?"}
    Q -->|No| C{"Large<br/>catalogue?"}
    Q -->|Yes| E{"Identified?"}
    C -->|Yes| O1["O1<br/>Retrieve<br/>+ rank"]
    C -->|No| O2["O2<br/>Logistic"]
    E -->|No| O2
    E -->|Yes| O3["O3<br/>S/T uplift"]

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class Q,C,E decision;
    class O1,O2,O3 model;

If a retrieved product carries an incentive, use O1 for relevance and follow the incremental-effect branch separately for the incentive. A better product rank does not show discount lift.

O1 — Large catalogue: retrieve, then rank

• Model and why: Start with eligible popularity/content candidates. When scoring the whole catalogue costs too much, compare retrieval plus LightGBM LambdaRank. Use two-tower retrieval only when interaction volume and measured serving needs justify learned embeddings.
• Minimum data: For learned models: customer–item interactions, item metadata, candidate and exposure/position logs, and meaningful relevance labels with query/session groups.
• Promotion check: Improve retrieval recall, ranking quality and latency at the same eligible candidate budget. Check cold-start coverage. Continue with retrieval and ranking.

O2 — Small table or unidentified effects: predict response

• Model and why: Fit logistic regression, then compare one CatBoost or LightGBM response model. That gives an inspectable baseline and a nonlinear challenger. Comparing predicted response scores cannot identify what an intervention caused.
• Minimum data: Pre-decision customer/offer/context features, observed assignments or exposures, and mature response labels including non-responses, with a deployment-matched holdout.
• Promotion check: Compare log loss, rare-event PR metrics, calibration and serving cost on the same split. Without effect identification, keep claims predictive and collect randomised offer/no-offer data. Redemption prediction is not uplift. Follow the predictive workflow and collection design.

O3 — Identified effects: estimate uplift and choose a profit policy

• Model and why: For randomised data, compare arm profit with a regularised S-learner or T-learner, respecting assignment probabilities and strata. For observational data, they need a defensible confounder-adjustment set, overlap (each action used for similar customers), and effect or value estimates that this design justifies. Raw historical arm averages do not show uplift. The target is each action's difference from control, over a small supported action set including no offer.
• Minimum evidence: Mature net outcomes and supported action/control assignments with known randomised probabilities, or a defended observational design that handles confounding, overlap, treatment definition and interference.
• Promotion check: Require held-out incremental contribution, support and uncertainty checks, then a policy experiment. Compare a cross-fitted DR-learner or causal forest only when arm sizes and heterogeneity justify it. Continue with uplift and profit policy and policy evaluation.

A bandit is a separate step. Consider one only once these work: assignment probabilities logged after constraints, action support, joins to mature rewards, safe exploration and a randomised holdout. A bandit cannot supply missing causal identification. Use the bandit prerequisites after the fixed-policy baseline.

Model comparison ladder

The table below is my suggested order of work. You still choose the model on your own held-out data, costs and latency budget.

Predictive baseline: logistic regression, then one boosted tree

Model response from pre-decision context and offer attributes. Compare it with rules and offer-level historical rates. CatBoost suits tables with categorical customer, product and offer attributes, because it handles them directly. LightGBM is another good candidate with explicit categorical handling. Choose on validation quality, memory, training cost, serving latency and team familiarity, since neither wins everywhere. CatBoost's categorical-feature documentation and LightGBM's advanced topics describe how each one works.

Measure log loss, PR-AUC for rare conversions, and reliability curves by action and customer segment. Calibrate on a separate, later validation sample, or with a split that respects time and account identity. After any negative sampling or class weighting, evaluate at the original deployment prevalence. Calibration matters because profit uses the size of a probability, not just its rank. The scikit-learn calibration guide explains why calibration data must be separate from model fitting.

For a small table, an optional tabular foundation-model comparison can be worthwhile. Use the version-specific guidance in Chapter 3. Compare on the same future opportunities, per-action calibration and serving budget. Good outcome prediction does not make it a causal learner.

This model is a good predictive baseline. Past offers were targeted at different customers for a reason, and comparing its predictions for two such offers does not remove that reason. Label a profit simulation built on these predictions as model-based and unvalidated causally. Feature importance or SHAP helps inspect a prediction. It cannot show which features cause customers to respond.

Incentive selection: randomized uplift and profit policy

Run a multi-arm campaign with no offer and the candidate actions. Apply the common eligibility rules, then randomise at a stable account or customer level. At each assignment, record the known probability of every feasible action. Stratifying by a few pre-decision segments can improve balance, but the logged probabilities must reflect that design. Every action the new policy might take needs enough support in its context.

Start with average profit per arm and its uncertainty. When the treatment-effect signal is weak, a segment or no-offer policy can beat a personalised model. Then compare:

• S-learner: one outcome model with the assigned action as an input. It shares information across arms, but if the main outcome signal dominates, it may learn little treatment variation.
• T-learner: one outcome model per arm, using regularised CatBoost/LightGBM or another suitable learner. Subtract the no-offer estimate. It is easy to inspect but can be noisy in small arms.
• X-learner: a useful comparison for a binary treatment/control problem with unequal arm sizes. Whether it helps depends on how the data was generated. Künzel and colleagues' meta-learner paper finds that no meta-learner is best in every case.
• DR-learner: combines outcome and assignment models with cross-fitting. It can handle several categorical treatments. Choose and validate the nuisance (helper) and final models separately. With a properly randomised design, give the known assignment probabilities to estimators that accept them. EconML's DR guide covers multi-treatment estimation and its assumptions.
• Causal forest: an optional comparison when you have enough data to study nonlinear effect differences and their uncertainty. Use an estimator and inference method suited to the treatment representation. Wager and Athey's causal-forest paper develops inference under stated assumptions. Those guarantees do not cover any forest trained on a CRM export.

Cross-fitting means predicting helper quantities for a row with models fitted on other rows. Keep related account records together, and leave the final future test period untouched. For observational data, a defensible causal analysis also needs no unmeasured confounding, overlap, a well-defined treatment, and a suitable interference assumption. A rich CRM table does not prove these. An agent's unlogged judgement about which customers need a discount can confound the data even when the response model has excellent AUC. Double robustness cannot repair an unobserved confounder, or an action never assigned in a customer context.

Optional research challengers: learn the effect or the policy?

A treatment-effect model is one part of a decision. You can also learn a restricted policy, such as a shallow decision tree, from cross-fitted estimates of each action's value. This can make a campaign easier to review while enforcing a contact budget. Athey and Wager's policy-learning paper, published in Econometrica in 2021, develops binary decisions under explicit identification and estimation assumptions, including binary treatment allocation and small continuous-treatment nudges. It is not a ready-made multi-arm budget allocator. For several offers, estimate each supported action's net value and cost, then use an allocation method that enforces your real constraints. A simple policy with reliable value can beat a more accurate effect model whose decision threshold is unstable.

This extension is outside the 25-hour starting project. Add it only after the randomised baseline and held-out evaluation work. A recent paper is a reason to test a candidate, not to replace them.

Large catalogue: retrieve, rank, then apply the policy

If scoring every product costs too much, first retrieve a manageable candidate set. A two-tower model embeds the customer/context and the item separately. Item vectors computed ahead of time make nearest-neighbour retrieval fast. A richer ranker with customer–item interactions comes next. The TensorFlow Recommenders retrieval tutorial shows this structure, and Google's YouTube paper describes candidate generation followed by ranking at large scale.

Train retrieval on observed customer–item interactions, with features available for those examples. State how you sample negatives. When item IDs repeat, another row's positive can become an accidental negative. The retrieval task API removes these duplicate-positive hits with remove_accidental_hits=True and candidate_ids, and corrects logits using negative-candidate sampling probabilities. Those probabilities describe the training sampler, not campaign assignment or causal exposure. Report retrieval over the eligible catalogue, or say how candidates were sampled. An in-batch metric is not full-catalogue Recall@K.

LightGBM LambdaRank is a reasonable candidate when you have grouped recommendation opportunities and meaningful relevance labels. Keep each query/session group whole during training and evaluation. Log what appeared, where, and which candidates were available: a click near the top of the screen differs from a product never shown. Correcting position bias needs a suitable exposure model or experiment. Adding position as an ordinary feature does not by itself give a causal relevance estimate.

For a new customer, use stated preferences and context with recent eligible popularity. For a new product or offer, use metadata and carefully budgeted exploration, not an ID-only embedding. Metadata must go into a tower trained to use it. Adding descriptions to an ID-only model at serving does not create content-based retrieval. The side-feature tutorial shows train/serve feature representations. Test on users and items first seen after the training cutoff, using only metadata available when they appeared. Version the model and its matching candidate index together, and measure index refresh lag. Check retrieval recall before ranking: a ranker can only pick products that retrieval returned. Apply eligibility during retrieval where possible and again before the final decision, especially when stock changes.

If incentives are attached to retrieved products, there are two decisions: product relevance, and whether its incentive adds profit. The incentive still needs its own randomised test.

5. Treat constraints as part of the policy

Before ranking, remove expired, ineligible, unavailable, non-consented or below-margin actions. Apply contact-frequency rules across channels and across offer, price and retention workflows. Recheck fast-changing constraints just before fulfilment. A model score cannot override those rules.

For a campaign across customers i, a simplified allocation problem is:

maximize sum_i delta(x_i, a_i)
subject to sum_i expected_incentive_cost(x_i, a_i) <= campaign_budget
           a_i is eligible for customer i
           at most one selected action per opportunity

When costs and values differ, picking each customer's best offer on its own can exceed the budget. Sorting by benefit/cost is a useful heuristic in some settings, not an exact general solution for several actions. If this matters, compare a feasible heuristic with a small integer-programming solution on a representative batch. Expected spending also does not cap realised spending, so use reservations and redemption limits for hard financial commitments.

For synchronous serving, reserve budget atomically before returning an offer. Otherwise concurrent requests can promise the same remaining money. Use a shared contact ledger to stop duplicate sends. Apply eligibility and resource constraints before the final randomised assignment, and log the probabilities of the mechanism you really ran. If a later reservation or delivery fails, keep the original assignment and record the executed action and failure separately. Do not invent a propensity for the replacement or give it the original probability. Intention-to-treat then measures assignment under your real delivery process. A delivered-only comparison needs extra identification. The serving flow puts this into practice.

6. Evaluate the policy honestly

When decision dates are available, split train, validation and final test by time. Leave a gap, or filter training rows, so their outcome and return windows close before the training cutoff. Tune models, calibration, policy thresholds and budget rules on validation. Use the final test once, on the frozen policy.

For response probabilities, keep class weighting and any sampling inside training folds. Calibrate and evaluate on deployment-representative arm data or documented design weights. A balanced conversion sample asks a different probability question. Calibration folds must respect the same time and account boundaries as model selection, as described in Chapter 3. Report positives and observation coverage by arm: a pooled reliability plot can hide an unsupported offer arm.

If production serves existing customers repeatedly, they can appear in several periods, provided features are available at the right time and labels do not overlap. You need not force every customer to be unique across a time-based test. When new customers matter, also report a separate account holdout for cold start.

AUUC or Qini is a diagnostic, not the final campaign objective. State whether it evaluates a binary treatment, one arm against control, or a multi-action policy, because these differ. Tune any uplift curve before the final test, so its chosen contact depth stays an independent result. For public data without margins, report conversion/visit policy estimates with clearly labelled cost sensitivity, not verified incremental profit.

Why ranking alone can select a losing campaign. In this made-up example, two accounts have true incremental contribution of ₹12 and ₹6 before contact costs. Contacting either costs ₹10. Capacity allows at most two contacts. One model estimates effects of ₹12 and ₹6 and contacts only the first account: actual net value ₹2. Another estimates ₹24 and ₹12 and contacts both: actual net value ₹2 − ₹4 = −₹2. Both rank the accounts the same way. The error is in effect size and the cost-based threshold, which a ranking curve cannot see.

The August 2026 UpliftBench preprint studies this gap and metric sensitivity across outcome regimes. Its results are dataset-specific. Some threshold comparisons reuse the evaluation data, and its intervals measure split variability. Treat it as a reason to choose thresholds independently, not as a universal model ranking. On a validation experiment, inspect arm differences or DR estimates within effect-score bins fixed in advance, with counts and uncertainty. This checks effect sizes for groups, not any one person's counterfactual. Choose the threshold there, then evaluate the frozen policy on a separate experiment or test sample. Picking the best of many policies on one test set makes the winner look better than it is, even when each value estimator is valid.

For a fixed target policy pi and net reward r from the accounting contract, inverse-propensity scoring estimates policy value with a weight pi(a|x) / p_logged(a|x). Here a_i is the recorded assignment, including assignments whose delivery failed, and r_i is its eventual observed outcome. The target policy must stay within the logged feasible set and action/context support. In a held-out randomised sample, a doubly robust estimator combines reweighting with an outcome model:

V_DR(pi) = mean_i [sum_a pi(a|x_i) * Q_hat(x_i, a)
                  + pi(a_i|x_i) / p_logged(a_i|x_i)
                    * (r_i - Q_hat(x_i, a_i))]
incremental value = V_DR(pi) - V_DR(no_offer)

Take nuisance predictions from a separate fit or cross-fitting. Inspect weights, action/context support and effective sample size. For nonnegative weights w, the diagnostic effective sample size is (sum w)^2 / sum(w^2). Large weights can make an estimate too unstable to use. Clipping weights trades variance for bias, so report the threshold and sensitivity. Without extra assumptions, this estimate cannot identify a target action with zero support. Dudík, Langford and Li's policy-evaluation paper explains the direct-model and importance-weighting trade-off behind doubly robust evaluation.

This estimates one decision at a time, over the logged opportunities. Include pre-decision history, feasible actions and resource state when they affect assignment. A new policy may change future contact eligibility, budget use, stock or customer behaviour, so future states can differ from the log. This formula cannot give the value of the whole adaptive campaign. Evaluate the complete policy in a controlled experiment, or justify a sequential design separately. Coordinate simultaneous price, offer and retention actions through the shared delivery plan. Separate per-action probabilities are not, in general, the probability of their joint assignment.

Compute confidence intervals at the assignment unit, such as the account, and keep cluster or stratum structure. If the logging policy changed through online learning, use inference suited to adaptive data. Intervals that treat rows as independent can mislead with repeated customers, campaign-wide shocks or adaptive assignment. For policy minus baseline, use paired uncertainty instead of subtracting two unrelated interval endpoints.

Evaluate profit and contact rate by action, tenure, sparse history, channel, region where relevant, and other pre-decision business segments. Show support counts as well as scores. If the apparent winner changes with a small accounting assumption, a slightly later time period, or reasonable weight clipping, the result needs more proof.

7. Use experiments to establish business impact

There are two separate experiments. The first collects randomised arm data to learn which offers work. The second tests the complete proposed policy, with its eligibility, delivery, budgets and no-offer choices, against the current one. A policy chosen offline stays a proposal until the second experiment supports its value.

Before running either, specify the population, assignment unit, outcome horizon, primary metric, smallest useful effect, sample-size/power assumptions, guardrails, duration and stopping rule. For several arm comparisons, state the multiple-comparison correction. For sequential monitoring, use a valid sequential method, not repeated stops on an ordinary p-value. Large profit outliers and cluster assignment affect power, so plan with historical outcome distributions and the real assignment design, not an arbitrary “run it for two weeks” rule.

For the primary intention-to-treat analysis, compare groups by randomised assignment. Comparing only delivered, opened, exposed or redeemed offers selects on post-assignment behaviour and can break randomisation. Keep delivery failure as a diagnostic. Before accepting the lift estimate, check assignment counts, missing outcomes and sample-ratio mismatch. Microsoft's SRM guide shows how biased inclusion can reverse an experiment's conclusion.

Shared household coupons, limited stock, contact-centre queues and marketplaces can create interference, where one customer's treatment affects another. Decide whether account, household, store, geographic cluster or a carefully designed time-block experiment fits the mechanism, then adjust inference and power for that unit. A time-block design may need washout for carryover. Microsoft's pre-experiment guidance discusses stable identifiers and network effects. Cluster assignment also changes precision and the effect being estimated.

8. Add bandits only when the foundation is ready

A contextual bandit learns which action to choose while deliberately trying alternatives. It can help with frequent decisions, a manageable action space and reliably observed rewards. It is an advanced, optional extension of this project.

Before considering one, show:

• A feasible action set, including no offer, and a bounded or suitably modelled reward tied to net contribution, not clicks alone.
• An exploration budget, a durable randomised holdout, and the exact probability of the selected assignment, recorded after eligibility and budget rules and kept if execution later fails.
• Correct delayed-reward joins, complete follow-up for mature zeros and returns, and a policy for late or missing feedback. “Not received yet” is different from zero reward.
• Offline support and uncertainty diagnostics, plus a valid method for evaluating adaptive logs.
• A kill switch and deterministic fallback with working cost, consent and fatigue controls.

For changing action sets, Vowpal Wabbit's contextual-bandit tutorial covers action-dependent features and action/cost/probability logging. Exploration methods such as epsilon-greedy or Thompson sampling must stay within feasible actions. The right one depends on the reward model, traffic and acceptable cost.

A bandit sees only the chosen action's reward. Past deterministic assignments give no overlap for the alternatives. A fixed reward horizon matters too: a bandit optimised for clicks can learn to buy clicks while losing money. With long delays, changing price or stock, or large effects on future customer state, a fixed randomised experiment with periodic policy refreshes may be easier to trust. If you move to reinforcement learning for long sequences, justify the state, reward, transition evidence and evaluation design separately.

9. Serve, log, reconcile, and roll back

For a scheduled campaign, start with a batch decision job if that meets the business need. Use an online API when context, stock or eligibility changes during the customer journey. Both should share the same accounting, rule evaluation and logging logic.

The flow shows a new decision. A retry replays the saved decision and reuses or completes its idempotent reservation.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Offer decision serving flow
    accDescr: Eligible requests pass through scoring, resource checks and durable logging before execution, with a no-offer fallback and mature-outcome review.
    R["Request + available<br/>features"] --> E{"Eligible?"}

    M["Score actions<br/>including no offer"] --> B{"Resources?"}

    L["Persist decision<br/>+ probabilities"] --> D["Reserve + deliver once"]
    D --> O["Mature profit<br/>+ guardrails"]

    E -->|Yes| M
    E -->|No| F["No offer fallback"]
    M -->|Error| F
    B -->|Yes| L
    B -->|No| F
    F --> L

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class R,L,O data;
    class M model;
    class E,B decision;
    class D execution;
    class F neutral;

The eligibility gate checks eligibility, contact rules and input freshness. The resource gate checks budget and stock before the final action draw. Either gate failing, or a model error, selects no offer. On both paths, save the original assignment, chosen action and real post-constraint assignment probabilities. Then recheck resources and reserve budget, atomically and idempotently, before delivery. In this order, a crash during reservation or delivery still leaves a saved decision to recover.

If the execution recheck fails, append the no-offer result and reason, keeping the original assignment. Record assignment and execution separately. Assignment probabilities belong to the assigned action, not the one that overrode it. Mature outcomes feed review and controlled policy updates.

An example POST /recommend-offer request contains a customer/account ID, channel, decision time and idempotency key. The response contains decision_id, action ID and catalogue version, expiry, reason code and policy version. Keep debug estimates in restricted logs. Internal causal or model terms need not appear in the offer copy. A repeated request with the same key must return the same decision, reserving budget and sending a message only once.

Save the decision before delivery, and use a durable outbox or similar retry design. Record the candidate set, scores, real selection probabilities, assignment and any fallback. A sample log record might be:

{
  "decision_id": "d_123",
  "customer_id": "c_456",
  "decision_at": "2026-10-08T06:00:00Z",
  "feature_snapshot": "snapshot_abc",
  "eligible_actions": ["no_offer", "reminder", "coupon_10"],
  "selected_action": "coupon_10",
  "action_probabilities": {"no_offer": 0.25, "reminder": 0.25, "coupon_10": 0.5},
  "policy_version": "offer_policy_v3",
  "catalog_version": "catalog_v7",
  "experiment_id": "offer_collection_01",
  "budget_reservation_id": "reservation_789",
  "fallback_reason": null
}

These probabilities show one collection policy, not a split to use everywhere. Log the real probabilities from the selection mechanism you built. In a deterministic policy, the chosen action has probability one and the others zero, which limits later evaluation. For a multi-item slate, log the joint or sequential selection probabilities your evaluation method needs. Item scores that look independent are not a slate propensity.

On a timeout, stale features, invalid output, exhausted budget or a failed stock check, return the defined safe rule-based decision or no offer. Do not reuse an expired coupon. Log the cause, and keep retry behaviour intact. Monitor input freshness, delivery, p95 latency, contact-limit and budget breaches, action mix, observed costs, outcome maturity, calibration, complaints and experimental incremental value. Drift alone is a reason to investigate, not an automatic reason to retrain.

Roll out in steps: log replay, shadow scoring, an A/A logging check where you can, and a limited randomised policy test. Increase traffic only once the predeclared criteria are met and outcomes have matured. Rollback restores the previous policy and rules, not just the model binary. Version the training cutoff, feature definitions, catalogue, calibration, constraints and policy so any decision can be rebuilt.

10. Choose practice data without overstating the result

Criteo's official page describes an advertiser leakage problem in the original release and links the corrected one. Use the corrected version, keep assignment proportions when sampling, record the file checksum, and keep enough examples in both arms. Treat treatment as the assignment. exposure happens after assignment, so do not use it to filter the causal sample or as a baseline feature. The public features are anonymised, and non-uniform sampling hides the original incrementality. Benchmark estimates do not translate into the original advertiser's business lift. The original assignment and sampling logs are unavailable, and the released arm proportions are not the original logging propensities. For any benchmark policy estimate, state the assignment and sampling assumptions, test sensitivity, and validate the estimator on known-propensity simulations. The release lacks the dated campaign history, multiple coupon actions and profit accounting that the full retail decision contract needs.

For an undated benchmark, use a suitable held-out split and say that it cannot test future-period performance. You can add a dated simulator to exercise feature availability and deployment logic, but keep its results separate. Check each dataset's terms before use. Publish code, a manifest and download instructions instead of casually bundling source data.

Practise with the reference lab before a benchmark. The reference lab's evaluation report compares risk ranking with known-effect ranking at a fixed contact budget. Its simulated randomised experiment keeps failed deliveries in their assigned arm. Use it to check incremental-value arithmetic, budget allocation and the intention-to-treat analysis. Its effects are simulator assumptions, not evidence for a real offer.

What to build

An offer decision system that is clear about what it can show. Extend your existing project where the data fits, or use the corrected uplift benchmark plus a clearly labelled operational simulator. Keep the 25-hour scope to a few actions and one main boosted-tree comparison. Two-tower retrieval, causal forests and bandits are optional extensions once the core system works.

Deliver these:

1. A decision contract and accounting example, covering no offer, constraints, horizon and guardrails.
2. A data manifest and feature/label builder that checks decision-time availability, maturity and outcome coverage, plus one versioned net-reward definition.
3. Rules and arm-mean baselines, logistic regression and one boosted tree, and a supported uplift or value-policy comparison where the data allows.
4. An evaluation report with split dates or the benchmark's limits, support counts, calibration, policy value and uncertainty, cost sensitivity, and results for sparse-history customers.
5. A batch job or API with an offer catalogue, idempotency, shared contact controls, budget reservations, decision/outcome logs and a tested fallback.
6. A policy experiment memo and rollout/rollback runbook, including the proof still needed.

Useful correctness checks test real failure modes. A late-arriving event stays out of an old feature snapshot. An immature or mature-but-missing outcome is not called negative. An ineligible or negative-value offer is not sent. Concurrent requests stay within a hard budget, and retries send only once. Changing the profit formula changes the chosen action in the worked example. Check propensity normalisation over the real feasible set, and reject logged assignments with impossible zero probability. Simulate a failed reservation after assignment: keep the original action and probabilities, log the no-offer execution separately, and include the account in the assignment analysis.

Before you call it complete

• You can tell relevance, response probability and incremental profit apart with one example.
• Every input and outcome traces back to its decision and availability time. Missing outcomes are reported, not quietly turned into zeros.
• You have an explicit no-offer option, a comparison with the current policy, and reproducible eligibility.
• You state whether effects come from randomisation, from defended observational assumptions, or are only predictive/simulated. You do not claim observed individual treatment effects.
• Your frozen policy has a held-out evaluation, uncertainty, action support and accounting sensitivity. A result without enough support stays a proposal.
• Delivery, budget, fatigue, retries and fallbacks work together, and one decision can be replayed.
• You can explain how a policy experiment shows impact, and what would stop or reverse a rollout.

A project can be complete as a portfolio exercise while its deployment hypothesis is still untested. A business rollout is complete only when enough experiment results, operational controls and ongoing monitoring support it. Make that difference clear in the README.

Back to Chapter 9  ·  Next: Price recommendation

About 25 hours for the learning project.  ·  Chapter 9  ·  Index  ·  Resources

The short version. Start with an interpretable demand model, compare it with a boosted tree, and choose among prices the business has approved. The hard part is estimating demand if you changed the price. Accurately predicting past prices or sales does not, by itself, answer that question.

My suggested first approach is a log-price Poisson or negative-binomial GLM, then a CatBoost or LightGBM demand challenger, with a separate constrained profit optimiser. This is a practical starting point, not a universal ranking. Your evidence for price effects, available price variation, demand volume and operational constraints decide which model you can safely use.

In this guide: Data contract · Model choice · Causal evidence · Price policy · Delivery · Practice lab

Your project plan

Use the splitting and API habits from Chapter 3 and the serving and monitoring habits from Chapter 6. Real pricing experiments, approvals and waiting for returns take extra calendar time.

Read the causal primer first. The schedule assumes prepared opportunity data, your earlier environment and an existing API boundary. Rebuilding data from scratch or a new serving integration needs extra setup time. Finish the approved fixed-price baseline, supported demand comparison and constrained grid before the optional 2026 research extensions.

1. Define the decision before the model

Choose one unit, such as SKU × store × next day, and a fixed decision horizon. Say whether the system recommends a posted price, a markdown, a subscription tier, a quote or a negotiated price, since each has different actions, exposures and outcomes. This guide assumes one shared posted price for a product and market over a stated interval.

Separate three questions:

Write the intervention as do(price = p): set the price while keeping the specified offer, placement and availability policy fixed. Decide whether the objective is contribution profit, clearance sell-through, revenue under a margin floor, or a longer-horizon customer objective. No single objective fits every case. Agree the currency, tax treatment and horizon with the business owner.

For a one-period SKU decision, a useful simplified objective is:

profit(p, x) = (net_unit_receipt(p, x) - variable_unit_cost(x))
              × E[min(D(p, x), available_inventory) | do(price = p)]
              - expected_return_cost(p, x)
              - price_change_cost(p, x)

p* = argmax profit(p, x), over approved and supported prices only

Here D is latent customer demand (true demand, before stock limits) and x holds what is known at decision time. Count refund losses once: inside net receipts or in a separate return term. Include fulfilment, payment fees and other variable costs consistently. Fixed costs may cancel between prices, but state that assumption. With scarce inventory, E[min(D, inventory)] generally differs from min(E[D], inventory), so a mean-only estimate can misprice a product with limited stock.

A daily optimiser can also sell stock cheaply today that would be worth more tomorrow. Start with products where a one-period approximation is credible. Add a multi-period inventory model only when replenishment, perishability or customers waiting for a better price require it. Dynamic pricing with limited supply shows why capacity changes the decision problem.

2. Build a table of opportunities, including non-purchases

Transactions alone miss people who saw the price and chose not to buy. Build a row for every eligible product-market-period, using posted-price and availability logs to create zero-sales rows. A missing row is not automatically zero demand.

Measure exposure carefully. For visit-level conversion, a customer who does not buy is a valid negative example. For aggregate counts with unequal exposure, a GLM can use a log-exposure offset. The scikit-learn Poisson example shows the related rate-target and exposure-weight set-up. Zero exposure means no chance to observe demand and cannot go through log(0).

Exposure can itself respond to price. Search placement, traffic or customer visits may change after a price change. Conditioning on that realised exposure can hide part of the total price effect. State whether you estimate conversion given exposure or total period sales. For a total effect, use pre-change exposure forecasts or model traffic jointly.

Sales are not demand during a stockout. Selling the last ten units tells you demand was at least ten, not exactly ten. Keep censored periods and mark them. When they matter, use a censored likelihood or an availability-aware choice/demand model. An uncensored subset is fine for a narrow first baseline if you state its selection limits. Dropping stockouts can leave mostly low-demand periods. Availability also changes substitution across products, as Conlon and Mortimer show.

Check support for inventory levels too. If comparable observations were always capped at ten units, they cannot show the demand tail needed to value an inventory of twenty without extra assumptions. Lower-stock decisions may need the tail only up to their own cap. Mark model-based extrapolation clearly and abstain when the needed tail or censoring assumptions are unvalidated.

Include day of week, holidays, seasonality, item age, category, store and lagged demand, using only information available before the recommendation. Compute rolling features separately inside each validation history. A future realised competitor price, end-of-day inventory, post-change page views or a future return flag is not a pre-decision feature. Event time and availability time must both come before the decision: a backfilled Monday transaction first loaded on Wednesday was unavailable to Tuesday's recommendation.

Promotions, loyalty benefits and offer recommendations change the effective price. Track them together, including who funds a discount. Optimising posted price and discount separately can spend the same margin twice. Cross-item prices and availability matter for substitutes and complements, so begin with an isolated category and test that assumption.

3. Use a model ladder with clear promotion conditions

Model-selection decision tree

Choose a demand estimator first; a separate constrained optimiser picks among approved prices. “Logs ready?” means usable demand, price exposure, cost and inventory histories. “Effect + support?” needs defensible identification and observations near the candidate prices, not a good forecasting score. For identified, uncensored demand, fit the GLM baseline before judging whether its remaining errors justify a more flexible challenger.

1. Check data, evidence and inventory readiness

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Check readiness for price-response modeling
    accDescr: Repair missing logs, retain forecasting when causal support is absent, and separate inventory extensions before continuing to the GLM baseline.
    R{"Logs<br/>ready?"}
    I{"Effect +<br/>support?"}
    S{"Stock<br/>issues?"}
    A["A<br/>Repair<br/>logs"]
    B["B<br/>Forecast<br/>only"]
    C["C<br/>Inventory<br/>extension"]
    G["Fit GLM<br/>Next tree"]

    R -->|No| A
    R -->|Yes| I
    I -->|Yes| S
    I -->|No| B
    S -->|Yes| C
    S -->|No| G

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class R,I,S decision
    class A data
    class B neutral
    class C,G model

The letters link each outcome to its requirements below. “Stock issues?” covers sales censored by stockouts and decisions tied to future inventory or replenishment, which need a separate extension. Having a model does not authorise a new price, and more rows cannot repair missing identification or price support. The “Fit GLM” path continues in the promotion tree below.

2. Decide whether the GLM needs a challenger

Follow this tree after the readiness checks above. Fit the GLM first, then use forward error analysis to decide whether a boosted comparison is worth running. Choosing a challenger does not show it beats the baseline.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Compare a boosted demand challenger with the GLM
    accDescr: Retain the interpretable GLM unless nonlinear errors justify comparing a boosted demand challenger on forward validation.
    G["Fit log-price GLM"]
    N{"GLM errors<br/>nonlinear?"}
    D["D<br/>Keep GLM"]
    E["E<br/>Boosted<br/>challenger"]

    G --> N
    N -->|No| D
    N -->|Yes| E

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class G,D,E model
    class N decision

• A · Repair the logs; keep the current approved policy. Model: no new price-response model yet. Missing exposures, costs or inventory make the decision target unreliable. Required: posted prices, exposed opportunities including non-purchases, timestamped costs and stock/availability. Proceed when reconciliation and readiness checks pass, and forecast only what the records support. See the data contract.
• B · Forecast only; gather intervention evidence. Model: a seasonal/count demand forecast, keeping the current approved pricing policy. Forecasting history cannot identify demand under a price change. Required for promotion: a bounded, approved price experiment or another defensible identification design, with local support for the proposed grid. Promote only within that identified, supported price region. See price-effect identification and the experiment design.
• C · Add a censored-demand/inventory extension only when supported. Model: a justified latent-demand distribution, with censor-aware estimation when sales are censored, plus a separate inventory optimisation model when decisions interact over time. Realised sales alone cannot reveal demand above available stock. Required: stock and availability histories, exposure, price/cost state, identifiable price effects and a validated observation/censoring model. Dynamic decisions also need replenishment and transition evidence. Promote after validating the observation model and demand-distribution calibration under the inventory rules; otherwise abstain from the unsupported price change. See the stock-censored research and simulator extension.
• D · Start with a regularised log-price Poisson or negative-binomial GLM. It gives an interpretable count-demand baseline, handles zero counts and can model exposure explicitly. Required: repeated product-market observations, valid exposure denominators, uncensored demand for this baseline, credible supported price variation and documented identification assumptions. Check forward performance, dispersion and uncertainty. See the elasticity baseline.
• E · Compare CatBoost or LightGBM with that GLM. Rich features and nonlinear seasonal/product interactions can justify a flexible demand challenger. Required: the same identified price evidence and support, features available at decision time, and a held-out forward comparison of accuracy, calibration, response curves and operating cost. Keep the GLM if the challenger adds nothing useful. See the boosted demand challenger.

Only a supported demand response moves on to the constrained price policy. There, feasibility, predictive risk and selection-adjusted improvement gate every candidate before selection. This tree organises the existing project work. Inventory and other advanced extensions stay outside its 25-hour practice budget.

These promotion gates are engineering recommendations. Model documentation sets out supported losses and estimator assumptions, not which model wins on your data.

The elasticity baseline

For counts, fit the conditional mean instead of taking the log of observed zero sales:

log E[units | p, x] = log(exposure)
                     + item_effect + store_effect + calendar_effect
                     + beta × log(p) + other_pre_decision_features

Here beta is the elasticity of the conditional mean: a 1% price increase goes with roughly a beta% change in demand, holding the modelled features fixed. An estimate of -1.4 suggests about a 1.4% decrease. It is a causal elasticity only under the identification assumptions in the next section.

A Poisson log-link handles zero counts naturally. Check for overdispersion (variance above the mean); a negative-binomial model may fit it better. Use partial pooling or regularisation when many products have little price variation. If you fit log-log regression on positive sales instead, state that zeros are excluded and handle retransformation bias. log1p(units) loses the usual elasticity meaning. statsmodels documents the GLM families and links.

Check how your implementation handles dispersion. In statsmodels' negative-binomial GLM family, you supply alpha; choosing the family does not estimate it. Estimate or tune dispersion on training data with a suitable count model, then check the tails on held-out data. The family documentation states this contract.

The boosted demand challenger

CatBoost is a useful challenger when product, store and category identifiers matter most. LightGBM suits larger tabular panels. Both support Poisson and quantile objectives, as the CatBoost loss documentation and LightGBM parameters show. Choose by careful validation and operating needs, not a generic leaderboard.

Use a count loss for units. For zero-heavy, nonnegative continuous spending, compare a Tweedie or two-part model where suitable. A hurdle model can separate whether someone buys from how many units, but the extra complexity needs validation. Keep the response scale explicit: a library may return a raw log score or a transformed mean, depending on prediction settings.

Inspect demand curves across the supported price grid in real customer-market contexts. A monotonic constraint (demand falls as price rises) can encode a reasonable own-price assumption, but it cannot fix confounding. Check that your objective supports that constraint in the pinned library version. Flexible trees extrapolate poorly beyond observed prices, and a smooth curve does not prove a price change will work.

A boosted Poisson-loss model estimates a conditional mean. That objective alone does not validate a Poisson demand distribution, and a few fitted quantiles are not a complete distribution either. For inventory or downside-risk calculations, supply a coherent nonnegative demand distribution or a justified tail estimator, and validate dispersion, tail calibration and interval ordering. Keep uncertainty in the fitted mean separate from future demand variation.

When advanced models earn their place

For strong substitutes, estimate category demand and the no-purchase outside option, instead of maximising each SKU's apparent profit separately. A choice model's assumptions and feasibility matter as much as its fit. Personalised individual pricing is a separate product-policy decision that a pooled product-market demand model does not need.

Use a bandit only when exploration has been approved for the defined market and price grid. Start with a randomised fixed-grid experiment. Later, a contextual bandit can give more exposure to promising allowed prices while still exploring. Log the chosen action's real probability after all filters, inventory checks and policy constraints. The Vowpal Wabbit tutorial explains the context/action/reward/probability contract. A bandit optimising immediate purchases can lose money after discounts, returns or longer-term effects, so define a matured contribution reward.

Delayed feedback changes the learning contract. Declare a fixed return/settlement horizon and an allowance for late data. Keep recent decisions pending instead of giving them zero reward or updating only on quick outcomes. A whole decision cohort matures under one rule, and its reward may still miss later returns, so state that limit. Delays or missing outcomes can depend on price and reward. The peer-reviewed ICML study of unrestricted delay distributions separates reward-independent from reward-dependent delays, and ordinary bandit updates are not automatically valid in both. Start with scheduled updates on mature cohorts and an incumbent fallback. More elaborate delay-aware learning is an optional extension.

For a faster purchase or revenue proxy, individual-level correlation with settled profit is not enough. Check, with uncertainty, whether the randomised price effects on the proxy point the same way as those on matured contribution. The August 2026 delayed-feedback workshop paper by Lee and colleagues examines reward alignment and learning efficiency separately. Its marketplace example is underpowered, later overall lift is null and positive subgroup findings are exploratory, so it does not validate a retail pricing reward. Compare against an approved fixed-price policy, and confirm any proxy-driven policy with experimental settled-profit outcomes.

Optional depth: what the 2026 research adds

E[min(D, I) | price, context] = sum from q=1 to I of P(D >= q | price, context)

4. Identify the price effect before optimizing it

Managers often lower prices when they expect weak sales and raise them when demand is strong. Past prices then carry demand information the model may not see. Price and sales can rise together even when raising the price reduces demand. Fixed effects, SHAP and better forecast scores do not fix this on their own.

Prefer a randomised price experiment within a small approved grid, stratified by product, market and relevant calendar blocks. Run a concurrent current-policy control. Record assignment, realised exposure, overrides and outcomes. Define the primary effect as assignment to the pricing policy, so overrides cannot quietly change the population being compared. Fisher, Gallino and Li give a controlled retail example that uses randomised prices to address this endogeneity problem. Its reported improvement is specific to that setting, not a forecast for yours.

For observational estimation, write down an identification argument before choosing the estimator:

• Adjustment/DML: all common causes of price assignment and demand are recorded, enough price variation remains within comparable contexts, and treatment and outcome definitions are consistent. Avoid conditioning on anything the price change itself affects. DML cross-fitting reduces some estimation biases, but it cannot reveal missing confounders. The EconML DML documentation also sets out structural assumptions. A continuous-treatment causal forest estimates effects within those assumptions, and a local marginal effect is not automatically a complete nonlinear demand curve.
• Instrumental variables: the instrument changes price, is independent of unobserved demand shocks given the stated controls, and affects demand only through price. Report first-stage strength, the population and variation identified, and any extra functional-form or monotonicity assumptions. A competitor's price or your cost is not automatically a valid instrument, because common shocks, advertising and availability can create other paths to demand. EconML's instrument guide describes this distinction. A more recent competitive-pricing study, Jiang and Li, combines experiments and instrumental estimation in its own setting.
• No credible identification: use the model only as a forecast or analyst simulator, keep the approved current policy, and design an experiment. Label counterfactual numbers as unvalidated scenarios. Adding CausalForestDML to the code does not change that evidence gate.

For continuous price, support means real variation near the proposed price within comparable contexts, not just lying between the global minimum and maximum. If every item was always priced differently, cross-item variation cannot identify an item's own-price effect. Plot price variation by SKU, store and season, and show which decisions must abstain.

5. Validate forecasting, response and business value separately

Prediction validation. Use forward rolling windows on the SKU-store panel. Use a later final period once; tune only on earlier windows. Add held-out stores or products when deployment must serve unseen groups. A group split alone cannot prevent temporal leakage, and a time split alone cannot test cold-start products. Fit encoders and transformations within each training window. Use dependence-aware folds for causal nuisance models, since library defaults may assume independent rows.

Report seasonal-baseline, GLM and challenger results on the same rows. Use count deviance and MAE, with bias and observed-versus-predicted totals by SKU, store, season, stock availability and price band. MAPE breaks at zero demand. WAPE is undefined when total actual demand is zero and can hide poor low-volume performance. Keep uncensored demand checks apart from realised sales checks, and state what is being forecast.

Response validation. On randomised or otherwise identified held-out data, compare demand and profit by assigned price, show uncertainty, and inspect monotonicity and heterogeneous effects. Report how much supported price variation remains after controls. Good MAE under the current price policy does not validate a new price curve.

Uncertainty. Keep three things apart: variation in future customer demand, uncertainty about the estimated causal mean response, and uncertainty about the policy's profit improvement. Quantile models can estimate demand prediction intervals; check their coverage, width, pinball loss and calibration on later data and operational slices. Check distributional count models for tail behaviour and stockout probability too. A refit/cluster bootstrap or estimator-specific inference can measure response-estimation uncertainty under the relevant assumptions.

Ordinary conformal coverage relies on assumptions that temporal drift may break, so rolling coverage checks and adaptation are needed. Gibbs and Candès study adaptive conformal inference under shift. Neither conformal prediction nor quantile calibration gives causal validity or guaranteed coverage for an unsupported candidate price.

Coverage after choosing a price. Per-price average coverage may not hold once a policy picks different prices in different contexts. The July 2026 Zheng and Jin preprint studies this with policy-coupled coverage under i.i.d. logging, supported identified actions and known assignment probabilities, separating model fitting, policy learning and calibration. It is optional research. Validate the frozen policy's selected-action coverage on independent identified data with a justified weighting or design strategy. Pooled coverage under the current policy is not enough, and a multiplicity adjustment for mean improvement does not solve this predictive-risk problem.

A made-up counterexample makes this concrete. Two prices each miss five of the same 100 contexts, on separate sets, so each has 95% coverage. A policy that picks each price in exactly the contexts it misses has only 90% selected-price coverage. These are constructed outcomes, not observed customer counterfactuals, but an optimiser can unintentionally favour poorly covered contexts.

Policy validation. Compare against the current pricing rule, with matured contribution profit per randomised unit and horizon as the primary outcome. Also track sell-through, revenue, stockout rate, returns, complaints, repeat purchase and constraint violations. Estimate uncertainty at the assignment unit: thousands of customer events in one treated market are not thousands of independent treatment assignments.

Offline policy evaluation needs the candidate policy's actions to have support under the logged assignment mechanism, and valid reward/assignment assumptions. On a finite price grid, IPS and doubly robust estimates can help. Inspect propensity errors, extreme weights, effective sample size and interval width. Dudík, Langford and Li describe the doubly robust approach, which cannot repair missing support or unmeasured confounding. Generic IPS on logs from a deterministic current policy cannot evaluate a different deterministic price. Continuous-price evaluation needs suitable density or smoothing assumptions, and binning prices afterwards does not create randomised data. A model-only profit simulation is not an observed lift.

Evaluating a fixed learned policy differs from evaluating an adaptive learner that updates as feedback arrives. The learner's actions depend on its history, so a one-shot IPS/DR score for its final policy misses cold-start learning, pending rewards and inventory depletion. For those claims, use a justified sequential evaluation design or a bounded live trial.

6. Optimize a constrained grid, with an explicit abstain path

Choose a small price grid approved by the relevant business owners. Apply constraints before ranking and again after selection:

• Price floors/ceilings, positive unit contribution where required, and a margin floor.
• Maximum change from the current price, minimum time between changes, rounding and price endings.
• Inventory, replenishment, contractual and campaign restrictions.
• Consistent customer-facing price/quote rules and any applicable organisational policies.
• Valid cost, competitor and feature timestamps, and candidate prices within identified support.

Decide what happens when the current price itself is no longer allowed. A fallback must come from a currently approved policy. If there is none, return review_required with no executable price. The optimiser cannot invent an exception to restore availability.

The policy flow

Before comparing expected profits, pass every candidate through feasibility, local support, predictive risk and a selection-adjusted improvement rule. Each check keeps only passing prices. The identification gate uses the design and assumptions in Section 4, not a predictive score.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Price recommendation policy
    accDescr: Every approved price must pass support, feasibility, predictive-risk and selection-adjusted improvement checks before selection; otherwise use a valid approved fallback or request review.

    checks["Readiness + identification<br/>Gate every candidate"]
    passing{"Any pass?"}
    best["Highest mean profit"]
    recheck{"Still valid?"}
    apply["Log, then apply once"]
    fallback["Valid fallback<br/>or review"]

    checks --> passing
    passing -->|Yes| best
    passing -->|No| fallback
    best --> recheck
    recheck -->|Yes| apply
    recheck -->|No| fallback

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px
    class best model
    class checks,passing,recheck decision
    class apply execution
    class fallback neutral

The first stage checks input readiness and causal identification, then tests every approved price for feasibility, local support, predictive risk and selection-adjusted mean improvement. If a precondition fails, no candidate passes. Pick the highest expected profit among the survivors only, recheck the current state, and log durably before delivery. Revalidate the approved fallback before using it. If none is still valid, execute nothing and return review_required.

A worked example

Assume a cost of ₹60 per unit, enough inventory, no extra fees or returns, and the following illustrative identified mean demand for one day. These made-up values show the arithmetic, not an elasticity estimate from a public dataset.

Under these assumptions, ₹110 has the highest expected contribution profit: ₹150 above the current policy, about 3.26%. ₹90 maximises the table's revenue, a different objective. A 5% change limit from ₹100 would rule ₹110 out. If uncertainty about the profit difference still includes an unacceptable loss, a conservative rule would keep the approved fallback, even though ₹110 has the highest estimated mean.

Try it yourself. A supplier cuts the unit cost to ₹50 and the demand estimates are unchanged. Which approved price now has the highest expected contribution profit, and what should the policy recommend?

Use paired estimates of profit difference for a conservative improvement bound. Subtracting the ends of two unrelated prediction intervals does not give a confidence interval for lift. Choose minimum improvement and loss thresholds before seeing experimental results. To pair estimates, refit on the same bootstrap sample, or take the same parameter-posterior draw, for every price. This captures covariance in the estimated mean responses. Randomised price data can identify each price's marginal outcome distribution under the design assumptions, not one person's joint potential outcomes across prices.

To estimate model uncertainty, refit the demand and nuisance estimators with a dependence-aware resampling scheme. Resampling fixed predictions alone misses that fitting uncertainty. Label a posterior bound as conditional on its model and prior, not as a distribution-free confidence guarantee. If live monitoring tests for improvement repeatedly, fix an analysis schedule in advance or use justified sequential inference. A fixed-sample interval does not allow unlimited peeking.

# Pseudocode: response inference and predictive risk are separate dependencies.
def recommend(context, policy, response_model):
    fallback = policy.valid_fallback(context)  # Can be None: then review_required.
    if not context.features_fresh or not context.costs_valid:
        return abstain(fallback, reason="stale_or_invalid_inputs")
    if not response_model.price_effect_identified:
        return abstain(fallback, reason="unvalidated_price_effect")

    candidates = [
        p for p in policy.approved_prices(context)
        if policy.allows(p, context)
        and response_model.has_local_support(p, context)
    ]
    if not candidates or fallback is None:
        return abstain(fallback, reason="no_supported_feasible_action")
    if not response_model.has_local_support(fallback, context):
        return abstain(fallback, reason="cannot_compare_with_fallback")

    # Each refitted bootstrap model (or parameter draw) estimates each price's
    # marginal demand distribution. Integrate that distribution to calculate
    # expected profit, applying inventory and costs once. No paired customer
    # outcomes across counterfactual prices are assumed.
    fits = response_model.inference_fits(context)
    mean_profit = policy.integrated_profit_by_fit(
        fits, prices=candidates + [fallback], context=context)
    passing_prices = {}
    for price in candidates:
        # Predictive variation governs operating risk, such as stockouts or
        # a future-period loss limit. Check every candidate before selection.
        demand = response_model.predictive_demand_distribution(price, context)
        future_profit = policy.predictive_profit_distribution(demand, price, context)
        if not policy.operating_risk_acceptable(future_profit):
            continue
        mean_gain = mean_profit[price] - mean_profit[fallback]
        if not policy.mean_improvement_supported(
                mean_gain, candidate_family=mean_profit):  # Selection-adjusted rule.
            continue
        passing_prices[price] = future_profit
    if not passing_prices:
        return abstain(fallback, reason="no_candidate_passes_evidence_and_risk")
    best = max(passing_prices, key=lambda p: mean_profit[p].mean())
    if not policy.allows(best, context):
        return abstain(fallback, reason="constraints_changed")
    return decision(best, alternatives=candidates, mean_profit_summary=mean_profit,
                    predictive_risk_summary=passing_prices[best],
                    evidence=response_model.evidence_version)

The support check and improvement rule are contracts you implement, not magic confidence scores. Define their diagnostics, estimation assumptions and thresholds in the policy card, and record both gates there. A full future-profit distribution supports risk decisions; statistical confidence in mean lift is a different calculation. Validate the marginal predictive distributions used for risk, and let the mean-improvement inference account for picking the best of several candidates. operating_risk_acceptable must reject an unvalidated distribution or inventory regime, so a mean forecast cannot quietly pass as a predictive-risk certificate. Include estimation uncertainty and plausible cost/return scenarios in the risk sensitivity checks. Apply the adjustment to the full supported, feasible candidate family, and do not narrow it after seeing risk or profit estimates. State may change after scoring, so revalidate cost, inventory and policy version when applying a recommendation.

7. Serve recommendations that can be replayed

A daily batch job is usually enough for a first posted-price system. Version the response model, policy configuration and price application separately. Use an API when operators or downstream systems need a consistent recommendation contract.

{
  "request_id": "price-2026-10-08-001",
  "sku_id": "SKU-17",
  "market_id": "STORE-3",
  "decision_at": "2026-10-08T08:00:00+05:30",
  "horizon_hours": 24,
  "currency": "INR",
  "current_price_minor": 10000,
  "cost_snapshot_id": "cost-17-v4",
  "inventory_snapshot_id": "stock-3-v21",
  "policy_version": "pricing-v2"
}

Validate currency, positive prices, horizon, timestamps and snapshot ownership. Resolve approved prices and constraints on the server, so the caller cannot relax them. Money uses integer minor units, with currency-specific rules defined in the contract.

Return decision_id, status (recommended, fallback, review_required), recommended or null price, validity interval, supported alternatives, expected contribution, uncertainty semantics, reason codes, model version, policy version and snapshot versions. A demand interval must state its unit, horizon and nominal coverage. A mean-lift confidence interval must state the comparison policy and inference method. Avoid a generic percentage called “confidence.”

Save the pre-decision feature snapshot, grid, eliminated candidates and reasons, response estimates, selected price, assignment probability, operator override, applied price and actual exposure. Join purchases and matured returns using decision_id plus the decision unit and time interval. Log the action actually taken even if an operator overrides it, and keep the original experimental assignment for the planned analysis. Make retries idempotent, and stop an expired recommendation from overwriting a newer price.

Save a pending application record before delivery. Tie the idempotency key to the exact price, unit, interval and release, then check the current price and version atomically when applying it. Reject a changed or expired state instead of overwriting a newer decision, and reconcile an unknown delivery result before retrying. The response's alternatives are diagnostic, not permission to apply a price that failed a gate.

An operator override may have an unknown selection probability. Mark it clearly, and do not reuse the original randomisation probability as the override's propensity. These records can stay in an intention-to-treat analysis, but offline evaluation of actual actions needs a justified way to handle them.

Monitor stale inputs, cost changes, price application failures, recommendation/application gaps, abstention rate, supported-action coverage, price oscillation, margin-floor breaches and latency. Once outcomes mature, add profit, bias, demand interval coverage, returns, repeat purchase and complaints. Show outcome completeness by decision cohort, so a recent cohort does not look more profitable just because its returns have not arrived.

Set named owners and triggers for rolling back to the approved fallback, such as a sudden margin drop, broken stock logging or too many constraint violations. Retraining is a reviewed response to evidence, not the automatic answer to every drift alert.

8. Design an experiment around the market

Prices can affect neighbouring products, customer search and a competitor's response. A customer may also wait to buy until the next price change. Choose an assignment unit that contains the important interference: store, market or category clusters may work better than randomising each visit. If time switchbacks fit, randomise the blocks and allow for carryover, washout and replenishment cycles. Bojinov, Simchi-Levi and Zhao study switchback design under carryover assumptions. Simply alternating prices by day is not a valid randomised design.

Write a pre-experiment memo covering the action grid, allocation unit, primary profit metric, minimum detectable effect, number of independent clusters or blocks, duration, return-maturity window, customer-policy constraints and stop rules. Keep the same primary analysis if results disappoint. Analyse cross-item effects at category or market level when that is the business objective. A short-run best response to competitors differs from a lasting market effect, as competitors may react after the first test.

9. Build a bounded lab before proposing production

Use synthetic data with a known demand mechanism for the intervention lab. Public retail transactions can teach cleaning and forecasting, but they generally cannot show what a price change would cause. UCI Online Retail has transaction timestamps, quantities, unit prices and cancellations. Its documented schema lacks exposure to unpurchased prices, inventory and randomised price assignment, so do not label a regression on that dataset a validated optimal-price engine.

The simulator below is a specification for you to build; the repository does not include it. Practise first with the reference lab. Its pricing example has one product, three prices and assumed conversion probabilities, written to a price comparison and a pricing figure. It checks only the contribution arithmetic: it estimates no demand and has no confounded prices, stores, seasonality or stockouts.

For the 25-hour project, keep your generator to 20 products, three stores, 140 days and four approved prices per product. Use seeded generation and save the generator configuration. Model isolated products first, and leave assortment and multi-period inventory optimisation as extensions.

1. Generate exposure and count demand with a known log-price effect, store/product effects and seasonality. Hide the true parameters from the training pipeline. Use ample inventory for the core training, calibration and evaluation experiment, and check that no demand is censored. Observed units then equal demand, so the baseline has a valid uncensored target.
2. Generate two clearly labelled assignment regimes: randomised approved prices with known probabilities, and manager-chosen prices that depend on a hidden demand shock. The second shows confounding and is not causal training data. Include zero-sales rows and return delays, with some outcomes deliberately incomplete at the final observation date.
3. Fit the seasonal baseline, log-price GLM and one boosted demand model on forward windows. Keep separate later windows for calibration and testing. Compare observed sales forecasting and identified response estimation separately. Cluster uncertainty at the simulated assignment unit, and document the independence built into the generator.
4. Implement the price-grid optimiser, then compare its decisions with the simulator's known expected-profit optimum over the same feasible actions. Report simulated regret and abstention, plus matured profit. Score it on independent simulation draws, not on the optimiser's own predictions. Check predictive coverage for the selected actions as well as by price. The simulator supplies those outcomes without claiming observed individual counterfactuals. Freeze model, calibration and policy before the final test.
5. Run a separate stockout stress scenario with finite inventory and sales observed as min(latent_demand, inventory). Keep those records out of the uncensored baseline's training data, and show that it abstains outside its approved inventory regime. A censor-aware model is an optional extension: it estimates latent demand, and the optimiser applies the inventory cap once. Avoid training on capped sales as if they were demand and then capping that forecast again. In a failure exercise, remove exposure logs or stockout flags and show why validation rejects the data. Request an out-of-support price and change inventory after scoring, to test fallback and execution revalidation.
6. Serve one recommendation request, save its decision log and replay it from saved versions. Write an experiment memo for replacing synthetic identification with a real approved trial.

Deliver a reproducible generator, feature/data contract, split diagram, comparison table, response curves, interval-calibration report, optimiser, API example and policy card. Label every result “synthetic” or “observational” as appropriate. The simulator's known elasticity says nothing about a real retailer.

Before increasing scope

You are ready to explain the project when you can answer: what evidence identifies the price effect, which candidate prices are supported, why the selected price improves the stated objective, and what the service does when any of those conditions fails.

Previous: Offer recommendation  ·  Next: Churn  ·  Back to Chapter 9

About 25 hours for the learning project.  ·  Chapter 9  ·  Index  ·  Resources

The short version. Start with a documented churn definition, historical customer snapshots, a time-based backtest and logistic regression. My suggested first serious challenger is CatBoost for mixed categorical customer data, or LightGBM when its training and serving profile suits the workload. Calibrate the chosen classifier on a later, untouched cohort. Use survival analysis when follow-up is incomplete or the business needs several time horizons. To choose actions, use randomised retention experiments and uplift modelling. A model that predicts who will leave cannot, on its own, tell you who a discount will save.

These are starting recommendations, not one model that always wins. Choose the simplest candidate that improves the operational metric across future cohorts and that your team can maintain. Read Chapter 3 before training and Chapter 6 before connecting the result to a customer workflow.

In this guide: Decision and data · Model selection · Survival · Retention effects · Workflow · Completion gates

Your five-session plan

The hours cover a reproducible prototype and a written experiment plan. A retention trial needs real customers, permission to run the campaign, enough observations and time for outcomes to mature. Another afternoon with a public CSV cannot replace those.

The schedule assumes prepared histories, the earlier environment and a reused delivery boundary. Rebuilding source data or a new CRM integration needs its own time. Read the causal primer, finish the fixed-horizon baseline, and add survival or treatment-effect modelling when the question and evidence call for it. The reference lab gives a reproducible synthetic fallback and keeps simulated outcomes clearly apart from real campaign results.

If you only have a customer-level benchmark without dates, do the model-comparison exercise and report its limits. Avoid inventing prediction dates or claiming a temporal backtest. For the stronger project, use transaction or subscription histories.

1. Define the decision before choosing the model

Churn means different things in different businesses. Subscription cancellation is an observed event. A retailer usually sees only missing purchases and defines an inactivity proxy. These targets need separate names, horizons and evaluation.

These are example definitions. Set inactivity windows from buying cadence, seasonality and the time available to act. A 60-day gap can be normal for an annual gift buyer. Record whether churn means voluntary cancellation, payment failure, non-renewal or any termination. Payment recovery often deserves its own model and workflow. Define grace periods, refunds, pauses, account merges and reactivation. When a cancelled subscriber returns, start a new episode.

Keep the target ahead of an actionable decision. A cancellation reason entered after departure is leakage. A cancellation notice submitted before scoring can be a valid signal for a clearly defined cancellation-rescue workflow, but it changes the prediction problem. Keep that workflow out of a model that claims to predict unannounced churn.

Write down the comparator too: existing rules, random allocation at the same capacity, or the usual retention policy. "Higher accuracy" is not a decision objective.

2. Build a table that represents what was known then

The primary key is (customer_id, episode_id, as_of). This is a landmark snapshot: what you knew about an eligible customer at one decision time. The same customer can have many rows. They are correlated observations, not many independent customers.

Use two clocks when the source allows it. event_time records when an event happened. available_at records when the scoring system could first use it. A feature needs both to be at or before as_of. A payment dated Monday but loaded on Thursday was not available to Tuesday's score. Historical dimensions must also use the version available then, not today's plan or account status copied backwards.

Keep these fields alongside the features:

Build it in this order. Reconstruct eligible accounts at each scoring time. Join only the historical events and dimensions available then, and compute features. Join future outcomes into a separate label table and mark incomplete labels as unknown. Split by decision time, and fit all learned transformations inside the training folds. Save the query, extraction date, source versions and counts of excluded rows.

For fixed-horizon classification, use a cohort whose labels have all matured. Avoid mixing early known positives with immature, presumed negatives. If the outcome feed is complete through day 100, the latest fully observed 30-day snapshot is day 70. A snapshot on day 85 with no cancellation yet has an unknown label. Completeness means what had arrived when you trained, not the calendar date. Suppose a day-99 cancellation for the day-70 snapshot only arrived on day 103. The feed was then not complete through day 100 for a model retrained on day 100, so that label was unavailable to it. Use the completeness watermark recorded at training time. An ingestion allowance moves the eligible cutoff further back to cover such delays.

Good first features are simple aggregates whose source you can explain.

Run feature ablations and inspect suspiciously strong predictors. Drop post-churn status, churn reason, a supplied churn score, future redemption and hindsight lifetime value. A supplied "customer value" field needs an as-of definition before use. Avoid giving plan names an arbitrary order. CatBoost supports categorical inputs and advises against one-hot encoding them yourself. Its encoding still cannot repair a wrong time split or a column derived from the future. CatBoost categorical-feature documentation.

Earlier campaigns also change what the target means. A classifier trained on customers who received a mix of past campaigns estimates risk under that historical mix, not untreated risk. Keep the campaign records and report this limit. When the decision needs a no-campaign comparator, use a randomised control group.

3. Use a model ladder that matches the available evidence

Model-selection decision tree

Use the prediction tree to choose a risk formulation, then the action tree if the workflow will recommend a retention intervention. These stages compose: an R1 classifier or R2 survival model can supply risk for triage alongside an A1 action-effect model. Risk alone does not show who an offer will save, and causal modelling does not need a separately trained risk classifier.

1. Choose the prediction formulation

“Usable” means you can reconstruct eligibility, feature availability and outcome observation at a real decision time. Incomplete follow-up is usable for survival only when event or censoring times and the observation limit are recorded. A missing date is not a censored duration.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Choose a churn prediction formulation
    accDescr: Repair missing dates or outcomes; use mature fixed-horizon classification, or survival when recorded censoring or event timing matters.
    D{"Usable dates<br/>and outcomes?"}
    D -->|No| R0["R0<br/>Repair data"]
    D -->|Yes| Q{"Censoring or<br/>event timing?"}
    Q -->|No| R1["R1<br/>Classifier"]
    Q -->|Yes| R2["R2<br/>Survival"]

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    class D,Q decision;
    class R0 data;
    class R1,R2 model;

2. Add the retention-action stage

Use this second tree to decide whether to contact a customer, or which offer to deliver. Action effects need a randomised comparison, or a defensible identified design with its assumptions and overlap checked. A contact-response history chosen by the old policy is not enough on its own.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Add evidence for a retention action
    accDescr: Without identified action effects, use risk for triage and collect a randomised pilot; with valid evidence, evaluate uplift or doubly robust action models.
    E{"Action effects<br/>identified?"}
    E -->|No| A0["A0<br/>Triage + pilot"]
    E -->|Yes| A1["A1<br/>Uplift / DR"]

    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    class E decision;
    class A0 execution;
    class A1 model;

Read each leaf as a starting recommendation with an evidence gate, not an automatic release.

Model ladder

Fit and calibrate the classifier

For the first classifier, start with unweighted log loss and a small tuning budget. Keep class weighting only if it improves the operational tradeoff. Tune tree depth or leaves, minimum leaf size, learning rate, regularisation and the number of trees on earlier validation cohorts. Run early stopping there too, not on the final test. Apply the same feature availability rules to every challenger. Sequence models are an extra experiment, for when event histories hold information your aggregates miss and you have enough data to validate the added complexity.

For smaller tables, you can compare a released tabular foundation model using the Chapter 3 protocol. Report calibration and contact-capacity performance on the same future cohort, including sparse-history customers. Use a bounded tuning and inference budget. A leaderboard average does not decide this queue's value.

Handle imbalance inside the training process. Avoid SMOTE or undersampling before splitting. If you test resampling, apply it only inside each training fold. Keep the natural class balance in validation, calibration and test data. This avoids both leakage and evaluation on an invented prevalence. imbalanced-learn's resampling pitfalls.

Class weighting and sampling can change probability estimates even when ranking improves. LightGBM warns that is_unbalance and scale_pos_weight can give poor individual class-probability estimates. Use one or the other, not both together. LightGBM objective parameters.

After choosing and freezing the classifier, fit a sigmoid calibrator on a later matured cohort that the classifier did not train on. Isotonic calibration is a challenger when you have enough examples for a flexible fit. Assess reliability plots, log loss and Brier score on the untouched final cohort. A Brier score mixes calibration and discrimination, so inspect the reliability plot separately. A decimal from the API is not a probability until you check its calibration. Calibrators need out-of-sample predictions, and refitting the classifier afterwards can break the fitted mapping. scikit-learn calibration guide.

4. Validate the next campaign, with its real training cutoff

Use expanding or rolling chronological backtests. At each simulated retraining time, use only features and labels available by that cutoff. Then score a later cohort with that cohort's historical feature state. A chronological row split alone is not enough if training labels reach into the validation period and were unknown when that model would have been trained.

For a transparent first project, reserve successive time blocks for these purposes:

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Chronological churn validation
    accDescr: Separate successive cohorts for model selection, frozen-model calibration, policy selection and final untouched testing.
    subgraph Selection["Earlier cohorts"]
        direction LR
        T["Fit model"] --> V["Select<br/>+ freeze"]
    end
    subgraph Preparation["Later mature cohorts"]
        direction LR
        C["Calibrate"] --> P["Freeze<br/>policy"]
    end
    subgraph Holdout["Untouched future cohort"]
        E["Final test"]
    end
    Selection --> Preparation
    Preparation --> Holdout

    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class T,V,C model;
    class P decision;
    class E neutral;

Validation sequence. Calibration and capacity or policy selection use separate later cohorts. The classifier, calibrator and operating policy are all fitted without the final future cohort.

Leave enough time for outcomes to mature before each fitting or policy-selection cutoff. Small datasets may instead use carefully designed forward out-of-fold predictions for calibration and policy selection. Document exactly which earlier data each fitted component saw. Keep a final future period untouched. If you have only enough events for two blocks, report the resulting uncertainty and simplify the model instead of quietly tuning on the final test.

Match customer overlap to deployment. Weekly scoring usually includes existing customers seen in earlier training periods. Their earlier history is legitimate if it was available at the simulated training time. Putting every customer into exactly one lifetime fold answers a different question: performance on unseen customers. When onboarding or new-account generalisation matters, also report a customer-disjoint, time-respecting test. A random row split across monthly snapshots is unsuitable for the main deployment backtest. Group-only cross-validation does not guarantee chronological validity either.

Overlapping historical feature windows can stay: the past usage they share can be legitimate. The danger is future labels, future information, near-duplicate decisions and uncertainty estimates that treat repeated customers as independent. Use an embargo or purging (a gap between training and test periods) where label intervals and the retraining protocol require it. Justify its length from horizon, ingestion delay and data dependencies.

Report by month, tenure, plan, acquisition channel and relevant operating segment. Check both the repeat-customer and new-customer cohorts. For repeated snapshots, estimate uncertainty by resampling customers. Where shared calendar shocks matter, use time blocks or another dependence-aware method. State the resampling unit. A narrow interval from bootstrapping single rows means little when one customer appears twelve times.

Say whether each result weights scoring opportunities or unique accounts. One churn event can label several earlier weekly snapshots. That suits a recurring scoring task, but it does not create several independent churn events or saved accounts. Evaluate capacity on the actual decision cadence after cooldowns and exclusions, and count realised contribution once in the economic follow-up.

5. Measure the queue the team can actually work

Accuracy rarely helps when churn is uncommon. Average precision (AP) is a useful ranking summary. If you report "PR-AUC", say whether it means AP or trapezoidal area under the precision-recall curve, because they are computed differently. The scikit-learn AP definition weights precision by changes in recall. Average precision documentation.

ROC-AUC remains a valid discrimination measure, but it does not tell you whether precision is usable at the team's capacity. Precision depends on prevalence, so compare AP and precision-recall curves on comparable populations and show each cohort's churn rate.

An arithmetic example, not a benchmark result. In a matured cohort of 10,000 accounts, 500 churn. A queue of 1,000 contains 200 of them. Precision is 200 / 1,000 = 20%, capture is 200 / 500 = 40% and lift is 20% / 5% = 4. This is good risk concentration. It says nothing about how many of those customers a retention offer would save.

For a purely diagnostic alert with stated false-positive and false-negative losses, you can derive a probability threshold from those losses. For a retention campaign, use expected incremental value, action eligibility and capacity. Choose its operating point on the policy-selection cohort, freeze it and evaluate it on the final cohort. Choose at most k feasible actions with positive expected benefit, even if that leaves a quota unfilled. When budgets or action costs differ, use a constrained allocation instead of a single risk threshold.

6. When to use survival analysis

If you observe a subscriber for ten days without churn, you know only that they lasted those ten days, not their 30-day outcome. Survival analysis stores an event indicator and the observed time until churn or right censoring (observation ended before churn was seen). It can use partial follow-up without inventing a negative classification label. scikit-survival introduction.

Choose the time origin explicitly: acquisition for a lifecycle model, or as_of for a landmark model predicting remaining lifetime among customers still active then. If you model lifetime from acquisition but the data start later, handle delayed entry instead of pretending you saw customers from signup. Historical tenure alone is not an event time. Censor at the last date the event feed is known complete, not the date a CSV was downloaded.

Both estimation and evaluation need defensible censoring assumptions. Record why observation ended and whether censoring depends on measured customer history or unobserved risk. A planned study end differs from losing visibility after a migration. A more flexible survival model cannot recover unobserved outcomes by itself. When simple independent censoring is implausible, document conditional modelling or sensitivity assumptions.

Start with Kaplan-Meier for a population baseline and penalised Cox for a compact conditional model. Cox proportional hazards assumes proportional relative hazards for the covariates in its specified form, so check that assumption and model fit. Nonlinear terms or stratification may help, but a coefficient does not identify a causal reason to churn. lifelines survival regression.

Compare a random survival forest when interactions or changing relative risk over time matter. It estimates survival functions without imposing Cox proportional hazards. A forest's generic predict output is a risk score, not a 30-day probability. Get a survival function and compute 1 - S(30) for the chosen origin and defined all-cause event. For a particular cause with competing events, use the cumulative-incidence approach below. Random survival forest guide.

Gradient boosting with a Cox loss can learn nonlinear feature effects while keeping the proportional-hazards structure. It does not automatically fix every violation of that assumption. Other survival losses, including accelerated-failure-time approaches, have different assumptions. Name the fitted loss, not just "XGBoost survival" or "survival boosting". Survival boosting guide.

For changing usage, a practical first approach is to rebuild features at each scoring landmark and predict the next horizon. More advanced models use start-stop intervals or a discrete-time hazard with covariates known at the start of each interval. Keep the customer's later usage trajectory out of an earlier prediction. Time-varying Cox needs interval data. Predicting ahead also needs an explicit assumption about future covariates, because the library cannot know them. Keep repeated landmarks or intervals together for uncertainty calculations. lifelines time-varying survival guide.

Evaluate survival on future cohorts using IPCW concordance for ranking, time-dependent AUC at the decision horizons and Brier score / integrated Brier score (IBS) for predicted survival probabilities. Harrell's concordance can be optimistic with heavy censoring. IPCW means inverse probability of censoring weighting. It relies on a suitable censoring mechanism and enough follow-up. Marginal censoring estimates need independence assumptions. If censoring depends on customer characteristics, consider conditional modelling and sensitivity checks. Choose supported time ranges in advance, keep estimated censoring survival above zero, and estimate censoring from the designated training or reference sample. IBS needs survival probabilities, not an arbitrary risk score. Survival evaluation guide, time-dependent AUC assumptions, IBS documentation.

Competing events change the probability target. If closure or a known service migration rules out voluntary churn within the episode, keep separate event codes. Start with an Aalen–Johansen cumulative-incidence baseline for the probability of voluntary churn before competing events. With covariates, cause-specific hazard models for all causes can be combined into that probability. A single cause-specific Cox survival complement is not enough. Censoring competing events and reporting 1 - Kaplan–Meier instead gives a net-risk quantity, not the observed probability of voluntary churn before another cause. Use cause-compatible evaluation and calibration. A disappearance with unknown cause is loss to follow-up and needs censoring assumptions. It is not automatically a known competing event. Competing-risk guide, Aalen–Johansen implementation.

Check calibration at the operational horizon. For a fitted lifelines regression model, survival_probability_calibration gives a censor-aware, out-of-sample calibration curve at t0, with ICI and E50 summaries. It is a diagnostic, not a drop-in calibrator for every survival library or a competing-risk evaluator. Avoid turning unfinished follow-up into negative labels or simply dropping censored customers. If you fit a recalibration mapping, use a separate cohort and keep survival probabilities coherent across horizons before testing on the final future cohort.

Survival handles observation limits. It does not turn observational customer histories into retention experiments.

7. Risk is not the expected effect of an offer

This is the most important distinction in the use case. Two field experiments in Eva Ascarza's Retention Futility found that the customers at highest risk were not necessarily the best targets for retention. It supports testing who responds instead of treating risk ranking as a retention policy. It does not promise the same treatment effects in your company. Ascarza, 2018.

For a churn outcome Y = 1, define the action effect with a helpful sign:

tau_a(x) = P(churn by H | do(no incremental contact), x)
           - P(churn by H | do(action a), x)

Positive tau_a means the action reduces churn. It is a conditional average treatment effect among customers with similar observed information, not proof that one named person will be saved. Historical response rates among contacted customers cannot identify it, because the earlier team may have chosen whom to contact using risk, value or information missing from the dataset.

Design a randomised pilot among the eligible population. Include no incremental retention contact, the current policy and the candidate actions you can actually deliver. Randomise by customer or account, or by household or another cluster when actions spill over. Avoid giving the same customer conflicting treatments in simultaneous campaigns. Record assignment probabilities, assignment, delivery, cost, redemption and matured outcomes separately. Evaluate the main trial by assigned treatment, including failed deliveries, using intention to treat. Put the churn horizon, follow-up margin, minimum worthwhile effect, stopping rules and sample size assumptions in the plan before reading results. Measure complaints, unsubscribes, discount cost and longer-term retention as well as the first renewal.

With randomised assignment as treatment, outcome learners estimate effects of assignment with that delivery process, including nondelivery, not effects among recipients only. A change in channel, delivery success, overrides or capacity can change those effects. Evaluate the revised workflow instead of carrying old numbers over. Follow the shared experiment protocol for missing outcomes and dependent assignment.

My suggested modelling sequence once a valid pilot exists:

1. Estimate average action effects with uncertainty and compare them with no contact. A null average can still hide useful heterogeneity, but a subgroup or targeting-policy claim needs its own prespecified or independent evaluation. If neither level shows credible benefit, investigate the intervention and delivery before scaling personalised targeting.
2. Fit a simple T-learner: one regularised logistic or boosted-tree outcome model per arm. Subtract the predicted churn probabilities in the sign convention above. Compare it with an S-learner and segment-level effects, since a tiny treatment arm cannot support elaborate heterogeneity. The T/S/X learner framework offers different bias and variance tradeoffs, not a guaranteed winner. Künzel et al., 2019.
3. When sample size and overlap support it, compare a cross-fitted doubly robust learner or a suitable causal forest. Use the known randomised propensities where supported, keep nuisance fitting and policy evaluation separate, and check effect stability. Doubly robust methods still need identification and overlap. They cannot fix unrecorded confounding in a historical campaign. EconML doubly robust learning.
4. On an independent randomised evaluation cohort, compare the frozen targeting policy with the current policy, equal-capacity risk targeting and no incremental contact. Report uplift or Qini curves with their sign and normalisation. The decision metric is incremental contribution at the actual budget, with a confidence interval. For unequal assignment probabilities, use a suitable propensity-weighted or doubly robust policy-value estimator, and inspect overlap and weight variance. Repeat in a prospective controlled rollout.

A margin example, using hypothetical known effects. Assume avoiding a churn event creates ₹1,500 of incremental contribution over the chosen follow-up, and an assigned action costs ₹100 in total, whatever the response. Check this simplifying value assumption too.

At a one-contact capacity, B gives the higher expected value despite its lower churn risk. Include discounts given to customers who would have stayed anyway. If the discount is paid only on redemption, use expected redemption cost instead of a fixed assigned cost. If an offer only delays churn, a short-horizon rescue probability times a long-horizon value can mislead. Evaluate realised incremental contribution over a suitable follow-up directly.

Try it yourself. In the margin example, how large must segment A's absolute churn reduction be for the ₹100 action to break even? If the action cost falls to ₹50, which segments have positive expected value, and which comes first at a one-contact capacity?

For value forecasts, use revenue minus fulfilment, service, incentive and relevant contact costs, with a stated horizon and discounting assumption. Build those forecasts from information available at as_of, and handle incomplete future observation with care. A total-spend or CLTV column computed after departure is hindsight. Current revenue times a risk score is not incremental lifetime value. Keep causal benefit estimation separate from model explanations. SHAP can describe a classifier's score, but cannot show that fixing a feature will keep the customer.

8. Connect the decision to a working retention workflow

Most churn decisions suit batch scoring. Start with a daily or weekly job that builds point-in-time features, scores eligible accounts and writes a versioned decision table. Add an API only when a live workflow needs fresher information. An endpoint does not make the model more actionable.

%%{init: {"theme":"base","themeVariables":{"fontFamily":"Arial, sans-serif","fontSize":"16px","textColor":"#17324D","primaryColor":"#F5F7FA","primaryTextColor":"#17324D","primaryBorderColor":"#8796A8","lineColor":"#718096","secondaryColor":"#EFF6FF","tertiaryColor":"#E9F6F2","clusterBkg":"#F8FAFC","clusterBorder":"#D6DEE8","edgeLabelBackground":"#FFFFFF"},"flowchart":{"htmlLabels":true,"nodeSpacing":18,"rankSpacing":28,"curve":"linear","padding":10,"subGraphTitleMargin":{"top":8,"bottom":12}}}}%%
flowchart TB
    accTitle: Churn evidence and retention workflow
    accDescr: Historical snapshots predict risk, randomised pilots estimate action effects, and a constrained policy delivers actions for mature-outcome review.
    subgraph Risk["Risk prediction"]
        direction TB
        D["Eligible<br/>snapshots"] --> M["Risk or<br/>survival"]
    end
    subgraph Effects["Action evidence"]
        direction TB
        T["Randomised<br/>pilot"] --> A["Action<br/>effects"]
    end
    subgraph Delivery["Retention workflow"]
        direction TB
        P["Value +<br/>constraints"] --> C["CRM queue<br/>or API"]
        C --> O["Mature outcome<br/>review"]
    end
    Risk --> Delivery
    Effects --> Delivery

    classDef data fill:#EFF6FF,stroke:#3973AC,color:#17324D,stroke-width:1.5px;
    classDef model fill:#EFF2FD,stroke:#596FC2,color:#17324D,stroke-width:1.5px;
    classDef decision fill:#FFF5DF,stroke:#A8751A,color:#17324D,stroke-width:1.5px;
    classDef execution fill:#E9F6F2,stroke:#23866E,color:#17324D,stroke-width:1.5px;
    classDef neutral fill:#F5F7FA,stroke:#8796A8,color:#17324D,stroke-width:1.5px;
    class D,T data;
    class M,A model;
    class P decision;
    class C execution;
    class O neutral;

Retention workflow. Risk scores and intervention evidence serve different purposes. The pilot includes a randomised no-contact comparator. The policy compares incremental value and feasible actions before delivery. Assignment, delivery and cost logs join to matured outcomes for governed model and policy review, including the option of no incremental contact.

Export decision_id, account, as_of, horizon, expiry, calibrated risk, model and feature versions, assigned action, policy version and a workflow reason. If an effect estimate exists, store its sign convention, action and expected incremental value too. Join the eventual outcome back to the original decision. Make the write idempotent so a retry cannot create duplicate contacts or spend the same budget twice.

Include the experiment or assignment key, actual assignment probability and relevant coordinator state from the shared decision record. Keep the assigned action after delivery overrides, and log execution separately.

Recheck eligibility at execution. The account may have cancelled since scoring, lost contact eligibility or already received an offer. Enforce suppression lists, cooldowns, budget, agent capacity and mutually exclusive campaigns. Log the action actually delivered separately from the proposed action. Label explanation text as a model explanation, so agents do not read it as a verified cause of churn. If an LLM drafts message copy, keep it downstream of the validated decision and evaluate it as its own feature.

For stale or missing required data, use a documented fallback such as the existing service workflow or no incremental campaign. Route payment issues to payment recovery instead of automatically subsidising every high-risk account. Keep a manual escalation path for account questions the score cannot resolve.

Monitor at three speeds:

Document data and calibration triggers for retraining, the available-label cutoff, model owner, rollback artefact and campaign stop owner. Shadow a challenger before it allocates contacts. Review policy changes as well as model changes, because a new discount or altered renewal terms changes the outcome distribution. Keep a randomised control group when measuring continuing impact. Without it, the model can look worse simply because successful interventions reduced churn among its high-risk customers. A drifting input distribution is a reason to investigate, not proof that a replacement model is better.

9. Choose a dataset without overstating what it proves

IBM documents a fictional-company sample and separates provided churn labels from supplied churn scores and churn reasons, which makes it useful for a leakage exercise. Dataset variants have different fields, so check the dictionary instead of copying a public notebook. UCI documents Iranian Churn as first-nine-month feature aggregates with churn status at month twelve, a three-month planning gap. That supports its stated customer-table task, without per-customer event dates.

Online Retail provides invoice generation times, with no ingestion or version history. A snapshot lab that assumes immediate availability must state that assumption and test simulated late arrivals. This release cannot verify historical production availability. Limit matured inactivity labels to the documented observation window and the customers you can identify. These are practice datasets with different limits. Keep the dataset licence, attribution, checksum and provenance in the project README.

What to build and what counts as complete

Build a retention decision service with an evidence report. You create these deliverables in your own project. This roadmap does not supply them:

• decision-contract.md: population, episode rules, event, horizon, label maturity, capacity, comparator, costs and action owner.
• A versioned snapshot query and data dictionary, with a saved example showing that a late event cannot change features available to an earlier decision.
• A reproducible training and chronological backtest command, source hashes, dependency versions and a table comparing rules, logistic regression and the chosen challenger.
• A calibration report and capacity curve with future-cohort results, segment counts and dependence-aware uncertainty. Use survival diagnostics too if that is your target formulation.
• A batch CRM export or validated API, idempotent decision writes, eligibility rechecks and action logs linked to decision IDs.
• A randomised pilot plan, and an uplift and margin report only when valid intervention data exist. Label a simulated campaign as a simulation and a historical association as an association.
• Monitoring and rollback instructions, plus checks for feature availability, label maturity, duplicate decisions, exhausted budgets and contact cooldowns.

A learner can finish the first gate and show the second with a local integration. The last two need more than a public benchmark. If the data do not support a gate, name the missing evidence and give a concrete plan to collect it.

Before you move on

• You can explain why an immature non-event is not a negative label.
• You can replay an old decision using only the information available then.
• You can justify customer overlap and the training-label cutoff in your validation design.
• You can distinguish discrimination, calibration, risk concentration and incremental action benefit.
• You can say when survival changes the target formulation and what censoring assumptions you used.
• You can show the actual queue capacity, its expected costs and the no-contact option.
• You can identify which claimed result needs a live trial, and who owns that trial and rollback.

Previous: Price recommendation  ·  Back to Chapter 9  ·  Resources

Chapter 9  ·  Offers  ·  Prices  ·  Churn

All three cases share this plan. It turns the research into a reproducible prototype and a reviewable production proposal. Work through the gates that apply to you, and complete the live experiment gate only after a real experiment has run. The 15 shared hours assume a small project reusing the Chapter 6 deployment work.

Use the causal primer to write the estimand before you choose a policy estimator. The reference lab is a prepared starting point. Its synthetic data lets you inspect the mechanics, but cannot show a live business effect or replace the integration work these gates ask for.

In this plan: Decision brief  ·  Data contract  ·  Policy evidence  ·  Experiment  ·  Delivery  ·  Completion

1. Define the decision and economics

Write a one-page brief before choosing a model:

Measure contribution instead of revenue, and retained value instead of churn risk. If cost or treatment data is missing, state the narrower predictive question you can answer and collect the missing data.

2. Create the data and label contract

Store timestamps in UTC and keep the local calendar that business features need. Define the unique snapshot key. Use customer IDs as join keys, and as features only for a deliberate reason. For each snapshot at as_of, record the feature lookback, latest allowed availability time, label window, label completion time and training cutoff.

For fixed-horizon targets, require label_available_at <= training_cutoff and verified outcome coverage through label_end at fitting time. Apply any documented reporting allowance too. A passed calendar horizon is not enough when the outcome feed is late or missing.

For survival targets, keep the event indicator and observed event or censoring time instead. Use only records available at the training cutoff and a verified observation limit. Partial follow-up can stay under the censoring assumptions, but an unfinished horizon is not a negative fixed-horizon label. Document the supported evaluation horizons and the censoring/reference-sample protocol.

An example historical aggregation, assuming available_at is when an event became usable and each event has an immutable ID:

SELECT s.customer_id, s.as_of,
       COUNT(e.event_id) AS purchases_30d,
       COALESCE(SUM(e.net_contribution), 0) AS contribution_30d
FROM snapshots AS s
LEFT JOIN purchase_events AS e
  ON e.customer_id = s.customer_id
 AND e.event_time >= s.as_of - INTERVAL '30 days'
 AND e.event_time < s.as_of
 AND e.available_at <= s.as_of
GROUP BY s.customer_id, s.as_of;

This is a query pattern, not a complete feature system. Deduplicate upstream and check timezone types and join cardinality. Version corrections instead of overwriting history with today's values. A margin finalised later needs its own availability and version record. Use the same definitions for replay and serving. Retrieval tools still leave you to check what “available” means (Feast point-in-time joins).

For experiments, keep eligible opportunities even when no action was delivered. Keep assignment, delivery and outcome in separate tables. Assignment defines the main analysis group, and delivery explains implementation failures. Report the main result on everyone assigned, not only redeemers, buyers, contacted responders or other groups selected after treatment.

3. Train, compare and freeze a release

Compare the business-as-usual rule, a simple statistical baseline and one challenger. Use your guide's split and keep all tuning and preprocessing out of the final test. When rows depend on each other, report uncertainty at the customer, market or time-block level that matches the decision. Evaluate both the model and the policy built from it.

Save a release manifest like this, with real values instead of placeholders:

release_id: example-2026-10-08-001
code_commit: <commit>
data_snapshot: <immutable-source-version>
feature_query_version: <version>
training_cutoff_utc: <timestamp>
label_contract: <version>
feature_schema: <version>
model_artifact: <path-and-checksum>
calibrator_artifact: <path-or-null>
policy_config: <version>
experiment_config: <version-or-null>
dependency_lock: <lockfile-checksum>
evaluation_report: <path>
approved_actions: <versioned-action-set>
fallback_policy: <version>

This manifest is enough for a first project. MLflow's model registry adds lifecycle management when a team needs it. A deployment must load the matching feature schema, model, calibrator and policy together. Changing a threshold or budget counts as a policy release.

4. Evaluate an action policy honestly

Report predictive metrics separately from causal and policy results. An offline comparison of people who did and did not get a discount can mostly reflect how the old team chose them. If randomised logged actions support the target policy, evaluate that frozen policy on held-out assignments. For a frozen deterministic target policy on a discrete action set, an inverse-propensity estimate is:

estimated_value(policy) = mean(
    I[logged_action == policy(context)] * net_outcome / logging_probability
)

For a stochastic target policy, replace the indicator with the target probability of the logged action, given the same context and feasible set.

The logging probability is the chance of that action under the actual assignment policy, given context and candidate eligibility. It is not the model's confidence. Inspect overlap, extreme weights and effective sample size, since a deterministic old policy cannot evaluate actions it never tried. State the uncertainty and any clipping bias. Doubly robust estimation can reduce some errors, but it cannot repair missing action support or arbitrary unobserved confounding (Dudík, Langford and Li, 2011).

Here net_outcome is the observed, mature net contribution after actual costs for every eligible assignment unit, including no purchase and no offer. A contacted nonbuyer can have negative contribution. It is measured, not a predicted score or a baseline-subtracted uplift. The formula evaluates an assignment policy when logged_action records assignment. Keep that original assignment for intention-to-treat after human overrides or delivery failures. Executed actions need their own justified propensities, so do not reuse the original probability for an override chosen in an unknown way. Constraints applied before the random draw belong in the known assignment mechanism. A delivery failure after the draw is a separate record, not a reason to invent a new randomised probability.

For combined actions, log the joint assignment mechanism. An action can be a tuple of offer, price and retention choice, each with its no-change option. Its propensity is the probability of that tuple under the coordinator, given eligible candidates and pre-decision inventory, budget, contact history and other assignment state. Multiplying three marginal probabilities is valid only when the design makes the choices conditionally independent, which mutually exclusive or budget-sharing choices usually are not. For staged random draws, use the recorded conditional probability at each stage, given earlier choices. Their product is the tuple probability. Keep the coordinator version and assignment history to reproduce it.

This estimate covers one contextual decision over the stated outcome horizon. It does not identify a sequential campaign policy, where contacts change later eligibility or prices use up shared inventory. That needs a design that handles sequences and interference, or a prospective experiment of the whole policy. Conditioning on recorded stock and budget cannot show a new policy's effect on their future distribution, and a chain of probabilities for one compound action cannot evaluate a policy over time. Sequential off-policy methods need supported decision histories, correct assignment probabilities and a suitable dependence model. Jiang and Li's ICML 2016 paper develops a sequential doubly robust estimator under its stated trajectory assumptions. When allocations interact across customers, start with a controlled test of the whole coordinator.

If assignment probabilities adapt as results come in, keep their value at each draw and use inference valid for that adaptive design. Ordinary independent-row intervals can fail even with known probabilities. Hadad et al., 2021 (author preprint) studies policy-evaluation inference for adaptive experiments. Check its conditions, because logging propensities alone is not enough.

For continuous prices, this discrete formula applies only to a deliberately logged, finite price grid. A new continuous-price policy needs density-based or structural methods with more assumptions. Prefer a bounded randomised experiment to unsupported extrapolation. Uplift ranking scores alone cannot show net business value.

5. Design the experiment before launch

Write the hypothesis, control policy, target population, primary outcome, minimum detectable effect, power and sample-size assumptions, assignment unit, duration and stopping rule. Use pre-treatment variance estimates and allow for repeat observations, clustering, delayed labels and attrition. Count independent assignment units, because rows are not automatically independent. A retention test needs its full outcome horizon, and a pricing test needs mature returns and costs.

A power sanity check. For two equal independent arms, a continuous outcome and a fixed two-sided 5% test with 80% power (the chance of detecting the minimum effect), a normal approximation gives:

n_per_arm ≈ 2 × (1.96 + 0.84)^2 × outcome_variance / minimum_effect^2

With an invented contribution standard deviation of ₹200 per eligible account and a minimum useful mean difference of ₹5, this is about 25,088 accounts per arm. Halving the effect needs roughly four times as many accounts. This is a planning illustration, not a campaign-size rule. Heavy tails, unequal arms, multiple comparisons, covariate adjustment, attrition and clustered assignment change the calculation. Use design-based simulation or a suitable power method with representative pre-treatment data. Pricing market-days and repeat contacts are not independent accounts. Allow time for outcomes to mature after the last assignment.

Keep assignments stable where repeat exposure matters. If inventory, household sharing or market competition cause interference, consider cluster randomisation or a switchback design (switching units between arms over time), with its own carryover assumptions. Prespecify important segment checks and multiplicity handling. For ongoing peeking, use a valid sequential method. A daily dashboard does not turn fixed-horizon intervals into valid stopping rules.

Where feasible, run an A/A test (both groups treated alike) or a replay to check instrumentation. Reconcile assignment with delivery and outcome joins, and diagnose sample-ratio mismatch before reading lift. Microsoft documents the pre-experiment design practices and SRM checks. Use intention-to-treat as the main analysis, report actual delivery separately and explain any extra estimand.

Analysis by assignment still needs defensible outcomes. Report follow-up completeness by arm and recover missing outcomes where possible. Keep unreachable accounts in, and leave unknown margins missing instead of zero. Predeclare justified missing-data, censoring and sensitivity methods. If treatment affects whose outcome you observe, randomisation alone cannot prevent bias (Lee's sample-selection analysis). Separate a verified zero-purchase outcome from an unobserved one, and postpone the impact claim when evidence is thin.

For high-variance outcomes, a predeclared covariate adjustment such as CUPED can improve precision. Use only covariates available before assignment, not purchases or engagement that treatment affects. June 2026 CUPED research examines adjustment and valid variance estimation. Cross-fit learned adjustments where required, and keep the assignment and clustering structure in inference. Check coverage in an A/A simulation, and update the power calculation with justified variance estimates instead of assuming a fixed reduction.

Compare against the current business policy, not only a deliberately weak model. Look for a meaningful contribution result with acceptable guardrails, since a significant click-rate gain is a different claim. Investigate missing telemetry and immature labels before interpreting any result.

Decide how results generalise before expanding the pilot. An experiment among eligible existing customers in one region shows an effect for that population and period. In a new population, compare eligibility, action support, baseline outcomes, costs and delivery capacity. If those change much, replicate in stages: a larger model cannot make a transportability claim valid. Keep a control or run a new valid experiment when expanding the action policy.

6. Deliver and log the decision

Start with batch scoring for daily campaigns or retention queues. Use an API only when a decision needs current session or inventory context. Both paths need schema validation, authorisation, deterministic eligibility checks, freshness limits, deadlines and fallbacks.

The decision record should contain:

Keep late outcomes in their own table, joined by stable keys. Store scores and reason codes needed for debugging, with restricted access and retention periods. Keep customer secrets out of general request logs. At execution, recheck inventory, contact fatigue and campaign and price validity. Reserve budget transactionally before delivery. Retries must never send duplicate incentives or apply a price change twice.

Release a reservation only after confirming the benefit was finally not delivered or issued. Reconcile uncertain timeouts so an issued benefit keeps its reservation. Log assignment, proposed action, executed action and override mechanism separately, marking unknown execution propensities.

When evidence is missing, return the currently allowed baseline with a reason. If even that baseline is invalid, return review_required and execute nothing. A new action chosen by extrapolation makes an unreliable fallback. Rehearse stale features, empty candidates, exhausted budget, missing inventory, invalid input, timeout and duplicated delivery.

7. Monitor the service, policy and outcomes

Use a small dashboard with an owner and a response for each alert:

Record baseline levels and set alert limits from business tolerance and observed variability. Tutorial thresholds are placeholders. Keep a suitable holdout or control so the model is not credited with changes in the world. Monitor subgroup performance and allocation against the project's customer policy.

Retrain on a documented schedule or justified trigger after checking the data. Compare the challenger on recent mature cohorts and a stable benchmark, then shadow or canary it. Update calibration and policy evaluation with the exact released model. Roll back the whole compatible release or switch to the baseline, and keep the incident records.

8. Package the project and check completion

Below is a target layout for the project you build. You create these artifacts yourself:

business-ml-project/
  README.md                 # question, result, evidence level and limits
  pyproject.toml, uv.lock
  contracts/                # input, label, cost and policy definitions
  src/                      # extraction, features, train, score, decide, feedback
  configs/                  # approved actions, budgets and release config
  tests/                    # meaningful data, policy and delivery checks
  reports/                  # model/policy comparisons and experiment memo
  artifacts/                # versioned manifests; large/private files excluded from Git
  operations/               # dashboard specification, rollout and rollback runbook

Document commands that create data, train, evaluate, replay decisions and start serving from a clean environment. A notebook can explore, but the delivered pipeline must run without manual cell ordering. Use synthetic examples for restricted data and state how they were generated.

An inconclusive or negative experiment is a completed analysis. Explain the result and keep the baseline when the evidence does not justify replacing it. End to end means the loop reaches a defensible decision, which may be to collect more data.

Back to Chapter 9  ·  Resources

A small, complete project accompanies the book. Synthetic subscription data becomes historical features, a calibrated churn model, an independent evaluation and a constrained decision exercise. It runs locally on Python 3.10 or later, using only the standard library. It makes no network requests and needs no API key, GPU, private customer data or package installation.

Every customer, policy document, intervention and result is simulated. The lab shows implementation and reasoning, not real-world model quality or business impact.

Run it

From the repository root:

python3 labs/ground_truth_lab.py all --output labs/output
python3 -m unittest discover -s labs/tests -p 'test_*.py' -v

The first command takes a few seconds on an ordinary laptop. When it finishes, open labs/output/reports/evaluation.md. It reports two separate decisions:

• Model gate. On this simulator the calibrated candidate passes for seed 42 and for every seed from 1 to 20, because the generator deliberately resembles the fitted model. defer appears when the evidence is weaker. To see it, set the three outcome coefficients in generate to zero. ReleaseGateTests builds the defer and regression paths directly.
• Contact experiment. For seed 42 the intention-to-treat interval includes zero, so the result is inconclusive.

Each of these outcomes is a valid, completed experiment. Keep it and explain it, rather than changing the test protocol until a score you like appears.

The two stages can also run separately:

python3 labs/ground_truth_lab.py generate --output labs/output --seed 42
python3 labs/ground_truth_lab.py evaluate --output labs/output
python3 labs/ground_truth_lab.py --help

generate overwrites the lab's named data files in the chosen output directory. Use a dedicated directory, keep earlier results when comparing experiments, and keep your own data elsewhere. Before using existing data, evaluate checks that the manifest lists exactly the lab's source files, with matching hashes (see Checks that stop a run). Record the exact Python version when comparing numeric results, since the last floating-point bit can differ across platforms. Fixed-seed data and numeric results reproduce within one supported environment.

What exists when you finish

output/
  manifest.json                # source hashes, seed, cohort clocks and data contract
  data/
    events.csv                 # event time and first-available time
    snapshots.csv              # one historical row per unique customer
    outcomes.csv               # horizon, availability and verified coverage
    assignments.csv            # separate randomized intervention population
    experiment_outcomes.csv    # includes nonbuyers and delivery failures
    documents/                 # three fictional policies for retrieval/extraction practice
  reports/
    model.json                 # train-only scaling, coefficients, calibrator, thresholds
    test-predictions.csv       # paired baseline/candidate probabilities
    evaluation.json            # full machine-readable evidence
    evaluation.md              # explained results and decision
    price-comparison.csv       # assumed response and contribution
    release-manifest.json      # hashes of source, data, model and evaluation
    data-card.md
    model-card.md
    incident-report.md
  figures/
    calibration.svg
    pricing.svg

Small, checked-in reference results show the kind of output the command creates. They are examples from the seed-42 simulator, not claims about a deployed product. Rerun the command to get a release manifest that matches the source you used. The reference uses a flat layout, and one command rebuilds it (see Regenerate the checked-in reference).

Read the filled-in evaluation, data card, model card and incident rehearsal. The same lab command draws the calibration and pricing figures in the evaluation report.

Follow the project through the book

The lab covers Chapters 2 and 3 in depth. For the other chapters it gives a smaller starting point, and the table says how small.

The lab has no LLM and no neural-network trainer. Its documents and fixtures are a low-cost bridge into those chapters' independent exercises. A hosted-model experiment needs its own permission, budget, model version and evaluation record.

Read the implementation in this order

1. features_at: include an event only if it happened within the historical window and was usable by the decision time. Every synthetic customer has a late event and a future event that must be excluded. The boundary rules for both clocks are deliberate.
2. generate and load_rows: 300 training customers, 150 calibration customers and 150 final test customers. Cohorts are chronological and separate. Labels mature before each relevant cutoff, and each cohort's labels mature before the next cohort's decisions.
3. fit_model: feature means, scales and logistic weights use training data only. A separate cohort fits sigmoid calibration and picks thresholds with false-positive cost 2 and false-negative cost 10. A sigmoid fit cannot promise calibration after a population shift.
4. evaluate: compare a training-prevalence baseline with the frozen challenger. Bootstrap the paired log-loss differences per customer. The gate, release_gate, is fixed in advance and needs an upper 95% bound below −0.01 and no worse Brier score. An uncertain gain, a worse Brier score and a regression all defer. This is a simulated-prototype gate, not launch approval.
5. judge_fixture: calculate precision, recall and false acceptance from a hand-built 50-case confusion matrix. These numbers teach denominators. They do not measure a real judge.
6. business_examples: see why conversion, risk, value and contribution answer different questions. Price response and individual treatment effects are known simulator assumptions. For the randomised experiment, sample_ratio_check runs before intention_to_treat, which reports the assigned-arm difference with its standard error, 95% interval and power check.
7. DecisionLedger: reserve a fixed contact budget atomically, reject conflicting retries, keep a reservation while delivery is unknown, and refund only confirmed nondelivery. The local ledger cannot promise exactly-once external effects or replace a durable production outbox.

Checks that stop a run

evaluate refuses to report rather than quietly producing a number. Each refusal exits with status 2 and a Lab error: message, and the tests cover each one.

Two decisions are reported rather than raised: the model gate (pass_simulated_gate or defer) and the contact experiment (inconclusive when the 95% intention-to-treat interval includes zero, otherwise a simulated positive or negative effect). The experiment section also reports the minimum detectable effect at 80% power and the sample size needed for the simulator's designed effect, using the power approximation in section 5 of the delivery plan.

Worked decisions

Price: a product costs 10. At prices 20, 25 and 30, assumed conversion is 0.30, 0.24 and 0.15. Expected contribution per opportunity is (price − cost) × conversion: 3.00, 3.60 and 3.00. Conversion peaks at 20, but contribution peaks at 25. Those response probabilities require identification in a real pricing problem.

Retention: a high-risk segment has churn risk 0.65 but a contact only increases retention by 0.02. At retained contribution 100 and contact cost 4, its expected incremental value is −2. A saveable segment has risk 0.35 and a retention effect of 0.18, giving +14. A budget of eight funds two contacts. Risk ranking gives −4, and known-effect ranking gives +28. These are oracle illustrations, not estimates the churn classifier learned.

Inventory: demand fixed at 10, and demand equally likely to be 0 or 20, both have mean 10. With stock 10, expected served demand is 10 in the first case and 5 in the second. Using min(mean demand, stock) ignores the spread of demand, which the second case needs.

Exercises, hints and explained answers

1. A purchase happened yesterday but arrived tomorrow

Should it enter today's historical features? Explain both clocks before running the code.

2. The challenger passes the synthetic gate

What can you claim, and what evidence is still missing?

3. A contact request timed out

You reserved four units. Can you refund now and send a new decision ID?

4. Turn the policy corpus into an evaluation set

Before choosing an LLM, write six questions: two answered directly, two that need correct number extraction and two unsupported by the documents. Specify expected source spans and an abstention rule. A deterministic mock response is enough to test your evaluation code.

Evidence boundaries

This reference is deliberately easy to inspect. The generator resembles the fitted model, which real projects rarely do. It assumes independent customers, fully observed outcomes and fixed cohorts. The separate experiment randomises assignment at probability 0.5 and keeps failed deliveries in their assigned arm. For seed 42 its intention-to-treat difference is about −0.2 units, with a 95% interval of about −6.9 to +6.4. The result is inconclusive even though the simulator's designed effect is about +2.7 units per assigned account. About 400 accounts per arm cannot detect an effect that small. The power approximation asks for roughly 4,900. With some other seeds the interval excludes zero. Those estimates, roughly 7 to 11 units, overstate the designed effect, because only large observed differences clear an underpowered test. The lab makes no rollout claim from any of them.

With repeated customers, changing policies, missing outcomes, limited inventory or spillovers, adapt the design and uncertainty method before reading results. Keep a no-action baseline, report negative findings, and gather better data when yours cannot support a decision.

Regenerate the checked-in reference

labs/reference/ holds the seed-42 output in a flat layout. The report and its two figures sit side by side, so its image links are calibration.svg and pricing.svg rather than ../figures/.... The website publishes the folder unchanged. Edit those files only by regenerating them. From the repository root, rebuild all ten with:

python3 labs/ground_truth_lab.py reference

The command runs generate and evaluate in a temporary directory and writes the reference set to labs/reference/ (--output and --seed default to that folder and 42). Every other byte matches an ordinary run. release-manifest.json records the SHA-256 of ground_truth_lab.py, so any edit to the script, even a comment, changes the reference. ReferenceTests runs the command into a temporary folder and checks for the same file names and identical bytes. When it fails, regenerate, review the diff and commit the script and reference together.

Sums use math.fsum, which is correctly rounded on every supported Python, so the numbers do not depend on the Python minor version. A platform's math library could still differ in the last digit. If the reference test fails only on a new platform, report it instead of regenerating.

Evidence level: synthetic prototype. Decision: pass_simulated_gate. Contact experiment: inconclusive. It shows no real customer performance, causal benefit or production readiness.

Frozen protocol

One unique customer per row. Cohorts in time order: 300 training, 150 calibration and 150 final-test customers. Training labels mature before calibration snapshots, and calibration labels mature before test snapshots. Both event time and availability time limit features. Standardisation and logistic weights use training data only. A separate calibration cohort fits a sigmoid and picks cost thresholds for both baseline and candidate. The final cohort is opened once, after those choices are frozen.

Threshold-dependent costs use the listed policies. The release gate compares probability quality only. It does not show that the candidate's action policy improves real outcomes.

Candidate-minus-baseline mean log-loss difference: -0.199816. Paired percentile-bootstrap 95% interval: [-0.289922, -0.115393], using 1000 draws, with unique customers as the independent resampling unit. These finite-sample bootstrap estimates hold only for this simulator and cohort design.

Gate, set in advance: upper interval bound below −0.01 and no worse Brier score. A defer result stands. Do not tune on the final test until it passes.

Judge arithmetic fixture

Positive means an acceptable answer. Human labels against a hypothetical judge on 50 hand-built cases: TP=26, FN=4, FP=3, TN=17. Precision is 26/29=0.8966. Recall is 26/30=0.8667. False acceptance among unacceptable answers is 3/20=0.15. This is an arithmetic fixture, not a measured model judge. For a sensitive task, three false accepts may be too many, even when overall agreement looks good.

Business decisions

Prices 20, 25 and 30 have assumed conversion probabilities 0.30, 0.24 and 0.15. With unit cost 10, expected contribution per opportunity is 3.00, 3.60 and 3.00. Price 20 wins on conversion. Price 25 wins on contribution. The probabilities are simulator assumptions. Observational price correlations could not prove them.

For six hypothetical accounts and a contact budget of eight, risk ranking picks the two high-risk accounts for expected incremental contribution -4.00. Known-effect ranking picks the two saveable accounts for 28.00. These are oracle illustrations. A real policy needs identified effects and independent policy evaluation.

The separate randomised simulation assigns contact with probability 0.5. Failed deliveries stay in their assigned arms (intention to treat). Assigned arms: 374 control and 426 contact accounts. The 385 contact-arm deliveries are reported separately. Sample-ratio check before reading lift: chi-square 3.380, p = 0.0660 against the designed allocation. The alarm, set in advance, is p < 0.001.

Experiment decision: inconclusive. The interval includes zero, so this run cannot support a claim of benefit or harm. The simulator's designed effect, known here only because the data are simulated, is 2.706750 per assigned account. Detecting it with the book's delivery-plan approximation, 2 × (1.96 + 0.84)^2 × variance / effect^2, needs about 4,912 accounts per arm. This experiment is underpowered for that effect. A real plan fixes its size before launch, from pre-treatment variance and a minimum worthwhile effect chosen upfront.

Limitations and next experiment

The generator deliberately resembles its fitted model and uses independent synthetic customers. Real data have measurement errors, censoring, repeated users, selection and shifting treatment effects. Before a real pilot, document permissions and coverage, identify effects, size a prospective test for power, check delivery and outcome instrumentation, set guardrails in advance and name an owner. See the full evaluation, release manifest, data card, model card and incident rehearsal.