Branch C — Data Scientist

Go deeper on four things.

• Experimentation. Estimands, randomisation units, power and minimum detectable effect before you run, sample-ratio mismatch, multiple comparisons, novelty and carryover, interference in marketplaces, variance reduction and guardrail metrics. Repeated observations from one user are not independent experimental units.
• Causal inference. Potential outcomes, confounders and colliders, difference-in-differences, instrumental variables, regression discontinuity, matching, and sensitivity analysis.
• The classical statistics the book deliberately left for later.
• Communication. Turn a metric, its uncertainty and its limitations into a clear decision memo.

In applied work, separate prediction from treatment effects before choosing a causal method. Chapter 9 adds primary research, identification assumptions and a shared experiment protocol for the three business cases. The annotated list is in resources/ under Branch C — Data Scientist depth: Kohavi, Tang and Xu on trustworthy online experiments (companion site at experimentguide.com), and two books free online, Cunningham's Causal Inference: The Remix and Huntington-Klein's The Effect.

For each experiment, write down in advance who is eligible, the unit randomised, treatment and control, assignment logging, the outcome window, the intended estimand and the stopping rule. Analyse by assignment for an intent-to-treat effect. Dropping failed deliveries or non-responders after assignment can break the comparison. Reconcile assignment, exposure and outcomes, and look into any sample-ratio mismatch before reading a lift. Match the uncertainty to clustered or repeated units. Use a fixed analysis horizon or a well-founded sequential method. Checking ordinary p-values repeatedly and stopping when one crosses 0.05 changes the error rate.

Include one variance-reduction experiment in the depth work. Compare unadjusted estimates with CUPED, using only covariates the treatment cannot affect. Then run simulated A/A tests to check interval coverage. Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED, a preprint submitted 17 June 2026, examines choices of adjustment and variance estimation, including multi-arm and two-stage sampling. It is recent method research, not a promise that any adjustment makes every business experiment more precise.

For observational work, state the identification argument before choosing an estimator. Difference-in-differences needs a credible comparison and trend assumptions. Instruments need relevance and exclusion. Regression discontinuity needs a cutoff design you can defend. An uplift model, a doubly robust learner or a flexible predictor cannot supply missing randomisation, overlap or control of unmeasured confounding. Use sensitivity checks that suit the design, and write down what remains unidentifiable. Chapter 9 connects these choices to offer, price and retention actions.

Capstone, pick one:

• An experiment analysis with a well-supported causal claim and sensitivity checks.
• A metric system for a real product, with a decision memo.
• A data engineering project on your Chapter 2 sources and snapshots, showing validated SQL, lineage and reliable metric delivery.

Choose the one closest to your target role's actual work.

If you have no randomised data of your own, run the reference lab. It writes a simulated experiment to data/assignments.csv and data/experiment_outcomes.csv in its output directory, with assignment probabilities and failed deliveries kept in their assigned arm. Practise the intent-to-treat analysis, sample-ratio check and uncertainty there, and label the result as simulated.

Interview preparation: practise live SQL, statistics, product reasoning and experiment design, putting the target employer's actual rounds first. Pair a dashboard with a memo that links its metrics, uncertainty and limitations to a decision.

A simple six-week plan (about 12 hours a week):