Go deeper on four things.
- Agent harnesses. State, tool contracts, durable execution, limited retries and verification. Add sub-agents or memory when a measured task needs them. Make external actions idempotent (safe to repeat), and put human review at the decisions that need it.
- Evaluation as a speciality. Judge the whole trajectory as well as the final answer. Check tool-call correctness and multi-turn behaviour, and keep a judge calibrated over time.
- Reliability and safety. Prompt injection (hostile instructions hidden in content the model reads), privilege boundaries, escalation, rate limits and per-user cost caps. Test allowed and forbidden actions together. A system that refuses every task avoids execution mistakes but fails the product goal.
- The product layer. Design for a component that is sometimes wrong, and capture user feedback that feeds your evaluation set.
For depth, apply the strongest Chapter 5 and Chapter 6 resources to your capstone, within the depth budget. Read the Anthropic engineering blog's agent sequence end to end, then 12-Factor Agents. Apply Husain and Shankar's evals material to your own traces. Add the dated research and official security guidance from this chapter's resource list. Recommending these sources is not a claim that any agent framework guarantees reliability.
The ICML 2026 paper Towards a Science of AI Agent Reliability organises reliability around consistency, robustness, predictability and safety. Its arXiv version was submitted 18 February and revised 2 June 2026. For your capstone, repeat typical tasks, perturb inputs, test escalation and permissions, and measure recovery after tool failures. Report completion, action errors, latency and cost together. Record harness, model and evaluator versions, because the same final answer can hide a very different sequence of tool actions. Use the OWASP LLM Top 10 2026, published 3 August 2026, to choose relevant attack cases. A risk checklist alone does not show that your defence works.
Capstone, pick one:
- A reproducible evaluation of one important failure mode.
- An agentic system with trajectory evaluation and a small user pilot.
- An open-source tool or contribution, with a reproduced problem and documented validation. Start from a bug or gap you hit in a library your Chapter 5–6 app uses, and show the change working in that app.
Keep the work to 30 hours. Record real pilot feedback when available. Adoption, novelty and a maintainer's merge decision are separate outcomes.
Interview preparation: practise an AI systems design discussion, then confirm each employer's rounds and tool rules using Chapter 8. Good practice prompts: a triage system for ten thousand tickets a day, showing it works, halving its cost, recovering from a high error rate, and whether the task needs an agent. Use your artifacts to support the tradeoffs. These prompts are practice. Real formats vary.
A simple six-week plan (about 12 hours a week):
| Week | Depth (about 7 h) | Capstone (about 5 h) |
|---|---|---|
| 1 | Re-read Chapter 5 eval resources; instrument one live trace end to end | Pick capstone; write README with success metric |
| 2 | Anthropic agent sequence + 12-Factor Agents | Ship rough version; collect and label a small diverse trace sample |
| 3 | Reliability paper and eval material applied to your traces | Check trajectory scorer against human labels; repeat tasks and report uncertainty |
| 4 | Relevant injection cases from OWASP 2026; least-privilege tools | Test permissions, idempotency and recovery; fix top failure mode and re-measure |
| 5 | Cost and latency table from Chapter 6 | Harden deploy; add kill switch |
| 6 | Write interview stories from the capstone | Polish README; publish write-up |