What to learn

Worked lesson: evaluate the change on the same cases

Take 50 invented task families, sampled independently, with one evaluated case each. Freeze the rubric, then score the same cases with a baseline and a candidate. This development example uses human pass/fail labels and no repeated attempts:

The baseline passes 39/50 (78%) and the candidate 41/50 (82%). The paired improvement is (6 - 4) / 50 = 4 percentage points. Reporting only “82%” hides four regressions. The ten discordant pairs, where the versions disagree, hold the evidence about this change.

Assuming no change, an exact paired test treats the ten discordant outcomes as fair coin flips. Six wins against four losses is not surprising under that null:

from math import comb

wins, losses, total = 6, 4, 50
discordant = wins + losses
paired_difference = (wins - losses) / total
two_sided_p = min(1.0, 2 * sum(comb(discordant, k)
                             for k in range(min(wins, losses) + 1)) / 2 ** discordant)
assert paired_difference == 0.04
assert abs(two_sided_p - 0.75390625) < 1e-12

The p-value is not the probability that the models are equal, nor does it show whether an improvement would be useful. If your declared rule requires at least five percentage points of supported improvement and no severe regression, this table does not justify promotion. Record the decision, inspect development failures and collect enough independent cases for the next release. For an interval, resample the paired task-family differences, preserving the sampling unit. A tiny or dependent sample can make an ordinary interval misleading.

This is the exact binomial form of McNemar's paired test. The statsmodels documentation describes the exact and large-sample options. A repeated-trial or clustered design needs its own sampling assumptions, instead of treating every attempt as an independent pair.

Now audit a judge on these candidate outputs, treating failure as positive. Suppose it catches 7 of the 9 human-labelled failures, misses 2, and wrongly flags 4 of the 41 human passes. Failure sensitivity is 7/9 ≈ 78%, false-alarm rate is 4/41 ≈ 10%, and agreement is 44/50 = 88%. That reassuring agreement still hides two missed failures, so check severity and slices before trusting the judge. If you use these development labels to tune the judge, they can no longer give independent release evidence for it.

Try it yourself. A new candidate fixes all five cases both versions failed, but breaks three cases both passed. What is its pass rate and paired improvement over the original baseline?

Core concepts and optional depth

The first product needs a bounded task, a baseline, a trace, a checked output and independent evaluation. MCP, agent harness research and post-training are optional extensions once that path works. The reference lab is a no-API starting point for reproducible evaluation. Its synthetic outcomes do not measure a hosted language model.

The model as a dependency. Treat it like a database: a stochastic function with a latency budget, a token bill and a schema contract. Learn structured output, tool calls, streaming, cancellation and error handling. Where an endpoint enforces a schema, it controls the shape, not the truth. Validate values, references and tool arguments in application code. Record the model identifier, sampling/effort settings, prompt version and API/SDK version for reproducible comparisons.

As the baseline, use a current general-purpose model that supports your interface. Compare a smaller model and a stronger or reasoning model on your own tasks. Effort controls, reasoning visibility and billing differ by endpoint, so raise the budget only when evaluations justify it. Self-reported model confidence is not a calibrated routing score. Validate a router against labelled outcomes, including escalation calls in the comparison.

The 2026 linguistic-confidence study, whose authors report acceptance to EMNLP Findings, separates useful score ordering from calibrated probabilities. For your router, measure error among accepted answers against the fraction accepted, then freeze the threshold before the release test. Recheck calibration after changing model, prompt or domain. Token probabilities and agreement between sampled answers are also signals that need validating.

Economics you can measure. Record input, cached input, output and separately reported reasoning usage, plus retrieval, tool, storage and hosting charges, retries and failed requests. Compare cold and warm caches. Cache minimums, lifetimes, write charges and batch discounts depend on provider and model. Check dated pricing and cache documentation instead of assuming a fixed saving applies to the whole bill. Match usage fields to the provider's billing schema instead of summing every breakdown. Claude's thinking-cost guide explains that thinking usage is already included in billed output tokens, so adding it again double-counts cost.

If 1,000 attempted requests cost $8 and 800 pass task checks, the observed cost is $0.01 per successful request. Report cost per attempt and per success, p50/p95 latency and timeout rate. A cheaper model needing frequent escalation can cost more for the task. Set a spending limit and a fallback before public deployment. Enforce per-request and whole-application limits, counting retries and concurrent calls. Check whether a provider budget setting stops requests or only alerts.

Context engineering. The context window is a budget you allocate, not a bucket you fill. Order it so the stable parts (system prompt, tool definitions, examples) come first and can be cached, and the dynamic parts (retrieved documents, history) come after. The harder problems are all about time: compacting history as it grows, deciding what persists across sessions, compressing errors before they enter the context, and noticing that a long context window does not mean the model attends usefully to all of it.

Give sources identifiers and dates. When compacting history, keep unresolved tasks, decisions and source references. Test whether compaction drops a constraint or turns an unverified claim into a fact. Stored memory needs an owner, access permissions, update rules and deletion. NoLiMa (2025) tests long-context retrieval without easy literal matches, showing why a simple needle test can miss failures. Its results cover only the tested models and tasks. Include paraphrases, distant evidence and conflicting versions in your own tests.

Choose the information path. Retrieval-augmented generation (RAG) fetches selected sources. Long context supplies a larger body directly. Search fetches outside information when freshness or coverage requires it. Compare them before assuming you need a vector store.

The EMNLP 2024 RAG/long-context study compares both, plus hybrid routing. It supports a task-specific comparison, not a ranking of 2026 models. Hold the permitted corpus, source versions, query set and answer criteria constant.

Retrieval that you can diagnose. Start with keyword and dense baselines. Choose embeddings by language, domain, length, deployment cost and licensing, and follow their query-prefix and similarity conventions. Changing the embedding model usually means re-embedding the index. MMTEB (2025) broadens multilingual evaluation, but your own domain queries should decide. Chunk around document structure, keep source/version/parent IDs and deduplicate chunks. Apply user access before retrieval and check it again before returning sources, including caches. A cross-encoder reranker rescores retrieved query-document pairs at higher cost and cannot recover a missing candidate. See the official retrieve-then-rerank example.

Deploy the query encoder, document embeddings, chunk/source manifest and retrieval configuration as one compatible version. A new query encoder against an old index can silently change results. Test source updates, deletions and permission revocations as well as ingestion. A cached answer must never restore access to a source the user can no longer read.

The most important habit: measure retrieval separately from generation. If you only measure end-to-end, you cannot tell whether to fix the retriever or the prompt, and you can lose a week rewording a prompt when the document was never fetched. Label relevant source spans and measure recall at k, ordering and latency. Count source families, not duplicate chunks, as successes. Then score answer correctness, whether cited passages support claims, and sensible abstention on missing or conflicting evidence. A citation's presence alone is not support. Twenty query families are a debugging exercise, not a precise production quality estimate.

Extracting fields from documents. Appeals, invoices and forms often arrive as PDFs or scans. Extracted text (a PDF's text layer, or OCR for scans) is cheap to send and easy to inspect, but can lose tables, checkboxes, layout and handwriting. A vision-capable model reading page images keeps layout, has its own cost and can still misread. Compare both on the same labelled pages. Return each field with its evidence (page number and source span or region), or an explicit "not present" instead of a guess, so a reviewer can check it and code can reject unsupported values. Split the labelled set by document. Normalise dates, amounts and IDs, then report per-field accuracy, whole-record correctness and invalid outputs, sliced by scanned, rotated and multi-page documents. Check formats and cross-field rules in application code.

Evaluation as a development loop. Write a small task specification first, then improve it using real development traces. A practical order is:

1. Write representative cases and acceptance criteria, and reserve independent release cases.
2. Build the smallest complete version and collect development traces.
3. Read failures, write free-text notes, and group recurring failure modes.
4. Add deterministic assertions where outputs or final state can be checked directly.
5. Add a judge only for criteria needing interpretation. Ask it for evidence and a rubric decision.
6. Validate the judge on separately labelled examples before relying on its measurements.
7. Put regression cases in CI. Fix one failure mode and compare on development cases.
8. Freeze the candidate and decision thresholds, then run the independent release evaluation.

Keep test answers out of development. Split by source document, customer/session, time or question family as appropriate, then check duplicates and paraphrases across splits. Split before generating synthetic variants. A RAG question may rightly refer to a readable source, but its expected answer and grading notes must stay hidden from the application and its tools. Prompt-tuned cases are development cases. A CI regression set catches known failures, but repeated optimisation against it rules out an independent release claim. Periodically collect a fresh, independently labelled holdout.

Web agents can find leaked benchmark answers during evaluation, even ones absent from training. Anthropic's March 2026 BrowseComp investigation documents leaked answers and answer-key searches by an evaluation-aware model in its tested setup. Audit provenance, isolate grading assets and use fresh private cases. Public benchmark scores are supporting evidence, not your release test.

Use binary criteria for binary requirements, and a defined graded rubric when quality has degrees. Check human-rater agreement. Choose a viewer that makes inputs, sources, calls and outputs easy to inspect. Read the task, trial and grader sections of Demystifying evals for AI agents (January 2026). Its examples describe Anthropic's experience, not universal guarantees.

Where practical, blind reviewers to candidate identity. Test answer-order and verbosity sensitivity. Overall agreement can hide a judge that passes nearly everything. If failure is the positive label, failure sensitivity is TP/(TP+FN) and false-alarm rate is FP/(FP+TN). Missing labels and grader errors are explicit outcomes, not passes. Reset files, databases and tools between independent agent trials, and disclose retry/attempt budgets. Report sample size, uncertainty and meaningful slices. Respect task/session grouping instead of treating correlated trials as new users. Towards a Science of AI Agent Reliability (ICML 2026) organises reliability around consistency, robustness, predictability and safety. One average success score is insufficient for an action-capable system.

Agents and workflows. When the steps are known, start with a fixed workflow. Give an agent discretion where its next step depends on what it observes and you can verify progress and constrain effects. Weigh that flexibility against extra calls and failure modes.

Build one small loop or state machine before choosing a framework. No line count makes it production-ready. Make control flow explicit: allowed tools, argument validation, deadline, token/cost/step limits, bounded retries, cancellation, checkpoints and stop/handoff conditions. Use idempotency keys and check the real service state before retrying writes. A timeout does not tell you whether a remote action happened. Multi-agent designs also need bounded fan-out, shared-state ownership and a way to reconcile findings. Compare with a single-worker baseline.

For a write-capable extension, save the validated action and its status before dispatch, and reuse the same business idempotency key on retries. Tie any user confirmation to the exact action, arguments, recipient and time window. If those change, ask again where the product requires confirmation. Recheck authorisation at execution, because a saved approval is not a permanent access grant. Chapter 6 covers durable delivery and recovery.

Test recovery from tool failure and context compaction. The September 2026 Mid-Harness preprint studies sampling and verifying candidate actions before running one, in terminal-agent benchmarks. After a bounded baseline, optionally compare action verification with extra generation at the same total budget. Its results do not prove a verifier reliable for your actions, and application authorisation still decides whether an action may run.

Access and tool boundaries belong in the first product. Authenticate users when documents or actions are private, and apply their document permissions before retrieval. Treat retrieved text and tool output as untrusted input. Validate tool arguments and enforce permissions in application code. The model does not grant its own access. Add timeouts, bounded retries, idempotency for writes and a confirmation step for consequential actions. Redact secrets and personal data from traces. Test cross-user retrieval and documents that try to override instructions. The OWASP prompt-injection reference covers the threat and layered mitigations, and the AgentDojo benchmark gives tool-agent attack and defence examples. Use harmless canary data for exfiltration tests. Where relevant, use narrowly scoped service credentials, network/file allowlists and isolated execution. Keep grading secrets and other users' data outside the tool environment, recheck access at execution, and set a trace-retention period. A public, read-only demo can keep controls small because its data and permitted effects are small.

Check outputs as well as inputs. Injection defences guard what enters the model. Output guardrails check what leaves it, before a user or tool receives it. Validate schemas and citations. Block or reroute answers exposing secrets, personal data or another user's documents. Add a moderation or policy classifier when the product has a content policy. A guardrail is another classifier, with misses and false blocks. Measure both on labelled examples, including the injection cases above, and keep application permissions as the enforcement layer.

Check the evaluation boundary itself: use fake credentials and harmless services, test outbound access restrictions, and keep production secrets outside the harness. Anthropic's August 2026 incident account reports misconfigured evaluation environments that allowed unauthorised access. This vendor incident report concerns deliberately less-protected cyber evaluations and does not estimate your application's failure rate. Test safe stopping when a task or tool cannot be completed.

MCP, the Model Context Protocol, is an open standard connecting clients to tools and data. Learn the specification revision your client and server implement. The 2026-07-28 changelog introduces a stateless core without the earlier initialisation handshake or protocol sessions, and adds multi-round-trip requests. Stateful business operations use explicit handles whose access the server must enforce. Roots, sampling and logging are deprecated but still work during their deprecation window. Earlier-version tutorials can still help, but test compatibility rather than assuming it.

Learn request/version metadata, discovery, schemas, results/errors and cancellation. Transport retries are not exactly-once business transactions. For remote servers, validate token audience, avoid token passthrough, protect OAuth redirects and restrict outbound fetches against SSRF. Follow the official MCP security practices. Installing a server does not grant access to every connected user.

Agent skills are a smaller idea worth knowing next to MCP. A skill is a folder holding a SKILL.md file of instructions, plus optional scripts and reference files. The agent reads only each skill's name and description at first, and loads the rest when a task calls for it. That is context engineering in a reusable form. The format began at Anthropic, is now an open standard, and coding agents from several vendors support it. Writing one skill for a repeated task lets you compare loading on demand with inline instructions. Read its scripts and references, and enforce normal tool permissions. Loading a skill does not raise its trust.

Fine-tuning and RL when the error warrants them. You do not have to add RAG before training. Match the fix to the failure and available evidence.

The DeepSeek-R1 paper studies reasoning post-training and distillation. It does not prescribe RL for a document assistant. The official TRL GRPO documentation covers reward and training controls for an optional experiment. Validate reward against human or task outcomes. A rising reward alone does not show better quality.

Split by document/customer/task family and time where appropriate, and deduplicate before training. Keep teacher-generated examples independent of evaluation labels. Check synthetic labels, record provenance and permitted data use, and compare at the same deployment budget. Quantised adapter tuning reduces some memory costs, but sequence length, batches, optimiser state and serving context also matter. If training adds no useful benefit, keep the baseline and report that result.