Resources for the chapter. Use selected documentation sections to finish the same application within the 55-hour plan. Full courses and extra infrastructure are optional. The hours below are learner estimates, not provider commitments.
Official guidance was checked on 9 October 2026. Pin tool versions, record your workload and check current pricing before spending. Project or vendor benchmarks and feature announcements do not show reliability, cost or performance for your service.
Browse: Start here · Optional depth · Keep for reference
Start here
Docker — Get started
Docker documentation team · Official documentation · 4–6 hours
Build and run your application's container, with a multi-stage build where useful. Inspect layers, startup, health checks and secret injection. Record the image digest and deploy that artifact. A working local container is the first deliverable. Hosting credits are not assumed.
FastAPI — Testing
FastAPI maintainers · Official tutorial · 2–4 hours of selected API/testing work
Use the request-validation and testing sections with your existing endpoint. Test normal and invalid inputs, model readiness, timeouts and the chosen fallback. Keep a recorded request that links the served output to the compatible feature/model release. A working endpoint does not prove model quality. Use the lifespan guide to load and clean up model resources. Test that traffic reaches only a ready release, and that overload has a bounded deadline and fallback.
GitHub Actions — Building and testing Python
GitHub documentation team · Official documentation · 3–5 hours
Pin Python and dependencies, run tests and save reports as artifacts. Add deterministic data and permission checks, plus a bounded evaluation job. A known broken schema or forbidden action should always fail. Stochastic quality comparisons need repeated runs and a declared decision rule.
MLflow — Experiment tracking
MLflow project · Official documentation · 2–3 hours
Track code, data and configuration identifiers, metrics and artifacts for your existing model. Use one run to rebuild its training and evaluation. A dashboard is not lineage. Pin the client and server versions, and capture the exact data and split manifest.
MLflow — LLM and agent evaluation
MLflow project · Official documentation · 2–3 hours
Use versioned cases, recorded traces and code-based or model-based scorers to compare two application versions. When the final answer is not enough, include retrieved evidence and tool actions. Check a judge against human-labelled examples. An evaluation API will not check the labels or the rubric for you.
MLflow — Regression testing and CI/CD
MLflow project · Official documentation · 1–2 hours
Connect a small evaluation suite to pytest and inspect the recorded failing case. Use exact assertions for deterministic requirements, and predeclared statistical gates for uncertain quality measures. If you keep adding failed cases to the development suite, keep a separate release set.
Langfuse documentation
Langfuse maintainers · Official documentation · 2–3 hours
An alternative tracing and evaluation stack to inspect. It is optional. Trace one request across retrieval, model and tools, and join latency, tokens, cost and feedback. Set sampling, redaction, access and retention explicitly. Compare hosted and self-hosted use under current terms without assuming a fixed free-tier quota.
OpenTelemetry GenAI semantic conventions
OpenTelemetry project · Official specification repository · 1–2 hours
This repository now defines GenAI spans, metrics and events. The old OpenTelemetry semconv/gen-ai
documentation page is marked moved and unmaintained. Use the repository to design interoperable
telemetry. Pin the convention and instrumentation versions while the interfaces change. By default,
avoid recording raw prompts or tool arguments that can contain sensitive data.
OpenAI — Prompt caching
OpenAI · Official developer documentation · 1 hour plus measurement
Study exact-prefix reuse, and the minimum, retention and read/write pricing for your model. Place reusable content where it can match, then measure cache hits, cost and latency. Caching rules differ across model generations. Budgeting on one universal token threshold or discount is unsafe. If your application uses another API, read your provider's equivalent.
OpenAI — Batch API
OpenAI · Official developer documentation · 1 hour plus measurement
For supported asynchronous work, the current guide gives a 50% discount and a 24-hour completion
window. A batch can expire with completed and unfinished requests. Reconcile output and error files
by custom_id, not position, and retry only unfinished work. Measure actual turnaround. Use batch
requests only when the product's deadline allows them. Record the model and price date in your cost
table.
Optional depth
Add a tool when it meets a requirement in your current system.
vLLM — Serving benchmark
vLLM maintainers · Official documentation · 2–4 hours with a load test
Measure the supported model on your hardware with realistic input and output lengths. Record request rate, concurrency, time to first token, time per output token, end-to-end latency, errors and throughput at the service objective. Keep runtime settings and warmup. Peak throughput from an overloaded queue is not the capacity you can offer users.
PagedAttention
Woosuk Kwon and colleagues · Primary paper first released in September 2023 · 2–3 hours
Read the KV-cache memory-management design behind the original vLLM work. Explain why wasting less cache memory can allow more useful batching, then measure your current engine. The paper's speedups depend on its baselines and hardware. They do not describe every current model or deployment.
vLLM — Speculative decoding
vLLM maintainers · Official documentation · 1–2 hours plus measurement
The current guide aims to cut inter-token latency for memory-bound, medium-to-low-QPS workloads. Check supported methods and feature incompatibilities, then compare accepted draft tokens, quality and latency under load. Extra draft work may not pay off. Keep the baseline if the measured workload does not benefit.
SGLang documentation
SGLang maintainers · Official documentation · 2–4 hours of selected sections
An alternative serving engine to benchmark when its supported models and hardware fit. Read the deployment and performance guidance, pin the runtime, and reuse the load and quality tests from vLLM. Choose by measured task performance and operating needs, not a project headline.
Ray Serve
Ray maintainers · Official documentation · 3–5 hours
Look at deployment composition, scaling and request handling when the application needs several services or replicas. Compare with your existing simple API before adding a distributed runtime. Rehearse failure and recovery. A configuration that scales a demo does not prove capacity or resilience under your workload.
Feast — Point-in-time joins
Feast maintainers · Official documentation · 1–2 hours
Read about historical feature retrieval against entity timestamps and TTL. Build one training snapshot and check it holds only information available at the decision. Point-in-time retrieval is necessary, but it may also need explicit ingestion and availability timestamps and mature labels. A warehouse query can handle the first version. A feature store is optional.
DVC — Pipeline files
DVC maintainers · Official documentation · 2–3 hours
Declare stages, inputs, parameters and outputs for a small reproducible pipeline. Record a data artifact's actual version, not only its path, and confirm another environment can fetch it. DVC is one option. For a first project, an immutable-storage manifest can also be enough.
Dagster — Assets
Dagster maintainers · Official documentation · 2–3 hours
Model extraction, snapshots, training and scores as explicit dependencies with freshness and failure handling. Make one upstream stage fail on purpose, and check downstream actions do not quietly use stale outputs. Use the simplest scheduler that meets your recovery and lineage needs. This is an optional implementation choice.
Evidently documentation
Evidently maintainers · Official documentation · 2–3 hours
Create a report on feature/prediction changes and outcome quality once labels mature. Define a reference period and useful slices, and allow for sample size and seasonality. Input drift is a reason to investigate. It does not prove lost business value or show that automatic retraining will fix the cause.
MLOps Zoomcamp
DataTalks.Club instructors and contributors · Free course repository · Selected modules here
Use the tracking, deployment, monitoring and best-practices material to close a gap in your existing classical-ML pipeline. Follow the repository's environment and project instructions, pin its commit, and check current dependency APIs. A full course project is extra work, not a requirement in these five weeks.
LLM Zoomcamp
DataTalks.Club instructors and contributors · Free course repository · Selected modules here
Use the evaluation, monitoring and application material for your existing LLM product. Adapt its exercises to your documents and failure cases instead of opening another tutorial repository. Cohort dates and hosted services change. What lasts is your versioned application, evaluation and feedback loop.
Keep for reference
MLflow — Model registry workflows
MLflow project · Official documentation · 1–2 hours
Use versions and aliases when several releases need managed promotion. Keep a release manifest with compatible preprocessing, feature contract, model, calibrator, policy and evaluation. If the rest of the decision interface changed, switching a model alias is not enough. Load a complete compatible release and pin it for each request or batch. Artifact rollback still respects current access revocations and operating constraints. For write-capable extensions, the transactional-outbox guide explains the database/message consistency problem and duplicate-delivery handling. Destination idempotency and reconciliation are still needed. Chapter 9's delivery plan gives the business-action and mature-outcome gates.
PyTorch — Reproducibility
PyTorch documentation team · Versioned official documentation · 1 hour
Pin environment and device, control relevant random sources, and declare a reproducibility tolerance that repeated runs support. Reproducibility across releases or platforms is not guaranteed. Keep full training checkpoints and immutable data versions alongside code.
Hidden Technical Debt in Machine Learning Systems
D. Sculley and colleagues · Peer-reviewed NeurIPS 2015 paper · 1–2 hours
A foundational account of data dependencies, feedback loops and system maintenance. Map one dependency and one feedback loop in your own product, then add a check or owner. Use it to reason about current systems. It predates today's LLM tooling and does not prescribe a modern stack.
Site Reliability Engineering
Google SRE authors · Free online book · 2–4 hours of selected chapters
Read about service objectives, monitoring and incident response. Turn a product requirement into an actionable latency or error target, and rehearse the rollback. A model-quality signal with delayed labels needs different proof and timing from an immediate availability failure.
OWASP LLM Top 10 2026
OWASP GenAI Security Project · Official community guidance, 3 August 2026 · 2–3 hours
Pick threats that match your application's tools and data boundaries, then write attack cases and check the recorded actions. Track both authorised-task completion and forbidden-action failures. The risk list is guidance. It does not certify that a system resists injection or misuse.
NIST AI RMF — Generative AI Profile
NIST · Official July 2024 profile, NIST AI 600–1 · 1–2 hours of selected sections
Use the profile to find risks, owners and measurement gaps that matter for your application. Connect the chosen risks to evaluation and incident handling, instead of copying a checklist into a model card. This is risk-management guidance, not a substitute for applicable obligations.
European Commission — AI Act guidance
European Commission · Official guidance · Lookup reference
For an EU-facing product, check intended use, your provider or deployer role, and the current applicable text and implementation schedule with the responsible owner. Separate an enacted obligation from a proposal or political agreement. Record the assessment. A general chapter summary cannot classify every system or show compliance.