Incident response and capstones
Combine system, model, retrieval, serving, safety, cost, and evidence habits in operator-style incidents.
What you will practice
Triage elevated latency; Investigate inference errors; Diagnose retrieval regression; Review a prompt rollout incident.
Use the worked examples when needed, then test your understanding without them.
- 01Triage elevated latencyLatency triage starts with a scoped symptom, then traces its largest contributor. Separate the evidence from the hypothesis and prefer a reversible next action.Practice
- 02Investigate inference errorsInference error triage distinguishes rejected inputs from runtime failures. Group by status and request ID before choosing a retry or configuration change.Practice
- 03Diagnose retrieval regressionA retrieval regression should be compared against corpus, embedding, filters and index revisions. Find the changed boundary before tuning unrelated generation settings.Practice
- 04Review a prompt rollout incidentA prompt incident can appear as a semantic regression rather than a process crash. Compare revisions and evaluation evidence while preserving a known-good rollback.Practice
- 05Investigate GPU pressureGPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.Practice
- 06Trace a cost spikeA cost spike should be explained with comparable units and a measured driver. Increased retries can multiply work even when successful request counts remain flat.Practice
- 07Review a guardrail regressionA guardrail regression needs the failing examples and the exact policy/configuration revision. Keep the affected boundary gated while diagnosing the failure.Practice
- 08Handle a stale indexA stale-index incident needs freshness evidence and a safe update plan. Stage a validated index and keep the prior working artifact until the replacement is accepted.Practice
- 09Prepare a model rollbackA model rollback restores a compatible, previously verified release tuple. Preserve the failing candidate evidence so recovery does not erase the investigation.Practice
- 10Build a complete evidence bundleAn incident bundle must connect the symptom, versions, traces, case results and action. Keep sensitive raw data out of routine shared artifacts.Practice
- 11Write an incident reviewAn incident review separates confirmed causes, contributing factors and unresolved questions. Record corrective actions with owners instead of inventing certainty.Practice
- 12AI operations final capstoneThe final operator decision combines contract, evidence, quality, safety and recovery. A production-looking answer is not sufficient when a required gate fails.Practice