Skip to content
Clairos Learn

180 lessons · 12 courses · open practice

AI Operations
Learn it by doing it.

Connect model behavior to configuration, data, evaluations, retrieval, runtime health, and operational evidence.

Start your first labContinue learningOpen sandbox
0 / 180 complete

Complete the exercise. Explain the result. Apply the pattern.

Your course map stays open. Learn in order or jump to the problem you have.

Your course map

Each course builds on the last. Every lesson has a working example, a practice task, and feedback.

01AI systems foundationsUnderstand models, requests, components, failure boundaries, and the operator's evidence-first role.15 lessons0 / 15 complete02CLI and workspace hygieneBuild careful command, path, redaction, configuration, and evidence habits for AI operations work.12 lessons0 / 12 complete03Models, artifacts, and runtimesInspect model manifests, versions, artifacts, runtime compatibility, accelerators, and rollout evidence.16 lessons0 / 16 complete04Inference requests and APIsInspect bounded loopback requests, response envelopes, status codes, timeouts, retries, and streaming signals.16 lessons0 / 16 complete05Prompt and configuration operationsVersion prompts and configuration, compare changes, validate variables, and prepare controlled rollbacks.14 lessons0 / 14 complete06Evaluations and datasetsOperate golden sets, evaluation runs, thresholds, regressions, slices, annotations, and release evidence.18 lessons0 / 18 complete07Retrieval and embeddingsInspect document ingestion, chunks, embeddings, indexes, retrieval quality, freshness, and grounded evidence.16 lessons0 / 16 complete08Serving, containers, and resourcesInspect services, containers, Kubernetes resources, GPU signals, health, configuration, and rollout readiness.17 lessons0 / 17 complete09Observability and tracingUse logs, metrics, traces, request identifiers, latency percentiles, and error budgets to explain behavior.15 lessons0 / 15 complete10Safety, privacy, and governanceReview guardrails, redaction, data boundaries, access, retention, model cards, and operational approvals.15 lessons0 / 15 complete11Cost, capacity, and reliabilityInspect token cost, capacity, concurrency, caching, fallback strategy, load signals, and reliability tradeoffs.14 lessons0 / 14 complete12Incident response and capstonesCombine system, model, retrieval, serving, safety, cost, and evidence habits in operator-style incidents.12 lessons0 / 12 complete

Find a specific skill

  1. 01Map an AI applicationAn AI application is a chain of components, not just a model. Trace the request from input validation through retrieval, inference and response checks; each hop can fail independently.Practice
  2. 02Separate model and applicationModel weights produce predictions; the application owns routing, authentication and presentation. A change to one layer does not automatically change the other.Practice
  3. 03Trace a request lifecycleA request lifecycle has separate waiting, retrieval, inference and validation stages. Compare durations within the same request rather than combining unrelated log lines.Practice
  4. 04Name the serving boundaryA serving boundary defines which process accepts inference requests and which dependencies it calls. A loopback listener is not evidence that a service is publicly reachable.Practice
  5. 05Read an inference contractAn inference contract specifies required fields, supported model identifiers and response shape. Validate the request before blaming a model for a rejected envelope.Practice
  6. 06Distinguish training and inferenceTraining updates learned parameters; inference uses a selected artifact to generate output. Generation settings can change inference behavior without changing the weights.Practice
  7. 07Identify deterministic componentsApplication arithmetic, validation and routing can be deterministic even when text generation is variable. Keep hard constraints in code instead of trusting generated prose to enforce them.Practice
  8. 08Recognize probabilistic outputA probabilistic system can return different valid wording for the same task. Compare task quality over repeated cases rather than demanding identical strings when exact wording is not a requirement.Practice
  9. 09Separate data and instructionsRetrieved documents are data, not trusted instructions. Keep the application policy separate and do not let a quoted document authorize tools or change the user task.Practice
  10. 10Locate an operational failureLocate the first boundary that failed. An HTTP request rejected by a gateway has not necessarily reached the model, so inspect the gateway error before changing generation parameters.Practice
  11. 11Read a system dependency mapDependencies form a directed map. A failed shared dependency can affect several healthy callers; confirm the dependency evidence instead of restarting every caller.Practice
  12. 12Define an operator signalAn operator signal names a quantity, unit and observation window. A bare latency number cannot support an incident claim until its aggregation and time range are clear.Practice
  13. 13Use evidence before actionSeparate observations from hypotheses. A correlation with a deployment is a useful lead, but logs, traces and a controlled comparison are needed before claiming cause.Practice
  14. 14Write a system baselineA reproducible baseline records artifact, configuration, dataset and environment versions together. Keeping only an average score makes later comparisons ambiguous.Practice
  15. 15Foundations review checkpointA sound system review connects one observed failure to a component, its evidence and a reversible next step. Do not turn a system map into an unsupported diagnosis.Practice
  16. 16Orient inside the AI labThe current working directory controls relative paths. Confirm your location before opening or writing a case file; an absolute path starts at the filesystem root.Practice
  17. 17Read the workspace layoutA workspace separates source artifacts, mutable scratch and generated reports. Make experiments in scratch so an investigation does not overwrite the baseline it is meant to compare.Practice
  18. 18Inspect redacted environment dataRedacted configuration proves which keys are present without revealing their values. A redaction marker is not a credential and should not be pasted into an authentication request.Practice
  19. 19Protect local configurationConfiguration containing credentials needs an appropriate local permission boundary. In this exercise mode 600 gives the owner read/write access and no group or other access.Practice
  20. 20Name files predictablyPredictable filenames let another operator relate a note to a case and version. Names should describe what was captured without embedding secrets or sensitive user content.Practice
  21. 21Compare a configuration copyCompare the intended configuration with the candidate before activation. A one-line change can still alter behavior substantially, so read the value and its context.Practice
  22. 22Search without exposing secretsSearching logs should minimize unnecessary disclosure. Select the needed fields and redact credentials before sharing an evidence bundle.Practice
  23. 23Capture bounded command outputBound captured output by relevant scope, not by silently discarding inconvenient evidence. A short recent log window is useful when its limits are recorded in the note.Practice
  24. 24Build an evidence directoryAn evidence directory should group related observations and leave source artifacts unchanged. A directory name can carry a case ID while each file names one type of evidence.Practice
  25. 25Verify a manifest checksumA checksum tests byte identity against a trusted expected value. Matching hashes do not by themselves establish the safety or provenance of the expected artifact.Practice
  26. 26Write an operator noteA useful operator note distinguishes observation, interpretation and next action. Include identifiers and limits so the next person can reproduce the investigation.Practice
  27. 27Workspace hygiene checkpointWorkspace hygiene protects evidence as well as credentials. A clean handoff keeps the baseline, redacts secrets and records the reversible next action.Practice
  28. 28List model inventoryA model inventory is an index of artifacts and their intended roles. Identify the active version and retained rollback separately; a listed artifact is not necessarily loaded.Practice
  29. 29Read a model manifestA manifest records an artifact contract: format, architecture, tokenizer and context assumptions. Read the manifest before selecting a runtime or an inference envelope.Practice
  30. 30Compare model versionsA version comparison must hold the task and evaluation inputs constant. A better aggregate result from another dataset is not a controlled improvement over the current model.Practice
  31. 31Inspect artifact metadataArtifact metadata explains what was downloaded or built. Format, size and quantization are useful checks, but an artifact name alone is not a provenance guarantee.Practice
  32. 32Verify an artifact checksumChecksums detect byte differences relative to a reference. The reference must itself be trusted, and a mismatch should stop activation until its cause is understood.Practice
  33. 33Match model and runtimeA runtime must support the model architecture and storage format. Insufficient memory and unsupported architecture are different failures and need different responses.Practice
  34. 34Read context limitsInput tokens and reserved output share a context budget. Check the combined planned use rather than comparing the prompt alone with the limit.Practice
  35. 35Inspect tokenizer metadataToken IDs depend on the tokenizer paired with the model. A tokenizer mismatch can change segmentation and special tokens even when the input text is identical.Practice
  36. 36Compare quantization profilesQuantization changes representation and memory use, and it can affect task quality. Measure both resource behavior and held-out quality for the intended workload.Practice
  37. 37Read memory requirementsWeight size is not the total serving memory requirement. Leave space for the KV cache, runtime buffers and concurrent work before declaring an artifact a fit.Practice
  38. 38Inspect accelerator availabilityAccelerator availability has several layers: device visibility, driver/runtime support and free memory. A visible GPU does not prove that the model runtime can use it.Practice
  39. 39Spot a runtime mismatchMismatch diagnosis starts with explicit compatibility evidence. When a runtime rejects an architecture, changing prompt text cannot repair the loader contract.Practice
  40. 40Choose a rollback artifactA rollback artifact should be known-good, retained and compatible with the current runtime and configuration. The oldest available file is not automatically a safe rollback.Practice
  41. 41Review model provenanceProvenance connects an artifact to its source, license and revision. Preserve that chain alongside the hash so another operator can explain what was activated.Practice
  42. 42Build a model baselineA serving baseline includes model, tokenizer, runtime, quantization and workload. Record these together before tuning so changes can be attributed rather than guessed.Practice
  43. 43Models and runtimes checkpointModel release readiness combines compatibility, capacity and quality evidence. Passing one dimension is not permission to ignore another failed gate.Practice
  44. 44Check gateway healthA health endpoint can describe process liveness without proving model readiness. Read the scope of the endpoint before using its status to approve traffic.Practice
  45. 45List served modelsA served-model list describes identifiers the gateway accepts. Use the exact served identifier rather than a local filename or an unrelated registry name.Practice
  46. 46Read a request envelopeA request envelope combines model selection, messages and generation constraints. Invalid field types should be corrected at the contract boundary before retrying.Practice
  47. 47Read a response envelopeA response envelope contains output plus metadata such as completion status and usage. Inspect the finish reason before treating partial output as a complete answer.Practice
  48. 48Inspect token usageUsage accounts for input and generated tokens separately. Compare the sum with the recorded total and keep the tokenizer/model context with any estimate.Practice
  49. 49Distinguish client and server errorsClient and server status classes point to different boundaries. A malformed request calls for correction, while a transient server failure may warrant a bounded retry policy.Practice
  50. 50Reason about timeoutsTimeouts must identify which layer stopped waiting. If the client deadline is shorter than normal server work, the client can time out while the server keeps processing.Practice
  51. 51Review retry boundariesRetry policies need a maximum attempt count, delay and retryable failure set. Unbounded retries can amplify overload and duplicate work.Practice
  52. 52Inspect a request identifierA request identifier ties a client operation to gateway and runtime evidence. Record the identifier rather than copying the whole prompt into ordinary logs.Practice
  53. 53Follow a trace identifierA trace identifier connects spans across component boundaries. Use the shared trace ID and parent relationships to reconstruct one request path.Practice
  54. 54Compare batch and single requestsBatch execution has per-item outcomes. An accepted batch request does not mean every item completed successfully, so preserve item IDs and partial failure details.Practice
  55. 55Read a streaming event shapeStreaming chunks are increments, not necessarily complete messages. Assemble them in order and require the documented completion signal before marking the response done.Practice
  56. 56Inspect rate-limit metadataRate-limit metadata describes a specific quota scope and reset window. It does not authorize an unbounded retry loop or guarantee immediate capacity after a guessed delay.Practice
  57. 57Review idempotency guidanceIdempotency is an endpoint contract, not a property to assume. Reusing a key is useful only where the service documents how duplicate requests are handled.Practice
  58. 58Build an API evidence bundleAn API evidence bundle needs request IDs, status, timing, versions and a redacted contract example. It should explain the failure without exporting credentials or user prompts.Practice
  59. 59Inference API checkpointAPI incident review distinguishes validation, readiness, transport and overload failures. Classify the observed boundary before choosing correction, waiting or a bounded retry.Practice
  60. 60List prompt versionsA prompt registry separates the active revision from drafts and rollback candidates. A listed draft should not receive traffic until its rollout is deliberate.Practice
  61. 61Inspect a prompt manifestA prompt manifest records ownership, variables and intended use. Missing metadata makes future evaluation and rollback harder even when the prompt text itself is readable.Practice
  62. 62Separate system and user inputTrusted application instructions and untrusted user input belong in distinct roles. Quoted user text should not gain authority merely because it resembles an instruction.Practice
  63. 63Validate prompt variablesPrompt-variable validation catches missing substitutions before inference. An unresolved placeholder is a configuration defect, not useful context for the model to guess.Practice
  64. 64Compare prompt revisionsCompare prompt changes by meaning as well as line count. Removing an evidence requirement can weaken grounding even if the edited text is shorter and cleaner.Practice
  65. 65Review a prompt rolloutA canary rollout limits exposure while collecting comparable evidence. Decide promotion against predefined gates rather than the novelty of a new prompt.Practice
  66. 66Locate a prompt ownerOwnership identifies who can explain, approve and roll back a prompt change. A named owner is useful only when the escalation and change process are recorded.Practice
  67. 67Inspect configuration precedenceConfiguration precedence decides which value is effective when several sources define a key. Read the documented order rather than assuming the nearest file always wins.Practice
  68. 68Read a redacted environmentA redacted environment can expose configuration names and safe flags while withholding secrets. Preserve this separation when making notes or screenshots.Practice
  69. 69Detect configuration driftDrift is a difference between intended and effective configuration. Confirm the effective runtime value before deciding whether the source or the deployment needs repair.Practice
  70. 70Prepare a prompt rollbackPrompt rollback should restore a known prompt/configuration pair. Changing only one component can leave incompatible variables or validation rules in place.Practice
  71. 71Write a change noteA change note records why a revision exists, what evidence supports it and how to reverse it. Avoid vague success claims that cannot be checked against an evaluation.Practice
  72. 72Review prompt observabilityPrompt observability links outcomes to a revision without routinely logging sensitive text. Revision IDs and bounded metrics make comparisons possible with less disclosure.Practice
  73. 73Prompt operations checkpointA prompt release needs a reproducible manifest, passing evaluation and rollback. Syntax validity alone cannot show whether the new instruction achieves the intended task.Practice
  74. 74Inspect a golden datasetA golden dataset is a versioned set of representative evaluation cases. Keep its intended coverage and split separate from the data used to tune a candidate.Practice
  75. 75Read evaluation casesAn evaluation case needs a task input, expected behavior and scoring rule. A reference answer alone is not enough when multiple valid outputs exist.Practice
  76. 76Validate a dataset schemaSchema validation checks the dataset shape before a run starts. Missing case identifiers or labels should be surfaced instead of silently excluded from the denominator.Practice
  77. 77Count evaluation recordsRecord counts help catch accidental truncation or filtering. Reconcile loaded, evaluated, skipped and failed cases so an aggregate score has a trustworthy denominator.Practice
  78. 78Run a baseline evaluationA baseline evaluation freezes model, prompt, dataset and scoring configuration. A rerun is comparable only when the relevant conditions are recorded.Practice
  79. 79Read aggregate scoresAggregate scores compress many case outcomes. Inspect the denominator, metric direction and important slices before treating one number as a release verdict.Practice
  80. 80Inspect failed casesFailed cases are diagnostic evidence. Group failures by behavior and inspect examples before changing a prompt, model or threshold.Practice
  81. 81Compare evaluation runsControlled comparisons require the same cases and scoring rules. Separate an actual model/prompt improvement from a change in the evaluation itself.Practice
  82. 82Set a release thresholdA release threshold should be chosen before looking at candidate results. Include critical failure bounds, not only a single average target.Practice
  83. 83Review metric tradeoffsPrecision and recall answer different questions. More retrieved items can improve recall while introducing irrelevant results; select the tradeoff for the actual task.Practice
  84. 84Inspect a data sliceSlice evaluation checks behavior on a meaningful subset. Keep slice size with the score and avoid generalizing a small or unrepresentative slice to all traffic.Practice
  85. 85Find annotation disagreementAnnotation disagreement can reveal an ambiguous rubric rather than a model defect. Resolve the criterion and preserve adjudication notes before using the label as ground truth.Practice
  86. 86Detect evaluation leakageEvaluation leakage happens when held-out information influences tuning or selection. A contaminated test set can exaggerate generalization performance.Practice
  87. 87Review dataset provenanceDataset provenance records origin, permission, transformations and version. Without it, a technically valid file may still be inappropriate or impossible to reproduce.Practice
  88. 88Build a regression reportA regression report identifies which cases worsened under a controlled comparison. Include the baseline and candidate versions and representative failures, not just a red badge.Practice
  89. 89Make a release recommendationA release recommendation is a decision against recorded gates. Separate evidence that passed, evidence that failed and any untested conditions.Practice
  90. 90Archive evaluation evidenceArchived evaluation evidence must preserve the reproducible inputs and results. Sensitive raw examples may need controlled storage rather than inclusion in a public report.Practice
  91. 91Evaluations checkpointA complete evaluation review reconciles coverage, comparable versions, critical slices and release thresholds. A score is evidence, not a substitute for that decision.Practice
  92. 92Map the retrieval pipelineA retrieval pipeline separates ingestion, chunking, embedding, indexing and querying. Locate where evidence disappears before changing the answer-generation prompt.Practice
  93. 93Inspect source documentsRetrieved evidence is only as useful as its source coverage and validity. Inspect the original document identity and revision before reasoning from a fragment.Practice
  94. 94Review chunk boundariesChunk boundaries should preserve enough local context to interpret a passage. Separating a condition from its exception can change the meaning of the retrieved evidence.Practice
  95. 95Read embedding metadataEmbeddings have a model identity and dimension contract. Query and document vectors must use a compatible embedding setup; equal storage format alone is not enough.Practice
  96. 96Inspect index statusAn index status must be read with document counts and readiness. A process being up does not guarantee that the index contains the expected records.Practice
  97. 97Check index freshnessFreshness compares the indexed revision with the source revision or update watermark. Wall-clock uptime does not establish that new documents were ingested.Practice
  98. 98Review retrieval resultsRetrieval results should be judged for relevance, coverage and source quality. A ranking position is not a guarantee that a passage supports the answer.Practice
  99. 99Compare similarity scoresSimilarity scores are ranking signals under a specific embedding and metric setup. They are not calibrated probabilities of truth or correctness.Practice
  100. 100Spot a missing citationA citation must connect a claim to source evidence that actually supports it. Merely including a document identifier somewhere in an answer is not adequate grounding.Practice
  101. 101Diagnose empty retrievalEmpty retrieval can come from filters, missing documents or an unavailable index. Inspect those boundaries before loosening access restrictions or inventing an answer.Practice
  102. 102Inspect duplicate chunksDuplicate chunks can crowd a result list and reduce source diversity. Review stable document/chunk identifiers before deciding whether two similar passages are redundant.Practice
  103. 103Review metadata filtersMetadata filters are part of retrieval correctness and access control. A relevant passage from the wrong tenant is not an acceptable result.Practice
  104. 104Compare retrieval configurationsCompare retrieval configurations on the same query set, corpus and relevance judgments. Otherwise a score change may come from different evidence rather than a better retriever.Practice
  105. 105Prepare an index rebuild planA safe index rebuild stages a candidate, validates counts and retrieval, and retains the current index until the candidate is accepted. Do not delete the only working index first.Practice
  106. 106Build retrieval evidenceA retrieval evidence bundle records query, filters, index and embedding revisions, selected source IDs and failure cases. Redact sensitive query content when sharing outside its boundary.Practice
  107. 107Retrieval checkpointRetrieval readiness requires fresh, accessible, relevant evidence as well as a working index. An answer should not hide a missing or unsupported evidence boundary.Practice
  108. 108Inspect the serving serviceA serving process has liveness, readiness and dependency state. Inspect the service before restarting it so the intervention addresses a measured condition.Practice
  109. 109Read gateway logsGateway logs help connect request failures with component state. Keep the request identifier and error category; do not infer root cause from severity alone.Practice
  110. 110List training containersA container list shows process-level state and names. A running container may still contain an application that has not passed readiness checks.Practice
  111. 111Inspect a containerContainer inspection reveals the actual image, mounts, limits and network setup. Compare these effective settings with the intended deployment manifest.Practice
  112. 112Read container resource useResource limits and current use must be interpreted together. Sustained memory close to a hard limit can lead to termination even when the host still has free memory.Practice
  113. 113Validate compose configurationConfiguration validation catches malformed deployment structure before activation. A syntactically valid Compose file still needs separate readiness and workload checks.Practice
  114. 114List Kubernetes workloadsPod readiness summarizes whether workloads can accept traffic under their probes. Running phase alone can coexist with zero ready containers.Practice
  115. 115Describe a serving podPod description brings together events, scheduling and container state. Use the reported reason to choose a focused investigation.Practice
  116. 116Read pod logsContainer logs describe a particular process instance. After a restart, retain the prior instance evidence when it is needed to explain the failure.Practice
  117. 117Inspect pod resourcesResource requests, limits and observed use serve different roles. Compare the observation with the relevant boundary instead of treating all resource numbers as the same quantity.Practice
  118. 118Review a deployment diffRead a deployment diff for behavioral changes, not just additions and deletions. A replica change affects capacity and cost and should be reviewed against readiness and load.Practice
  119. 119Inspect GPU availabilityGPU visibility and available memory are separate observations. Measure free memory and runtime attachment before deciding whether the next model can be loaded.Practice
  120. 120Match model memory to GPUServing memory includes weights, cache and working buffers. Compare their combined demand with available memory, not only the artifact size.Practice
  121. 121Read readiness evidenceReadiness should cover the dependencies needed for useful work. A process can listen on a port while a required index or model remains unavailable.Practice
  122. 122Inspect service dependenciesDependency checks identify what a service needs to work. Prefer repairing the failing dependency over restarting healthy callers without evidence.Practice
  123. 123Prepare a rollout reviewA rollout review combines the diff, compatibility, capacity, readiness and rollback. Explicitly record untested conditions rather than converting them into a pass.Practice
  124. 124Serving and resources checkpointServing acceptance requires the actual running version and successful readiness under the intended workload. Do not sign off from configuration alone.Practice
  125. 125Read an observability mapLogs, metrics and traces answer different questions. Combine a metric trend with a trace and targeted logs instead of treating any single view as the complete explanation.Practice
  126. 126Inspect gateway metricsMetrics need units, scope and a time window. Counters, rates and gauges should not be compared as though they measured the same thing.Practice
  127. 127Read latency percentilesA p95 latency describes a percentile within an observed window. It does not mean that every request or the average request has that latency.Practice
  128. 128Compare throughput signalsRequests per second and tokens per second measure different throughput. Workload length can change token throughput without the same change in request throughput.Practice
  129. 129Inspect error-rate evidenceAn error rate needs both failures and total eligible requests over a stated window. A count of failures without a denominator cannot show the rate.Practice
  130. 130Follow a request traceA trace follows one request through its spans. Use parent-child relationships and IDs so unrelated concurrent work does not get included in the incident path.Practice
  131. 131Find a slow spanCompare span duration with the total trace and distinguish sequential from overlapping work. A long span is a starting point for investigation, not automatic proof of CPU saturation.Practice
  132. 132Connect logs and tracesLogs and traces correlate through stable identifiers, not approximate timestamps alone. Keep the relevant request and span identifiers in diagnostic events.Practice
  133. 133Inspect token metricsToken metrics help distinguish input growth from output growth. A cost or latency increase can follow a longer context even when the response length remains stable.Practice
  134. 134Review model saturationSaturation combines utilization, queueing and service time. High utilization without queue growth can mean useful work; a growing queue gives the observation operational context.Practice
  135. 135Read an SLO windowAn SLO defines a target and window for a user-visible measure. Read the denominator and exclusions before comparing current behavior to the target.Practice
  136. 136Calculate error-budget postureError-budget posture compares observed bad events with the allowance implied by the SLO. Record the window and avoid mixing request-based and time-based budgets.Practice
  137. 137Build an incident timelineAn incident timeline preserves ordering, versions and the distinction between observations and interventions. Use timestamps with a shared timezone or offset.Practice
  138. 138Prepare an observability summaryAn observability summary should name the symptom, scope, supporting signals and uncertainty. A concise note is useful only if its key claims can be checked.Practice
  139. 139Observability checkpointA complete observation ties metric, unit, window and scope to a next investigation. Do not combine unrelated signals simply to make a compelling story.Practice
  140. 140Inspect guardrail policyA guardrail policy names risks, checks and escalation boundaries. A policy document is not proof that those controls are actually enabled in the serving path.Practice
  141. 141Review input redactionInput redaction limits sensitive data sent to a model or logged by an application. Verify the resulting payload rather than assuming that a redaction switch caught every field.Practice
  142. 142Review output redactionOutput redaction has to inspect the produced response, not just the original input. A model may repeat sensitive source material in a new wording or field.Practice
  143. 143Locate sensitive fieldsSensitive-field review depends on context and policy. Distinguish operational metadata from credentials and personal data before exporting a diagnostic record.Practice
  144. 144Check data retentionRetention describes how long data is kept and for what purpose. A retention setting should match policy and actual deletion behavior, not just a label in a form.Practice
  145. 145Inspect access boundariesAccess boundaries limit which subjects can read or change which resources. Relevance to a task does not grant permission to cross a tenant or role boundary.Practice
  146. 146Read a model cardA model card records intended uses, limits and evaluation context. It is an evidence source for review, not a guarantee of suitability for every deployment.Practice
  147. 147Review approved useApproved use is specific to the task, data and controls that were reviewed. A new high-impact use needs its own assessment rather than inheriting an unrelated approval.Practice
  148. 148Inspect refusal evidenceRefusal evidence should be evaluated against the actual policy and task. Over-refusal and missed refusals are different failure modes with different user impacts.Practice
  149. 149Review human escalationHuman escalation needs an identified recipient, a bounded reason and a safe hold state. A vague “ask someone” message is not an operational handoff.Practice
  150. 150Check audit metadataAudit metadata should record what changed, who authorized it and which versions were involved. Avoid collecting raw secrets simply to make an audit record look complete.Practice
  151. 151Separate safety and qualitySafety and task quality are separate evaluation dimensions. A useful answer can violate a data boundary, while a safe refusal can still be unhelpful for an allowed request.Practice
  152. 152Prepare a governance reviewGovernance review checks intended use, data permission, access, retention, testing and ownership together. Mark missing evidence explicitly rather than assuming it exists.Practice
  153. 153Write a risk summaryA risk summary should describe the failure, affected boundary, mitigation and residual uncertainty. Do not promise zero risk from a finite set of checks.Practice
  154. 154Safety and governance checkpointA safety checkpoint needs both policy and tested enforcement. Keep releases gated when a required control or approval is missing, even if the model output looks plausible.Practice
  155. 155Read token cost inputsToken cost inputs separate input and output rates, their units and the model revision. This case uses fictional training rates, not a provider price quote.Practice
  156. 156Estimate request costA token cost estimate multiplies each token count by its corresponding per-token rate. Convert per-million rates before summing input and output estimates.Practice
  157. 157Compare model cost profilesModel cost comparison should use the same workload and required quality. A cheaper request that repeatedly fails the task may not be the better operating choice.Practice
  158. 158Inspect cache effectivenessCache effectiveness depends on hit rate, eligible work and correctness. A high hit rate is harmful when responses come from stale or improperly shared entries.Practice
  159. 159Read concurrency limitsConcurrency limits bound in-flight work; they do not directly state requests per second. Throughput also depends on service time and workload size.Practice
  160. 160Estimate capacity headroomCapacity headroom compares measured demand with a relevant tested limit. State the workload and measurement assumptions instead of calling all unused resources interchangeable.Practice
  161. 161Inspect queue depthQueue depth is meaningful with arrival and completion rates. A growing queue can signal overload even while workers remain healthy.Practice
  162. 162Compare latency and costCost and latency tradeoffs must respect task quality and service targets. Choose among configurations that pass the required gates rather than optimizing a single metric.Practice
  163. 163Review fallback routingFallback routing needs explicit eligibility and limits. A fallback must preserve task, privacy and compatibility boundaries rather than serving any response at any cost.Practice
  164. 164Plan graceful degradationGraceful degradation retains a safe useful subset when dependencies fail. Make the reduced behavior explicit rather than pretending the full service is healthy.Practice
  165. 165Read reliability targetsReliability targets need a measure and window. Keep user-visible success separate from process uptime so an alive but unusable system is not counted as healthy.Practice
  166. 166Prepare a capacity noteA capacity note records observed demand, tested limits, bottlenecks and the next safe experiment. Include uncertainty rather than promising unmeasured scale.Practice
  167. 167Review cost anomaliesCost anomalies require separating request volume, token length, model selection and retries. Compare those drivers before treating an invoice increase as a provider error.Practice
  168. 168Cost and reliability checkpointA cost/reliability decision should preserve quality, latency, privacy and bounded failure behavior. Resource savings do not justify silently breaking another contract.Practice
  169. 169Triage elevated latencyLatency triage starts with a scoped symptom, then traces its largest contributor. Separate the evidence from the hypothesis and prefer a reversible next action.Practice
  170. 170Investigate inference errorsInference error triage distinguishes rejected inputs from runtime failures. Group by status and request ID before choosing a retry or configuration change.Practice
  171. 171Diagnose retrieval regressionA retrieval regression should be compared against corpus, embedding, filters and index revisions. Find the changed boundary before tuning unrelated generation settings.Practice
  172. 172Review a prompt rollout incidentA prompt incident can appear as a semantic regression rather than a process crash. Compare revisions and evaluation evidence while preserving a known-good rollback.Practice
  173. 173Investigate GPU pressureGPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.Practice
  174. 174Trace a cost spikeA cost spike should be explained with comparable units and a measured driver. Increased retries can multiply work even when successful request counts remain flat.Practice
  175. 175Review a guardrail regressionA guardrail regression needs the failing examples and the exact policy/configuration revision. Keep the affected boundary gated while diagnosing the failure.Practice
  176. 176Handle a stale indexA stale-index incident needs freshness evidence and a safe update plan. Stage a validated index and keep the prior working artifact until the replacement is accepted.Practice
  177. 177Prepare a model rollbackA model rollback restores a compatible, previously verified release tuple. Preserve the failing candidate evidence so recovery does not erase the investigation.Practice
  178. 178Build a complete evidence bundleAn incident bundle must connect the symptom, versions, traces, case results and action. Keep sensitive raw data out of routine shared artifacts.Practice
  179. 179Write an incident reviewAn incident review separates confirmed causes, contributing factors and unresolved questions. Record corrective actions with owners instead of inventing certainty.Practice
  180. 180AI operations final capstoneThe final operator decision combines contract, evidence, quality, safety and recovery. A production-looking answer is not sufficient when a required gate fails.Practice