Skip to content
Clairos Learn

Incident response and capstones · Lesson 6 of 12 · 8 min

Trace a cost spike

A cost spike should be explained with comparable units and a measured driver. Increased retries can multiply work even when successful request counts remain flat.

Mastery 0/3Run itExplain itApply it
learner@clairosSimulated AI Ops lab
learner@clairos:~ $
↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Understand the pattern

A cost spike should be explained with comparable units and a measured driver. Increased retries can multiply work even when successful request counts remain flat.

Interpret the evidence

Investigate and bound retry amplification from 1020 to 3000 attempts.

A tempting mistake

Flat successful request counts prove total work is unchanged.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result
$ cat /opt/ai-lab/cases/aiops-incidents-capstones-006.json
{
  "fixture": "Fictional training data; not a live measurement.",
  "successful_requests": 1000,
  "attempts_before": 1020,
  "attempts_after": 3000,
  "mean_tokens_unchanged": true
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .attempts_after /opt/ai-lab/cases/aiops-incidents-capstones-006.json
3000

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'bound-retry-amplification' > /home/learner/lab/aiops-incidents-capstones-006-decision.txt
(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “bound-retry-amplification” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Trace a cost spike” justify?

Apply it elsewhere

A colleague concludes: “Flat successful request counts prove total work is unchanged.” What should you do?

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Review a guardrail regressionTry an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

  1. 01Investigate GPU pressureGPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.Practice
  2. 02Review a guardrail regressionA guardrail regression needs the failing examples and the exact policy/configuration revision. Keep the affected boundary gated while diagnosing the failure.Practice
  3. 03Handle a stale indexA stale-index incident needs freshness evidence and a safe update plan. Stage a validated index and keep the prior working artifact until the replacement is accepted.Practice