Incident response and capstones · Lesson 5 of 12 · 7 min

Investigate GPU pressure

GPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.

Open the lab Course map

Mastery 0/3Run itExplain itApply it

learner@clairos:~ $

↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Your objectives

Hints are always available. Wrong turns never cost XP.

1
Read the case evidence
Read /opt/ai-lab/cases/aiops-incidents-capstones-005.json. Separate the recorded facts from your assumptions.
Hint & worked example
GPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.
The complete case JSON is read successfully.
```
cat /opt/ai-lab/cases/aiops-incidents-capstones-005.json
```
Common mistake to avoid: The weights fit, so every out-of-memory error must be unrelated.
2
Isolate the deciding signal
Select the kv_cache_gib field from the case JSON. What does it establish—and what does it not establish?
Hint & worked example
Review context/concurrency cache demand because the total planned use is 17 GiB on 16 GiB.
A successful JSON query isolates kv_cache_gib.
```
jq .kv_cache_gib /opt/ai-lab/cases/aiops-incidents-capstones-005.json
```
Common mistake to avoid: Using a different case or an unrelated signal as the justification.
3
Record your operational decision
Choose the justified code: reduce-cache-demand, change-without-evidence, or ignore-the-signal. Save that one code followed by a newline in /home/learner/lab/aiops-incidents-capstones-005-decision.txt.
Hint & worked example
Review context/concurrency cache demand because the total planned use is 17 GiB on 16 GiB.
The decision artifact contains the correct code, not just a matching command in history.
```
printf '%s\n' 'reduce-cache-demand' > /home/learner/lab/aiops-incidents-capstones-005-decision.txt
```
Common mistake to avoid: The weights fit, so every out-of-memory error must be unrelated.

Understand the pattern

GPU pressure diagnosis separates weights, KV cache, concurrent requests and other processes. Reducing a measured source of memory demand is safer than guessing at arbitrary flags.

Interpret the evidence

Review context/concurrency cache demand because the total planned use is 17 GiB on 16 GiB.

A tempting mistake

The weights fit, so every out-of-memory error must be unrelated.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result

$ cat /opt/ai-lab/cases/aiops-incidents-capstones-005.json

{
  "fixture": "Fictional training data; not a live measurement.",
  "weights_gib": 8,
  "kv_cache_gib": 7,
  "buffers_gib": 2,
  "gpu_gib": 16
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .kv_cache_gib /opt/ai-lab/cases/aiops-incidents-capstones-005.json

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'reduce-cache-demand' > /home/learner/lab/aiops-incidents-capstones-005-decision.txt

(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “reduce-cache-demand” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Investigate GPU pressure” justify?The weights fit, so every out-of-memory error must be unrelated.Make the change now and gather the missing evidence afterward.Review context/concurrency cache demand because the total planned use is 17 GiB on 16 GiB.

Apply it elsewhere

A colleague concludes: “The weights fit, so every out-of-memory error must be unrelated.” What should you do?Treat this fixture as a live measurement.Reject that conclusion; return to the case evidence and its stated limits.Accept it because the training command completed.

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Trace a cost spike Try an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

Case methodology: evidence, limits, decision