Skip to content
Clairos Learn

Serving, containers, and resources · Lesson 13 of 17 · 9 min

Match model memory to GPU

Serving memory includes weights, cache and working buffers. Compare their combined demand with available memory, not only the artifact size.

Open the labCourse map
Mastery 0/3Run itExplain itApply it
learner@clairosSimulated AI Ops lab
learner@clairos:~ $
↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Understand the pattern

Serving memory includes weights, cache and working buffers. Compare their combined demand with available memory, not only the artifact size.

Interpret the evidence

Reduce the 15 GiB combined demand before loading into 14 GiB free memory.

A tempting mistake

Eight GiB of weights means the complete workload needs eight GiB.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result
$ cat /opt/ai-lab/cases/aiops-serving-resources-013.json
{
  "fixture": "Fictional training data; not a live measurement.",
  "weights_gib": 8,
  "kv_cache_gib": 5,
  "buffers_gib": 2,
  "free_gib": 14
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .free_gib /opt/ai-lab/cases/aiops-serving-resources-013.json
14

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'reduce-serving-memory' > /home/learner/lab/aiops-serving-resources-013-decision.txt
(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “reduce-serving-memory” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Match model memory to GPU” justify?

Apply it elsewhere

A colleague concludes: “Eight GiB of weights means the complete workload needs eight GiB.” What should you do?

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Read readiness evidenceTry an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

  1. 01Inspect GPU availabilityGPU visibility and available memory are separate observations. Measure free memory and runtime attachment before deciding whether the next model can be loaded.Practice
  2. 02Read readiness evidenceReadiness should cover the dependencies needed for useful work. A process can listen on a port while a required index or model remains unavailable.Practice
  3. 03Inspect service dependenciesDependency checks identify what a service needs to work. Prefer repairing the failing dependency over restarting healthy callers without evidence.Practice

Case methodology: evidence, limits, decision