Skip to content
Clairos Learn

Incident response and capstones · Lesson 9 of 12 · 6 min

Prepare a model rollback

A model rollback restores a compatible, previously verified release tuple. Preserve the failing candidate evidence so recovery does not erase the investigation.

Mastery 0/3Run itExplain itApply it
learner@clairosSimulated AI Ops lab
learner@clairos:~ $
↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Understand the pattern

A model rollback restores a compatible, previously verified release tuple. Preserve the failing candidate evidence so recovery does not erase the investigation.

Interpret the evidence

Restore the verified m3/r2 tuple while retaining candidate evidence.

A tempting mistake

Mix the old model with an arbitrary runtime and delete the failure logs.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result
$ cat /opt/ai-lab/cases/aiops-incidents-capstones-009.json
{
  "fixture": "Fictional training data; not a live measurement.",
  "rollback_model": "m3",
  "rollback_runtime": "r2",
  "pair_verified": true,
  "candidate_evidence_saved": true
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .pair_verified /opt/ai-lab/cases/aiops-incidents-capstones-009.json
true

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'restore-verified-tuple' > /home/learner/lab/aiops-incidents-capstones-009-decision.txt
(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “restore-verified-tuple” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Prepare a model rollback” justify?

Apply it elsewhere

A colleague concludes: “Mix the old model with an arbitrary runtime and delete the failure logs.” What should you do?

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Build a complete evidence bundleTry an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

  1. 01Handle a stale indexA stale-index incident needs freshness evidence and a safe update plan. Stage a validated index and keep the prior working artifact until the replacement is accepted.Practice
  2. 02Build a complete evidence bundleAn incident bundle must connect the symptom, versions, traces, case results and action. Keep sensitive raw data out of routine shared artifacts.Practice
  3. 03Write an incident reviewAn incident review separates confirmed causes, contributing factors and unresolved questions. Record corrective actions with owners instead of inventing certainty.Practice