Incident response and capstones · Lesson 4 of 12 · 6 min

Review a prompt rollout incident

A prompt incident can appear as a semantic regression rather than a process crash. Compare revisions and evaluation evidence while preserving a known-good rollback.

Open the lab Course map

Mastery 0/3Run itExplain itApply it

learner@clairos:~ $

↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Your objectives

Hints are always available. Wrong turns never cost XP.

1
Read the case evidence
Read /opt/ai-lab/cases/aiops-incidents-capstones-004.json. Separate the recorded facts from your assumptions.
Hint & worked example
A prompt incident can appear as a semantic regression rather than a process crash. Compare revisions and evaluation evidence while preserving a known-good rollback.
The complete case JSON is read successfully.
```
cat /opt/ai-lab/cases/aiops-incidents-capstones-004.json
```
Common mistake to avoid: Increase exposure to p8 because the process remains healthy.
2
Isolate the deciding signal
Select the unsupported_claims_after field from the case JSON. What does it establish—and what does it not establish?
Hint & worked example
Review rollback to p7 while preserving p8 failure examples for diagnosis.
A successful JSON query isolates unsupported_claims_after.
```
jq .unsupported_claims_after /opt/ai-lab/cases/aiops-incidents-capstones-004.json
```
Common mistake to avoid: Using a different case or an unrelated signal as the justification.
3
Record your operational decision
Choose the justified code: rollback-prompt-candidate, change-without-evidence, or ignore-the-signal. Save that one code followed by a newline in /home/learner/lab/aiops-incidents-capstones-004-decision.txt.
Hint & worked example
Review rollback to p7 while preserving p8 failure examples for diagnosis.
The decision artifact contains the correct code, not just a matching command in history.
```
printf '%s\n' 'rollback-prompt-candidate' > /home/learner/lab/aiops-incidents-capstones-004-decision.txt
```
Common mistake to avoid: Increase exposure to p8 because the process remains healthy.

Understand the pattern

A prompt incident can appear as a semantic regression rather than a process crash. Compare revisions and evaluation evidence while preserving a known-good rollback.

Interpret the evidence

Review rollback to p7 while preserving p8 failure examples for diagnosis.

A tempting mistake

Increase exposure to p8 because the process remains healthy.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result

$ cat /opt/ai-lab/cases/aiops-incidents-capstones-004.json

{
  "fixture": "Fictional training data; not a live measurement.",
  "prompt_before": "p7",
  "prompt_after": "p8",
  "unsupported_claims_before": 1,
  "unsupported_claims_after": 9,
  "rollback_available": true
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .unsupported_claims_after /opt/ai-lab/cases/aiops-incidents-capstones-004.json

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'rollback-prompt-candidate' > /home/learner/lab/aiops-incidents-capstones-004-decision.txt

(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “rollback-prompt-candidate” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Review a prompt rollout incident” justify?Review rollback to p7 while preserving p8 failure examples for diagnosis.Increase exposure to p8 because the process remains healthy.Make the change now and gather the missing evidence afterward.

Apply it elsewhere

A colleague concludes: “Increase exposure to p8 because the process remains healthy.” What should you do?Accept it because the training command completed.Treat this fixture as a live measurement.Reject that conclusion; return to the case evidence and its stated limits.

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Investigate GPU pressure Try an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

Case methodology: evidence, limits, decision