Skip to content
Clairos Learn

Evaluations and datasets · Lesson 11 of 18 · 8 min

Inspect a data slice

Slice evaluation checks behavior on a meaningful subset. Keep slice size with the score and avoid generalizing a small or unrepresentative slice to all traffic.

Open the labCourse map
Mastery 0/3Run itExplain itApply it
learner@clairosSimulated AI Ops lab
learner@clairos:~ $
↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Understand the pattern

Slice evaluation checks behavior on a meaningful subset. Keep slice size with the score and avoid generalizing a small or unrepresentative slice to all traffic.

Interpret the evidence

Investigate six passes out of ten long-document cases despite the stronger overall score.

A tempting mistake

Use the overall score to dismiss the long-document failures.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result
$ cat /opt/ai-lab/cases/aiops-evals-datasets-011.json
{
  "fixture": "Fictional training data; not a live measurement.",
  "slice": "long documents",
  "slice_cases": 10,
  "slice_passed": 6,
  "overall_passed": 94
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .slice_cases /opt/ai-lab/cases/aiops-evals-datasets-011.json
10

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'inspect-long-documents' > /home/learner/lab/aiops-evals-datasets-011-decision.txt
(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “inspect-long-documents” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Inspect a data slice” justify?

Apply it elsewhere

A colleague concludes: “Use the overall score to dismiss the long-document failures.” What should you do?

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Find annotation disagreementTry an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

  1. 01Review metric tradeoffsPrecision and recall answer different questions. More retrieved items can improve recall while introducing irrelevant results; select the tradeoff for the actual task.Practice
  2. 02Find annotation disagreementAnnotation disagreement can reveal an ambiguous rubric rather than a model defect. Resolve the criterion and preserve adjudication notes before using the label as ground truth.Practice
  3. 03Detect evaluation leakageEvaluation leakage happens when held-out information influences tuning or selection. A contaminated test set can exaggerate generalization performance.Practice

Case methodology: evidence, limits, decision