Skip to content
Clairos Learn

Evaluations and datasets · Lesson 3 of 18 · 5 min

Validate a dataset schema

Schema validation checks the dataset shape before a run starts. Missing case identifiers or labels should be surfaced instead of silently excluded from the denominator.

Open the labCourse map
Mastery 0/3Run itExplain itApply it
learner@clairosSimulated AI Ops lab
learner@clairos:~ $
↑ history · Tab complete · Ctrl+C cancel · Esc, Tab to leaveExit 0

Understand the pattern

Schema validation checks the dataset shape before a run starts. Missing case identifiers or labels should be surfaced instead of silently excluded from the denominator.

Interpret the evidence

Repair the two missing case IDs before treating this as a clean 100-case run.

A tempting mistake

Drop the invalid cases silently and report a perfect dataset.

All names, measurements, prices, versions, and thresholds in this exercise are fictional training data. The evidence supports this bounded scenario, not a claim about a live provider.

Worked example and expected result
$ cat /opt/ai-lab/cases/aiops-evals-datasets-003.json
{
  "fixture": "Fictional training data; not a live measurement.",
  "records": 100,
  "missing_case_ids": 2,
  "schema_valid": false
}

Exit status: 0. The command completed successfully in the training fixture.

$ jq .missing_case_ids /opt/ai-lab/cases/aiops-evals-datasets-003.json
2

Exit status: 0. The command completed successfully in the training fixture.

$ printf '%s\n' 'repair-case-ids' > /home/learner/lab/aiops-evals-datasets-003-decision.txt
(no output; exit 0)

Exit status: 0. The command completed successfully in the training fixture.

Try a variation

Name the assumption that would have to change for “repair-case-ids” to stop being the justified action. Do not silently replace missing evidence with a guess.

Explain it

What does the evidence in “Validate a dataset schema” justify?

Apply it elsewhere

A colleague concludes: “Drop the invalid cases silently and report a perfect dataset.” What should you do?

Keep the momentum.

Complete the lab, then explain and apply the pattern.

Next: Count evaluation recordsTry an incident drill

Anonymous lesson and mastery progress lasts for this session. Sign in to keep synced lesson XP between sessions; mastery stays device-local. No speed bonus, lost lives, or hint penalties.

Keep learning

  1. 01Read evaluation casesAn evaluation case needs a task input, expected behavior and scoring rule. A reference answer alone is not enough when multiple valid outputs exist.Practice
  2. 02Count evaluation recordsRecord counts help catch accidental truncation or filtering. Reconcile loaded, evaluated, skipped and failed cases so an aggregate score has a trustworthy denominator.Practice
  3. 03Run a baseline evaluationA baseline evaluation freezes model, prompt, dataset and scoring configuration. A rerun is comparable only when the relevant conditions are recorded.Practice

Case methodology: evidence, limits, decision