Evaluations and datasets
Operate golden sets, evaluation runs, thresholds, regressions, slices, annotations, and release evidence.
What you will practice
Inspect a golden dataset; Read evaluation cases; Validate a dataset schema; Count evaluation records.
Use the worked examples when needed, then test your understanding without them.
- 01Inspect a golden datasetA golden dataset is a versioned set of representative evaluation cases. Keep its intended coverage and split separate from the data used to tune a candidate.Practice
- 02Read evaluation casesAn evaluation case needs a task input, expected behavior and scoring rule. A reference answer alone is not enough when multiple valid outputs exist.Practice
- 03Validate a dataset schemaSchema validation checks the dataset shape before a run starts. Missing case identifiers or labels should be surfaced instead of silently excluded from the denominator.Practice
- 04Count evaluation recordsRecord counts help catch accidental truncation or filtering. Reconcile loaded, evaluated, skipped and failed cases so an aggregate score has a trustworthy denominator.Practice
- 05Run a baseline evaluationA baseline evaluation freezes model, prompt, dataset and scoring configuration. A rerun is comparable only when the relevant conditions are recorded.Practice
- 06Read aggregate scoresAggregate scores compress many case outcomes. Inspect the denominator, metric direction and important slices before treating one number as a release verdict.Practice
- 07Inspect failed casesFailed cases are diagnostic evidence. Group failures by behavior and inspect examples before changing a prompt, model or threshold.Practice
- 08Compare evaluation runsControlled comparisons require the same cases and scoring rules. Separate an actual model/prompt improvement from a change in the evaluation itself.Practice
- 09Set a release thresholdA release threshold should be chosen before looking at candidate results. Include critical failure bounds, not only a single average target.Practice
- 10Review metric tradeoffsPrecision and recall answer different questions. More retrieved items can improve recall while introducing irrelevant results; select the tradeoff for the actual task.Practice
- 11Inspect a data sliceSlice evaluation checks behavior on a meaningful subset. Keep slice size with the score and avoid generalizing a small or unrepresentative slice to all traffic.Practice
- 12Find annotation disagreementAnnotation disagreement can reveal an ambiguous rubric rather than a model defect. Resolve the criterion and preserve adjudication notes before using the label as ground truth.Practice
- 13Detect evaluation leakageEvaluation leakage happens when held-out information influences tuning or selection. A contaminated test set can exaggerate generalization performance.Practice
- 14Review dataset provenanceDataset provenance records origin, permission, transformations and version. Without it, a technically valid file may still be inappropriate or impossible to reproduce.Practice
- 15Build a regression reportA regression report identifies which cases worsened under a controlled comparison. Include the baseline and candidate versions and representative failures, not just a red badge.Practice
- 16Make a release recommendationA release recommendation is a decision against recorded gates. Separate evidence that passed, evidence that failed and any untested conditions.Practice
- 17Archive evaluation evidenceArchived evaluation evidence must preserve the reproducible inputs and results. Sensitive raw examples may need controlled storage rather than inclusion in a public report.Practice
- 18Evaluations checkpointA complete evaluation review reconciles coverage, comparable versions, critical slices and release thresholds. A score is evidence, not a substitute for that decision.Practice