What actually changed?
Separate task behavior from prompt, rubric, data and scoring-scale changes.
Exploring
How do we know an evaluator improved — rather than drifted?
EvalScope keeps evaluation inputs, reviews, and outputs connected. This page explores how to tell a real evaluator improvement from a change in data, prompts, or scoring.
THREE QUESTIONS EVERY EVALUATOR CHANGE SHOULD ANSWER
Who judged it?
Audit judge consistency and failure modes instead of treating a score as an oracle.
Would the choice still hold?
Measure the consequence when an update changes a ranking or recommendation.
A REPRODUCIBLE VALIDATION PATH
How to validate an evaluator change.
Freeze a baseline, introduce one controlled change, test it on held-out tasks, inspect the evidence, and record the decision with its artifacts. Preserved raw outputs and explicit claims keep any apparent improvement reviewable.
01
Freeze a baseline
Tasks, environments and raw artifacts.
02
Create variants
One controlled change at a time.
03
Test held-out tasks
Held-out tasks and declared metrics.
04
Inspect the evidence
Review evidence before accepting a score.
05
Record the decision
Keep the selection and its rationale.
What supports this work today.
Labels separate available capabilities, external evidence, and open research directions.
Built into EvalScope
Versioned benchmark metadata, structured judge contracts, output artifacts, local reports and API-first evaluation workflows.
Research directions
Self-evolving harness evaluation, evaluator drift guardrails and selection-regret protocols. These are research directions, not product claims.
Traceable agent records
Preserve requests, tool calls and outcomes alongside the task result.
Versioned benchmark catalog
Browse benchmark metadata and evaluation versions across modalities.
Linked run artifacts
Reports, predictions, reviews and configs remain linked.