EvalScope
中文Use with AI

OPEN-SOURCE EVALUATION INFRASTRUCTURE

Evaluate every AI system — with evidence you can inspect.

One evidence path for LLMs, VLMs, agents, benchmarks, load tests, and the artifacts behind every decision.

Use with AIView on GitHub
run-evaluation.sh
export DASHSCOPE_API_KEY='…'
evalscope eval \
  --model qwen-plus \
  --api-key "$DASHSCOPE_API_KEY" \
  --datasets gsm8k \
  --limit 5
Real EvalScope qwen-plus GSM8K results

ONE SYSTEM, THREE MEASUREMENT SURFACES

Make every AI release legible.

01

Evaluation

LLM, VLM, AIGC and RAG tasks with sample reviews, judge contracts, versioned semantics, and portable artifacts.

02

Agent & Harness

Measure requests, tools, tokens and outcomes across native agent evaluations and external harness workflows.

03

Performance

Turn concurrency, latency, RPS, TTFT and throughput into a repeatable service decision.

EVIDENCE, NOT A SCORECARD

One reproducible path from input to decision.

Inputs, predictions, reviews, reports, configuration, and versioned metadata remain connected so a result can be inspected instead of merely repeated.

EvalScope architecture from inputs through native evaluation, agent and serving workflows, to inspectable artifacts.
1

Inputs & Config

TaskConfig
Model APIs
Local Models
Benchmarks
Custom Data
2

Native Evaluation

Registry
Data Adapter
Model
Evaluator
Prediction
Review / Judge
Metrics & Aggregation
3

Inspectable Artifacts

Logs
Configs
Predictions
Reviews
Reports
Dashboard

Agent & Harness

Agent Loop
Tools & Environment
Trace & Outcome
Review / Judge

Serving Performance

Workload
Endpoint
Load Runner
TTFT · RPS · Throughput

Fig. 01EvalScope Architecture

AGENT & HARNESS

Trace the work behind an agent answer.

Keep model requests, tool calls, token use and the submitted outcome in one inspectable task record.

Task contextTool evidenceFinal outcome
Open Agent details
Light Agent Trace with model requests and tool calls

PERFORMANCE

Choose capacity with evidence, not a single number.

Compare concurrency, latency, TTFT, throughput and success rate in one run record, then keep the charts with their configuration.

Open performance details
Light performance charts and run comparison

BENCHMARK CATALOG

Find the benchmark that fits the question.

Browse task type, modality, metrics and evaluation version before you run. The catalog stays tied to generated documentation and runnable metadata.

Browse all benchmarks
250+benchmarks in the registry
LLMVLMAgentAIGC

EVIDENCE VIEWS

Follow every metric back to its evidence.

Open Visualization details
Light Dashboard overview
Light evaluation overview
Light performance overview
Light Agent Trace overview

GET STARTED

Start with an agent or one command.

Start a small API-backed evaluation with a coding agent or from the terminal. Both paths keep the same evidence.

Need endpoint setup, task prompts or troubleshooting? Continue with the complete guide.

Open full setup guide

USE WITH AI

AI evaluation handoff

Use the EvalScope project skill to evaluate qwen-plus on five GSM8K samples. Ask me for the model or endpoint configuration you need before running it, then report the results and where the outputs were saved.

COMMAND LINE

01

Install service toolsAdds the CLI and local dashboard.

02

Run five samplesUses the API key from your environment.

03

Inspect artifactsOpens reports, predictions and configs from ./outputs.

quickstart.sh
pip install 'evalscope[service]'
export DASHSCOPE_API_KEY='…'
evalscope eval \
  --model qwen-plus \
  --api-key "$DASHSCOPE_API_KEY" \
  --datasets gsm8k \
  --limit 5
evalscope service --outputs ./outputs