Anthropic Adds Plugin Evaluation Framework to Claude Code
Claude Code now features built-in evaluation tools allowing developers to benchmark plugin performance against a no-plugin baseline and gate CI workflows.
Claude Code's new evaluation command benchmarks plugin performance against unassisted model runs.
- Anthropic has introduced the
claude plugin evalcommand in Claude Code v2.1.269 and later to systematically benchmark plugins and skills. - The framework executes test cases in isolated environments and scores outputs using checks like regex, tool invocation verification, or secondary LLM judges.
- Runs include an automatic "no-plugin" baseline comparison, calculating the exact performance delta (Δ) provided by the plugin.
- Developers can scaffold suites using
claude plugin eval initand enforce score thresholds within continuous integration pipelines.
Automated Testing for Claude Code Extensions
As agentic development workflows expand, developers increasingly rely on custom plugins and skills to tailor model behavior. To help authors verify that these extensions actually improve task completion, Anthropic has released a native evaluation framework accessible through the Claude Code documentation and announced via ClaudeDevs on X. The new claude plugin eval command line utility allows creators to benchmark their tools, diagnose regressions, and quantify the specific value added by their code.
The evaluation framework requires Claude Code version 2.1.269 or higher. Designed for authors managing standard plugin.json manifests or skills-directory plugins, the system focuses on behavioral scoring rather than basic schema validation, which remains handled separately by claude plugin validate.
How the Evaluation Pipeline Works
An evaluation suite lives inside an evals/ directory located at the root of a plugin repository. Each test scenario is defined in its own subdirectory containing a realistic prompt alongside designated graders.
When an evaluation is triggered, Claude Code starts a clean, isolated, non-interactive session containing only the target plugin. The model attempts to resolve the provided prompt until it reaches completion or encounters preset execution limits, such as turn counts or timeouts. Once the session concludes, graders evaluate the output, inspecting Claude's final response, the command transcript, or any files generated during the process.
Grading Mechanisms
Graders operate as pass/fail checks and can be configured with relative weights. Supported grader types include:
- Pattern matching: Inspecting transcripts or generated files using regular expressions.
- Tool verification: Confirming whether specific tools or Model Context Protocol (MCP) servers were invoked during the session.
- Model-based evaluation: Utilizing a secondary model call to score replies against a predefined rubric.
Because agentic execution can exhibit variance, Claude Code executes each test case three times by default. A run's score reflects the weighted proportion of passing graders, and the overall score for a case is calculated as the mean across runs. Cases pass if their average meets the designated threshold, which defaults to 1.0.
Measuring Value Against a No-Plugin Baseline
A central feature of the framework is the automatic no-plugin baseline comparison. In agentic development, an agent might complete a prompt correctly relying purely on its underlying model knowledge rather than the installed plugin. If a test case achieves a perfect score both with and without an extension installed, the extension provides no demonstrable benefit for that task.
To measure true impact, claude plugin eval runs two evaluation arms by default: one with the plugin loaded ("WITH") and one without ("W/OUT"). The tool computes the difference between these scores, denoted as Delta (Δ), explicitly displaying what capability the plugin contributes. Suites can also integrate mocks for external systems, such as mocked MCP servers or conversation replays, ensuring test reproducibility without unpredictable external side effects.
Test Scaffolding and Continuous Integration
To reduce the friction of defining evaluation suites manually, the tool includes an interactive bootstrapping command: claude plugin eval init. Running this command opens a guided session where Claude analyzes the plugin structure, prompts the author regarding desired outcomes, generates edge-case prompts, and configures appropriate graders.
For teams maintaining shared extensions, evaluation suites can be integrated directly into CI/CD pipelines. Workflows can execute the test suite in headless mode and configure minimum passing thresholds, gating pull requests to ensure code modifications or underlying model updates do not introduce behavioral regressions. Because judge-scored graders and agent runs utilize real model calls using the developer's credentials, execution costs map directly to standard account usage limits or API billing.
Enjoyed this?
Get more posts like this delivered to your inbox.
