How to Evaluate Agent Skills: Do They Actually Improve Results?

Your team shipped a skill for your API. It installs cleanly, and in the demo the coding agent writes integration code that looks right. Then someone asks whether agents integrate the product more correctly with the skill, or whether it only adds tokens and a maintenance job. Nobody can show the answer.

To find out, compare equivalent integration tasks under controlled conditions and measure correct outcomes alongside cost. A successful skill load is not evidence that the integration succeeded. A skill earns its place when the same task, run with and without it, produces more correct integrations at a cost you accept.

One skill can now reach many agents. Anthropic's post on donating Agent Skills to the Agentic AI Foundation (October 22, 2026) says "More than 70 agent platforms, including ChatGPT, Cursor, GitHub Copilot, and VS Code, have integrated Agent Skills." Every result cited here comes from a team that published its own runs, and each has limits.

Name the three things under test

Every result describes three parts at once. The first is the product-specific skill at a specific version. The second is the agent and runtime: the model, plus the harness program that runs it with its tools, settings, and limits. The third is the integration task. Change any one and you are running a different experiment. Per the Agent Skills overview, activation happens "When a task matches a skill's description," so the runtime decides when your instructions arrive. Anthropic's skill authoring guide says "Test your Skill with all the models you plan to use it with."

Reading a skill's SKILL.md before installing it is a cheaper first filter, the approach behind our ranking of Claude skills for documentation. Reading shows what a skill claims to do; the paired run shows whether it changes results on your task.

Define "better" before you run anything

For an API team, better means a working integration against current, supported product behavior. Write task-specific acceptance checks before any scoring, so nobody moves the bar on the release checks after seeing a result.

Keep two numbers apart. An assertion pass rate counts individual checks. A whole-task success rate counts integrations that work end to end. A run can pass four of five checks and still ship a webhook handler that accepts forged requests, so record both.

The Agent Skills guide on evaluating skills asks for concrete evidence before a check counts as a pass, and says to "Remove or replace assertions that always pass in both configurations." Samuel Berthe makes the same point: a check that passes with and without the skill is testing common knowledge. Keep the answer out of the prompt, too. Tessl found that 30% of its generated tasks leaked a meaningful amount of the grading criteria into the task description.

That removal advice is about measuring what the skill adds. Those assertions, the guide says, "inflate the with-skill pass rate without reflecting actual skill value." The guide does not address release checks. Our recommendation: some checks guard what an integration must never break, such as rejecting a tampered webhook payload. Keep those in your acceptance checks even when the runs with and without the skill both pass them. Leave them out when you compare the two pass rates. Claude Code's plugin evals exist in part to "Catch regressions when you change the plugin or a new model is released."

(opens the full-size image in a new tab)

Run two separate experiments: activation and outcome

A skill can fail by never loading, or by loading and still leading to a broken integration. Test each on its own. The first experiment checks automatic activation on realistic prompts. The guide on optimizing skill descriptions starts from "A skill only helps if it gets activated." It recommends about 20 queries, 8 to 10 that should trigger and 8 to 10 that should not. Run each one three times to start, and split the queries about 60/40 into a train set and a validation set. Make the should-not-trigger prompts near misses, such as a question about a neighboring product.

The second experiment checks output quality when the skill is in use. Here you may tell the agent to load the skill, because you are testing what the instructions do. Explicitly telling an agent to load a skill cannot prove that normal users will trigger it. Tessl measured unforced activation at "41%" (929 of 2,286 tasks) and forced activation at "98%" (2,240 of 2,286).

Activation alone also proves little about results. In Vercel's evals of Next.js 16 API tasks, explicit instructions "improved the trigger rate to 95%+ and boosted the pass rate to 79%," which still trailed a docs index in the same test.

Record activation for every run as yes, no, or unknown. If the run transcript shows the agent reading the skill, record yes. If the runtime gives no evidence, record unknown; do not assume your runtime exposes skill loads. A commenter in a Hacker News thread on a skill eval tool wrote that the agent "can sometimes not pull the skill even if the trigger conditions are there."

Set up a fair paired run

Each test case has two arms: one run with the skill and one without. The evaluating guide calls this the core pattern: run each test case once with the skill and once without it, or with a previous version, so you have a baseline to compare against. Anthropic's skill-creator runs one subagent with the skill and one without per test case. NVIDIA's SkillEvaluator gives the design rule: "the only experimental variable is whether the skill is installed."

Hold these constant across both arms:

• the task inputs and the starting repository state
• permissions and tool access
• current product docs, reachable in both arms
• the model, runtime settings, and limits such as turns or time

Change one thing only: whether the skill is available, or which version of it.

Isolate every run: "Each eval run should start with a clean context." Block leakage into the baseline, too. Berthe's rule is "Never load sibling skills. They leak overlapping guidance and inflate the 'without' score." Check that no skill folder sits in the repository the baseline agent opens.

Repeat the tasks that matter, because runs vary. The evaluating guide says "Start with 2-3 test cases" and, for that stage, "focus on the raw pass counts and the delta," because standard deviation "is only meaningful with multiple runs per eval." NVIDIA's catalog mostly stopped at one attempt per skill: of its skills "with published results, 85% ran one attempt," and the post reports no confidence intervals. Use a two or three task pilot to find failures, and treat its numbers as signals, not statistical proof.

Pick three tasks and write the checks first

Start where the agent struggles. Anthropic's authoring guide says "Build evaluations first," beginning with "Identify gaps: Run Claude on representative tasks without a Skill." Then pick three task types:

• A realistic setup task: an integration a new customer builds with your current SDK.
• A failure and recovery task: the agent starts from a broken integration and has to find and fix the cause.
• A boundary task: a request your product does not support, where the right answer is to say so and point to the supported route.

Write the acceptance checks before scoring, and add cases as failures show up. Keep the exploratory cases you tune against separate from held-out release checks. Run those only when you decide, since a skill tuned on the cases that judge it looks better than it is. The evaluating guide adds detailed assertions after the first runs; freezing the checks and holding cases back for the release decision is our recommendation for a fair evaluation.

Illustrative scenario: webhook verification for a payments API

This illustrative table applies the three task types to a hypothetical payments API. The outcome columns are blank because nobody has run it.

A row passes only when every check holds; the worksheet records checks passed separately.

Record what each run cost

The evaluating guide puts cost next to quality, because "a skill that dramatically improves output quality but triples token usage is a different trade-off than one that's both better and cheaper." Record elapsed time, tokens if your runtime reports them, retries, and human interventions. Write unavailable when a measurement is missing. Raw token count is not dollar cost, so convert it with your own pricing.

Skill changes can move cost a lot. In Wix's runs comparing skills with docs, the code snippets in one skill were missing an export declaration, "resulting in a 39% token increase over the docs runs." That is a single case.

In a single-attempt example, NVIDIA reports that one skill, cuopt-install, raised token use from 25,227 to 55,582 (120.3%), and names no cause.

Diagnose the failure before you fix it

Each failure class points to a different fix.

Permission gaps and bad checks are fixed in the test setup; the other four classes are fixed in the skill.

SignatureAPI's agent skills docs describe a CI check that "fails whenever an identifier a skill mentions stops existing in the spec," which catches stale names without testing outcomes. PostHog's notes on writing agent skills report that "The agent's effectiveness declined as skill descriptions filled the agent's context window."

If an LLM judge grades your runs, read a sample of the graded transcripts yourself.

"The agent can already read our docs. Why add a skill?"

This is the strongest objection, and the evidence gives it weight. In Vercel's Next.js 16 tests, a baseline with no docs and the skill at default behavior both scored "53%," while an "AGENTS.md docs index" scored "100%." Vercel built the eval itself, scored it on Build, Lint, and Test, and did not state the eval count, model, or harness.

Wix ran 250 evaluations, with each condition run 3 times per task. Their baseline reached docs through llms.txt and web fetch. The agent graded its own work, and the page does not state the agents, models, or year.

On CLI tasks, docs-optimized runs reached "85% completion vs 78% for skills-only runs." On REST API tasks, the two tied at 80%.

The research points both ways. SkillsBench (v4) covers 87 tasks in 8 domains across 18 model and harness configurations. It found "Curated Skills raise the average pass rate from 33.9% to 50.5%."

The same benchmark shows the limits. Software engineering was among the domains that "benefit least," and "13 of 87 tasks show negative Skills deltas." A separate study of skills in the wild found that when agents had to retrieve skills from a large pool, "the benefits of skills are fragile."

So skills can help on average and can also make a task worse. None of this settles your case, which is why the test answers the objection: the baseline arm is the agent reading your docs. If that arm already integrates correctly, the skill adds nothing on that task. Never run the baseline without docs, because a skill measured against Vercel's no-docs "53%" looks better than it would against the docs your users' agents can read.

If formal evaluation sounds like too much work, start small: the evaluating guide begins with two or three test cases and raw pass counts.

Make the call: keep, revise, or remove

The evaluating guide frames the decision: "The delta tells you what the skill costs (more time, more tokens) and what it buys (higher pass rate)." The guide adds that a skill that doubles token usage for a 2-point improvement may not be worth it. Claude Code's plugin evals keep two questions apart: a case passes when its with-plugin score meets the threshold, and "The with-minus-without delta is reported but never changes the exit code." Use the delta to judge what the skill buys, and the acceptance checks to decide whether a given version is safe to release.

• Keep the skill when every must-never-break check passes and the with-skill arm produces more whole-task passes on held-out tasks. Activation must also work on realistic prompts, at a cost you accept.
• Revise it when failures trace to activation, stale guidance, or instructions you can fix, then rerun the same tasks.
• Remove it when the docs-reachable baseline matches or beats it, or the gain does not cover the tokens, time, and upkeep.

Write the test boundary next to the decision (tasks, skill and docs versions, model, runtime, number of runs) and what remains unknown, such as prompts and runtimes you did not test. The product or DevEx lead owns the decision; whoever runs the evaluation owns the record.

Keep the record current

Version the skill, the docs, the model, and the runtime together in every record, and assign one owner for the skill. Rerun the relevant checks when any of the four changes, ideally in the same review step where your team edits skills (see our guide to Claude Code for technical writers).

Anthropic's skill authoring guide says there is no built-in way to run its example evaluations. The skill-creator automates much of the loop, and a spreadsheet is a fine start. The result holds for the tasks, versions, model, and runtime you tested. Safety and the next model or product release need their own checks.

The worksheet

Use one row per run. The two arms of a case share a case_id.

Copy the blank header row into a spreadsheet:

case_id,prompt,expected_result,acceptance_checks,arm,skill_version,docs_version,model_runtime_settings,input_snapshot,activation_evidence,outcome,checks_passed,elapsed_time,tokens,retries,human_interventions,evidence_link,failure_class,decision

Start with one task

Choose one real integration task your customers build, write its acceptance checks, and run it with and without the skill using the worksheet in this post. The Agent Skills guide on evaluating skills covers the run mechanics.

Your docs are half of this test, because the baseline arm is an agent reading them. EkLine applies the same before-and-after test to those docs: it starts with a baseline of what agents say about your product, and after you ship fixes you re-run the same questions and tasks and compare.

Sources

• Evaluating skill output quality, Agent Skills
• Optimizing skill descriptions, Agent Skills
• Agent Skills overview, Agent Skills
• Donating Agent Skills to the Agentic AI Foundation, Anthropic
• Skill authoring best practices, Claude Platform docs
• skill-creator SKILL.md, anthropics/skills on GitHub
• Test plugins with evals, Claude Code docs
• EkLine, EkLine
• AGENTS.md outperforms skills in our agent evals, Vercel
• We Ran 250 AI Agent Evals to Find Out if Skills Beat Docs, Wix Engineering
• SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks, arXiv, v4
• How Well Do Agentic Skills Work in the Wild, arXiv
• Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator, NVIDIA Technical Blog
• A Proposed Framework For Evaluating Skills, Tessl
• Agent Skills are an API Design Problem, not a Documentation Problem, Samuel Berthe
• What nobody tells you about writing agent skills, PostHog newsletter
• Agent Skills and Plugin, SignatureAPI docs
• Show HN: Agent-skills-eval, Hacker News