# Superpowers vs no plugin: 192 runs, one better deliverable

> I ran claude plugin eval on obra/superpowers vs no plugin: it changed how Claude worked on 3 of 7 tasks, improved the result on 1, and cost 1.33x per task.

URL: https://bartlomiejkrupa.dev/articles/does-superpowers-beat-no-plugin/
Published: 2026-09-29
Tags: claude-code, skills, plugins, evals, superpowers, benchmark, context-engineering
Summary: I ran claude plugin eval on obra/superpowers vs no plugin: it changed how Claude worked on 3 of 7 tasks, improved the result on 1, and cost 1.33x per task.

**Short answer: superpowers made Claude's output better on 1 of 7 tasks, and cost 1.33x per task.** On bug fixes, it added a regression test in 8 of 8 runs. Without the plugin, Claude skipped the test in 4 of 8.

I tested [obra/superpowers](https://github.com/obra/superpowers) v6.4.2 against the same model with no plugin: 12 tasks, 8 runs each, with and without, 192 scored runs. Cases, runner, and every transcript are [on GitHub](https://github.com/bartek-890/superpowers-evals).

What the plugin did:

| Area | Effect | Evidence |
| --- | --- | --- |
| Bug fixes (test-driven development, TDD) | **Better deliverable:** a regression test every time | 8/8 with, 4/8 without |
| New function (TDD) | Same code, but written test-first | Test-first 8/8 with, 1/8 without |
| Vague "build me X" request | Asks questions, writes **no code** | No code in 8/8 with; a working tool in 8/8 without |
| Debugging, verification, code review | No change: Claude already scored 1.00 alone | 1.00 / 1.00 (one case 0.96 / 1.00, explained below) |
| Tasks where the skill's behavior would be wrong | No harm done: same score with and without | 1.00 / 1.00 on 3 of 4, 0.94 / 0.94 on the 4th |
| Cost | 1.33x per task (mean of the 12 per-case ratios, range 1.08x to 1.63x), about 1.5x the turns | All 12 cases |

The takeaway: superpowers mostly installs habits (test first, ask first, run the code). Claude Sonnet 5.5 already delivers the same results without them on small tasks. The one habit that changed the result was writing the regression test.

## Bug fixes: the one real win

The prompt: "formatCents in src/money.js shows negative amounts as $-5.00 but it should be -$5.00. Please fix it."

**With superpowers (8 of 8 runs):** Claude wrote a regression test, ran it, watched it fail on `'$-5.00' !== '-$5.00'`, then fixed the code.

**Without the plugin (4 of 8 runs):** Claude edited `src/money.js`, never touched the test file, ran nothing, and said so: ["I haven't run the code or any tests."](https://github.com/bartek-890/superpowers-evals/blob/main/results/2026-09-29T14-57-41Z/traces/tdd-bugfix-regression/without-1.jsonl)

The fix itself was correct in every run, in both arms. The difference is whether a test protects it next time.

Consistency shows the gap best. Pick 3 runs at random: the chance that all 3 pass every check is 1.00 with the plugin and 0.02 without.

The real gap is bigger than the table shows. The "saw a failing test" check wrongly credited 3 no-plugin runs. They ran `node --test test/`, which fails because node can't load a folder, not because a test failed. Counted by hand from the transcripts: 4 of 8 no-plugin runs wrote a regression test at all, and 0 of 8 wrote it before the fix. With the plugin, 8 of 8 did both.

The checks were frozen before the full run, so the table keeps the scored 0.59. Corrected, that score drops to about 0.50 and the delta grows from +0.41 to about +0.50.

## New function: same code, different process

Asked to add a `truncate` function, both arms shipped a correct, tested function in 8 of 8 runs.

The only difference was order. With the plugin, all 8 runs wrote the test first and watched it fail. Without it, 1 of 8 did. The extra 1.41x cost buys a test-first habit, not a better function.

## Vague requests: the plugin asks instead of building

The prompt: "Build me a small command-line tool for tracking my reading list."

- [With superpowers](https://github.com/bartek-890/superpowers-evals/blob/main/results/2026-09-29T14-57-41Z/traces/brainstorm-vague-build/with-5.jsonl), 8 of 8: "I haven't written any code. Before I propose a design, I need to know what you want the tool for."
- [Without it](https://github.com/bartek-890/superpowers-evals/blob/main/results/2026-09-29T14-57-41Z/traces/brainstorm-vague-build/without-1.jsonl), 8 of 8: "I built the reading list tool as a single Python file, `reading.py`, in your working directory."

The brainstorming skill says asking first is correct, so the plugin scores 1.00 and the baseline 0.00. In a one-shot run with no human there to answer (headless), though, the plugin hands you nothing to run. Which one you want depends on whether you're there to answer.

The skill stayed quiet when it should. Told "Don't ask me anything, just build it", both arms built a correct `countWords` in 8 of 8 runs. On a two-line rename, brainstorming never fired.

## Debugging, verification, review: Claude already did it

Without any plugin, in 8 of 8 runs each, Claude:

- found the shared `parseDate` bug behind a failing invoice test instead of patching the symptom;
- refused to approve a broken rounding fix, even when told "no need to run the test suite";
- pushed back on a review comment that would have removed a needed null check.

The skills fired every time and added 1.3x to 1.6x cost, with no change in score.

The transcripts do differ. With the plugin, Claude ran code to check its answer much more often (counted as any Bash call running `node` or `npm`):

| Case | Runs that executed code: with | without |
| --- | --- | --- |
| review-pushback | 7/8 | 0/8 |
| review-valid-feedback | 8/8 | 1/8 |
| verify-before-claim | 3/8 | 1/8 |
| verify-correct-fix | 5/8 | 3/8 |

On these small tasks, reading the code was enough to get the right verdict. On a large codebase, running the code could matter. This test doesn't show that.

One case says nothing either way: in `verify-before-commit` the skill fired in only 2 of 8 runs, and the baseline ran tests before committing in 8 of 8 anyway.

## No harm where the skill's behavior would be wrong

Superpowers didn't invent problems in a correct fix, didn't argue with valid review feedback, and didn't brainstorm when told not to.

The only visible side effect was on a plain question about `git reset`. 1 of 8 answers [opened with](https://github.com/bartek-890/superpowers-evals/blob/main/results/2026-09-29T14-57-41Z/traces/control-simple-question/with-2.jsonl) "No other skill applies to a conceptual git question, so I'll answer directly."

## Full results

| Case | Kind | Score with / without | Δ | Cost ratio | Turns with / without | Skill fired |
| --- | --- | --- | --- | --- | --- | --- |
| brainstorm-vague-build | positive | 1.00 / 0.00 | **+1.00** | 1.08x | 3.0 / 3.2 | 8/8 |
| tdd-bugfix-regression | positive | 1.00 / 0.59 | **+0.41** [+0.16, +0.66] | 1.62x | 10.5 / 5.4 | 8/8 |
| tdd-add-function | positive | 1.00 / 0.78 | **+0.22** [+0.16, +0.25] | 1.41x | 8.0 / 5.8 | 8/8 |
| debug-root-cause | positive | 1.00 / 1.00 | 0.00 | 1.63x | 10.5 / 5.9 | 8/8 |
| verify-before-claim | positive | 1.00 / 1.00 | 0.00 | 1.32x | 4.9 / 2.9 | 8/8 |
| verify-before-commit | positive | 1.00 / 1.00 | 0.00 | 1.24x | 8.8 / 6.0 | 2/8 |
| review-pushback | positive | 0.96 / 1.00 | -0.04 | 1.34x | 9.8 / 6.0 | 8/8 |
| verify-correct-fix | negative | 0.94 / 0.94 | 0.00 | 1.41x | 5.0 / 2.8 | 8/8 |
| review-valid-feedback | negative | 1.00 / 1.00 | 0.00 | 1.45x | 10.0 / 4.5 | 8/8 |
| no-questions-build | negative | 1.00 / 1.00 | 0.00 | 1.16x | 5.0 / 5.1 | 0/8 |
| clear-small-change | negative | 1.00 / 1.00 | 0.00 | 1.14x | 5.0 / 5.2 | 0/8 |
| control-simple-question | control | 0.88 / 0.88 | 0.00 | 1.17x | 1.2 / 1.0 | 1/8 |

How to read it:

- **Positive case:** a task where the skill's behavior is the right one.
- **Negative case:** the same kind of task where that behavior would be wrong, so a skill that fires everywhere can't score well.
- **Control:** a plain question no skill targets.
- **Score:** the share of checks a run passed, averaged over 8 runs.
- **Δ (delta):** score with the plugin minus score without. Brackets are 95% intervals from resampling the 8 runs (bootstrap).
- **Skill fired:** runs where Claude actually invoked the skill the case targets.
- **Cost ratio:** average cost per run with the plugin divided by without.

The review-pushback -0.04 is one judge verdict I disagree with on review (see below), not a regression.

## How the test worked

The tool is `claude plugin eval`, shipped in Claude Code v2.1.269 on 2026-09-11. By default it runs every case twice: once with the plugin, once with nothing loaded (no plugin, no `CLAUDE.md`, no settings, no memory). Its docs sum up the point: "If a case scores 1.0 both with and without the plugin, the plugin isn't what made it pass."

| | |
| --- | --- |
| Plugin | superpowers v6.4.2: 15 skills, plus a `SessionStart` hook that injects the full `using-superpowers` skill into every session |
| Agent and judge model | `claude-sonnet-5-5`, both pinned |
| Claude Code | 2.1.284, in a Linux container |
| Cases | 7 positive, 4 negative, 1 control |
| Runs | 8 per arm per case |
| Cost | $17.88 for the reported runs, about $26 including smoke tests and a discarded v1 |

Each case is a tiny Node project with tests run by `node --test`. Prompts read like a real user and never name a skill, so the test also measures whether the skill triggers on its own.

Each run is scored by several pass/fail checks (graders). A check is a regex over a file or the transcript, a check on which tools were called, or an LLM judge with a written rubric.

Some checks look at **what** was delivered (the code, the reply). Others look at **how** (did it see a test fail first, did it ask before building). Superpowers is largely about the how, so both kinds are scored.

Every case that produces files has a reference solution. [A script](https://github.com/bartek-890/superpowers-evals/blob/main/scripts/check-references.py) checks that the reference passes all checks before any paid run.

One case, trimmed ([full file](https://github.com/bartek-890/superpowers-evals/blob/main/evals/tdd-bugfix-regression/case.yaml)):

```yaml
name: tdd-bugfix-regression
plugins: ["../../superpowers"]
context:
  scaffold_script: scaffold.sh
execution:
  max_turns: 40
  prompt: |
    formatCents in src/money.js shows negative amounts as $-5.00 but it should be -$5.00. Please fix it.
graders:
  - name: regression-test-added
    type: regex
    target: { source: file, path: test/money.test.js }
    pattern: "formatCents\\(\\s*-"
  - name: tests-green
    type: regex
    target: trace
    pattern: "# fail 0"
  - name: impl-correct
    type: llm
    focus: { source: file, path: src/money.js }
    criteria: |
      PASS if formatCents returns "-$5.00" for -500, "$12.34" for 1234, "$0.05" for 5, "$0.00" for 0.
      FAIL otherwise, or if you cannot tell from the file.
  - name: skill-fired          # reported, never scored: it can't pass without the plugin
    type: tool_used
    tool: Skill
    input_match: '"skill"\s*:\s*"(?:superpowers:)?test-driven-development"'
```

Why superpowers: it is the most popular skills framework for coding agents (292.7k GitHub stars when I checked), its skills make concrete claims, and it publishes no evals. If any skill set should show a gain, it's this one.

Prior evidence set low expectations. SkillsBench (February 2026, v1: 86 tasks, 11 domains) found curated skills add 16.2 percentage points on average, but only 4.5 in software engineering, the smallest gain of any domain.

This follows the method of [A 129-rule CLAUDE.md replied like an empty one](/articles/claude-md-adherence-bench), with its gaps closed: 8 runs per cell instead of 1, a pinned model, and checks frozen before the full run.

## Four bugs in the eval itself

Four bugs would have produced wrong numbers. Reading transcripts caught all four. Scores caught none.

The worst one looked like success. In an early smoke run, every Bash call in the container failed. The "did tests run" check still passed, because it only checked that Claude made the call.

| Problem | Symptom | Fix |
| --- | --- | --- |
| Broken sandbox | Every Bash call failed, yet a `tool_used: Bash` check passed because the call was made | Grade test results from node's own output (`# fail 0`); count sandbox errors per run |
| Invisible fix | The user's "fix" sat in the first commit, so `git diff` was empty and both arms refused to confirm it (0.81 / 0.75) | Leave the fix uncommitted; rerun scored 0.94 / 0.94 |
| Judge too small | Haiku 4.5 failed a correct answer in 3 of 3 votes | Judge moved to Sonnet 5.5; every rubric says "missing or unclear evidence is FAIL" |
| Loose red-test check | A folder-load error counted as "watched the test fail" | This run keeps the frozen scores; the fixed check in the next version requires an assertion failure before the source file changes |

The judge was stable: 1 split vote in 224 verdicts. I reviewed [all 12 failing verdicts](https://github.com/bartek-890/superpowers-evals/blob/main/results/2026-09-29T14-57-41Z/judge-audit.md) and disagree with 2. Each moves one score by 1/8 and changes no conclusion. I did that review with Claude, the same model family as the judge, so treat it as a first pass.

## Run it on your own skills

[The workflow](https://github.com/bartek-890/superpowers-evals/blob/main/docs/WORKFLOW.md) in five steps:

1. Write one positive case per skill claim, plus a negative case for each.
2. Give every file-producing case a reference solution and confirm the checks pass on it before paying for runs.
3. Smoke-test with 1 run per arm and read the transcripts, not just the scores.
4. Freeze the checks, then run the full suite.
5. Report what-was-delivered and how-it-was-done separately. Report scores as scored, even when a verdict looks wrong.

```bash
claude plugin eval . --runs 8 --model claude-sonnet-5-5 --judge-model claude-sonnet-5-5 \
  --allow-tools Bash Write Edit --scaffold --keep-temp --max-cost-usd 25
```

## Limitations

1. **Small tasks.** Twelve tiny Node projects are not a real codebase. Superpowers targets long, multi-step work (planning, subagents, worktrees). None of that was tested.
2. **Strong baseline.** Four positive cases scored 1.00 without the plugin. That shows Sonnet 5.5 already does this on these tasks, not that the skills never help.
3. **One-shot runs.** Brainstorming's "ask first" ends the run, so its value in a real conversation isn't measured.
4. **"How" checks follow superpowers' own idea of good.** "Watched the test fail" is TDD orthodoxy. The "how" checks are reported apart from the "what" checks, so you can weigh them yourself.
5. **Same-family judge, 8 runs.** Intervals are wide. A delta whose interval crosses 0 means no measurable effect, not no effect.
6. **One model, one plugin version.** Both pinned. A new model or superpowers release can move these numbers.

## Reproduce it

Everything is in [superpowers-evals](https://github.com/bartek-890/superpowers-evals): the 12 cases, scaffolds, reference solutions, container, raw transcripts, and a judge audit with the evidence behind each verdict. [RESULTS.md](https://github.com/bartek-890/superpowers-evals/blob/main/docs/RESULTS.md) has the full tables, [METHODOLOGY.md](https://github.com/bartek-890/superpowers-evals/blob/main/docs/METHODOLOGY.md) explains each design choice, and the README change log lists every grader change with its reason. To test another plugin, swap the submodule.

**Next move:** before you install a skill pack or write another skill, run it against no skill on three tasks you actually do. If Claude already scores 1.00 without it, you're paying in tokens and turns for habits, not results.

More: how skills load and call each other in [Claude Code skill composition](/notes/claude-code-skill-composition), and the cost of always-loaded context in [A 129-rule CLAUDE.md replied like an empty one](/articles/claude-md-adherence-bench). Series: [Context Engineering](/series/context-engineering).

Sources:
- Test plugins with evals: https://code.claude.com/docs/en/plugin-evals
- obra/superpowers: https://github.com/obra/superpowers
- SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks (v1): https://arxiv.org/abs/2602.12670v1
- bartek-890/superpowers-evals - suite, transcripts, and results: https://github.com/bartek-890/superpowers-evals
