✓ Prove It
View the code →

Lesson 7 · Proof the pack works

A pack that tells your agent to prove its work had better prove its own. And there's a good reason not to trust your impression of any AI tool, this one included: in METR's 2025 randomised trial, experienced open-source developers took 19% longer on tasks when they used AI tools, while estimating afterwards that the tools had made them about 20% faster. METR has since published updated 2026 data, and the study page now says the 2025 results are out of date, so read them as a warning about impressions, not as a current speed number. The warning still stands: how a tool feels and what it does can point in opposite directions.

So this lesson gives you three ways to check Prove It: a test suite you can run in seconds, the demo tasks run with and without the plugin, and the knobs to tune or switch off each piece in your own projects.

1. Run the test suite

git clone https://github.com/georgehanu/prove-it.git
cd prove-it
npm test

No install step: the suite uses Node's built-in test runner and the hooks have no dependencies. You need Node 22 or newer. It takes under half a minute and prints more than a hundred passing tests.

There are two files:

The scenarios cover each decision a hook can make (block, ask, deny, add a note, stay silent), each .prove-it.json key, and inputs that should trigger nothing. They don't cover every command or edit an agent could come up with; the Limits section of each lesson lists the gaps I know of. What's in the fixtures:

When you change a rule, add a scenario first and watch it fail. The fixtures are plain JSON; copy one and edit it.

2. Run the demo tasks, with and without

The tests prove the hooks make the right decision for a given input. They don't prove an agent behaves better. For that, run the demo tasks yourself (setup in tasks/README.md). Each takes a few minutes:

  1. Copy only demo/ to a temp folder, git init, commit. Do this twice. The task files stay outside the copy: they list what a proven answer shows, and an agent that reads that first makes the comparison worthless.
  2. In the first copy, start your agent without Prove It and paste only a task's request.
  3. In the second, start it with Prove It (claude --plugin-dir path/to/prove-it/plugin, or installed) and paste the same request.
  4. Compare the endings against the task's "What a proven answer looks like" list. Did it run the checks after its last edit? Did the reply map each requirement to evidence? Did a test change get past you?

Run each pair more than once if you can. Agents vary from run to run; one good run without the plugin, or one bad run with it, tells you little.

What a headless run does

You can script the comparison with claude -p (non-interactive). Two things behave differently there, and it's worth knowing before you read the results:

Add --debug-file hooks.log to see every hook decision. The per-session event log that the Stop hook reads is at <CLAUDE_PLUGIN_DATA>/sessions/<session id>.events.jsonl, where CLAUDE_PLUGIN_DATA is the plugin's folder under ~/.claude/plugins/data/. Files untouched for 7 days are deleted.

3. Tune it: .prove-it.json

Put a .prove-it.json in your project root. Every key is optional:

{
  "checks": ["\\bnpm (run )?test\\b", "\\bmake verify\\b"],
  "testGlobs": ["**/*.test.*", "e2e/**"],
  "codeGlobs": ["src/**/*.{js,ts,tsx}"],
  "ignoreGlobs": ["**/node_modules/**", "**/generated/**"],
  "diffBudget": 600,
  "testGuard": "ask",
  "requireProof": true,
  "disable": ["shortcut-check"]
}
Key Default What it does
checks built-in list Regexes for commands that count as a check. When set, they replace the built-in list, and the Stop hook's message lists them. They are matched against the command with quoted text, heredoc bodies and -m/-e arguments removed, and against each part between &&, ; and pipes, so ^make verify also matches cd api && make verify, and git commit -m "make verify" matches nothing.
testGlobs *.test.*, *.spec.*, *_test.*, test_*.py, test/, tests/, __tests__/, spec/ ... Which files guard-tests protects. Replaces the default list.
codeGlobs common code extensions Which edits count as code for the Stop hook and shortcut-check. It doesn't narrow testGlobs: a test outside codeGlobs is still guarded.
ignoreGlobs node_modules, dist, build, coverage, vendor, .git, *.min.js Never treated as code. Replaces the default list.
diffBudget 400 Changed lines (against HEAD, plus new files) before the budget note. New untracked files count up to 200 files and 2 MB, never inside ignoreGlobs, node_modules, dist, build, .next, vendor or target; the note says when some were left out.
testGuard "ask" "deny" refuses weakening edits instead of asking.
requireProof true Once checks have passed, the Stop hook also wants a Proof block in the reply (a heading named "Proof", or **Proof** on its own line). false turns off only this rule; the check rules stay on.
disable [] Hook names to switch off, or "all".

A glob without a / matches the file name anywhere, like .gitignore. The file is read on every hook call, so changes apply straight away. If it isn't valid JSON, the defaults are used and the error is logged.

Turning things off

What's left to you

The pack makes the agent run checks, protects the tests, flags shortcuts, and hands you a brief. It can't decide whether a test checks the right thing, whether the root cause is really the root cause, or whether the change is what your users needed. Those are still your calls. The aim is that you spend your review time on them, and not on finding out whether the tests were run.

If the pack helped, or got in your way, I'd like to hear which piece and why.