Feature matrix¶

Every CLI command and flag, every hook event per agent, every MCP tool, and the end-to-end agent self-test, run against throwaway projects. Generated by python bench/feature_matrix.py.

114 / 114 passed · keelyard 0.1.0 · commit 9360853 · Python 3.13.5 · Linux · 2026-10-14 · 15s

Expected exit codes: 0 ok, 1 error, 2 usage error, 3 check not passed. Rows that test an error path pass when keelyard fails cleanly with the right code.

setup¶

	case	command	exit	ms
✓	init --dry-run changes nothing	`keelyard init --dry-run --json`	0	164
✓	init --no-agents	`keelyard init --no-agents --json`	0	156
✓	init --agent claude (hooks, MCP, skill, commands, CLAUDE.md)	`keelyard init --agent claude --json`	0	150
✓	init twice is idempotent	`keelyard init --agent claude --json`	0	166
✓	install codex	`keelyard install codex --json`	0	169
✓	install opencode	`keelyard install opencode --json`	0	118
✓	install unknown agent is a usage error	`keelyard install cursor --json`	2	92
✓	doctor	`keelyard doctor --json`	0	130
✓	status	`keelyard status --json`	0	121

rewind¶

	case	command	exit	ms
✓	snap picks up the installed files	`keelyard rewind snap --json`	0	156
✓	snap with no change	`keelyard rewind snap --json`	0	127
✓	snap after an edit	`keelyard rewind snap --json`	0	151
✓	list	`keelyard rewind list --json`	0	120
✓	list -n 1	`keelyard rewind list -n 1 --json`	0	135
✓	show	`keelyard rewind show 4 --json`	0	128
✓	fork	`keelyard rewind fork 1 ../fork --json`	0	122
✓	to	`keelyard rewind to 1 --json`	0	155
✓	to an unknown step	`keelyard rewind to 999 --json`	1	132
✓	show a non-number	`keelyard rewind show x --json`	2	92

proof¶

	case	command	exit	ms
✓	add	`keelyard proof add adds numbers has a Cart --json`	0	136
✓	status (open) exits 3	`keelyard proof status --json`	3	128
✓	check needs evidence	`keelyard proof check 1 --json`	1	135
✓	check unknown id	`keelyard proof check 9 x --json`	1	122
✓	check	`keelyard proof check 1 test_add passes --json`	0	122
✓	test after (pass)	`keelyard proof test after --cmd python -m pytest -q -p no:cacheprovider --json`	0	459
✓	test with a missing tool exits 3	`keelyard proof test after --cmd no-such-tool-xyz --json`	3	118
✓	shot without url	`keelyard proof shot --json`	1	113
✓	report	`keelyard proof report --json`	0	141

context¶

	case	command	exit	ms
✓	run (pass-through exit)	`keelyard run --json -- python -c print('hi')`	0	151
✓	run a failing command	`keelyard run --json -- python -c import sys; sys.exit(4)`	4	135
✓	run a missing command	`keelyard run --json -- no-such-tool-xyz`	127	139
✓	run with nothing	`keelyard run --json`	2	119
✓	filter --for	`keelyard filter --for pytest -q --json`	0	119
✓	recall unknown id	`keelyard recall abc --json`	1	114
✓	recall bad id	`keelyard recall ../etc --json`	1	167
✓	code outline	`keelyard code outline app.py --json`	0	123
✓	code outline missing file	`keelyard code outline nope.py --json`	1	129
✓	code map	`keelyard code map --json`	0	121
✓	code find	`keelyard code find Cart --json`	0	118

memory¶

	case	command	ms
✓	index	`keelyard memory index --json`	166
✓	stats	`keelyard memory stats --json`	139
✓	search	`keelyard memory search websocket --json`	172
✓	search with odd input	`keelyard memory search NEAR( "* --json`	123

usage¶

	case	command	ms
✓	usage --by day	`keelyard usage --json`	121
✓	usage --by model	`keelyard usage --by model --json`	124
✓	usage --by agent	`keelyard usage --by agent --json`	127

swarm¶

	case	command	exit	ms
✓	swarm run missing file	`keelyard swarm run nope.json --json`	1	203
✓	swarm run bad JSON	`keelyard swarm run app.py --json`	1	186
✓	swarm plan	`keelyard swarm plan fix add for strings --json`	0	221
✓	swarm plan --subagents	`keelyard swarm plan where is add defined --subagents`	0	213

arena¶

	case	command	exit	ms
✓	arena run missing spec	`keelyard arena run nope.toml --json`	1	119
✓	arena render without results	`keelyard arena render nowhere --json`	1	124

bench¶

	case	command	exit	ms
✓	bench leaves the timeline alone	`keelyard bench --json`	0	302

cli¶

	case	command	exit	ms
✓	--version	`keelyard --version`	0	89
✓	no command prints help	`keelyard`	0	91
✓	unknown command	`keelyard frobnicate --json`	2	90
✓	reinstall codex (rewind removed it)	`keelyard install codex --json`	0	123
✓	reinstall opencode	`keelyard install opencode --json`	0	123
✓	uninstall codex keeps AGENTS.md for opencode	`keelyard uninstall codex --json`	0	114
✓	uninstall opencode removes the AGENTS.md block	`keelyard uninstall opencode --json`	0	139

hooks¶

	case	command	ms
✓	claude SessionStart	`keelyard hook claude SessionStart`	161
✓	claude UserPromptSubmit	`keelyard hook claude UserPromptSubmit`	116
✓	claude PostToolUse Read	`keelyard hook claude PostToolUse`	122
✓	claude PreToolUse Read (repeat read denied)	`keelyard hook claude PreToolUse`	106
✓	claude PostToolUse Write (snapshot)	`keelyard hook claude PostToolUse`	130
✓	claude PostToolUse Bash (output filtered)	`keelyard hook claude PostToolUse`	146
✓	claude PreCompact	`keelyard hook claude PreCompact`	126
✓	claude Stop (blocks without proof)	`keelyard hook claude Stop`	123
✓	codex PostToolUse apply_patch	`keelyard hook codex PostToolUse`	128
✓	codex Stop	`keelyard hook codex Stop`	128
✓	opencode PostToolUse edit	`keelyard hook opencode PostToolUse`	144
✓	unknown event is ignored	`keelyard hook claude Future`	140
✓	garbage stdin is ignored	`keelyard hook claude Stop`	125
✓	cwd outside any project is ignored	`keelyard hook claude Stop`	150

MCP tools¶

Called over stdio by the official MCP client. Error rows check that bad input is refused.

	tool	arguments	result
✓	`status`	`{}`	ok
✓	`workflow`	`{'name': 'plan'}`	ok
✓	`proof_requirements`	`{'items': ['adds numbers']}`	ok
✓	`proof_requirements`	`{'items': []}`	Error executing tool proof_requirements: 1 validation error for proof_requirementsArguments
✓	`proof_check`	`{'requirement_id': 1, 'evidence': 'test_add passes'}`	ok
✓	`proof_check`	`{'requirement_id': 7, 'evidence': 'x'}`	Error executing tool proof_check: no requirement 7
✓	`proof_run_tests`	`{'phase': 'after', 'command': 'python -m pytest -q -p no:...`	ok
✓	`proof_run_tests`	`{'phase': 'later'}`	Error executing tool proof_run_tests: 1 validation error for proof_run_testsArguments
✓	`proof_screenshot`	`{}`	Error executing tool proof_screenshot: no url given and [proof].app_url not set
✓	`proof_status`	`{}`	ok
✓	`rewind_list`	`{'last': 5}`	ok
✓	`rewind_diff`	`{'step': 1}`	ok
✓	`rewind_diff`	`{'step': 0}`	Error executing tool rewind_diff: 1 validation error for rewind_diffArguments
✓	`rewind_to`	`{'step': 1}`	ok
✓	`read_file`	`{'path': 'app.py'}`	ok
✓	`read_file`	`{'path': 'app.py'}`	ok
✓	`read_file`	`{'path': '../../etc/passwd'}`	Error executing tool read_file: ../../etc/passwd is outside the project root
✓	`read_file`	`{'path': '.keelyard/config.toml'}`	Error executing tool read_file: refusing to read keelyard internal state
✓	`run_compact`	`{'command': 'python -c "print(\'ok\')"'}`	ok
✓	`recall`	`{'output_id': 'nope'}`	Error executing tool recall: unknown output id 'nope'. Hint: use the id from the last line of run_co
✓	`outline`	`{'path': 'app.py'}`	ok
✓	`project_map`	`{}`	ok
✓	`find_symbol`	`{'name': 'add'}`	ok
✓	`usage_report`	`{'by': 'model'}`	ok
✓	`route_task`	`{'task': 'design the concurrency architecture', 'n_files'...`	ok
✓	`swarm`	`{'tasks': [{'kind': 'ask', 'prompt': 'where is add?', 'fi...`	ok
✓	`swarm_plan`	`{'goal': 'fix add so it handles strings'}`	ok
✓	`swarm_verify`	`{'answer': 'app.py:2`return a + b`', 'files': ['app.py']}`	ok
✓	`swarm_verify`	`{'answer': 'app.py:99 nothing', 'files': ['app.py']}`	ok
✓	`memory_search`	`{'query': 'websocket', 'this_project_only': False}`	ok

Agent self-test¶

keelyard selftest: a scripted agent completes a task through MCP and Claude Code hook payloads.

	step	detail	ms
✓	mcp: server starts and lists tools	13 tools, all described	0
✓	hooks: session start + prompt	baseline snapshot, 84-token brief injected	318
✓	proof: requirements recorded		5
✓	context: find_symbol + deduplicated read	cart.py:1: def total(prices, discount=0):	13
✓	memory: finds the fix from a past session	1 hit(s)	9
✓	swarm: helper answer is citation-checked	1/1 verified	40
✓	proof gate: blocks 'done' without evidence	- Requirement #1 not verified: total() applies the discount	279
✓	rewind: wrong edit undone	restored #1, undo with #2	39
✓	proof: tests pass, gate lets the agent finish	exit 0, report report.html	309
✓	context: run_compact output	89 chars	20