Bokken to Shinken: The Week My AI Framework Became a Real Blade

Most open-source projects that promise to help you build AI agents never become trustworthy tools. They stay beautiful wooden swords. You can swing them in a demo and show them to colleagues — but when a real developer runs npx your-thing on a clean machine, the blade either isn’t there, can’t be verified, or quietly installs something subtly different from what you tested.

That gap matters more for agent tooling than for almost anything else you npx. These packages don’t print “hello world” — they drop skills, instructions, and guardrails into a repo and then steer an autonomous agent inside it. If the artefact a stranger pulls from the registry isn’t provably the artefact you built, you haven’t shipped a tool. You’ve shipped an unsigned instruction set for someone else’s automation. Supply-chain attacks on developer CLIs are no longer hypothetical; “it worked from my checkout” is not a security model.

In the dōjō you don’t start with live steel. You start with a bokken — a wooden sword: the shape, the weight, the arc of the cut, but no edge. For three months that was my Copilot Agents Dojo. It had skills, personas, a gate, a control plane — and you installed it by pointing npx at a raw GitHub tarball. Fine for practice. Not a real cut.

This week I made it shinken — a live blade: published to npm, signed by the pipeline, behind a gate that’s allowed to say no. Below is the deep version of each fix — the symptom, the mechanism, the fix, and why it generalises — and then the four principles that tie them together. Steal whichever you need; you don’t have to touch my repo.

Cut #1 — The jammed scabbard: npm sanitises your bin path

A katana’s first move is koiguchi wo kiru — breaking the seal so the blade leaves clean. My blade was welded into its scabbard, and I almost shipped it that way.

Symptom. The package declared its entry point the way a thousand tutorials show it:

"bin": { "copilot-dojo": "./dist/cli.js" }

Tests passed. npm pack produced a flawless tarball. But npm publish emitted one line between two green checkmarks:

npm warn publish "bin[copilot-dojo]" script name dist/cli.js was invalid and removed

Mechanism. npm normalises the manifest at publish time — not in the tarball — and its bin handling rejects values it considers unsafe. A bin entry becomes a symlink in node_modules/.bin/; to stop packages planting or escaping that directory, npm sanitises the path and drops entries it doesn’t like, the leading-./ form among them. The result is the nastiest kind of bug: the file is still inside the tarball, so every local check passes, but the registry manifest has no bin record. npx copilot-dojo resolves nothing.

This is why it’s invisible. npm pack && tar -tzf *.tgz shows the tarball’s package.json — which still has your ./ — so the artefact looks correct. The divergence only exists in the published manifest.

Fix — one character of absence:

"bin": { "copilot-dojo": "dist/cli.js" }

Set it with the tool, not by hand, so you can’t reintroduce the prefix: npm pkg set bin.copilot-dojo=dist/cli.js.

How to catch it before a stranger does. Local builds lie here; only the registry tells the truth.

npm publish --dry-run 2>&1 | grep -i warn      # the warning is the whole story
# after publishing, read it back from the registry, not your repo:
npm view copilot-dojo bin
# and actually draw the blade on a clean machine:
docker run --rm -it node:22 npx copilot-dojo@latest --help

Why it generalises. This is most dangerous for exactly the class of tool AI developers reach for via npx: zero-install CLIs. The package “installs” and then can’t be run, and the failure surfaces on someone else’s machine, in their voice, as your tool is broken. Treat publish warnings as errors until proven boring, and verify every CLI release from the registry, in a clean container, as a different user than yourself.

Cut #2 — The smith’s signature: provenance via OIDC, not trust

An expert appraising a katana slides off the handle and reads the mei — the maker’s signature chiselled into the bare tang. It’s how you know the blade is what it claims, by whom it claims, and not a convincing fake in good fittings.

Software grew a mei, and it’s stronger than a chisel mark because no human holds the pen.

What provenance actually gives you. Publishing with npm publish --provenance generates a SLSA build-provenance attestation that binds the published artefact to the exact source commit and the exact workflow that built it. The signing is keyless: GitHub Actions mints a short-lived OIDC identity token, Sigstore’s Fulcio issues an ephemeral signing certificate for that identity, and the signature is recorded in Rekor, a public append-only transparency log. There’s no long-lived key to leak and no human in the signing path — which is the point. It lands you around SLSA Build Level 2–3: tamper-evident, non-falsifiable after the fact, publicly auditable.

The 403 I hit, and the better answer. My first publish was refused:

403 — Two-factor authentication or a granular access token
with bypass enabled is required to publish.

I’d authenticated with an ordinary web-session token — a library card at the foundry door. A scoped granular access token got me through, but the token still exists, which means it can still leak. The genuinely better path is trusted publishing: bind the npm package to the GitHub repo + workflow and publish with no token at all — the OIDC identity is the credential. Fewer secrets, fewer rotations, smaller blast radius.

The setup is small:

# .github/workflows/release.yml
permissions:
  id-token: write        # lets the job mint the OIDC token Sigstore signs with
  contents: read
steps:
  - run: npm publish --provenance --access public

How your users check the mei — without trusting you:

npm audit signatures        # verifies registry signatures + provenance
npm view copilot-dojo dist.attestations

The verification you want to see — a registry signature and an attested build, not just one of them:

audited 1 package in 0s
1 package has a verified registry signature
1 package has a verified attestation

…or just the green provenance badge and “Built and signed on GitHub Actions” line on the package page.

Why it generalises. When someone installs your agent framework, provenance lets them cryptographically confirm the bytes weren’t altered between your commit and their machine — by a compromised registry mirror, a typo-squat, or a man-in-the-middle. For a package that configures an autonomous agent, that’s not polish; it’s the difference between trust me and check it yourself. Caveats worth knowing: provenance needs a public repo and CI-based signing (you can’t meaningfully attest a publish from your laptop), and the release tag must match the published version or the attestation is a lie with a certificate.

Cut #3 — The gatekeeper: make the gate fail closed

Old Japan had sekisho — barrier checkpoints. You didn’t simply walk to the next province; a gatekeeper checked your papers, and you passed only when everything was in order.

My framework’s sekisho is one command, verify.sh, that must come back fully green before anything heads for release. It checks the things a self-modifying agent framework can quietly break: spec invariants, the requirements-to-skills traceability matrix, JSON manifest validity, that every GitHub Action is pinned to a SHA, and that the skill smoke-tests pass.

The bug was the worst kind: a false negative. Two inspections had gone amber. The traceability check tripped over a teaching fixture it should have waved through — annoying, but loud. The dangerous one was silent: the smoke-test step globbed skills/*/tests, found nothing (the real suite lived in top-level tests/), and reported “no tests — pass.” A hundred tests sat one directory over, never run, while the gate flashed green.

That’s the asymmetry that matters. A false positive (the gate fails something fine) costs you an annoyed five minutes. A false negative (the gate passes something broken) costs you a shipped defect and a user’s trust. A gate that passes because it forgot to look is worse than no gate — it’s the done-gate that lies about finishing, wearing a uniform.

The fix is a principle, not a patch: absence of a result is a failure, not a pass.

# fail closed: finding zero tests is an error, never a silent success
mapfile -t suites < <(discover_tests)            # top-level tests/ + skills/*/tests
if (( ${#suites[@]} == 0 )); then
  echo "✗ gate: no test suites discovered — refusing to pass"; exit 1
fi
run_tests "${suites[@]}"                          # 107 tests, 0 skips

# exemptions are explicit, marked, and logged — never implicit
[[ -f "$dir/.teaching-fixture" ]] && note "exempt (teaching fixture): $dir"

  Before:  verify.sh → "all clear" (looked nowhere) → ship the flaw
  After:   verify.sh → 107 tests · 11 checks · 0 skips → THEN forge

Why it generalises. In CI/CD for any agentic system, the release gate is the single source of truth for “this is shippable” — and the agent itself may be the thing editing the code under test. So the default must be deny: unknown state fails, exemptions are explicit and auditable, and “nothing to check” is treated as “couldn’t verify,” never “verified.” Skips, continue-on-error, and empty globs are where green lies hide.

Cut #4 — Honest inventory: generate the truth, don’t type it

A samurai’s discipline starts with the inventory: know exactly what you carry, to the blade. My README was quietly lying. The badge said 29 skills; the real count was 32. The version badge was a release behind. The install examples pointed at a --ref v1.1 tag that did not exist — so anyone copy-pasting the quick-start was reaching for a sword that wasn’t on the rack.

None of this is glamorous, and all of it is trust. A reader who hits one broken command in your getting-started assumes the other ninety-nine are broken too, and they’re not wrong to.

The root cause is architectural, not careless: any fact a human maintains by hand is already drifting; it just hasn’t been caught yet. The fix is to make the docs derive from the code, with a single source of truth and a CI check that fails on drift.

# regen-skills-index.sh — the skill directory IS the source of truth
count=$(find skills optional-skills -name SKILL.md | wc -l | tr -d ' ')
generate_index > skills.md                        # the catalog
sed -i -E "s/skills-[0-9]+/skills-$count/" README.md   # the badge

# CI fails the build if the committed output is stale:
git diff --exit-code skills.md README.md \
  || { echo "✗ docs drifted from the skill index — run regen"; exit 1; }

The badge now reads whatever the build counts. The version stamp comes from package.json. The catalog is generated from the SKILL.md frontmatter. Nothing in the docs is a number I typed once and forgot. (Other ecosystems have idioms for this — <!-- generated --> regions, package.json scripts, a tiny Node/TS indexer — but the rule is constant: the artefact that contains the truth emits the docs.)

Why it generalises. As an AI framework grows — more skills, personas, tools — hand-maintained docs stop being a chore and become a liability: they misrepresent the agent’s actual capability surface, which is exactly the thing a user is trying to reason about before they trust it. Generated docs make “what does this do” answerable by the build, not by my memory.

Cut #5 — Name the blades: automate tags or they’re folklore

Great swords have names — Honjo Masamune, Kogarasu Maru. My releases had names in the changelog — Hardened Dojo, Self-Improving Dojo, Open-Door Dojo — and three of them had no git tag at all. The changelog claimed a history the repository couldn’t prove. A named blade with no signature on the tang is a story, not a record.

So I signed the tangs: every named release now has a matching annotated tag (git tag -a, which carries a message, author, and date — unlike a lightweight tag that’s just a moving pointer), retro-applied to the commit where it actually landed, plus a GitHub Release. This week’s work became Green-Belt Dojo — the first forged to live steel.

  v1.0.0  Hardened Dojo
  v1.1.0  Self-Improving Dojo     ← tang was blank; signed (銘 / mei)
  v1.2.0  Open-Door Dojo          ← tang was blank; signed
  v1.3.0  Green-Belt Dojo         ← the live blade (真剣 / shinken)

The real fix is to remove myself from the loop. A tag I have to remember to push is a tag I will eventually forget. Wire releases to conventional commits plus an automated release tool — release-please, semantic-release, or Changesets for monorepos — and the version bump, changelog entry, git tag, and GitHub Release all fall out of the merge. The human decides what the change means (feat:, fix:, feat!:); the machine decides the number and never forgets the tag.

One trade-off worth naming, because it bit me: this repo carries two artefacts — the framework and the copilot-dojo npm package — so it uses two tag namespaces. Framework releases are vX.Y.Z; the package publishes on copilot-dojo-vX.Y.Z, and only the package prefix triggers the publish workflow. That separation keeps a docs-only framework release from firing an npm publish, but it’s a sharp edge: tag with the wrong prefix and you either publish nothing or publish by surprise. In a monorepo, make the tag-to-pipeline mapping explicit and boring.

The Release Forge: four principles under the five cuts

Strip the metaphor and the five cuts collapse into four principles. They’re not about swords or npm — they’re how you make any agentic system trustworthy to a stranger.

1. Fail closed. Unknown state is failure, not success. Empty test discovery, a skipped step, a swallowed error — all of them are “couldn’t verify,” never “verified.” The gate’s entire value is its ability to say no.
2. Single source of truth. Every fact a human maintains by hand is already drifting. Counts, badges, versions, catalogs — derive them from the code and let CI fail on drift. Memory is not a data store.
3. Cryptographic truth over human memory. “I built it from this commit” is a claim. A Sigstore attestation in a transparency log is a proof. Where trust is the product, sign the artefact and let strangers verify without trusting you.
4. Verify from the outside. Test the published thing on a clean machine, not the thing in your checkout. npx in a fresh container, npm audit signatures, the registry’s manifest — the only honest test runs where your local state can’t help.

Here’s the same shift as a before/after — bokken on the left, shinken on the right:

Dimension	Bokken (practice)	Shinken (live steel)
Install	npx a raw GitHub tarball	npx copilot-dojo init from the registry
Identity	”trust my checkout”	SLSA provenance, signed via OIDC, in Rekor
Release gate	green even when it looked nowhere	fails closed: 107 tests, 0 skips, or no ship
Docs	hand-typed counts, drifting	generated from the skill index, drift = CI fail
History	named in the changelog, untagged	annotated tags + releases, automation-driven
Failure mode	”worked on my machine”	verifiable on a machine that never met me

The forge checklist (steal this)

No metaphor — just the commands. Run these against any CLI you publish, in order. If one fails, you have a bokken, not a shinken.

# Cut #1 — does the bin survive publishing? (registry lies live here)
npm pkg get bin                                   # no leading ./ on values
npm publish --dry-run 2>&1 | grep -i 'warn.*bin'  # must be empty
npm view <pkg> bin                                # read it back from the registry

# Cut #2 — is the artefact signed and attested?
npm view <pkg> dist.attestations                  # provenance present?
npm audit signatures                              # verifies signature + provenance

# Cut #3 — does the gate fail closed?
./verify.sh; echo "exit=$?"                        # non-zero on ANY gap, incl. 0 tests found

# Cut #4 — have the docs drifted from the code?
./scripts/regen-skills-index.sh && git diff --exit-code   # must be clean

# Cut #5 — does every named release have an annotated tag?
git for-each-ref --format='%(refname:short) %(objecttype)' refs/tags  # want 'tag', not 'commit'

# The whole blade — draw it as a stranger would, on a clean machine:
docker run --rm -it node:22 npx <pkg>@latest --help

What this unlocks

Not the feature list. The character — and what that character gives the people downstream.

For three months this thing could be rehearsed but not wielded. Now a stranger can draw it on a clean machine with one command, and verify from the signature alone that what they drew is exactly what I forged. For users of any AI framework that’s the whole game: reproducibility (the same artefact every install), verifiable behaviour (the guardrails you audited are the guardrails that run), and a lower trust burden (you check the math instead of trusting the author). That’s what makes safe adoption possible — you don’t have to know me to rely on it.

It isn’t free. Live steel has to be maintained: the gate has to stay honest, the provenance chain has to keep signing, the generated docs have to keep regenerating, every release. A bokken you can leave on the wall. A shinken you keep oiled, or it rusts — and rust on a tool that steers autonomous agents isn’t cosmetic. That ongoing cost is the difference between a demo and a dependency.

The bokken taught the form. The shinken can finally cut — and someone other than me can trust the edge. The dōjō was never the destination. It was where you got ready to carry live steel.

The dojo is open source. It draws clean now — npx copilot-dojo init — and you can check the signature on the tang yourself with npm audit signatures.