verify-ladder

A governed artifact in the stable bundle. Replacing this file on your machine makes your next attestation come back Drifted, naming this path.

Description	Walk the verification ladder — a one-way ratchet — when delivering any component (invariant, feature, refactor, fix). ONE runner command walks it (unit → integration → doctests → cross-crate → smoke → security → CHANGELOG, per-rung PASS / FAIL / NOT-RUN). Use when someone asks to verify, test, validate a change, "make sure it works", confirm a fix works, run the suite, check before a commit, or sign off a phase gate — and BEFORE hand-rolling any bare `cargo test` sequence yourself. `cargo test --lib` alone is NEVER sufficient.
Arguments	[component or change to verify]
Tools it may use	[Bash, Read, Grep, Glob]
Installs at	skills/verify-ladder/SKILL.md
Mode	0644 — never executable
Size	12555 bytes
SHA-256	5b54aeb73d7e4e1f2044a749234a517a5ac19199dec921c0f8feb8b2c262f526 fetch the raw bytes

The complete file, verbatim

This is the entire SKILL.md, including its YAML frontmatter: byte-identical to what installs, not rendered and not reformatted. Hash exactly what is below and you get the digest above. The table is only a reading aid; this block is the artifact.

---
name: verify-ladder
description: Walk the verification ladder — a one-way ratchet — when delivering any component (invariant, feature, refactor, fix). ONE runner command walks it (unit → integration → doctests → cross-crate → smoke → security → CHANGELOG, per-rung PASS / FAIL / NOT-RUN). Use when someone asks to verify, test, validate a change, "make sure it works", confirm a fix works, run the suite, check before a commit, or sign off a phase gate — and BEFORE hand-rolling any bare `cargo test` sequence yourself. `cargo test --lib` alone is NEVER sufficient.
argument-hint: "[component or change to verify]"
allowed-tools: [Bash, Read, Grep, Glob]
---

# verify-ladder — the one-way verification ratchet

When delivering ANY component, walk every rung and **report each rung's result
individually**. A green `cargo test --lib` is rung 1 of 7 — never the whole
story. Once a rung is added to a phase gate it is never relaxed. That is what
"one-way ratchet" means: standards only tighten.

The ladder is language-shaped in its examples (Rust / Cargo), but the shape is
not: every rung maps onto any toolchain. See **Non-Rust projects** below.

---

## 1. The runner — one command, not seven

The single most important thing in this skill is not the ladder. It is that the
ladder is **cheaper to run than to skip**.

We learned this the hard way, and then counted it. In a keyword count over one
corpus of our own engineering session logs — a stretch of work where the ladder
was documented, agreed, and cited in the phase gate — those sessions produced
**hundreds of raw `cargo test` invocations and not one walk of the full ladder**.
The ratio is the point, and it is the only part of this you should carry
forward — we are not asking you to trust a statistic you cannot check. Nobody was defying the process. The process was seven
commands and the shortcut was one. People take the cheap path, so make the
correct path the cheap path.

Wrap the whole ladder in one runner with this interface:

```
verify-ladder <crate-dir> [more-crate-dirs ...] [--smoke "<command>"]
```

Contract the runner must satisfy:

- Walks rungs 1–7 in order and prints **one verdict line per rung**.
- **A verdict is only ever printed for something actually evaluated.** A rung
  whose verdict is derived must derive it from *recorded results*, never from
  control flow having reached a line. This is the rule that runners break.
- `NOT-RUN` is a first-class verdict, printed as loudly as `FAIL`. It is never
  collapsed into silence and never rendered as a pass.
- A red rung does **not** abort the walk. You want the whole picture in one
  pass, not a bisect through seven invocations.
- One directory = rung 4 reports `NOT-RUN`, with the reason. Several
  directories = rung 4's verdict is whatever those crates' rungs actually
  returned. Note the limit honestly: the runner cannot know whether the
  directories you passed are really the *dependents* of the changed crate.
  Choosing them correctly is on you, which is why the report names the crates
  it walked.
- Exit non-zero if any rung is `FAIL`. `NOT-RUN` does **not** move the exit
  code — so exit 0 means "nothing went red", not "the full ladder ran". Read
  the verdict lines, not just `$?`.

### How a runner lies

Almost every false pass has the same shape: a line printing `PASS` because the
interpreter got there, not because anything was checked. Rung 4 is the classic
site. "More than one directory was passed" is a fact about `argv` — it is not a
test result, and a runner that treats it as one will cheerfully report
cross-crate `PASS` while a crate it just walked is red. The fix is not more
rungs; it is deriving each derived verdict from the results array.

The mirror-image failure is quieter and just as bad: reporting `FAIL` for a
check the runner could not perform. An unreadable check is `NOT-RUN`. Collapsing
"could not evaluate" into either colored verdict destroys the distinction the
whole skill exists to protect.

### Reference skeleton

A starting point, not a product. **Preconditions:** a Cargo workspace, `git` on
`PATH`, and bash 3.2 or newer (macOS's system bash qualifies — the auto-detect
below deliberately avoids `mapfile`, which is bash 4+). Adapt the rung-6 hook to
whatever linter you actually have; without one, rung 6 correctly reports
`NOT-RUN` rather than pretending.

```bash
#!/usr/bin/env bash
# verify-ladder.sh — one command, seven rungs, an honest verdict per rung.
# Deliberately NOT `set -e`: a red rung must be RECORDED, not abort the walk.
set -uo pipefail

SMOKE_CMD=""
LINT_CMD="${VERIFY_LADDER_LINT:-}"   # e.g. export VERIFY_LADDER_LINT="make security-lint"
DIRS=()

while [ $# -gt 0 ]; do
  case "$1" in
    --smoke)
      # `shift 2` with only one argument left fails and the loop spins forever.
      if [ $# -lt 2 ]; then echo "--smoke needs a command" >&2; exit 2; fi
      SMOKE_CMD="$2"; shift 2 ;;
    *)       DIRS+=("$1");       shift   ;;
  esac
done

# No dirs given: every crate touched in the working tree. This reads TRACKED
# changes only — a brand-new, untracked crate must be passed explicitly.
if [ ${#DIRS[@]} -eq 0 ]; then
  while IFS= read -r d; do
    [ -n "$d" ] && DIRS+=("$d")
  done < <(
    git diff --name-only HEAD 2>/dev/null | while IFS= read -r f; do
      d=$(dirname "$f")
      while [ "$d" != "." ] && [ ! -f "$d/Cargo.toml" ]; do d=$(dirname "$d"); done
      [ -f "$d/Cargo.toml" ] && printf '%s\n' "$d"
    done | sort -u
  )
fi

RESULTS=()
FAILED=0

run_rung() {           # run_rung <label> <command...>
  local label="$1"; shift
  printf '\n=== %s ===\n' "$label"
  if "$@"; then
    RESULTS+=("PASS     $label")
  else
    RESULTS+=("FAIL     $label")
    FAILED=1
  fi
}

skip_rung() {          # skip_rung <label> <reason>
  RESULTS+=("NOT-RUN  $1  — $2")
}

if [ ${#DIRS[@]} -eq 0 ]; then
  echo "No crate directories given and none detected in the working tree." >&2
  exit 2
fi

for d in "${DIRS[@]}"; do
  run_rung "rung 1 unit         [$d]" cargo test --manifest-path "$d/Cargo.toml" --lib
  run_rung "rung 2 integration  [$d]" cargo test --manifest-path "$d/Cargo.toml" --tests
  run_rung "rung 3 doctests     [$d]" cargo test --manifest-path "$d/Cargo.toml" --doc
done

# Snapshot BEFORE any later rung can move $FAILED. Rung 4's verdict is DERIVED
# from the crates actually walked above — never asserted from the argument count.
CRATE_RUNGS_FAILED=$FAILED

if [ ${#DIRS[@]} -gt 1 ]; then
  if [ "$CRATE_RUNGS_FAILED" -eq 0 ]; then
    RESULTS+=("PASS     rung 4 cross-crate  — ${#DIRS[@]} crates walked above, all green")
  else
    RESULTS+=("FAIL     rung 4 cross-crate  — a rung failed in one of the ${#DIRS[@]} crates walked above")
  fi
else
  skip_rung "rung 4 cross-crate " "only one crate given; pass the dependents explicitly"
fi

if [ -n "$SMOKE_CMD" ]; then
  run_rung "rung 5 smoke" bash -c "$SMOKE_CMD"
else
  skip_rung "rung 5 smoke       " "no --smoke command supplied"
fi

if [ -n "$LINT_CMD" ]; then
  run_rung "rung 6 security" bash -c "$LINT_CMD"
else
  skip_rung "rung 6 security    " "no linter configured; run the security pass by hand"
fi

# An unreadable check is NOT-RUN — not a failing check, and certainly not a
# passing one.
if ! git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
  skip_rung "rung 7 CHANGELOG   " "not a git work tree; confirm the entry by hand"
elif { git diff --name-only HEAD; git ls-files --others --exclude-standard; } \
       | grep -qi 'CHANGELOG'; then
  RESULTS+=("PASS     rung 7 CHANGELOG    — entry present in the working-tree diff")
else
  RESULTS+=("FAIL     rung 7 CHANGELOG    — no CHANGELOG change in the working-tree diff")
  FAILED=1
fi

printf '\n--- verification ladder ---\n'
printf '%s\n' "${RESULTS[@]}"
exit "$FAILED"
```

---

## 2. The ladder — run in order, report each

1. **Unit** — `cargo test --lib`
2. **Integration** — `cargo test --tests`. Cross-module and real-process
   boundaries: peer-to-peer networking paths, end-to-end message-broker paths,
   anything that only breaks when two components actually talk.
3. **Doctests** — `cargo test --doc`. Catches public-API examples that bit-rot.
   Documentation that no longer compiles is a defect with a long half-life.
4. **Cross-crate** — `cargo test --tests` in **each crate that depends on the
   modified crate**. A green home crate hides downstream breakage; this is the
   rung people skip and the rung that catches the expensive class of bug.
5. **Smoke** — boot the binary. Observe boot ordering. Query the data store.
   Verify the expected rows land with the expected shape. "It compiles" is not
   "it works"; "the job exists" is not "the job succeeded".
6. **Security hardening** — verify the new lines do not (a) introduce a
   vulnerability, (b) expose existing or new code to exploitation, or
   (c) violate CVE / NVD / OWASP / NIST guidance. A severity-tiered lint pass is
   the workhorse here (see the companion `security-lint` skill); for non-trivial
   crypto or trust-boundary code, dispatch a dedicated security-review agent or
   a second human.
7. **CHANGELOG** — entry added, per your project's commit rule.

---

## 3. Reporting rules

- Report each rung as **PASS / FAIL / NOT-RUN** — explicitly, one line each.
  There is no fuzzy fourth state.
- **If a rung was not run, say so.** The absence of a result must never be
  allowed to read as a pass. This is the single rule the whole skill exists to
  protect: a report that stops early and says nothing is indistinguishable from
  a complete one, and every reader downstream inherits the error.
- **Phase-gate sign-off requires the FULL ladder.** Partial is not signed off.
- **Never relax a rung to make it green.** "Lower the bar" is not a valid unblock
  path — fix the code or escalate the blocker. Standards are a one-way ratchet;
  a rung, once added to a gate, is never removed to unstick a delivery.
- Your own conclusion is not a fact. "I believe the integration tests would
  pass" is `NOT-RUN`, not `PASS`.
- The rules above bind the runner exactly as they bind you. A tool that reports
  a rung it did not evaluate is the same defect as a human doing it, shipped at
  scale — so hold the runner to the ladder before you trust it to hold you.

---

## 4. Scope shortcuts — legitimate, but still stated

Shortcuts are allowed. Silent shortcuts are not.

- **Docs- or comment-only change:** rungs 1–5 are often genuinely N/A — **state
  that they are N/A and why**. Still do rung 6 (no secret leaked, no internal
  hostname or path published) and rung 7.
- **Fix in a leaf crate with no dependents:** rung 4 may legitimately be empty.
  Report "no dependent crates" — do not skip it silently.
- **Generated or vendored code:** name what you did not test and why the risk is
  accepted.

---

## 5. Non-Rust projects — map the rungs, then name the mapping

The rungs are toolchain-independent. Translate them and **say out loud which
translation you used**, so a reader can tell what was actually exercised:

| Rung | Rust | Python | TypeScript / Node | Go |
|---|---|---|---|---|
| 1 unit | `cargo test --lib` | `pytest tests/unit` | `vitest run` (unit) | `go test ./...` (short) |
| 2 integration | `cargo test --tests` | `pytest tests/integration` | `vitest run` (integration) | `go test -tags=integration ./...` |
| 3 doctests | `cargo test --doc` | `pytest --doctest-modules` | doc-example type-check / `tsd` | `go test` on `Example*` funcs |
| 4 cross-crate | dependent crates | dependent packages in the monorepo | dependent workspaces | dependent modules |
| 5 smoke | boot the binary | import-smoke + boot the service | start the server, hit a route | run the binary, hit a route |
| 6 security | lint + review | lint + review | `npm audit` + lint + review | `govulncheck` + lint + review |
| 7 CHANGELOG | entry | entry | entry | entry |

These are conventional invocations, not universal ones — a project with a
different layout or test runner will need different commands. Name the commands
you actually ran in the report; that is what makes the mapping checkable.

---

## 6. Pairs with

- **Your commit ritual** — the ladder runs *before* staging. Verify, then
  commit. Not the other way around.
- **`security-lint`** — the rung-6 workhorse.
- **A cost/spend preflight**, if you run paid infrastructure — that is a
  separate gate on deploys, not a rung on this ladder. Do not merge the two;
  they answer different questions and block different actions.

All skills · Back to the overview