Playbook

Full System Code Audit

A repeatable, skeptical, multi-agent audit methodology — run it against any project before it ships.

For review: this page is a self-contained methodology document shared for an independent LLM critique. Read it end-to-end and assess the rubric, the fan-out procedure, the self-debug protocol, and anything missing, weak, or contradictory.

Full System Code Audit ��� Playbook (v1.0)

Full System Code Audit — playbook (v1.0)

A strict, repeatable, LLM-operable protocol for auditing any project we build, before it ships. The audit is read-only and produces a prioritized, machine-checkable change list that becomes the future fix session's work queue.

This document is an execution protocol for an LLM agent, kept human-readable. Where it says "the agent," it means the auditing Claude session and its subagents. Follow the schemas and boundaries literally — they exist so the model cannot quietly skip work, overstate confidence, or emit plausible-but- unactionable findings.

When to run. Before a v1 / publishable cut; after a big milestone; before open-sourcing or handing to another user; when the docs are suspected of drift; or on a periodic cadence for anything in active development.

0. Purpose and non-goals

1. Audit Target Packet (fill before auditing)

If a value can't be discovered, write UNKNOWN and explain the search you did; then continue. Pin the audit to a commit SHA; never audit a moving working tree without recording dirty state.

## Audit Target Packet
- Repo path:
- Branch:
- Commit SHA:
- Dirty working tree: yes/no  (attach `git status --short`)
- Audit date:
- Primary language(s)/framework:
- Runtime / deploy target:
- Authoritative docs/specs (the source of truth when docs disagree):
- Build command:
- Test command:
- Lint / typecheck command:
- Probe command(s):
- Database / storage dependencies:
- External services:
- Explicitly excluded paths (vendored, generated, third-party):
- Live probes allowed: yes/no
- Network access allowed: yes/no
- Secrets access allowed: no by default

Values come from the project's CLAUDE.md / README where possible, not from inference. Language/build/test/probe commands feed the language adapter (Appendix A).

2. Read-only operating rules

Do not edit source, docs, config, migrations, lockfiles, generated files, repo metadata, or any external system. The audit produces findings only.

3. Evidence standard, severity, priority, confidence

Every finding cites concrete evidence. No vague "consider reviewing." Tag each finding with one or more evidence classes:

Severity = inherent impact:

Priority = recommended remediation order, distinct from severity. Rank by: severity + reachability + blast radius + likelihood + regression risk + fix- dependency order. Map to P0/P1/P2/P3 for the queue.

Confidence: confirmed | likely | plausible | needs-probe. Every high/critical finding must include either a reproduction/probe or a clear statement of why reproduction is unsafe/unavailable, plus a concrete reachability path.

Do not inflate style preferences into correctness findings.

4. Audit lenses

Every in-scope area is examined through every applicable core lens. Optional lenses become mandatory when the project has that surface.

Core (C1–C10):

Cross-cutting mandatory audits (dedicated agents, §7):

Product-readiness (P1–P3):

Optional lenses (mandatory when the surface exists): UI (accessibility, responsive, front/back type drift, client-side auth assumptions, loading/error/ empty states); AI/agent systems (prompt injection, tool-permission boundaries, untrusted tool output, exfil via prompts/logs, unsafe autonomous actions, eval coverage, guardrail bypass); API/spec compatibility (route inventory vs docs, status codes, error shapes, pagination semantics, contract drift); state-machine & invariants (impossible transitions, idempotent retries, duplicate/out-of-order events, partial failures).

5. Orientation (main session, cheap — do before fan-out)

Fill the Target Packet, then map the system. Standard command set (adapt per Appendix A):

pwd; git rev-parse --show-toplevel; git rev-parse HEAD; git status --short
find . -maxdepth 3 -type f | sort | sed 's#^./##'
rg -n "TODO|FIXME|HACK|XXX|BUG|DEPRECATED|TEMP|WORKAROUND" .
rg -n "process\.env|os\.environ|getenv|ENV\[|config|settings" .
rg -n "password|secret|token|api[_-]?key|private[_-]?key|credential" .

Then build these inventories (they scope the fan-out):

Read the central wiring yourself (config loader + entry point) — it anchors the product-readiness lenses.

6. Fan-out (parallel subagents, one per subsystem)

Dispatch general-purpose agents (they read files fully; Explore only reads excerpts) in a single message, multiple tool calls so they run concurrently. Assign by runtime path when critical behavior crosses directories — not blindly by directory. Give high-risk seams duplicate coverage (auth, persistence, migrations, billing/destructive actions, tenant isolation, queue/ event processing, external API/webhook handling).

Always include the three dedicated cross-cutting agents: interface-parity, docs-vs-reality, and productization/self-containment (lenses P1–P3 + the Appendix B grep + a proposed single-settings file).

Subagent assignment template:

You are auditing: <subsystem>
In-scope files: <explicit list>
Out-of-scope files: <explicit exclusions>
Required lenses: <lens IDs, incl. optional lenses that apply>
Known claims to verify: <claim list>
Required commands: <safe read-only commands>
Output: the Subagent output contract below. Completeness proof required.
Rules: read-only; do not edit/create repo files; do not start long-running
services or mutate shared DBs/caches without approval.

Subagent output contract — return exactly these sections:

1. Scope restatement (what you audited + exclusions)
2. Files fully read
3. Files searched/skipped and why
4. Commands / probes run  (and probes NOT run, and why)
5. Findings — each in the Finding schema (§9)
6. Claims checked and CONFIRMED true
7. Claims checked and CONTRADICTED
8. Highest-priority 5-bullet summary
9. Open assumptions / probes not run

"Do not claim a subsystem was audited unless all in-scope files were either read or explicitly excluded. Never invent a file, route, migration, setting, or test result — if not found, say not found. Do not mark something complete because an interface exists; trace at least one concrete call path."

7. Adversarial verification

The agent that produced a finding is not the one to confirm it.

  1. Fresh-context verifier per high/critical finding. Spawn a subagent that sees only the claim + cited code and is told to refute it (Writer/Reviewer split). Its disposition (upheld / refuted / needs-probe) is preserved in the final report — it is part of the output, not a throwaway.
  2. /code-review ultra as an independent deep pass. After fan-out, the operator runs /code-review ultra (multi-agent cloud review of the current branch) — or /code-review ultra <PR#> for a GitHub PR — as a second, independent, adversarial audit engine over the same diff/branch. It is user-triggered and billed; the audit agent cannot launch it itself — so the playbook surfaces it as a required operator step and the synthesis (§8) folds its findings into the finding queue (dedup by root cause, assign AUD- IDs, reconcile severity). If the working tree isn't a reviewable branch, commit a WIP branch or open a PR first so /code-review ultra has a target.

Constrain every verifier to correctness / stated-requirement gaps, not style — a critic asked to find problems always finds some.

8. Synthesis & dedupe (main session)

Write the report to <repo>/docs/audit-<YYYY-MM-DD>.md (or the repo's todo dir). Keep per-finding file:line. Executive summary first so a human owner can read the top in 10 minutes; detail after.

Final report schema:

# Audit Report
## 1. Verdict (is the code better or worse than the docs claim?)
## 2. Audit Target Packet
## 3. Executive risk summary
## 4. P0/P1 remediation queue
## 5. Findings by priority (Finding schema)
## 6. Interface-parity divergences (table)
## 7. Security / privacy findings
## 8. Data / migration findings
## 9. Productization / self-containment findings (+ config-drift table)
## 10. Docs-vs-reality findings (claim table)
## 11. Tests / probes missing
## 12. Claims that checked out
## 13. Files / areas audited
## 14. Files / areas NOT audited
## 15. Recommended fix order (numbered)
## 16. Appendix: commands / probes run  (+ /code-review ultra disposition)

9. Finding schema

### AUD-### — <short title>
- Severity: critical | high | medium | low
- Priority: P0 | P1 | P2 | P3
- Confidence: confirmed | likely | plausible | needs-probe
- Lens/category: <C#/P#/optional>
- Evidence class: STATIC | COMMAND | TEST | RUNTIME | DOCS | INFERRED
- Location: `file:line`  (or multi-file evidence + the command that connects them)
- What is wrong:
- Why it matters:
- Reachability / trigger:
- Suggested fix:
- Verification / probe to prove fixed:
- Verifier disposition (high/critical): upheld | refuted | needs-probe (by whom)

Format helpers by lens: security → attacker capability, entry point, vulnerable operation, impact, mitigation. concurrency → shared state, interleaving, lock/tx boundary, observable failure. parity → a table with both implementations and the specific divergent behavior. docs → doc claim and code reality side by side. performance → asymptotics tied to data size or a benchmark.

10. Acceptance criteria (audit is complete only when…)

11. Remediation protocol (for the future FIX session — not this one)

The audit session does not fix findings. A remediation session takes one finding ID at a time:

  1. Explore in plan mode (no edits) — summarize current behavior + the exact invariant before touching anything.
  2. State assumptions as verification questions (Chain-of-Verification), then answer each independently against code/spec — separate reads, not one hand-wave. The independence step stops self-confirmation.
  3. Reproduce first — write a failing test/probe that demonstrates the bug before any fix. If a direct failing test is hard, write one that passes on current (buggy) behavior, confirm it runs, then invert the assertion (AssertFlip).
  4. Fix, then behavior-probe — run the specific test/probe and paste actual output (curl the endpoint, read the body — not just "it builds").
  5. Deterministic gate — build + vet/lint + full test suite green (catches hallucinated APIs the model can't see; commands from Appendix A).
  6. Fresh-context adversarial review of the diff — /code-review (or ultra for high-stakes) sees only the diff + the finding's expected invariant, prompted to flag correctness/requirement gaps, not style.
  7. Only mark done when the probe output matches expectation — else park as "shipped, pending verification."

Remediation ticket (one per finding):

## AUD-001 — <title>
- Severity / Confidence / Files:
- Current behavior / Expected invariant:
- Failing test/probe to add:
- Fix approach:
- Commands to verify:
- Done condition:
- Regression risk / Rollback plan:  (mandatory if it touches migrations,
  config, destructive paths, auth, or data writes)

Finding status values (fix session updates the audit report): open | in-progress | fixed-pending-verification | verified | wontfix | invalid.

Rules: commit-level isolation (one finding or tight cluster per commit); don't broaden scope into unrelated refactors without approval; after two failed fix attempts, stop, summarize what failed, /clear, restart from the ticket.

Anti-patterns to refuse: declaring "fixed" without running it; self-review in the same polluted context; trusting recalled API signatures over the compiler/ docs; a reviewer that over-engineers style nits.

Primary sources: Anthropic Claude Code best practices (code.claude.com/docs/en/best-practices) + Chain-of-Verification (arXiv 2309.11495).

Appendix A — Language adapters (deterministic gate + discovery)

Swap the build/lint/test gate and discovery commands per language:

Appendix B — Owner/org-specific grep addendum (configurable, not hardcoded)

For P2 self-containment, grep the whole tree for owner-specific values and classify each hit as shippable code (bad) vs probe/doc (note). The list is parameterized from the org addendum, not baked here. Categories: local/private IPs, private domains, personal names/emails, hardcoded absolute paths, machine hostnames, cloud account IDs, webhook URLs, and named private services (e.g. the org's secrets manager, scheduler, notifier, home-automation host).

Appendix C — Worked example

The Axon homeserver audit (2026-07-06) is the reference run: axon/docs/audit-2026-07-06.md. It produced P0 data-loss findings, a 13-row memory-vs-postgres interface-parity divergence table, hot-path N+1s, SSRF/rate- limit findings, a productization change list with a proposed single settings file, and a docs-drift consolidation — in the report shape above. Use it to calibrate depth, not as a source of Axon-specific assumptions.

Quality bar: this is a strict execution protocol for an LLM, not soft human guidance. Schemas, boundaries, definitions, and completion criteria exist so the model cannot skip work, overstate confidence, or produce plausible-but-non- actionable findings.