Reproduce a verdict

Every packet on this site is an auditable artifact, and this page makes the audit repeatable. Reproducing a verdict does not mean re-running the whole assessment. It means re-deriving the load-bearing evidence under the same protocol the packet used, and checking whether the verdict survives your run. The procedure has four steps; franken_threed is the worked exemplar because its assessor independently re-executed the test suite: 61 of 61 green.

Assessors: the protocol is RULEBOOK.md  ·  Builders: the runnable kit is starter-kit/  ·  Researchers: the suite synthesis starts at synthesis/00-overview.md

Section map. This page is the RULEBOOK.md §0 item 1 procedure: a second reader with no prior context can evaluate it. Terms are defined above the steps, commands are exact, and every packet reference names file and line (verified against the shipped packet: method at line 5, strongest evidence at line 20, claim rows 4 through 6 at lines 93 to 95, the 76/79 characterization at line 110, reproduction cost at line 141). The worked exemplar is franken_threed; the closing checklist generalizes to any packet.

Six definitions. The pin is the exact commit hash the assessment froze on; every claim is evaluated at the pin, not at HEAD. A claim is a checkable assertion a project makes in public. The claim inventory is the packet's claim table: every substantive claim, its recorded status, and the evidence behind it. NODUS is the program's four-ring verdict scale (Invest, Pilot, Explore, Monitor). TRL is technology readiness level, scored 1 to 9. CI is continuous integration: the project's automated test runs. The rider is the license clause withholding all rights, including benchmarking and testing, from OpenAI, Anthropic, their affiliates, and anyone acting for them.

Step 1Pin the commit

Find the pin in the packet's header (the "Pinned commit" line in the packet's method section). Clone the repository and check out that exact commit. Never work at HEAD: the pin is the assessed artifact, and anything newer is a different assessment.

The franken_threed exemplar: pin 72dcaa1adbad6b3d552a071cc7db31a16ee4ba9d (2026-09-22 11:10:21 −0400), which was HEAD at assessment time. The assessor shallow-cloned the repository (depth 380, reaching the 2026-09-09 initial commit) and set the working tree to the pin. The packet warns that the pin was HEAD when assessed and a newer commit may exist by read time: re-pin before citing.

# franken_threed exemplar
git clone --depth 380 https://github.com/Dicklesworthstone/franken_threed.git
cd franken_threed
git checkout 72dcaa1adbad6b3d552a071cc7db31a16ee4ba9d

Step 2Re-run the checks

The project's own CI workflow is the reference: read .github/workflows/ in the pinned tree and reproduce its gates locally. Prefer the gates the packet marks as the strongest evidence; those are the ones whose failure would move the verdict.

The franken_threed exemplar: the "Marching cubes integration" workflow. The assessor reproduced it locally in three parts: install the toolchain dependencies, check out the oracle (the pinned three.js r186 at 148ef33ecb6d2502ff796d4554abd1549c95d519, recorded in upstream/PIN.md, with the oracle blob SHA-1 29a405be… verified before use), then run the three test suites with the oracle root set.

# franken_threed exemplar: the full CI trio, re-executed by the assessor
cd tools
npm ci --ignore-scripts
# sparse checkout of three.js at the pinned r186 commit (oracle);
# verify the oracle blob SHA-1 (29a405be…) before use
F3D_THREE_ROOT=/path/to/three-r186 node --test marching_cubes_numeric
# 23/23 pass
F3D_THREE_ROOT=/path/to/three-r186 node --test marching_cubes_adapter marching_cubes_bundle
# 38/38 pass — 61/61 green in ~21 s on Node v24
Record the wartsThe assessor hit one local-only hiccup: the bundle suite needs build/three.module.js alongside three.core.js in the sparse oracle checkout. That is a sparse-checkout artifact, not a repo defect, and it goes in your notes, not your verdict.

The same workflow was green on GitHub Actions at the pin (run created 2026-09-22T15:10Z, conclusion success). Your local re-run and the CI-observed run are two independent executions of the same suites: that is the reproduction.

Step 3Work the claim inventory

Open the packet's claim table. For each load-bearing claim (the ones the verdict section leans on), compare the packet's recorded status against what your re-run shows. Claims that survive independent execution get upgraded in confidence; claims that don't get demoted with the counter-evidence you found.

The franken_threed exemplar: claims 4, 5, and 6 (the differential harness description, the harness claim, and the CI-green-at-pin claim) were upgraded to Verified-executed after the assessor's 61/61 re-run. Claim 21 records the honest boundary: the compiler's own unit suites run 76/79 at the pin, and the 3 failures were characterized as stale refusal-contract and error-text tests, not miscompilations, so the project's CI gate covers the marching-cubes exemplar, not the compiler's full contract. The packet's verdict section states what the re-run did and did not prove: the numeric-specialization slice verifies at High confidence; the outcome claims (speedups, the full application pipeline) are explicitly disavowed or unmeasured by the maintainer.

Full procedure reference: franken_threed-assessment.md, method at line 5, strongest-evidence summary at line 20, claim inventory rows 4–6 at lines 93–95, the 76/79 characterization at line 110, and the measured reproduction cost at line 141.

Step 4Apply evidence tiers

Every claim in your re-derivation carries a tier and a confidence grade. A claim without both is a draft note, not a finding. These are the program's own tiers, from the assessment protocol (RULEBOOK.md), also taught on the method page:

1 · VerifiedConfirmed by direct inspection of a fresh clone, an API response, or a live page read by the analyst. Your local 61/61 re-run lands here, graded Verified-executed.
2 · CI-observedObserved executing on live CI pages. Attests the suite runs, not that it is green, unless pass/fail is legible.
3 · Maintainer claimAsserted in README/docs by the maintainer; not independently executed or reproduced.
4 · ExternalIndependent sources: APIs, papers, press, third-party benchmarks. Absence of coverage is reported as a finding.
5 · InferenceThe analyst's judgment. Always labeled; never presented as fact.

Confidence grades: High (multiple converging sources or direct inspection), Medium (single solid source, plausible), Low (thin evidence, extrapolation).

ChecklistThe general procedure, for any of the 44 packets

  • Locate the pin. Each packet's header records the pinned commit hash and its timestamp. Clone at that commit. If the packet warns the pin was HEAD at assessment time, note the possibility of drift and do not silently substitute HEAD.
  • Read the packet's method section first. It records exactly what the analyst executed (tools run, artifacts fetched, APIs queried) and the toolchain used. Match the environment where it matters (the exemplar ran on Node v24; runtimes change results).
  • Re-run the project's own checks, not your own. The .github/workflows/ files in the pinned tree are the reference implementation of the gates. Where a gate needs credentials you lack (private runners, live oracles), record a typed skip: which gate, why it could not run, and that its evidence tier stays at Maintainer claim or CI-observed, never upgraded.
  • Re-execute the strongest evidence first. Every packet has a "strongest evidence" summary. That is the load-bearing row of the claim inventory; if it reproduces, the verdict's foundation holds, and if it doesn't, you have found the most important sentence in your report.
  • Work the full claim inventory, row by row. For each claim, record: the packet's recorded status, your result, the tier your result earns, and any counter-evidence. Demotion needs counter-evidence, not vibes.
  • Honor the "Not done" list. Every packet records what the analyst did not attempt (the exemplar: no Rust toolchain installed, nothing in crates/ compiled, no browser ran, no benchmark reproduced). Your reproduction inherits those boundaries. Do not overclaim past them.
  • Record divergences explicitly. Toolchain versions, sparse-checkout artifacts, oracle verification steps, any step you skipped and why. A reproduction whose environment differs from the packet's without disclosure is a new claim, not a confirmation.
  • Check the verdict-change conditions. Each brief carries a "What would change the verdict, in order" section. Test the necessary condition first: if your re-run breaks it, the verdict moves; if it holds, the verdict stands and your work is corroboration.