Rigor practices worth copying

Engineering practices that make claims checkable, gathered from the 21 agent-stack evidence packs and the FrankenSuite corpus. Every practice is evidenced in a named project with a quoted source line, mapped to the items of the starter kit it strengthens, and marked with whether this repository adopts it. Of 136 practices, this repository has adopted 15, partly adopted 21, lists 23 as candidates, and marks 77 as not applicable; adopted and partial rows link their proof, and partial rows say what is done and what is missing. Adopted and partial claims about this repository were audited by a reviewer who did not write them; the audit record is in stack/reviews/.

TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Source file: stack/rigor-practices.tsv. Starter-kit items link to their entries in starter-kit/CHECKLIST.md.

PracticeWhat to copyEvidenced inAreasStarter kitOur status
RP-001Measured, justified CI job timeouts Size each job timeout-minutes from measured run history (about twice the slowest observed run) and write the measurement next to the number. modelcontextprotocol/inspectorSource: ecosystem/pickup/_evidence/mcp.md:26 roughly TWICE the slowest run observed MCP servers and clients none adopted.github/workflows/verify.yml:36; commit 0f14145
RP-002Concurrency cancel-in-progress for superseded PR runs Put CI in a concurrency group per pull request with cancel-in-progress, so a newer push cancels the stale run; keep push runs uncancelled. modelcontextprotocol/python-sdk, modelcontextprotocol/typescript-sdk, modelcontextprotocol/servers, browserbase/stagehand, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/mcp.md:29 on PRs (both SDKs, servers); python-sdk MCP servers and clients, Browser-use stacks, Realtime voice agents none adopted.github/workflows/verify.yml:22; commit 0f14145
RP-003Pin CI actions to commit SHAs Reference every third-party action by full commit SHA (tag in a comment) so a moved tag cannot change what CI runs. getzep/graphiti, huggingface/sentence-transformers, dottxt-ai/outlinesSource: ecosystem/pickup/_evidence/agent-memory.md:30 checkout/setup-python/setup-uv all pinned to SHAs Agent memory systems, Embedding model serving, Structured output and constrained decoding none adopted.github/workflows/verify.yml:41; .github/workflows/verify.yml:68; commit 0f14145
RP-004Scheduled runs so gates execute without pushes Add a nightly or weekly cron trigger so correctness, smoke, or benchmark jobs keep running when nobody pushes; drift then shows up on a date, not on the next unrelated change. trycua/cua, huggingface/text-embeddings-inference, michaelfeil/infinity, comet-ml/opik, cloudflare/sandbox-sdk, mlc-ai/xgrammar, meta-pytorch/torchtune, sgl-project/sglang, InternLM/lmdeploy, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/computer-use.md:32 scheduled live-environment smokes catch infra drift, not just unit failures Computer-use agents, Embedding model serving, Agent observability and tracing, Sandboxed code execution, Structured output and constrained decoding, Fine-tuning infrastructure, LLM inference engines, FrankenSuite none partialdone: weekly cron declared in the workflow; .github/workflows/verify.yml:15; commit 0f14145; missing: the first scheduled run has not fired yet
RP-005Least-privilege CI token by default Default the workflow token to permissions: contents: read and grant more only per job, with the reason in a comment. modelcontextprotocol/inspector, Lightning-AI/litgptSource: ecosystem/pickup/_evidence/mcp.md:26 with per-job overrides documented inline MCP servers and clients, Fine-tuning infrastructure none adopted.github/workflows/verify.yml:25; commit a298c99
RP-006Single required all-green status check Aggregate every job into one required check (re-actors/alls-green or a gate workflow) so branch protection names one status that a renamed or skipped job cannot bypass. modelcontextprotocol/python-sdk, mem0ai/mem0Source: ecosystem/pickup/_evidence/agent-memory.md:23 Single required status check + path-filtered builds MCP servers and clients, Agent memory systems none candidateApplies: main has no branch protection (gh api repos/JYeswak/franken-research/branches/main/protection returned HTTP 404 on 2026-09-24), so no status check is required at all. One job is not a required check.
RP-007Pin the whole toolchain, not just dependencies Commit lockfiles and toolchain files, install with the frozen or locked flag, and pin vendored cores and cross-repo dependencies by exact version or content hash. huggingface/text-embeddings-inference, huggingface/sentence-transformers, huggingface/text-generation-inference, ollama/ollama, Dicklesworthstone/franken_lean, Dicklesworthstone/franken_threed, Dicklesworthstone/franken_native_capsule, Dicklesworthstone/frankensimSource: ecosystem/pickup/_evidence/embedding-serving.md:28 Pin the whole toolchain, not just deps Embedding model serving, LLM inference engines, FrankenSuite none partialdone: esbuild pinned exactly, bun.lock installed frozen, bun pinned to 1.4.2; package.json:11; bun.lock; .github/workflows/verify.yml:4; .github/workflows/verify.yml:68; missing: node is pinned only to major 22, and the python3 and Chrome versions the gates use are neither pinned nor recorded
RP-008Supply-chain and secret gates as separate CI steps Run dependency audit, dependency firewall, and secret scanning as named gates of their own, before install touches the tree. getzep/graphiti, guardrails-ai/guardrails, Dicklesworthstone/franken_snowflakeSource: ecosystem/pickup/_evidence/agent-memory.md:30 Supply-chain hygiene in CI: pinned action SHAs, dependency firewall Agent memory systems, Guardrails and red-teaming, FrankenSuite none partialdone: the secret scan is its own step and runs before install; .github/workflows/verify.yml:58; .gitleaksignore; commit a298c99; missing: no dependency audit step and no dependency firewall
RP-009Verify tool and oracle integrity by hash before the suite runs Check the SHA-256 (or blob SHA) of the downloaded tool or oracle checkout before running anything with it; a mismatch fails the job. Dicklesworthstone/franken_threedSource: synthesis/uniqueness-catalog.md:192 The oracle checkout is verified by blob SHA-1 FrankenSuite Incumbent oracle pinned before implementation partialdone: the gitleaks tarball is sha256-checked before it runs; .github/workflows/verify.yml:53; commit a298c99; missing: Chrome gets only an existence check, and the canonical packets and site/assets/data.js the gates trust are not hashed before the suite runs
RP-010Regenerate derived artifacts in CI and fail on diff Rebuild generated files (types from the spec, OpenAPI from the server, bundles, ledgers) in CI and fail if the committed copy differs. modelcontextprotocol/typescript-sdk, langfuse/langfuse, qdrant/qdrant, Dicklesworthstone/franken_networkxSource: ecosystem/pickup/_evidence/observability.md:32 generate the spec from the server and diff it MCP servers and clients, Agent observability and tracing, Vector databases, FrankenSuite none adopted.github/workflows/verify.yml:76; commit a298c99; map bundle rebuilt from source and diffed
RP-011Public API or tool-surface diff gate on PRs Build the base and head versions and diff the exposed surface (tool names, schemas, endpoints), so any surface change becomes an explicit review item. github/github-mcp-server, qdrant/qdrantSource: ecosystem/pickup/_evidence/mcp.md:27 builds the server from base and head and diffs the exposed MCP tool surface MCP servers and clients, Vector databases none not applicableThis repository ships a static site, not an API or tool surface.
RP-012Docs-as-tests: documented commands and examples run in CI Execute README commands, doc snippets, and example scripts in CI so documentation cannot drift away from what actually runs. exa-labs/exa-py, deepset-ai/haystack, vibrantlabsai/ragas, guardrails-ai/guardrails, mit-han-lab/llm-awqSource: ecosystem/pickup/_evidence/web-search-apis.md:27 Copy this ruthlessly: example code rots unless CI runs it. Web search API layers, Retrieval-augmented generation (RAG) frameworks, Guardrails and red-teaming, Model quantization toolkits none partialdone: CI runs the README Reproduce commands; README.md:49; .github/workflows/verify.yml:3; missing: the README gate table ends at J while the script now has Gate K, and nothing fails when the README gate table disagrees with the script
RP-013Path-filtered, change-aware CI Run expensive jobs only when their paths change (paths-filter, paths-ignore for docs) and allow a labelled skip for costly evals. EleutherAI/lm-evaluation-harness, browserbase/stagehand, mem0ai/mem0, huggingface/text-embeddings-inference, jina-ai/readerSource: ecosystem/pickup/_evidence/eval-harnesses.md:26 CI validates only the benchmarks that changed LLM eval harnesses, Browser-use stacks, Agent memory systems, Embedding model serving, Web search API layers none not applicableThe whole gate chain runs in about a minute and the site is the docs; a docs-only skip would skip the gates that check them.
RP-014Reusable workflow composition Factor shared CI into workflow_call reusable workflows with pinned runtimes, called per package or per event. jina-ai/reader, trycua/cua, michaelfeil/infinitySource: ecosystem/pickup/_evidence/web-search-apis.md:32 reusable-workflow shape and the docs-skip Web search API layers, Computer-use agents, Embedding model serving none not applicableSingle workflow with a single job.
RP-015Per-package or per-integration CI workflows Give each package, SDK language, or framework integration its own workflow (or a discovered matrix entry) so ownership and failures stay local. modelcontextprotocol/servers, mem0ai/mem0, SuperMemoryAI/supermemory, trycua/cua, e2b-dev/E2B, comet-ml/opik, openlit/openlit, chroma-core/chroma, lancedb/lancedb, microsoft/autogen, firecrawl/firecrawlSource: ecosystem/pickup/_evidence/mcp.md:24 and fans out a per-server test matrix MCP servers and clients, Agent memory systems, Computer-use agents, Sandboxed code execution, Agent observability and tracing, Vector databases, Agent frameworks, Web search API layers none not applicableOne site, one package.
RP-016Cross-platform and multi-arch build matrix Build and test on every OS, architecture, runtime version, or hardware backend you ship for, with fail-fast off. kata-containers/kata-containers, ggml-org/llama.cpp, sgl-project/sglang, pytorch/ao, bitsandbytes-foundation/bitsandbytes, trycua/cua, microsoft/playwright-mcp, modelcontextprotocol/typescript-sdk, qdrant/fastembedSource: ecosystem/pickup/_evidence/sandbox-exec.md:59 Multi-arch build-check matrix (amd64/arm64/ppc64le/s390x/riscv64) Sandboxed code execution, LLM inference engines, Model quantization toolkits, Computer-use agents, Browser-use stacks, MCP servers and clients, Embedding model serving none candidateCI renders on ubuntu-latest only; the macOS reproduction path in the README is not exercised by CI.
RP-017Lint and format gates separate from test gates Run formatting and lint as their own required check, independent of heavy or hardware-bound CI, so PRs are reviewable without it. bitsandbytes-foundation/bitsandbytes, ggml-org/llama.cpp, huggingface/trl, Farama-Foundation/Gymnasium, OpenRLHF/OpenRLHF, deepspeedai/DeepSpeedSource: ecosystem/pickup/_evidence/rl-envs.md:22 Copy: formatting (ruff/black) as its own required check. Model quantization toolkits, RL environments and RLHF training none not applicableLittle code to format; copy quality is checked by Gate H instead.
RP-018Coverage reporting with a failure policy Upload coverage on every run and set fail_ci_if_error so a broken coverage upload is a red run, not a silent gap. jina-ai/reader, exa-labs/exa-py, noamgat/lm-format-enforcerSource: ecosystem/pickup/_evidence/web-search-apis.md:31 Pick the Jina strictness for a new project. Web search API layers, Structured output and constrained decoding none not applicableNo library code under test; line coverage is not a meaningful measure for this site.
RP-019CI failure monitoring, flaky-test tracking and auto-bisect Track CI failures over time, report flaky tests, and bisect automatically so a red main is traced to a commit without manual digging. sgl-project/sglang, temporalio/temporalSource: ecosystem/pickup/_evidence/inference-engines.md:28 CI failure monitoring and auto-bisect. LLM inference engines, Durable workflow orchestrators none not applicableRun volume is too small to have a flake population to track.
RP-020Retry only named transient failure classes Allow automatic reruns only for listed infrastructure errors (OSError, timeouts, HTTP 502/504, OOM), never for assertion failures. huggingface/trl, browser-use/browser-useSource: ecosystem/pickup/_evidence/fine-tuning.md:61 only for OSError/Timeout/HTTP 502/504/OOM messages Fine-tuning infrastructure, Browser-use stacks none not applicableverify.yml has no retry step and no flaky network test. Its one network call, the gitleaks download, is hash-checked and fails the run on error rather than retrying.
RP-021Privileged or expensive jobs only on explicit maintainer trigger Gate GPU, live-API, or dangerous e2e jobs behind a label, slash command, merge queue, or org-membership check instead of running them on every fork PR. cloudflare/sandbox-sdk, firecrawl/firecrawl, livekit/agents, axolotl-ai-cloud/axolotlSource: ecosystem/pickup/_evidence/sandbox-exec.md:57 Merge-queue + privileged-PR split (dangerous e2e tests only run with extra approval) Sandboxed code execution, Web search API layers, Realtime voice agents, Fine-tuning infrastructure none not applicableNo privileged or paid job exists and the workflow uses no secrets.
RP-022Tiered CI: fast tier on every PR, heavy tier gated or scheduled Split tests by marker or directory into a cheap always-on tier (unit, CPU) and heavy tiers (integration, GPU, e2e, live provider) that run on label, merge, or schedule; state the contract for each tier. axolotl-ai-cloud/axolotl, huggingface/trl, hiyouga/LlamaFactory, meta-pytorch/torchtune, langchain-ai/langgraph, getzep/graphiti, browserbase/stagehand, huggingface/text-embeddings-inference, UKGovernmentBEIS/inspect_ai, NVIDIA-NeMo/Guardrails, vllm-project/vllm, modelcontextprotocol/typescript-sdk, langfuse/langfuse-python, bitsandbytes-foundation/bitsandbytes, microsoft/graphrag, verl-project/verl, guidance-ai/guidance, weaviate/weaviate, livekit/agents, jina-ai/reader, temporalio/temporalSource: ecosystem/pickup/_evidence/fine-tuning.md:18 Tiered test pyramid: CPU unit tests on every PR, GPU e2e gated/scheduled Fine-tuning infrastructure, Agent frameworks, Agent memory systems, Browser-use stacks, Embedding model serving, LLM eval harnesses, Guardrails and red-teaming, LLM inference engines, MCP servers and clients, Agent observability and tracing, Model quantization toolkits, Retrieval-augmented generation (RAG) frameworks, RL environments and RLHF training, Structured output and constrained decoding, Vector databases, Realtime voice agents, Web search API layers, Durable workflow orchestrators none not applicableOne tier suffices: the full gate chain takes about a minute.
RP-023Deterministic scripted model and tool doubles shipped as a test API Ship fake models (and fake sandboxes and transports) that replay a scripted trajectory and that downstream users can import, so agent loops are tested with no network and no keys; one fake per scenario. openai/openai-agents-python, huggingface/smolagents, langchain-ai/langgraph, UKGovernmentBEIS/inspect_ai, EleutherAI/lm-evaluation-harness, guardrails-ai/guardrails, guidance-ai/guidance, livekit/agents, vercel/sandbox, tavily-ai/tavily-pythonSource: ecosystem/pickup/_evidence/agent-frameworks.md:19 Copy: make the fake model part of the SDK so downstream users can test agents deterministically too. Agent frameworks, LLM eval harnesses, Guardrails and red-teaming, Structured output and constrained decoding, Realtime voice agents, Sandboxed code execution, Web search API layers none not applicableThis repository makes no model or tool calls.
RP-024Strict scripted contract: unexpected or unconsumed steps fail Make drift between the test script and the actual trajectory a hard error in both directions: a call the script did not define, and a scripted step the run never used. openai/openai-agents-pythonSource: ecosystem/pickup/_evidence/agent-frameworks.md:22 turn drift between test script and agent trajectory into hard failures, not silent pass-throughs Agent frameworks none not applicableNo scripted doubles here.
RP-025Assert retry and backoff with scripted failure sequences Drive the client with a side-effect sequence (error, error, success) and assert the retry and backoff behaviour directly. huggingface/smolagentsSource: ecosystem/pickup/_evidence/agent-frameworks.md:24 Copy: assert retry/backoff behavior via side-effect sequences. Agent frameworks none not applicableNo retrying client code.
RP-026Record and replay external calls (VCR cassettes) Record LLM and HTTP interactions once and replay them in CI so unit tests are hermetic, cheap, and deterministic. NVIDIA-NeMo/Guardrails, langchain-ai/langchain, traceloop/openllmetry, lmnr-ai/lmnr-pythonSource: ecosystem/pickup/_evidence/guardrails.md:32 Copy: record once, replay in CI Guardrails and red-teaming, Retrieval-augmented generation (RAG) frameworks, Agent observability and tracing none not applicableThe gates call no model or HTTP API that could be recorded. The only network call is the pinned gitleaks download in CI.
RP-027Local deterministic test server or reference fixture instead of the live service Ship a local test server, or one kitchen-sink reference implementation, as the fixture every spec runs against, so core tests never depend on the live web. microsoft/playwright-mcp, modelcontextprotocol/servers, modelcontextprotocol/inspector, livekit/agents, inngest/inngest, cadence-workflow/cadenceSource: ecosystem/pickup/_evidence/browser-use.md:20 no external-network dependency for core specs Browser-use stacks, MCP servers and clients, Realtime voice agents, Durable workflow orchestrators none adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 9a99982; Gate I renders every page from file:// with no server and no network
RP-028Snapshot and byte-exact tests of full outputs Snapshot complete agent state, stream output, grammar text, or split text (or byte-compare shipped copies against the canonical ones) so unintended changes fail loudly. langchain-ai/langgraph, Azure-Samples/azure-search-openai-demo, ggml-org/llama.cpp, harbor-framework/harborSource: ecosystem/pickup/_evidence/agent-frameworks.md:25 Copy: snapshot full agent state/stream output to catch unintended trajectory changes. Agent frameworks, Retrieval-augmented generation (RAG) frameworks, Structured output and constrained decoding, LLM eval harnesses none adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 9a99982; Gate A byte-compares every shipped packet and the Rulebook with the canonical copy
RP-029Pinned baseline scores and golden numbers with an explicit tolerance Assert results against a pinned reference number (score, loss curve, tensor, retrieval metric) with a stated tolerance, one-sided where only regressions matter, and keep the ground truth checked in. vllm-project/vllm, AutoGPTQ/AutoGPTQ, meta-pytorch/torchtune, sgl-project/sglang, HKUDS/LightRAG, Azure-Samples/azure-search-openai-demoSource: ecosystem/pickup/_evidence/embedding-serving.md:21 One-sided assert: you only fail if Embedding model serving, Model quantization toolkits, Fine-tuning infrastructure, LLM inference engines, Retrieval-augmented generation (RAG) frameworks Incumbent oracle pinned before implementation not applicableNo numeric outputs of its own; Gate B recomputes the site statistics from data.js instead.
RP-030One conformance suite run against every backend or adapter Write the behaviour contract once and run the same suite over every backend, provider adapter, or bundled environment in CI, instead of ad hoc per-backend tests. langchain-ai/langgraph, dottxt-ai/outlines, vllm-project/vllm, Farama-Foundation/Gymnasium, Farama-Foundation/PettingZoo, mem0ai/mem0Source: ecosystem/pickup/_evidence/agent-frameworks.md:26 needs a backend-agnostic conformance harness, not per-backend ad-hoc tests Agent frameworks, Structured output and constrained decoding, RL environments and RLHF training, Agent memory systems Incumbent oracle pinned before implementation not applicableNo backends.
RP-031Official protocol conformance harness, pinned, separate from unit tests Adopt the protocol owner's conformance kit rather than hand-rolling protocol checks; pin its version in the workflow and keep it in its own job or repository. modelcontextprotocol/python-sdk, modelcontextprotocol/typescript-sdk, modelcontextprotocol/conformance, a2aproject/a2a-go, a2aproject/a2a-tckSource: ecosystem/pickup/_evidence/mcp.md:20 For a clean-room project: adopt the harness itself rather than hand-rolling protocol checks. MCP servers and clients, Agent-to-agent protocols (A2A class) Incumbent oracle pinned before implementation not applicableNo protocol implementation.
RP-032Named expected-failure baselines instead of silent skips List known-failing scenarios in a reviewed file, or as tests that must flip when fixed, so admitting a failure is explicit and a fix cannot go unnoticed. modelcontextprotocol/python-sdk, a2aproject/a2a-js, guidance-ai/guidanceSource: ecosystem/pickup/_evidence/mcp.md:21 Copy: named-baseline > silent skips. MCP servers and clients, Agent-to-agent protocols (A2A class), Structured output and constrained decoding Waivers are public, time-bounded, and recorded not applicableNo gate carries a known failure; every gate in verify.yml is hard.
RP-033Test every shipped spec revision and keep backward-compat fixtures Run conformance against each wire or spec version you still ship, and keep versioned fixtures (serialized agents, older-spec clients) so vN keeps working with vN-1. modelcontextprotocol/python-sdk, a2aproject/a2a-python, a2aproject/a2a-go, huggingface/smolagentsSource: ecosystem/pickup/_evidence/mcp.md:22 when the protocol versions its wire, test every shipped wire, not just latest MCP servers and clients, Agent-to-agent protocols (A2A class), Agent frameworks none not applicableNo versioned wire format.
RP-034Same behaviour tested on every binding, transport, runtime and language When a protocol or SDK has several bindings (JSON-RPC, gRPC, REST; Node, Deno, Bun; Python, Go, Java), test the same behaviours on each, with a named interop workflow and checked-in wire fixtures. a2aproject/a2a-go, a2aproject/a2a-python, agent-network-protocol/anp, firecrawl/firecrawl, milvus-io/milvus, openai/openai-agents-jsSource: ecosystem/pickup/_evidence/multi-agent-protocols.md:24 test the same behaviors on every binding, not just one Agent-to-agent protocols (A2A class), Web search API layers, Vector databases, Realtime voice agents none not applicableOne binding: static HTML.
RP-035Differential tests against an independent implementation or validator Compare outputs with a second implementation or a ground-truth validator on a shared corpus (HF transformers, a sibling SDK, Pydantic, the vendored incumbent) and treat divergence as the bug signal. vllm-project/vllm, ggml-org/llama.cpp, a2aproject/a2a-js, a2aproject/a2a-python, mlc-ai/xgrammar, e2b-dev/E2B, Dicklesworthstone/frankenredis, Dicklesworthstone/franken_threed, Dicklesworthstone/franken_numpySource: ecosystem/pickup/_evidence/inference-engines.md:22 Correctness via differential tests against HF transformers (perplexity/parity). LLM inference engines, Agent-to-agent protocols (A2A class), Structured output and constrained decoding, Sandboxed code execution, Model quantization toolkits, FrankenSuite Incumbent oracle pinned before implementation not applicableNothing here reimplements an incumbent; the corpus assesses others.
RP-036Unit-test the metrics, scorers and objective math themselves Test scorers, metrics, reward shaping, KL estimators, and loss aggregation as library code with fixture data, and make any shipped eval API run in the project's own CI. UKGovernmentBEIS/inspect_ai, EleutherAI/lm-evaluation-harness, run-llama/llama_index, deepset-ai/haystack, vibrantlabsai/ragas, confident-ai/deepeval, stanfordnlp/dspy, OpenRLHF/OpenRLHF, huggingface/trlSource: ecosystem/pickup/_evidence/rl-envs.md:18 Copy: gradient/shape-level tests for KL estimators, reward shaping, and loss aggregation LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks, Agent frameworks, RL environments and RLHF training none not applicableNo metric code.
RP-037Adversarial tests of your own grader, plus judge calibration sets Test that the grader cannot be fooled (spoofed PASS lines, truncated ids, skipped tests) and give any LLM judge its own labelled false-positive and false-negative calibration set, versioned apart from the attack corpus. SWE-bench/SWE-bench, centerforaisafety/HarmBench, microsoft/PyRITSource: ecosystem/pickup/_evidence/eval-harnesses.md:29 This is the single most copy-worthy practice: adversarial tests of your own scoring. LLM eval harnesses, Guardrails and red-teaming Gate self-test: deliberately break a gate candidateGate K3 checks that each source line contains its quote; a planted-bad TSV fixture that K3 must reject is not yet part of the gate chain.
RP-038Deterministic, model-free checker kept apart from generation Keep the code that decides PASS or FAIL importable and testable without any model, in a module separate from response handling or attack generation; prefer pinned tests in pinned containers over LLM judgment. SWE-bench/SWE-bench, ShishirPatil/gorilla, NVIDIA/garakSource: ecosystem/pickup/_evidence/eval-harnesses.md:32 Copy: the part that decides PASS/FAIL should be importable and testable without any model. LLM eval harnesses, Guardrails and red-teaming none adoptedsite/scripts/verify-site.sh; every gate is a deterministic script and no model call decides a PASS
RP-039Score the observable end state, not the agent's report Grade the resulting VM, repository, or rendered UI state (tests pass, spans render as a complete trace) rather than the transcript or an HTTP 200. xlang-ai/OSWorld, comet-ml/opik, Arize-ai/phoenix, SWE-bench/SWE-benchSource: ecosystem/pickup/_evidence/computer-use.md:22 State-based evaluation (score VM state, not chat) Computer-use agents, Agent observability and tracing, LLM eval harnesses none adoptedsite/scripts/verify-site.sh; commit 9a99982; Gate I loads each page in headless Chrome and fails on console errors or horizontal overflow
RP-040Explicit FAIL action for infeasible tasks Give infeasible tasks a defined correct answer (declare FAIL) so refusal is neither rewarded everywhere nor punished where it is right. xlang-ai/OSWorldSource: ecosystem/pickup/_evidence/computer-use.md:24 prevents reward-hacking by refusal Computer-use agents Anti-reward-hacking law adopted not applicableNo agent tasks are graded here.
RP-041Declarative task specs with LLM-judge criteria and a model matrix in config Write agent tasks as data (task, judge criteria, step budget), declare the model matrix in config, and run them as a reliability tier separate from unit tests. browser-use/browser-use, stanfordnlp/dspy, livekit/agents, pipecat-ai/pipecat, saharmor/voice-labSource: ecosystem/pickup/_evidence/agent-frameworks.md:27 declare the model matrix in config, not code Browser-use stacks, Agent frameworks, Realtime voice agents none not applicableNo agent tasks.
RP-042Virtual clock and time skipping in tests Run timing-dependent logic (timers, turn-taking, workflow sleeps) against a mocked clock that the test advances, so assertions never depend on wall time. temporalio/sdk-go, temporalio/sdk-java, temporalio/sdk-typescript, temporalio/temporal, cadence-workflow/cadence-go-client, livekit/agents, Dicklesworthstone/frankengraphdbSource: ecosystem/pickup/_evidence/voice-agents.md:22 fake clock so turn-taking/latency assertions Durable workflow orchestrators, Realtime voice agents, FrankenSuite none not applicableNo timing logic.
RP-043Deadlock detection: a workflow that blocks is a failure Put a deadlock-detection timeout on workflow coroutines so code that fails to yield fails the test instead of hanging it. temporalio/sdk-goSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:30 Deadlock detection for workflow coroutines (workflow code must yield; blocks are failures) Durable workflow orchestrators none not applicableNo concurrent workflow code.
RP-044Replay recorded histories, including a deliberately nondeterministic fixture Replay stored event histories (and agent-workflow histories) through the current code to catch nondeterminism, keep one fixture that must fail replay, and unit-test the journal itself as the durability primitive. temporalio/sdk-python, cadence-workflow/cadence-go-client, temporalio/temporal, restatedev/restate, triggerdotdev/trigger.dev, browser-use/browser-useSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:28 a deliberately nondeterministic fixture exists Durable workflow orchestrators, Browser-use stacks none not applicableNo stateful workflows.
RP-045Fault injection and network degradation in tests Inject faults (toxiproxy latency and drops, torn writes, cluster poisoning, noise and accent perturbations) through a harness, not by hand, and assert recovery. temporalio/temporal, cadence-workflow/cadence, livekit/agents, ServiceNow/eva, Dicklesworthstone/frankengraphdbSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:32 Fault-injection / cluster-poisoning / simulation harness for the engine itself Durable workflow orchestrators, Realtime voice agents, FrankenSuite none not applicableNo runtime to fault.
RP-046Stress, soak and load suites separate from functional tests Keep long-running stress, soak, pressure, and ingestion-load suites in their own tier with named workload shapes, off the PR critical path. chroma-core/chroma, qdrant/qdrant, kata-containers/kata-containers, mlc-ai/xgrammar, comet-ml/opikSource: ecosystem/pickup/_evidence/vector-dbs.md:28 Stress / soak suites separate from unit tests Vector databases, Sandboxed code execution, Structured output and constrained decoding, Agent observability and tracing none not applicableStatic site; no load to test.
RP-047Determinism and seeding assertions at every level Assert same seed gives the same trajectory at unit, checker, and vectorized levels, and requalify golden hashes across machines and build profiles. Farama-Foundation/Gymnasium, Farama-Foundation/PettingZoo, meta-pytorch/torchtune, Dicklesworthstone/frankensim, Dicklesworthstone/frankentui, Dicklesworthstone/frankensqliteSource: ecosystem/pickup/_evidence/rl-envs.md:17 Copy: same seed -> same trajectory assertions at unit, checker, and vector levels. RL environments and RLHF training, Fine-tuning infrastructure, FrankenSuite none not applicableNo stochastic code; build reproducibility is covered by the regenerate-and-diff row.
RP-048Standard benchmark harness in the repo, next to the code it measures Ship the throughput, latency, and quality harness in-tree (or in the library), with shared dataset configs and real workload shapes, not ad hoc scripts. vllm-project/vllm, sgl-project/sglang, InternLM/lmdeploy, huggingface/text-generation-inference, NVIDIA/TensorRT-LLM, ggml-org/llama.cpp, chroma-core/chroma, weaviate/weaviate, vespa-engine/vespa, trycua/cua, AutoGPTQ/AutoGPTQ, getzep/zep, mem0ai/memory-benchmarks, run-llama/llama_index, browserbase/stagehand, protectai/llm-guardSource: ecosystem/pickup/_evidence/inference-engines.md:18 Standardized serving benchmark harness (throughput/latency scripts, shared dataset registry). LLM inference engines, Vector databases, Computer-use agents, Model quantization toolkits, Agent memory systems, Retrieval-augmented generation (RAG) frameworks, Browser-use stacks, Guardrails and red-teaming none not applicableNo performance claims of its own.
RP-049Benchmarks unit-tested like code Test the benchmark CLI, timing functions, and parity between benchmark implementations so a broken harness cannot report a number. vllm-project/vllm, sgl-project/sglangSource: ecosystem/pickup/_evidence/inference-engines.md:19 Benchmarks are unit-tested like code. LLM inference engines none not applicableNo benchmark harness.
RP-050Performance regression as its own CI track, A/B against a baseline Run perf tests in a dedicated pipeline separate from correctness CI, as paired A/B comparisons against the baseline on PRs or nightly, with results kept as artifacts. vllm-project/vllm, sgl-project/sglang, firecracker-microvm/firecracker, mlc-ai/xgrammar, trycua/cuaSource: ecosystem/pickup/_evidence/sandbox-exec.md:39 perf tests run as A/B comparisons against baseline on PRs LLM inference engines, Sandboxed code execution, Structured output and constrained decoding, Computer-use agents none not applicableNo performance claims of its own.
RP-051Measure the overhead of the safety layer itself Benchmark what the isolation, constraint, or guard mechanism costs (seccomp filter cost, grammar compile and mask time, scanner latency) alongside what it prevents. google/gvisor, mlc-ai/xgrammar, vllm-project/vllm, protectai/llm-guardSource: ecosystem/pickup/_evidence/sandbox-exec.md:30 Seccomp-bpf filter benchmarking (cost of the isolation mechanism itself) Sandboxed code execution, Structured output and constrained decoding, Guardrails and red-teaming none not applicableNo safety layer at runtime.
RP-052Latency budgets asserted as tests Define latency metrics (time to first audio, boot-to-init, startup, snapshot restore) as tested code with a spec to assert against, not only as dashboard numbers. pipecat-ai/pipecat, livekit/agents, firecracker-microvm/firecracker, e2b-dev/E2BSource: ecosystem/pickup/_evidence/sandbox-exec.md:37 Boot-to-init time spec test: parse Realtime voice agents, Sandboxed code execution none not applicableNo latency-sensitive runtime.
RP-053Report quality delta together with the speedup Every optimization or quantization method reports the quality change (perplexity, downstream tasks) next to its speedup, from one command. pytorch/ao, NVIDIA/TensorRT-LLM, ggml-org/llama.cpp, EleutherAI/lm-evaluation-harnessSource: ecosystem/pickup/_evidence/quantization.md:25 reports perplexity delta, (c) reports kernel speedup, all one command Model quantization toolkits, LLM inference engines none not applicableNo optimizations to report.
RP-054Reproducible contender containers and parameter sweeps (report a frontier, not a point) Benchmark every contender in its own container against identical datasets and sweep index, query, and quantization parameters, publishing the recall-vs-throughput frontier. erikbern/ann-benchmarks, qdrant/qdrant, milvus-io/milvus, weaviate/weaviateSource: ecosystem/pickup/_evidence/vector-dbs.md:24 Docker-per-algorithm reproducibility for benchmarks Vector databases none not applicableNo benchmarks.
RP-055Security properties get dedicated adversarial tests Assert isolation and auth properties directly: jail uid/gid and permission bits, seccomp blocking the forbidden syscall set, known-vulnerability regression, signature paths with an independent reimplementation, audit events that must fire. firecracker-microvm/firecracker, e2b-dev/E2B, a2aproject/a2a-js, a2aproject/a2a-python, fetchai/uAgents, agent-network-protocol/anpSource: ecosystem/pickup/_evidence/sandbox-exec.md:27 verify the BPF filter actually blocks the forbidden syscall set Sandboxed code execution, Agent-to-agent protocols (A2A class) none not applicableNo isolation or auth surface.
RP-056Resource-limit and worst-case safety tests Test that timeouts, kills, rate limits, and compilation budgets are enforced, including pathological inputs that would stall the serving loop. firecracker-microvm/firecracker, e2b-dev/E2B, vllm-project/vllmSource: ecosystem/pickup/_evidence/structured-output.md:27 guards against pathological regexes DoS-ing the serving loop Sandboxed code execution, Structured output and constrained decoding none not applicableNo runtime resource limits.
RP-057Fuzzing and malformed-input batteries as maintained equipment Keep a fuzz corpus and targets for parsers, run them on a schedule, and assert that every malformed input class is refused. deepset-ai/haystack, oven-sh/bun, Dicklesworthstone/franken_node, Dicklesworthstone/frankenredis, Dicklesworthstone/franken_overlap, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/rag-frameworks.md:26 Fuzz corpus for parsers Retrieval-augmented generation (RAG) frameworks, FrankenSuite none not applicableNo parsers of untrusted input.
RP-058Property-based tests with explicit invariants Generate inputs with property strategies and assert named invariants (cross-version persistence, metamorphic identities) instead of example-only tests. chroma-core/chroma, Dicklesworthstone/frankenjax, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/vector-dbs.md:27 Property-based correctness testing (Hypothesis strategies + explicit invariants) Vector databases, FrankenSuite none not applicableNo library code.
RP-059UB and sanitizer lanes over the unsafe-dense code Run sanitizers, coverage, and Miri (with the aliasing model and its rationale recorded) as separate lanes over the code with the most unsafe. firecracker-microvm/firecracker, ggml-org/llama.cpp, oven-sh/bun, Dicklesworthstone/frankenfsSource: synthesis/vendor-port-learnings.md:680 G10. MIRI — UB detector over the unsafe-dense corner [Bun]. Sandboxed code execution, LLM inference engines, FrankenSuite none not applicableNo unsafe code.
RP-060Version and pin benchmark tasks and datasets; record the commit in every run Never mutate a benchmark silently: version task configs, pin dataset versions or upstream commits in run manifests, bundle third-party datasets with provenance, and keep old configs reproducible. EleutherAI/lm-evaluation-harness, harbor-framework/terminal-bench-1, xlang-ai/OSWorld, simular-ai/Agent-S, Skyvern-AI/skyvern, microsoft/PyRITSource: ecosystem/pickup/_evidence/eval-harnesses.md:25 Copy: never silently mutate a benchmark; bump a version. LLM eval harnesses, Computer-use agents, Browser-use stacks, Guardrails and red-teaming Incumbent oracle pinned before implementation partialdone: every packet is assessed at a pinned commit recorded in its header; RULEBOOK.md:68; packets/asupersync-assessment.md; missing: no benchmark task or dataset versioning and no per-run manifest recording the commit of each CI run
RP-061Separate benchmark data from harness code Keep the harness repository code-only and the datasets in their own versioned home, referenced by id. UKGovernmentBEIS/inspect_ai, UKGovernmentBEIS/inspect_evals, harbor-framework/terminal-bench-1, EleutherAI/lm-evaluation-harness, microsoft/graphragSource: ecosystem/pickup/_evidence/eval-harnesses.md:30 Copy: harness repo stays code; datasets live in their own versioned home. LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks none not applicableNo harness.
RP-062Registry as code enumerates the supported surface and drives the test matrix Declare supported models, evals, chunkers, or attack plugins as typed registry entries so the test matrix is enumerable and a new entry is a new file. qdrant/fastembed, openai/evals, HKUDS/LightRAG, promptfoo/promptfooSource: ecosystem/pickup/_evidence/embedding-serving.md:24 the supported-model contract and makes test matrices enumerable Embedding model serving, LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks, Guardrails and red-teaming none candidateThe 21 stack slugs are fixed by ecosystem/pickup file names; a registry file that Gate K reads would make that contract explicit.
RP-063Named attack families and labelled pass/fail cases in code Enumerate attack and evasion families as named modules with metadata, and keep false-positive and false-negative cases as parametrized test data, not a spreadsheet. promptfoo/promptfoo, NVIDIA/garak, protectai/llm-guard, microsoft/PyRITSource: ecosystem/pickup/_evidence/guardrails.md:28 Copy: enumerate attack families as named modules with per-module metadata Guardrails and red-teaming none not applicableNo guard to attack.
RP-064Tool-call gating as a separate guard family Treat guards on tool inputs and outputs as a category of their own, separate from prompt-text guards, with their own tests. meta-llama/PurpleLlama, NVIDIA-NeMo/GuardrailsSource: ecosystem/pickup/_evidence/guardrails.md:34 Copy: guardrails for tool input/output are a separate category from prompt text rails. Guardrails and red-teaming none not applicableNo tools exposed.
RP-065Test tree mirrors the source tree, with a shared test-utils module One test file or package per component, trainer, or encoder family, mirroring the source layout, with shared fixtures extracted into a test-utils package. run-llama/llama_index, deepset-ai/haystack, huggingface/trl, huggingface/sentence-transformers, microsoft/autogen, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/rl-envs.md:19 Copy: per-algorithm test files + a shared fixtures module. Retrieval-augmented generation (RAG) frameworks, RL environments and RLHF training, Embedding model serving, Agent frameworks, Realtime voice agents none not applicableTests are the gate chain in one script.
RP-066Smallest real instance end to end: tiny-model smoke and golden-path examples Run the full pipeline (config validation, load, train or quantize, save, evaluate or generate) on the smallest real model in single-digit steps, and run one golden-path example per format in CI. axolotl-ai-cloud/axolotl, huggingface/trl, meta-pytorch/torchtune, Lightning-AI/litgpt, hiyouga/LlamaFactory, mit-han-lab/llm-awqSource: ecosystem/pickup/_evidence/fine-tuning.md:34 Tiny-model smoke trains as the standard unit of correctness (steps in the single digits) Fine-tuning infrastructure, Model quantization toolkits none not applicableNo training or model pipeline.
RP-067Config validation as its own tested surface Expose config validation and normalization as an API and exercise it in every smoke test and per-recipe config test. axolotl-ai-cloud/axolotl, meta-pytorch/torchtune, Lightning-AI/litgptSource: ecosystem/pickup/_evidence/fine-tuning.md:49 Config validation as its own testable surface Fine-tuning infrastructure none partialdone: Gate K1 inside bun run verify rejects missing front-matter keys, bad enums and missing sections; site/scripts/verify-site.sh; commit 68d9523; missing: validation is not exposed as a separate API, and Gate K has not yet run on pushed CI
RP-068Keep hardware CI tractable: cache priming and CPU legs for distributed logic Prime model caches into every runner, cover distributed logic on CPU, and use free external hardware legs with canary-vs-control runs. axolotl-ai-cloud/axolotl, hiyouga/LlamaFactory, unslothai/unslothSource: ecosystem/pickup/_evidence/fine-tuning.md:55 Keeping GPU CI tractable: caching + model-cache priming + free external GPU legs Fine-tuning infrastructure none not applicableNo hardware-bound tests.
RP-069Assert telemetry against semantic conventions, including streaming span lifecycles Per instrumented provider, assert that spans follow the semantic conventions, that streaming spans close and carry token usage, and that the export path (masking, batching, propagation) is unit-tested. traceloop/openllmetry, openlit/openlit, langfuse/langfuse-python, lmnr-ai/lmnr-python, langchain-ai/langsmith-sdkSource: ecosystem/pickup/_evidence/observability.md:26 Copy: semantic-convention assertions per instrumented provider, not just Agent observability and tracing none not applicableNo telemetry.
RP-070Scheduled correctness jobs for facts that change underneath you Re-check externally changing facts on a schedule (model prices, upstream provider SDK changes) with their own job, instead of trusting the snapshot taken when the code was written. langfuse/langfuse, comet-ml/opik, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/observability.md:33 Copy: cost tracking needs its own scheduled correctness job Agent observability and tracing, Realtime voice agents none candidateThe 21 evidence packs are 2026-09-23 GitHub API snapshots (stars, push dates, file paths) with no scheduled re-check.
RP-071Post-deploy checks against the shipped artifact After every deploy, run the quality or smoke check against the live production artifact, labelled with the commit, not only against the branch build. firecrawl/firecrawl, trycua/cua, letta-ai/letta-code, Dicklesworthstone/franken_markdown_website, Dicklesworthstone/frankentui_websiteSource: ecosystem/pickup/_evidence/web-search-apis.md:22 every merge to main gets an automatic quality benchmark run on the shipped artifact Web search API layers, Computer-use agents, Agent memory systems, FrankenSuite none candidateDeploys to fr.zeststream.ai are manual and no check compares the served pages with the repository.
RP-072Live-provider tests wired to secrets, skipped on forks Test against the real endpoint in CI with repository secrets, path-triggered and skipped on forks, and keep a unit tier that intercepts transport for everything else. tavily-ai/tavily-python, exa-labs/exa-py, serpapi/serpapi-python, livekit/agents, NVIDIA-NeMo/Guardrails, guidance-ai/guidanceSource: ecosystem/pickup/_evidence/web-search-apis.md:25 test against the real endpoint in CI, with unit-only separation for the rest Web search API layers, Realtime voice agents, Guardrails and red-teaming, Structured output and constrained decoding none not applicableNo live provider.
RP-073Automated checks a contribution must pass before merge Document the automated checks a new benchmark or spec change must pass (policy gate script, spec lint, spelling, conventional commits) and run them before human review. UKGovernmentBEIS/inspect_ai, UKGovernmentBEIS/inspect_evals, a2aproject/A2ASource: ecosystem/pickup/_evidence/multi-agent-protocols.md:25 Copy: spec changes go through machine-checked lint before human review. LLM eval harnesses, Agent-to-agent protocols (A2A class) none partialdone: every pull request runs the full gate chain; .github/workflows/verify.yml:10; CONTRIBUTING.md; missing: main has no branch protection (HTTP 404 on 2026-09-24), so a red check does not block a merge
RP-074Change record enforced: changelog or release note per change Require a changelog entry when the version changes, or a release-note fragment per change, and automate the release record. mem0ai/mem0, deepset-ai/haystack, microsoft/graphrag, exa-labs/exa-pySource: ecosystem/pickup/_evidence/agent-memory.md:24 Changelog enforcement keyed to version bump Agent memory systems, Retrieval-augmented generation (RAG) frameworks, Web search API layers none candidateCHANGELOG.md is kept by hand; no check ties it to a version or tag.
RP-075Agent contributor playbook in the repository Keep AGENTS.md or CLAUDE.md with the build, test, and conduct rules agents must follow, next to the code. lancedb/lancedb, milvus-io/milvus, weaviate/weaviate, chroma-core/chroma, Dicklesworthstone/franken_whisper, Dicklesworthstone/franken_markdown, Dicklesworthstone/frankenfsSource: ecosystem/pickup/_evidence/vector-dbs.md:37 how these projects onboard agents; worth mirroring Vector databases, FrankenSuite Anti-reward-hacking law adopted candidateCONTRIBUTING.md covers humans; there is no AGENTS.md stating the 12 forbidden patterns for agents working in this repository.
RP-076Claim-to-proof registry with machine gates Bind every public claim to a machine-checkable key and a proof; the gate refuses wording above its evidence grade, demotes on digest mismatch, and fails CI on drift. Dicklesworthstone/franken_engine, Dicklesworthstone/frankengit, Dicklesworthstone/franken_markdown, Dicklesworthstone/franken_node, Dicklesworthstone/frankengraphdb, Dicklesworthstone/frankensim, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:7 Every public claim bound to a machine-checkable key and a proof script FrankenSuite Claim inventory registered before claims are made; Claim discipline enforced: README claims cross-checked against proofs partialdone: Gate A2 checks method-page claims against their packet line, K2 checks verdict quotes on their cited lines, K3 resolves proof tokens; site/scripts/verify-site.sh; commit 1ef0f8e; commit 68d9523; missing: no gate compares wording with its evidence grade or demotes a claim on a digest mismatch
RP-077Freshness decay on evidence Give every claim a maximum age since re-verification; the gate downgrades rows whose check has lapsed, so verified cannot silently mean verified once. Dicklesworthstone/franken_engineSource: synthesis/cross-pollination.md:15 the gate auto-downgrades rows whose re-verification lags FrankenSuite Demotion rules executed on schedule candidatePackets and evidence packs carry assessment dates, but no gate demotes a claim when its date ages past a ceiling.
RP-078Negative-evidence ledger with required entry classes and retry conditions Log rejections, retractions, and failed optimizations as first-class rows with mandatory classes and a retry condition, so a rejection is a paused hypothesis. Dicklesworthstone/frankengit, Dicklesworthstone/franken_numpy, Dicklesworthstone/frankensqlite, Dicklesworthstone/franken_tts, Dicklesworthstone/franken_networkx, Dicklesworthstone/frankenlibc, Dicklesworthstone/frankentorchSource: synthesis/cross-pollination.md:23 A ledger where rejections, retractions, and failed optimizations are first-class rows FrankenSuite Negative-evidence ledger initialized with its schema; Every falsified hypothesis banked with a retry predicate candidateRejected approaches for this repository are recorded only in commit messages and synthesis prose; there is no ledger file with retry predicates.
RP-079Anti-extrapolation: an unrun baseline is recorded as incomplete When the incumbent cannot be run within budget, record the comparison as incomplete or no admissible ratio; never extrapolate a ratio against a baseline that did not run. Dicklesworthstone/franken_overlap, Dicklesworthstone/franken_ttsSource: synthesis/cross-pollination.md:31 A baseline that exceeds the compute budget is recorded as FrankenSuite Incumbent oracle pinned before implementation partialdone: the Rulebook forbids citing an ungated number; RULEBOOK.md:80; missing: no rule that a baseline which could not be run is recorded as incomplete with no ratio
RP-080Committed statistical baseline gate with environment tagging Commit the benchmark baseline with per-row environment tags and captured hyperparameters, sign it off, and let a drift detector that demonstrably fires block regressions. Dicklesworthstone/asupersyncSource: synthesis/cross-pollination.md:39 Benchmark regression as a signed, waiver-free gate FrankenSuite none not applicableNo benchmarks of its own.
RP-081Per-optimization proof bundles Ship each accepted optimization as one reviewable bundle: SHA-pinned golden transcript, paired and reversed runs, and a keep or reject score gate. Dicklesworthstone/frankenredis, Dicklesworthstone/frankenjaxSource: synthesis/cross-pollination.md:47 Every accepted optimization ships a bundle FrankenSuite none not applicableNo optimizations.
RP-082Measurement-integrity four-rule gate Name the worker, name the harness, quote both estimators, and replicate before quoting; comparisons across different workers are invalid. Dicklesworthstone/frankentorch, Dicklesworthstone/frankenpandas, Dicklesworthstone/frankengraphdbSource: synthesis/cross-pollination.md:55 Name the worker, name the harness, quote both estimators, replicate before quoting FrankenSuite Receipts carry provenance; generation binding not applicableNo measurements of its own.
RP-083Result-class doctrine for performance verdicts Type every performance verdict: a self-speedup is maintenance; a campaign win needs the incumbent binary in the same invocation with A/A nulls in range; preregister the verdict rule. Dicklesworthstone/franken_whisper, Dicklesworthstone/franken_networkx, Dicklesworthstone/franken_overlap, Dicklesworthstone/frankenmermaidSource: synthesis/cross-pollination.md:63 SELF-SPEEDUP is maintenance, never a campaign win FrankenSuite Incumbent oracle pinned before implementation not applicableNo performance verdicts of its own.
RP-084Structured skip honesty: an absent oracle or unrun gate fails Emit a typed skip event instead of passing silently; make the absence of the oracle, the browser, or any scanned item a failure; refuse dormant lanes with a nonzero exit instead of a false green. Dicklesworthstone/franken_snowflake, Dicklesworthstone/frankenscipy, Dicklesworthstone/franken_tts, Dicklesworthstone/frankengitSource: synthesis/cross-pollination.md:71 Runs that cannot execute honestly emit a typed skip event instead of silently passing FrankenSuite none adopted.github/workflows/verify.yml:84; site/scripts/verify-site.sh; commit a298c99; commit 9a99982; no browser found is a FAIL, and zero data-stat slots or zero resolved links fail Gates B and E
RP-085Maturity vocabulary and dated maturity registry Label every capability on a ladder (source-present to production-admitted) where no lower label implies a higher one, and publish the dated distribution; partial never rounds up. Dicklesworthstone/franken_drone_geometry_reconstruction, Dicklesworthstone/frankensim, Dicklesworthstone/franken_ocr, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:79 no lower label implies a higher one FrankenSuite Proof taxonomy declared (provisional) candidateApplies: site/self/index.html scores this repository with the same TRL and ring vocabulary it applies to others (commit a298c99), but there is no dated maturity registry and no rule that a lower label never implies a higher one.
RP-086Evidence color and no-laundering composition Give evidence typed strengths and compose them so an aggregate can never carry a stronger color than its weakest component. Dicklesworthstone/frankensim, Dicklesworthstone/frankengraphdbSource: synthesis/cross-pollination.md:87 composition rules prevent weak evidence from being laundered into strong claims FrankenSuite Proof taxonomy declared (provisional) adoptedRULEBOOK.md:19; stack/METHOD.md; the verdict itself is always tier Inference, whatever its citations
RP-087Receipt-bound remote execution with self-disqualification Bind each remote batch to source hashes, job id, and raw logs; keep failed attempts; any code landing after the last executed gate opens a self-disqualification until the gate reruns. Dicklesworthstone/franken_alignment, Dicklesworthstone/franken_node, Dicklesworthstone/franken_remoteSource: synthesis/cross-pollination.md:95 code landing after the last executed gate opens a self-disqualification ledger FrankenSuite Receipts carry provenance; generation binding not applicableAll gates run on public GitHub Actions; there is no private runner to receipt.
RP-088Full-funnel verdict aggregation with selection pressure stated Aggregate every measured lane into typed verdicts, including dropped and undecidable ones, and quote the decidable-subset statistic with its selection pressure. Dicklesworthstone/frankenpandas, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:103 Aggregate every measured lane into typed verdicts FrankenSuite none not applicableNo measured lanes.
RP-089Adversarial self-audit: gates proven to trip on planted defects Plant defects (a laundered claim, a weakened gate, a one-byte drift) and record that the gate fails; keep a permanent self-test that proves the gates still trip, and a mutation-score floor where it fits. Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensympy, Dicklesworthstone/franken_code_browser, Dicklesworthstone/frankensim, Dicklesworthstone/franken_node, Dicklesworthstone/frankenredisSource: synthesis/cross-pollination.md:111 The evidence gate itself gets audited adversarially FrankenSuite Pre-commit honesty gate runs and self-tests with a canary; Gate self-test: deliberately break a gate partialdone: one-time planted defects recorded as failing: packet drift fails A and console.error fails I (9a99982), renamed data-stat and href fail B and E (a298c99), planted bad copies fail Gate K (68d9523); commit 9a99982; commit a298c99; commit 68d9523; missing: no permanent known-bad fixture that CI runs every time
RP-090Registry-agreement gate: parallel registries must agree exactly When several tables describe the same set (registry, slug map, factory, data file), fail the build unless they agree exactly. Dicklesworthstone/franken_agent_detectionSource: synthesis/uniqueness-catalog.md:28 the connector registry, tilde-expansion table, canonical-slug map, and factory registry must agree FrankenSuite none adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 1ef0f8e; Gate D requires briefs, packets and site/assets/data.js names to match one to one
RP-091Statistics computed from data, not typed Compute every displayed count from the data at build or render time and fail when static fallback text differs from the computed value. Dicklesworthstone/frankensim_website, Dicklesworthstone/frankentermSource: synthesis/uniqueness-catalog.md:331 Computed stats pipeline. FrankenSuite Claim discipline enforced: README claims cross-checked against proofs adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 1ef0f8e; Gate B recomputes every aggregate from site/assets/data.js and fails on bare N of 44 literals without a STAT source note
RP-092Disclaim as data: unmeasured targets are machine-readable as unmeasured Mark targets as targets and unmeasured values as unmeasured in the data itself (status fields, row-level basis labels), and count refusal sites, so absence of evidence is data rather than prose. Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensearch, Dicklesworthstone/franken_manimSource: synthesis/uniqueness-catalog.md:38 the absence of measurement is machine-readable data, not prose FrankenSuite Proof taxonomy declared (provisional) candidateCandidate verdict rows carry no machine field distinguishing measured from asserted beyond the tier label.
RP-093Release structurally blocked until the gate passes Make release a checked impossibility until the release gate passes (a release-check command that returns an error, a 100 percent declared-surface threshold with no downgrades). Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensqliteSource: synthesis/uniqueness-catalog.md:39 release is a checked impossibility, not an aspiration FrankenSuite Release gate defined: what blocks publication candidatev1.0.0 was tagged by hand after a green run; no workflow blocks a tag whose commit has not passed verify.
RP-094Upstream-anchored contracts Anchor each contract rule to the exact upstream file and line at the pin with an expected token, and fail CI when upstream drifts. Dicklesworthstone/franken_leanSource: synthesis/uniqueness-catalog.md:72 line naming the exact upstream file/line at the pin FrankenSuite Incumbent oracle pinned before implementation partialdone: K2 fails a verdict citation whose line no longer contains its quote, which is local citation integrity; site/scripts/verify-site.sh; commit 68d9523; missing: nothing re-fetches an upstream pin and fails CI when upstream moves
RP-095Reject fixture-only and synthetic artifacts as evidence Refuse simulated or fixture-only artifacts as backing for a performance or capability claim. Dicklesworthstone/franken_engine, Dicklesworthstone/franken_nodeSource: synthesis/uniqueness-catalog.md:65 ) are rejected by the gate as backing evidence FrankenSuite Claim discipline enforced: README claims cross-checked against proofs not applicableNo performance evidence of its own.
RP-096Publish rejected and failed artifacts instead of regenerating them Keep failed receipts, rejected release artifacts, stale not-green suite results, and the reason CI is disabled, visible with identifiers, rather than regenerating into green. Dicklesworthstone/franken_manim, Dicklesworthstone/frankensim, Dicklesworthstone/franken_markdown_website, Dicklesworthstone/franken_overlap, Dicklesworthstone/beads-for-frankentuiSource: synthesis/uniqueness-catalog.md:83 That artifact was rejected and is not attached FrankenSuite none partialdone: the repository and every CI run are public; .github/workflows/verify.yml:6; missing: no step uploads a failed run log as an artifact, and no red run has yet been observed to be retained
RP-097Machine-greppable honesty trailers on commits End each commit with what was actually run (or its verification level) in a fixed, greppable form, so history records evidence, not just intent. Dicklesworthstone/franken_surveillance_systemSource: synthesis/uniqueness-catalog.md:183 the pin's own trailer lists what was FrankenSuite Close only on cited evidence; close_reason required adoptedCONTRIBUTING.md:49; every commit subject ends with a verification level; convention only, not machine-checked in CI
RP-098Evidence tier on every statement Label each statement in plans and findings with its epistemic status (fact, design, hypothesis, target; or evidence tier plus confidence) so readers never mistake a target for a result. Dicklesworthstone/franken_surveillance_system, Dicklesworthstone/franken_nlpSource: synthesis/uniqueness-catalog.md:185 Every statement labeled FACT / DESIGN / HYPOTHESIS / TARGET / OPEN FrankenSuite Proof taxonomy declared (provisional) partialdone: the Rulebook requires a tier and confidence on every claim, and K2 enforces a tier on verdict bullets; RULEBOOK.md:19; site/scripts/verify-site.sh; commit 68d9523; missing: packet prose is not machine-checked statement by statement
RP-099Receipts carry provenance and generation binding Every receipt or fixture records commit, tool versions, host, worker, oracle script hash, and generation command; a receipt from another generation is not evidence. Dicklesworthstone/franken_remote, Dicklesworthstone/franken_snowflake, Dicklesworthstone/franken_manim, Dicklesworthstone/frankenpandas, Dicklesworthstone/frankenterm, Dicklesworthstone/frankengitSource: synthesis/uniqueness-catalog.md:279 All 1,387 conformance packets carry FrankenSuite Receipts carry provenance; generation binding partialdone: every packet header records repository, pinned commit, date and analyst method; RULEBOOK.md:68; missing: tool versions, host, worker, oracle script hash and generation command are not recorded
RP-100Machine-checked dependency admissibility Check the dependency tree against a declared diet (banned crates, franken-only, zero-external) in CI, with the admission rationale recorded per dependency. Dicklesworthstone/franken_snowflake, Dicklesworthstone/franken_alignment, Dicklesworthstone/franken_threed, Dicklesworthstone/frankensimSource: synthesis/uniqueness-catalog.md:175 Machine-checked dependency diet. FrankenSuite none candidateOne exact-pinned devDependency (esbuild) with no admission record or tree check.
RP-101Tolerance and floor ratchets: thresholds only tighten Enforce tolerances and conformance floors with ratchet semantics in CI, so a threshold can tighten but never loosen without an explicit gate change. Dicklesworthstone/frankenscipy, Dicklesworthstone/franken_markdownSource: synthesis/uniqueness-catalog.md:296 CI enforces the tolerance policy with ratchet semantics FrankenSuite Never weaken a gate to land a change; gate changes need two-direction evidence not applicableNo numeric thresholds in the gate chain.
RP-102Failure-mode taxonomy for honesty machinery Name the ways evidence culture fails (demo hardcoding, proof laundering, refusal farming, commit pumping, gate self-weakening, follow-up laundering) so reviewers can call them by name. Dicklesworthstone/franken_code_browser, Dicklesworthstone/franken_whisper, Dicklesworthstone/frankensimSource: synthesis/uniqueness-catalog.md:46 Coordinated-failure defect taxonomy. FrankenSuite Anti-reward-hacking law adopted candidateThe 12 forbidden patterns are in the starter kit, but this repository has no agent instructions file adopting them.
RP-103Bead closure requires evidence of minimum substance Reject tracker closures without an evidence-bearing reason of minimum length, and sweep old thin closures back open. Dicklesworthstone/frankentui, Dicklesworthstone/franken_nodeSource: synthesis/uniqueness-catalog.md:380 rejects bead closures without 80+ character evidence-bearing reasons FrankenSuite Close only on cited evidence; close_reason required; False-closure repair sweep not applicableThis repository keeps no bead graph of its own.
RP-104Typed proof grammar for claims Give each claim proof a typed form (test, path, identifier, command, count, manual) so proof kinds can be checked and counted. Dicklesworthstone/frankentuiSource: synthesis/uniqueness-catalog.md:379 The claims ledger uses a typed proof grammar FrankenSuite Proof taxonomy declared (provisional) partialdone: our_proof tokens have two resolvable forms, a path with optional line and a commit sha, which K3 resolves; site/scripts/verify-site.sh; commit 68d9523; missing: no typed kinds for test, identifier, command, count or manual proofs
RP-105SLO-as-code Keep performance budgets in a machine-readable file that gates enforce or that changes runtime posture on breach. Dicklesworthstone/frankentui, Dicklesworthstone/franken_drone_geometry_reconstruction, Dicklesworthstone/asupersyncSource: synthesis/uniqueness-catalog.md:382 frame budgets enforced by gates FrankenSuite none not applicableNo runtime performance budget.
RP-106In-place atomic swap per change, never big-bang or shadow A/B Each change replaces the old implementation with a thin shim into the new one and deletes the old code in the same commit, so the new code is exercised in place and main stays shippable. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:42 Each pull request replaces the FrankenSuite none not applicableNo port.
RP-107Shipping pilots and a trial run before scale Land the toolchain, CI, and rules first, prove the full pipeline on a few side-effect-free units with human-read drafts, and only then scale out. github/copilot-agent-runtime, oven-sh/bunSource: synthesis/vendor-port-learnings.md:231 Full pipeline (1 implementer + 2 adversarial reviewers + FrankenSuite none not applicableNo bulk port or generation run.
RP-108Leaves-inward waves, with temporary interop counted down to zero Order work from pure leaves inward to the most coupled core, move behaviour in waves (logic, state, orchestration, remove fallbacks, simplify), and count the temporary interop seam down to zero. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:67 T-C5. Temporary interop as a deliberately deletable seam. FrankenSuite none not applicableNo port.
RP-109Incremental rollout as last-mile validation Ship the port incrementally to a pre-release audience so regressions correlate with recent changes and are cheap to root-cause. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:72 Doing the porting incrementally over FrankenSuite none not applicableNo staged audience.
RP-110Protect the oracle from the agent: frozen E2E tests and two-key waivers Freeze the end-to-end suite as the contract; agents may not delete or weaken tests, update snapshots, raise baselines, or apply waiver labels without a second approver; zero tests skipped or deleted. github/copilot-agent-runtime, oven-sh/bun, Dicklesworthstone/frankenredis, Dicklesworthstone/frankengitSource: synthesis/vendor-port-learnings.md:85 The agent changing an implementation cannot FrankenSuite Never weaken a gate to land a change; gate changes need two-direction evidence; Waivers are public, time-bounded, and recorded candidateGate A keeps the shipped packets byte-identical to the canonical ones, but nothing requires a second approver to change the canonical packets, RULEBOOK.md or the gate script.
RP-111Split-context adversarial review, default-refute, with attribution Give the reviewer the diff and nothing else, tell it to assume the code is wrong, default verdicts to refuted unless it cites source and target lines, and attribute every caught bug in the fix commit. oven-sh/bun, github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:247 The reviewer gets the diff and nothing else FrankenSuite Plan independently reviewed before execution partialdone: the author of a verdict never reviews it, and K1 fails when reviewed_by equals author; stack/METHOD.md:83; site/scripts/verify-site.sh; commit 68d9523; missing: nothing gives the reviewer only the diff, defaults findings to refuted, or attributes caught bugs in fix commits
RP-112Line-by-line old-versus-new review by several models After each rebase, have one reviewer per model compare old and new code line by line for behavioural equality, flag leftover shims and changed E2E tests, and loop until every review is clean. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:98 a line-by-line comparison of the old TypeScript and the new Rust, confirming behavioral FrankenSuite Plan independently reviewed before execution not applicableNo port to compare.
RP-113Division of review labour Agents do exhaustive old-versus-new comparison, tests and static analysis check what is mechanically enforceable, and humans concentrate on architecture, contracts, risk, and whatever the other layers flag. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:108 reviewers concentrated on architecture, API contracts, risk, and any suspicious places FrankenSuite none candidateThis batch splits drafting, gating and independent review across agents, but the split is not written down for the repository.
RP-114Rebase conflict as a feature of atomic delete Delete old code in the same change that adds its replacement, so any incoming edit to already-ported code surfaces as a rebase conflict instead of being lost. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:114 T-C11. Rebase-conflict-as-a-feature from atomic delete. FrankenSuite none not applicableNo port.
RP-115Explorers fan out, one coordinator owns edits; child sessions own disjoint files Use subagents for independent read-only questions, keep mutation with the coordinating agent, give parallel writers disjoint file ownership, and write kickoff prompts that name siblings and what to leave alone. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:121 T-C12. Subagents for exploration, child sessions for parallel diffs. FrankenSuite none candidateThis v1.1 batch assigned exclusive file ownership per agent in the lead prompt; the rule is not recorded in the repository.
RP-116A failure seen twice becomes standing machinery When a failure mode recurs, turn it into a standing instruction, a reusable skill, an eval, a protected baseline, or a gate, not a one-off fix. github/copilot-agent-runtime, Dicklesworthstone/frankenfsSource: synthesis/vendor-port-learnings.md:140 it belongs in standing instructions, a reusable skill, an eval FrankenSuite none partialdone: Gate A2 names the audit failure that created it; site/BUILD-GATES.md:64; missing: not every gate names a recurring failure, and no rule says a repeat failure must become a gate
RP-117Translate first, redesign second Never change language and behaviour in the same change; land the faithful translation with its oracle green, then redesign in a separate change. github/copilot-agent-runtime, Dicklesworthstone/frankensqliteSource: synthesis/vendor-port-learnings.md:143 Translate first, redesign second. FrankenSuite none not applicableNo port.
RP-118Minimal permanent ABI surface with a dispatch interior Keep the permanent boundary tiny (a handful of exported functions) and route everything else through a dispatch table behind it. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:149 Adding, changing, or removing an API method touches the FrankenSuite none not applicableNo ABI.
RP-119Unsafe governance: counted, boundary-only, SAFETY-commented, compiler-enforced Count every unsafe block and keep them at interop boundaries, require an accurate SAFETY comment above each, push the unsafe boundary away from each access, and encode the style guide as lint bans. github/copilot-agent-runtime, oven-sh/bun, NVlabs/cuda-oxide, Dicklesworthstone/franken_native_capsule, Dicklesworthstone/frankentorchSource: synthesis/vendor-port-learnings.md:291 Use the compiler to enforce safety. Code comments do not FrankenSuite none not applicableNo unsafe code.
RP-120Classify every regression into named families Sort each regression into a small fixed taxonomy (incomplete migration, state and lifetime, contract mismatch, host boundary, incorrect oracle) and use the counts to steer review. github/copilot-agent-runtime, oven-sh/bunSource: synthesis/vendor-port-learnings.md:157 T-C18. The five regression families FrankenSuite none candidateSite defects found in audits are fixed and gated but not tallied by family.
RP-121Paired-operation completeness Declare every paired operation (abort and cancel, persist and project, register and deregister) and fail a change that touches one half without the other. github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:623 PAIR — Paired-operation completeness FrankenSuite none not applicableNo paired runtime operations.
RP-122Ambient and host-boundary inventory Inventory every ambient input (timezone, cwd, PATH, env, host tools) with its capture point and refresh policy, and test that changing it after capture changes nothing. github/copilot-agent-runtime, Dicklesworthstone/franken_nodeSource: synthesis/vendor-port-learnings.md:643 HOST — Ambient/host-boundary inventory FrankenSuite none candidateThe gate chain depends on the host Chrome, python3 and node versions; CI pins node 22 and bun 1.4.2 but Chrome floats with the runner image and nothing records the version used.
RP-123Porting rulebook adversarially reviewed before bulk work Write the idiom and type rulebook before any bulk work, have it adversarially reviewed, and make uncertainty markers greppable and budgeted: flagging is better than wrong output. oven-sh/bunSource: synthesis/vendor-port-learnings.md:220 adversarially reviewed before use FrankenSuite Plan independently reviewed before execution partialdone: stack/METHOD.md was revised after a stress-test before any verdict shipped; commit 16deb8c; missing: the reviewer findings are not in the repository, and the cited commit is itself marked pending
RP-124Pre-computed ownership table as an orthogonal source of truth Classify ownership of every pointer or handle in a cited table before the work, and have agents use the table verbatim instead of guessing locally. oven-sh/bunSource: synthesis/vendor-port-learnings.md:223 LIFETIMES.tsv — pre-computed ownership table as an orthogonal source of truth. FrankenSuite none not applicableNo ownership decisions.
RP-125Compiler errors as a work queue Write the checker output to a file grouped by unit and drain it with bounded agent teams, rather than letting each agent rediscover errors. oven-sh/bunSource: synthesis/vendor-port-learnings.md:236 Compiler errors as a work queue. FrankenSuite none not applicableNo compile step.
RP-126No stubbing to satisfy the checker Forbid stubs and justification-paragraph workarounds that make an error disappear; if the workaround needs a paragraph to defend, fix the code. oven-sh/bunSource: synthesis/vendor-port-learnings.md:243 comment to justify why the workaround is OK, the code is wrong FrankenSuite Anti-reward-hacking law adopted candidateCovered by the starter kit forbidden patterns, which this repository has not adopted in an agent instructions file.
RP-127Bounded loops with explicit IOUs, zero unresolved at the gate Give every agent loop a round cap, turn unfixables into structured blocked-on markers consumed by a later phase, and fail the phase gate on any left. oven-sh/bunSource: synthesis/vendor-port-learnings.md:333 Bounded loops with explicit IOUs: every phase had a round cap FrankenSuite Phase exit criteria defined before agents start not applicableNo agent loops run inside this repository.
RP-128Class fix followed by instance re-audit against the new code After fixing a bug class systemically, re-run the per-instance audit against the new code and record each instance as fixed or exempt. oven-sh/bunSource: synthesis/vendor-port-learnings.md:339 re-run the per-site audit against the FrankenSuite none candidateWhen a gate is added for a defect class, older pages are not separately re-audited beyond what the gate scans.
RP-129Compile-time layout assertions on serialized structures Put size, alignment, and offset assertions next to every FFI, on-disk, or serialized struct so cross-language drift breaks the build. oven-sh/bunSource: synthesis/vendor-port-learnings.md:341 Compile-time layout asserts: 148 FrankenSuite none not applicableNo binary layouts.
RP-130Methodology preservation: never squash the rigor away Keep rulebooks, ownership tables, and review-attribution commits in the main history or an archived appendix; a squash merge that deletes them erases the method. oven-sh/bunSource: synthesis/vendor-port-learnings.md:727 Methodology-preservation rule [Bun]. FrankenSuite none partialdone: the Rulebook versions its amendments, v11-manifest.md records the lineage, and main history is unsquashed; RULEBOOK.md:197; v11-manifest.md; missing: no merge policy prevents a squash merge
RP-131Unrepresentable before checked before tested Where a property is statically decidable, make the bug impossible to express (sealed types, borrow rules) and pin it with compile-fail tests; use runtime checks returning Option or Result for geometry- or device-dependent facts. NVlabs/cuda-oxide, NVlabs/cutile-rsSource: synthesis/vendor-port-learnings.md:466 The runtime-check vs compile-reject decision rule. FrankenSuite none not applicableNo typed code.
RP-132Preconditions validated once into a branded proof token Validate every precondition once against live conditions and return a token the safe action requires; anything without the token goes through an explicitly unsafe path. NVlabs/cuda-oxide, NVlabs/cutile-rsSource: synthesis/vendor-port-learnings.md:424 kernels without a contract expose only raw FrankenSuite none not applicableNo runtime actions.
RP-133Fail-closed verify barriers between pipeline stages Verify between each stage of a multi-stage pipeline and attach the failing stage and operation to the error, so a failure is attributable to the stage that broke it. NVlabs/cuda-oxide, Dicklesworthstone/franken_ocrSource: synthesis/vendor-port-learnings.md:452 fail-closed verify barriers between IR stages, with FrankenSuite none adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; each gate prints PASS or FAIL with its reason and the script exits nonzero if any gate fails
RP-134Headline numbers carry their scope caveat Never quote a headline number without the caveat its author attached (workload-specific, attributed cost only, input versus diff size); quote a disavowal when the source gives one. github/copilot-agent-runtime, oven-sh/bunSource: synthesis/external-validation-2026-09.md:21 That is a workload-specific result; the Rust runtime is not universally FrankenSuite none partialdone: the Rulebook requires quoting a source disavowal and forbids ungated numbers; RULEBOOK.md:80; missing: no rule that every headline number carries its author workload or cost caveat
RP-135Managed breaking migration: named opt-in features and advance warning Move deprecated stacks behind named opt-in features, label breaking changes per component, warn once per process before a default flips, and give a stable spelling for teams that pin the old stack. awslabs/aws-sdk-rustSource: synthesis/external-validation-2026-09.md:113 a once-per-process warning before a default flip FrankenSuite none not applicableNo public API to migrate.
RP-136Workflow files linted against a declared execution policy State in the repository which events may start CI and that every workflow must call the repository's own verify entry point; enforce both, and full-SHA action pins, with a checker that fails on any workflow file that breaks them. Dicklesworthstone/frankengitSource: updates/frankengit-2026-09-24.md:214 must delegate to repository-owned ./scripts/verify.sh FrankenSuite none candidateOur workflows run on push, pull_request and schedule by design; the transferable part is a gate that lints .github/workflows/*.yml for SHA pins, least-privilege permissions and delegation to a repository script, which no gate does today.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed