Source file: stack/rigor-practices.tsv. Starter-kit items link to their entries in starter-kit/CHECKLIST.md.
| Practice | What to copy | Evidenced in | Areas | Starter kit | Our status |
|---|---|---|---|---|---|
| RP-001Measured, justified CI job timeouts | Size each job timeout-minutes from measured run history (about twice the slowest observed run) and write the measurement next to the number. | modelcontextprotocol/inspectorSource: ecosystem/pickup/_evidence/mcp.md:26 roughly TWICE the slowest run observed |
MCP servers and clients | none | adopted.github/workflows/verify.yml:36; commit 0f14145 |
| RP-002Concurrency cancel-in-progress for superseded PR runs | Put CI in a concurrency group per pull request with cancel-in-progress, so a newer push cancels the stale run; keep push runs uncancelled. | modelcontextprotocol/python-sdk, modelcontextprotocol/typescript-sdk, modelcontextprotocol/servers, browserbase/stagehand, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/mcp.md:29 on PRs (both SDKs, servers); python-sdk |
MCP servers and clients, Browser-use stacks, Realtime voice agents | none | adopted.github/workflows/verify.yml:22; commit 0f14145 |
| RP-003Pin CI actions to commit SHAs | Reference every third-party action by full commit SHA (tag in a comment) so a moved tag cannot change what CI runs. | getzep/graphiti, huggingface/sentence-transformers, dottxt-ai/outlinesSource: ecosystem/pickup/_evidence/agent-memory.md:30 checkout/setup-python/setup-uv all pinned to SHAs |
Agent memory systems, Embedding model serving, Structured output and constrained decoding | none | adopted.github/workflows/verify.yml:41; .github/workflows/verify.yml:68; commit 0f14145 |
| RP-004Scheduled runs so gates execute without pushes | Add a nightly or weekly cron trigger so correctness, smoke, or benchmark jobs keep running when nobody pushes; drift then shows up on a date, not on the next unrelated change. | trycua/cua, huggingface/text-embeddings-inference, michaelfeil/infinity, comet-ml/opik, cloudflare/sandbox-sdk, mlc-ai/xgrammar, meta-pytorch/torchtune, sgl-project/sglang, InternLM/lmdeploy, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/computer-use.md:32 scheduled live-environment smokes catch infra drift, not just unit failures |
Computer-use agents, Embedding model serving, Agent observability and tracing, Sandboxed code execution, Structured output and constrained decoding, Fine-tuning infrastructure, LLM inference engines, FrankenSuite | none | partialdone: weekly cron declared in the workflow; .github/workflows/verify.yml:15; commit 0f14145; missing: the first scheduled run has not fired yet |
| RP-005Least-privilege CI token by default | Default the workflow token to permissions: contents: read and grant more only per job, with the reason in a comment. | modelcontextprotocol/inspector, Lightning-AI/litgptSource: ecosystem/pickup/_evidence/mcp.md:26 with per-job overrides documented inline |
MCP servers and clients, Fine-tuning infrastructure | none | adopted.github/workflows/verify.yml:25; commit a298c99 |
| RP-006Single required all-green status check | Aggregate every job into one required check (re-actors/alls-green or a gate workflow) so branch protection names one status that a renamed or skipped job cannot bypass. | modelcontextprotocol/python-sdk, mem0ai/mem0Source: ecosystem/pickup/_evidence/agent-memory.md:23 Single required status check + path-filtered builds |
MCP servers and clients, Agent memory systems | none | candidateApplies: main has no branch protection (gh api repos/JYeswak/franken-research/branches/main/protection returned HTTP 404 on 2026-09-24), so no status check is required at all. One job is not a required check. |
| RP-007Pin the whole toolchain, not just dependencies | Commit lockfiles and toolchain files, install with the frozen or locked flag, and pin vendored cores and cross-repo dependencies by exact version or content hash. | huggingface/text-embeddings-inference, huggingface/sentence-transformers, huggingface/text-generation-inference, ollama/ollama, Dicklesworthstone/franken_lean, Dicklesworthstone/franken_threed, Dicklesworthstone/franken_native_capsule, Dicklesworthstone/frankensimSource: ecosystem/pickup/_evidence/embedding-serving.md:28 Pin the whole toolchain, not just deps |
Embedding model serving, LLM inference engines, FrankenSuite | none | partialdone: esbuild pinned exactly, bun.lock installed frozen, bun pinned to 1.4.2; package.json:11; bun.lock; .github/workflows/verify.yml:4; .github/workflows/verify.yml:68; missing: node is pinned only to major 22, and the python3 and Chrome versions the gates use are neither pinned nor recorded |
| RP-008Supply-chain and secret gates as separate CI steps | Run dependency audit, dependency firewall, and secret scanning as named gates of their own, before install touches the tree. | getzep/graphiti, guardrails-ai/guardrails, Dicklesworthstone/franken_snowflakeSource: ecosystem/pickup/_evidence/agent-memory.md:30 Supply-chain hygiene in CI: pinned action SHAs, dependency firewall |
Agent memory systems, Guardrails and red-teaming, FrankenSuite | none | partialdone: the secret scan is its own step and runs before install; .github/workflows/verify.yml:58; .gitleaksignore; commit a298c99; missing: no dependency audit step and no dependency firewall |
| RP-009Verify tool and oracle integrity by hash before the suite runs | Check the SHA-256 (or blob SHA) of the downloaded tool or oracle checkout before running anything with it; a mismatch fails the job. | Dicklesworthstone/franken_threedSource: synthesis/uniqueness-catalog.md:192 The oracle checkout is verified by blob SHA-1 |
FrankenSuite | Incumbent oracle pinned before implementation | partialdone: the gitleaks tarball is sha256-checked before it runs; .github/workflows/verify.yml:53; commit a298c99; missing: Chrome gets only an existence check, and the canonical packets and site/assets/data.js the gates trust are not hashed before the suite runs |
| RP-010Regenerate derived artifacts in CI and fail on diff | Rebuild generated files (types from the spec, OpenAPI from the server, bundles, ledgers) in CI and fail if the committed copy differs. | modelcontextprotocol/typescript-sdk, langfuse/langfuse, qdrant/qdrant, Dicklesworthstone/franken_networkxSource: ecosystem/pickup/_evidence/observability.md:32 generate the spec from the server and diff it |
MCP servers and clients, Agent observability and tracing, Vector databases, FrankenSuite | none | adopted.github/workflows/verify.yml:76; commit a298c99; map bundle rebuilt from source and diffed |
| RP-011Public API or tool-surface diff gate on PRs | Build the base and head versions and diff the exposed surface (tool names, schemas, endpoints), so any surface change becomes an explicit review item. | github/github-mcp-server, qdrant/qdrantSource: ecosystem/pickup/_evidence/mcp.md:27 builds the server from base and head and diffs the exposed MCP tool surface |
MCP servers and clients, Vector databases | none | not applicableThis repository ships a static site, not an API or tool surface. |
| RP-012Docs-as-tests: documented commands and examples run in CI | Execute README commands, doc snippets, and example scripts in CI so documentation cannot drift away from what actually runs. | exa-labs/exa-py, deepset-ai/haystack, vibrantlabsai/ragas, guardrails-ai/guardrails, mit-han-lab/llm-awqSource: ecosystem/pickup/_evidence/web-search-apis.md:27 Copy this ruthlessly: example code rots unless CI runs it. |
Web search API layers, Retrieval-augmented generation (RAG) frameworks, Guardrails and red-teaming, Model quantization toolkits | none | partialdone: CI runs the README Reproduce commands; README.md:49; .github/workflows/verify.yml:3; missing: the README gate table ends at J while the script now has Gate K, and nothing fails when the README gate table disagrees with the script |
| RP-013Path-filtered, change-aware CI | Run expensive jobs only when their paths change (paths-filter, paths-ignore for docs) and allow a labelled skip for costly evals. | EleutherAI/lm-evaluation-harness, browserbase/stagehand, mem0ai/mem0, huggingface/text-embeddings-inference, jina-ai/readerSource: ecosystem/pickup/_evidence/eval-harnesses.md:26 CI validates only the benchmarks that changed |
LLM eval harnesses, Browser-use stacks, Agent memory systems, Embedding model serving, Web search API layers | none | not applicableThe whole gate chain runs in about a minute and the site is the docs; a docs-only skip would skip the gates that check them. |
| RP-014Reusable workflow composition | Factor shared CI into workflow_call reusable workflows with pinned runtimes, called per package or per event. | jina-ai/reader, trycua/cua, michaelfeil/infinitySource: ecosystem/pickup/_evidence/web-search-apis.md:32 reusable-workflow shape and the docs-skip |
Web search API layers, Computer-use agents, Embedding model serving | none | not applicableSingle workflow with a single job. |
| RP-015Per-package or per-integration CI workflows | Give each package, SDK language, or framework integration its own workflow (or a discovered matrix entry) so ownership and failures stay local. | modelcontextprotocol/servers, mem0ai/mem0, SuperMemoryAI/supermemory, trycua/cua, e2b-dev/E2B, comet-ml/opik, openlit/openlit, chroma-core/chroma, lancedb/lancedb, microsoft/autogen, firecrawl/firecrawlSource: ecosystem/pickup/_evidence/mcp.md:24 and fans out a per-server test matrix |
MCP servers and clients, Agent memory systems, Computer-use agents, Sandboxed code execution, Agent observability and tracing, Vector databases, Agent frameworks, Web search API layers | none | not applicableOne site, one package. |
| RP-016Cross-platform and multi-arch build matrix | Build and test on every OS, architecture, runtime version, or hardware backend you ship for, with fail-fast off. | kata-containers/kata-containers, ggml-org/llama.cpp, sgl-project/sglang, pytorch/ao, bitsandbytes-foundation/bitsandbytes, trycua/cua, microsoft/playwright-mcp, modelcontextprotocol/typescript-sdk, qdrant/fastembedSource: ecosystem/pickup/_evidence/sandbox-exec.md:59 Multi-arch build-check matrix (amd64/arm64/ppc64le/s390x/riscv64) |
Sandboxed code execution, LLM inference engines, Model quantization toolkits, Computer-use agents, Browser-use stacks, MCP servers and clients, Embedding model serving | none | candidateCI renders on ubuntu-latest only; the macOS reproduction path in the README is not exercised by CI. |
| RP-017Lint and format gates separate from test gates | Run formatting and lint as their own required check, independent of heavy or hardware-bound CI, so PRs are reviewable without it. | bitsandbytes-foundation/bitsandbytes, ggml-org/llama.cpp, huggingface/trl, Farama-Foundation/Gymnasium, OpenRLHF/OpenRLHF, deepspeedai/DeepSpeedSource: ecosystem/pickup/_evidence/rl-envs.md:22 Copy: formatting (ruff/black) as its own required check. |
Model quantization toolkits, RL environments and RLHF training | none | not applicableLittle code to format; copy quality is checked by Gate H instead. |
| RP-018Coverage reporting with a failure policy | Upload coverage on every run and set fail_ci_if_error so a broken coverage upload is a red run, not a silent gap. | jina-ai/reader, exa-labs/exa-py, noamgat/lm-format-enforcerSource: ecosystem/pickup/_evidence/web-search-apis.md:31 Pick the Jina strictness for a new project. |
Web search API layers, Structured output and constrained decoding | none | not applicableNo library code under test; line coverage is not a meaningful measure for this site. |
| RP-019CI failure monitoring, flaky-test tracking and auto-bisect | Track CI failures over time, report flaky tests, and bisect automatically so a red main is traced to a commit without manual digging. | sgl-project/sglang, temporalio/temporalSource: ecosystem/pickup/_evidence/inference-engines.md:28 CI failure monitoring and auto-bisect. |
LLM inference engines, Durable workflow orchestrators | none | not applicableRun volume is too small to have a flake population to track. |
| RP-020Retry only named transient failure classes | Allow automatic reruns only for listed infrastructure errors (OSError, timeouts, HTTP 502/504, OOM), never for assertion failures. | huggingface/trl, browser-use/browser-useSource: ecosystem/pickup/_evidence/fine-tuning.md:61 only for OSError/Timeout/HTTP 502/504/OOM messages |
Fine-tuning infrastructure, Browser-use stacks | none | not applicableverify.yml has no retry step and no flaky network test. Its one network call, the gitleaks download, is hash-checked and fails the run on error rather than retrying. |
| RP-021Privileged or expensive jobs only on explicit maintainer trigger | Gate GPU, live-API, or dangerous e2e jobs behind a label, slash command, merge queue, or org-membership check instead of running them on every fork PR. | cloudflare/sandbox-sdk, firecrawl/firecrawl, livekit/agents, axolotl-ai-cloud/axolotlSource: ecosystem/pickup/_evidence/sandbox-exec.md:57 Merge-queue + privileged-PR split (dangerous e2e tests only run with extra approval) |
Sandboxed code execution, Web search API layers, Realtime voice agents, Fine-tuning infrastructure | none | not applicableNo privileged or paid job exists and the workflow uses no secrets. |
| RP-022Tiered CI: fast tier on every PR, heavy tier gated or scheduled | Split tests by marker or directory into a cheap always-on tier (unit, CPU) and heavy tiers (integration, GPU, e2e, live provider) that run on label, merge, or schedule; state the contract for each tier. | axolotl-ai-cloud/axolotl, huggingface/trl, hiyouga/LlamaFactory, meta-pytorch/torchtune, langchain-ai/langgraph, getzep/graphiti, browserbase/stagehand, huggingface/text-embeddings-inference, UKGovernmentBEIS/inspect_ai, NVIDIA-NeMo/Guardrails, vllm-project/vllm, modelcontextprotocol/typescript-sdk, langfuse/langfuse-python, bitsandbytes-foundation/bitsandbytes, microsoft/graphrag, verl-project/verl, guidance-ai/guidance, weaviate/weaviate, livekit/agents, jina-ai/reader, temporalio/temporalSource: ecosystem/pickup/_evidence/fine-tuning.md:18 Tiered test pyramid: CPU unit tests on every PR, GPU e2e gated/scheduled |
Fine-tuning infrastructure, Agent frameworks, Agent memory systems, Browser-use stacks, Embedding model serving, LLM eval harnesses, Guardrails and red-teaming, LLM inference engines, MCP servers and clients, Agent observability and tracing, Model quantization toolkits, Retrieval-augmented generation (RAG) frameworks, RL environments and RLHF training, Structured output and constrained decoding, Vector databases, Realtime voice agents, Web search API layers, Durable workflow orchestrators | none | not applicableOne tier suffices: the full gate chain takes about a minute. |
| RP-023Deterministic scripted model and tool doubles shipped as a test API | Ship fake models (and fake sandboxes and transports) that replay a scripted trajectory and that downstream users can import, so agent loops are tested with no network and no keys; one fake per scenario. | openai/openai-agents-python, huggingface/smolagents, langchain-ai/langgraph, UKGovernmentBEIS/inspect_ai, EleutherAI/lm-evaluation-harness, guardrails-ai/guardrails, guidance-ai/guidance, livekit/agents, vercel/sandbox, tavily-ai/tavily-pythonSource: ecosystem/pickup/_evidence/agent-frameworks.md:19 Copy: make the fake model part of the SDK so downstream users can test agents deterministically too. |
Agent frameworks, LLM eval harnesses, Guardrails and red-teaming, Structured output and constrained decoding, Realtime voice agents, Sandboxed code execution, Web search API layers | none | not applicableThis repository makes no model or tool calls. |
| RP-024Strict scripted contract: unexpected or unconsumed steps fail | Make drift between the test script and the actual trajectory a hard error in both directions: a call the script did not define, and a scripted step the run never used. | openai/openai-agents-pythonSource: ecosystem/pickup/_evidence/agent-frameworks.md:22 turn drift between test script and agent trajectory into hard failures, not silent pass-throughs |
Agent frameworks | none | not applicableNo scripted doubles here. |
| RP-025Assert retry and backoff with scripted failure sequences | Drive the client with a side-effect sequence (error, error, success) and assert the retry and backoff behaviour directly. | huggingface/smolagentsSource: ecosystem/pickup/_evidence/agent-frameworks.md:24 Copy: assert retry/backoff behavior via side-effect sequences. |
Agent frameworks | none | not applicableNo retrying client code. |
| RP-026Record and replay external calls (VCR cassettes) | Record LLM and HTTP interactions once and replay them in CI so unit tests are hermetic, cheap, and deterministic. | NVIDIA-NeMo/Guardrails, langchain-ai/langchain, traceloop/openllmetry, lmnr-ai/lmnr-pythonSource: ecosystem/pickup/_evidence/guardrails.md:32 Copy: record once, replay in CI |
Guardrails and red-teaming, Retrieval-augmented generation (RAG) frameworks, Agent observability and tracing | none | not applicableThe gates call no model or HTTP API that could be recorded. The only network call is the pinned gitleaks download in CI. |
| RP-027Local deterministic test server or reference fixture instead of the live service | Ship a local test server, or one kitchen-sink reference implementation, as the fixture every spec runs against, so core tests never depend on the live web. | microsoft/playwright-mcp, modelcontextprotocol/servers, modelcontextprotocol/inspector, livekit/agents, inngest/inngest, cadence-workflow/cadenceSource: ecosystem/pickup/_evidence/browser-use.md:20 no external-network dependency for core specs |
Browser-use stacks, MCP servers and clients, Realtime voice agents, Durable workflow orchestrators | none | adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 9a99982; Gate I renders every page from file:// with no server and no network |
| RP-028Snapshot and byte-exact tests of full outputs | Snapshot complete agent state, stream output, grammar text, or split text (or byte-compare shipped copies against the canonical ones) so unintended changes fail loudly. | langchain-ai/langgraph, Azure-Samples/azure-search-openai-demo, ggml-org/llama.cpp, harbor-framework/harborSource: ecosystem/pickup/_evidence/agent-frameworks.md:25 Copy: snapshot full agent state/stream output to catch unintended trajectory changes. |
Agent frameworks, Retrieval-augmented generation (RAG) frameworks, Structured output and constrained decoding, LLM eval harnesses | none | adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 9a99982; Gate A byte-compares every shipped packet and the Rulebook with the canonical copy |
| RP-029Pinned baseline scores and golden numbers with an explicit tolerance | Assert results against a pinned reference number (score, loss curve, tensor, retrieval metric) with a stated tolerance, one-sided where only regressions matter, and keep the ground truth checked in. | vllm-project/vllm, AutoGPTQ/AutoGPTQ, meta-pytorch/torchtune, sgl-project/sglang, HKUDS/LightRAG, Azure-Samples/azure-search-openai-demoSource: ecosystem/pickup/_evidence/embedding-serving.md:21 One-sided assert: you only fail if |
Embedding model serving, Model quantization toolkits, Fine-tuning infrastructure, LLM inference engines, Retrieval-augmented generation (RAG) frameworks | Incumbent oracle pinned before implementation | not applicableNo numeric outputs of its own; Gate B recomputes the site statistics from data.js instead. |
| RP-030One conformance suite run against every backend or adapter | Write the behaviour contract once and run the same suite over every backend, provider adapter, or bundled environment in CI, instead of ad hoc per-backend tests. | langchain-ai/langgraph, dottxt-ai/outlines, vllm-project/vllm, Farama-Foundation/Gymnasium, Farama-Foundation/PettingZoo, mem0ai/mem0Source: ecosystem/pickup/_evidence/agent-frameworks.md:26 needs a backend-agnostic conformance harness, not per-backend ad-hoc tests |
Agent frameworks, Structured output and constrained decoding, RL environments and RLHF training, Agent memory systems | Incumbent oracle pinned before implementation | not applicableNo backends. |
| RP-031Official protocol conformance harness, pinned, separate from unit tests | Adopt the protocol owner's conformance kit rather than hand-rolling protocol checks; pin its version in the workflow and keep it in its own job or repository. | modelcontextprotocol/python-sdk, modelcontextprotocol/typescript-sdk, modelcontextprotocol/conformance, a2aproject/a2a-go, a2aproject/a2a-tckSource: ecosystem/pickup/_evidence/mcp.md:20 For a clean-room project: adopt the harness itself rather than hand-rolling protocol checks. |
MCP servers and clients, Agent-to-agent protocols (A2A class) | Incumbent oracle pinned before implementation | not applicableNo protocol implementation. |
| RP-032Named expected-failure baselines instead of silent skips | List known-failing scenarios in a reviewed file, or as tests that must flip when fixed, so admitting a failure is explicit and a fix cannot go unnoticed. | modelcontextprotocol/python-sdk, a2aproject/a2a-js, guidance-ai/guidanceSource: ecosystem/pickup/_evidence/mcp.md:21 Copy: named-baseline > silent skips. |
MCP servers and clients, Agent-to-agent protocols (A2A class), Structured output and constrained decoding | Waivers are public, time-bounded, and recorded | not applicableNo gate carries a known failure; every gate in verify.yml is hard. |
| RP-033Test every shipped spec revision and keep backward-compat fixtures | Run conformance against each wire or spec version you still ship, and keep versioned fixtures (serialized agents, older-spec clients) so vN keeps working with vN-1. | modelcontextprotocol/python-sdk, a2aproject/a2a-python, a2aproject/a2a-go, huggingface/smolagentsSource: ecosystem/pickup/_evidence/mcp.md:22 when the protocol versions its wire, test every shipped wire, not just latest |
MCP servers and clients, Agent-to-agent protocols (A2A class), Agent frameworks | none | not applicableNo versioned wire format. |
| RP-034Same behaviour tested on every binding, transport, runtime and language | When a protocol or SDK has several bindings (JSON-RPC, gRPC, REST; Node, Deno, Bun; Python, Go, Java), test the same behaviours on each, with a named interop workflow and checked-in wire fixtures. | a2aproject/a2a-go, a2aproject/a2a-python, agent-network-protocol/anp, firecrawl/firecrawl, milvus-io/milvus, openai/openai-agents-jsSource: ecosystem/pickup/_evidence/multi-agent-protocols.md:24 test the same behaviors on every binding, not just one |
Agent-to-agent protocols (A2A class), Web search API layers, Vector databases, Realtime voice agents | none | not applicableOne binding: static HTML. |
| RP-035Differential tests against an independent implementation or validator | Compare outputs with a second implementation or a ground-truth validator on a shared corpus (HF transformers, a sibling SDK, Pydantic, the vendored incumbent) and treat divergence as the bug signal. | vllm-project/vllm, ggml-org/llama.cpp, a2aproject/a2a-js, a2aproject/a2a-python, mlc-ai/xgrammar, e2b-dev/E2B, Dicklesworthstone/frankenredis, Dicklesworthstone/franken_threed, Dicklesworthstone/franken_numpySource: ecosystem/pickup/_evidence/inference-engines.md:22 Correctness via differential tests against HF transformers (perplexity/parity). |
LLM inference engines, Agent-to-agent protocols (A2A class), Structured output and constrained decoding, Sandboxed code execution, Model quantization toolkits, FrankenSuite | Incumbent oracle pinned before implementation | not applicableNothing here reimplements an incumbent; the corpus assesses others. |
| RP-036Unit-test the metrics, scorers and objective math themselves | Test scorers, metrics, reward shaping, KL estimators, and loss aggregation as library code with fixture data, and make any shipped eval API run in the project's own CI. | UKGovernmentBEIS/inspect_ai, EleutherAI/lm-evaluation-harness, run-llama/llama_index, deepset-ai/haystack, vibrantlabsai/ragas, confident-ai/deepeval, stanfordnlp/dspy, OpenRLHF/OpenRLHF, huggingface/trlSource: ecosystem/pickup/_evidence/rl-envs.md:18 Copy: gradient/shape-level tests for KL estimators, reward shaping, and loss aggregation |
LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks, Agent frameworks, RL environments and RLHF training | none | not applicableNo metric code. |
| RP-037Adversarial tests of your own grader, plus judge calibration sets | Test that the grader cannot be fooled (spoofed PASS lines, truncated ids, skipped tests) and give any LLM judge its own labelled false-positive and false-negative calibration set, versioned apart from the attack corpus. | SWE-bench/SWE-bench, centerforaisafety/HarmBench, microsoft/PyRITSource: ecosystem/pickup/_evidence/eval-harnesses.md:29 This is the single most copy-worthy practice: adversarial tests of your own scoring. |
LLM eval harnesses, Guardrails and red-teaming | Gate self-test: deliberately break a gate | candidateGate K3 checks that each source line contains its quote; a planted-bad TSV fixture that K3 must reject is not yet part of the gate chain. |
| RP-038Deterministic, model-free checker kept apart from generation | Keep the code that decides PASS or FAIL importable and testable without any model, in a module separate from response handling or attack generation; prefer pinned tests in pinned containers over LLM judgment. | SWE-bench/SWE-bench, ShishirPatil/gorilla, NVIDIA/garakSource: ecosystem/pickup/_evidence/eval-harnesses.md:32 Copy: the part that decides PASS/FAIL should be importable and testable without any model. |
LLM eval harnesses, Guardrails and red-teaming | none | adoptedsite/scripts/verify-site.sh; every gate is a deterministic script and no model call decides a PASS |
| RP-039Score the observable end state, not the agent's report | Grade the resulting VM, repository, or rendered UI state (tests pass, spans render as a complete trace) rather than the transcript or an HTTP 200. | xlang-ai/OSWorld, comet-ml/opik, Arize-ai/phoenix, SWE-bench/SWE-benchSource: ecosystem/pickup/_evidence/computer-use.md:22 State-based evaluation (score VM state, not chat) |
Computer-use agents, Agent observability and tracing, LLM eval harnesses | none | adoptedsite/scripts/verify-site.sh; commit 9a99982; Gate I loads each page in headless Chrome and fails on console errors or horizontal overflow |
| RP-040Explicit FAIL action for infeasible tasks | Give infeasible tasks a defined correct answer (declare FAIL) so refusal is neither rewarded everywhere nor punished where it is right. | xlang-ai/OSWorldSource: ecosystem/pickup/_evidence/computer-use.md:24 prevents reward-hacking by refusal |
Computer-use agents | Anti-reward-hacking law adopted | not applicableNo agent tasks are graded here. |
| RP-041Declarative task specs with LLM-judge criteria and a model matrix in config | Write agent tasks as data (task, judge criteria, step budget), declare the model matrix in config, and run them as a reliability tier separate from unit tests. | browser-use/browser-use, stanfordnlp/dspy, livekit/agents, pipecat-ai/pipecat, saharmor/voice-labSource: ecosystem/pickup/_evidence/agent-frameworks.md:27 declare the model matrix in config, not code |
Browser-use stacks, Agent frameworks, Realtime voice agents | none | not applicableNo agent tasks. |
| RP-042Virtual clock and time skipping in tests | Run timing-dependent logic (timers, turn-taking, workflow sleeps) against a mocked clock that the test advances, so assertions never depend on wall time. | temporalio/sdk-go, temporalio/sdk-java, temporalio/sdk-typescript, temporalio/temporal, cadence-workflow/cadence-go-client, livekit/agents, Dicklesworthstone/frankengraphdbSource: ecosystem/pickup/_evidence/voice-agents.md:22 fake clock so turn-taking/latency assertions |
Durable workflow orchestrators, Realtime voice agents, FrankenSuite | none | not applicableNo timing logic. |
| RP-043Deadlock detection: a workflow that blocks is a failure | Put a deadlock-detection timeout on workflow coroutines so code that fails to yield fails the test instead of hanging it. | temporalio/sdk-goSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:30 Deadlock detection for workflow coroutines (workflow code must yield; blocks are failures) |
Durable workflow orchestrators | none | not applicableNo concurrent workflow code. |
| RP-044Replay recorded histories, including a deliberately nondeterministic fixture | Replay stored event histories (and agent-workflow histories) through the current code to catch nondeterminism, keep one fixture that must fail replay, and unit-test the journal itself as the durability primitive. | temporalio/sdk-python, cadence-workflow/cadence-go-client, temporalio/temporal, restatedev/restate, triggerdotdev/trigger.dev, browser-use/browser-useSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:28 a deliberately nondeterministic fixture exists |
Durable workflow orchestrators, Browser-use stacks | none | not applicableNo stateful workflows. |
| RP-045Fault injection and network degradation in tests | Inject faults (toxiproxy latency and drops, torn writes, cluster poisoning, noise and accent perturbations) through a harness, not by hand, and assert recovery. | temporalio/temporal, cadence-workflow/cadence, livekit/agents, ServiceNow/eva, Dicklesworthstone/frankengraphdbSource: ecosystem/pickup/_evidence/workflow-orchestrators.md:32 Fault-injection / cluster-poisoning / simulation harness for the engine itself |
Durable workflow orchestrators, Realtime voice agents, FrankenSuite | none | not applicableNo runtime to fault. |
| RP-046Stress, soak and load suites separate from functional tests | Keep long-running stress, soak, pressure, and ingestion-load suites in their own tier with named workload shapes, off the PR critical path. | chroma-core/chroma, qdrant/qdrant, kata-containers/kata-containers, mlc-ai/xgrammar, comet-ml/opikSource: ecosystem/pickup/_evidence/vector-dbs.md:28 Stress / soak suites separate from unit tests |
Vector databases, Sandboxed code execution, Structured output and constrained decoding, Agent observability and tracing | none | not applicableStatic site; no load to test. |
| RP-047Determinism and seeding assertions at every level | Assert same seed gives the same trajectory at unit, checker, and vectorized levels, and requalify golden hashes across machines and build profiles. | Farama-Foundation/Gymnasium, Farama-Foundation/PettingZoo, meta-pytorch/torchtune, Dicklesworthstone/frankensim, Dicklesworthstone/frankentui, Dicklesworthstone/frankensqliteSource: ecosystem/pickup/_evidence/rl-envs.md:17 Copy: same seed -> same trajectory assertions at unit, checker, and vector levels. |
RL environments and RLHF training, Fine-tuning infrastructure, FrankenSuite | none | not applicableNo stochastic code; build reproducibility is covered by the regenerate-and-diff row. |
| RP-048Standard benchmark harness in the repo, next to the code it measures | Ship the throughput, latency, and quality harness in-tree (or in the library), with shared dataset configs and real workload shapes, not ad hoc scripts. | vllm-project/vllm, sgl-project/sglang, InternLM/lmdeploy, huggingface/text-generation-inference, NVIDIA/TensorRT-LLM, ggml-org/llama.cpp, chroma-core/chroma, weaviate/weaviate, vespa-engine/vespa, trycua/cua, AutoGPTQ/AutoGPTQ, getzep/zep, mem0ai/memory-benchmarks, run-llama/llama_index, browserbase/stagehand, protectai/llm-guardSource: ecosystem/pickup/_evidence/inference-engines.md:18 Standardized serving benchmark harness (throughput/latency scripts, shared dataset registry). |
LLM inference engines, Vector databases, Computer-use agents, Model quantization toolkits, Agent memory systems, Retrieval-augmented generation (RAG) frameworks, Browser-use stacks, Guardrails and red-teaming | none | not applicableNo performance claims of its own. |
| RP-049Benchmarks unit-tested like code | Test the benchmark CLI, timing functions, and parity between benchmark implementations so a broken harness cannot report a number. | vllm-project/vllm, sgl-project/sglangSource: ecosystem/pickup/_evidence/inference-engines.md:19 Benchmarks are unit-tested like code. |
LLM inference engines | none | not applicableNo benchmark harness. |
| RP-050Performance regression as its own CI track, A/B against a baseline | Run perf tests in a dedicated pipeline separate from correctness CI, as paired A/B comparisons against the baseline on PRs or nightly, with results kept as artifacts. | vllm-project/vllm, sgl-project/sglang, firecracker-microvm/firecracker, mlc-ai/xgrammar, trycua/cuaSource: ecosystem/pickup/_evidence/sandbox-exec.md:39 perf tests run as A/B comparisons against baseline on PRs |
LLM inference engines, Sandboxed code execution, Structured output and constrained decoding, Computer-use agents | none | not applicableNo performance claims of its own. |
| RP-051Measure the overhead of the safety layer itself | Benchmark what the isolation, constraint, or guard mechanism costs (seccomp filter cost, grammar compile and mask time, scanner latency) alongside what it prevents. | google/gvisor, mlc-ai/xgrammar, vllm-project/vllm, protectai/llm-guardSource: ecosystem/pickup/_evidence/sandbox-exec.md:30 Seccomp-bpf filter benchmarking (cost of the isolation mechanism itself) |
Sandboxed code execution, Structured output and constrained decoding, Guardrails and red-teaming | none | not applicableNo safety layer at runtime. |
| RP-052Latency budgets asserted as tests | Define latency metrics (time to first audio, boot-to-init, startup, snapshot restore) as tested code with a spec to assert against, not only as dashboard numbers. | pipecat-ai/pipecat, livekit/agents, firecracker-microvm/firecracker, e2b-dev/E2BSource: ecosystem/pickup/_evidence/sandbox-exec.md:37 Boot-to-init time spec test: parse |
Realtime voice agents, Sandboxed code execution | none | not applicableNo latency-sensitive runtime. |
| RP-053Report quality delta together with the speedup | Every optimization or quantization method reports the quality change (perplexity, downstream tasks) next to its speedup, from one command. | pytorch/ao, NVIDIA/TensorRT-LLM, ggml-org/llama.cpp, EleutherAI/lm-evaluation-harnessSource: ecosystem/pickup/_evidence/quantization.md:25 reports perplexity delta, (c) reports kernel speedup, all one command |
Model quantization toolkits, LLM inference engines | none | not applicableNo optimizations to report. |
| RP-054Reproducible contender containers and parameter sweeps (report a frontier, not a point) | Benchmark every contender in its own container against identical datasets and sweep index, query, and quantization parameters, publishing the recall-vs-throughput frontier. | erikbern/ann-benchmarks, qdrant/qdrant, milvus-io/milvus, weaviate/weaviateSource: ecosystem/pickup/_evidence/vector-dbs.md:24 Docker-per-algorithm reproducibility for benchmarks |
Vector databases | none | not applicableNo benchmarks. |
| RP-055Security properties get dedicated adversarial tests | Assert isolation and auth properties directly: jail uid/gid and permission bits, seccomp blocking the forbidden syscall set, known-vulnerability regression, signature paths with an independent reimplementation, audit events that must fire. | firecracker-microvm/firecracker, e2b-dev/E2B, a2aproject/a2a-js, a2aproject/a2a-python, fetchai/uAgents, agent-network-protocol/anpSource: ecosystem/pickup/_evidence/sandbox-exec.md:27 verify the BPF filter actually blocks the forbidden syscall set |
Sandboxed code execution, Agent-to-agent protocols (A2A class) | none | not applicableNo isolation or auth surface. |
| RP-056Resource-limit and worst-case safety tests | Test that timeouts, kills, rate limits, and compilation budgets are enforced, including pathological inputs that would stall the serving loop. | firecracker-microvm/firecracker, e2b-dev/E2B, vllm-project/vllmSource: ecosystem/pickup/_evidence/structured-output.md:27 guards against pathological regexes DoS-ing the serving loop |
Sandboxed code execution, Structured output and constrained decoding | none | not applicableNo runtime resource limits. |
| RP-057Fuzzing and malformed-input batteries as maintained equipment | Keep a fuzz corpus and targets for parsers, run them on a schedule, and assert that every malformed input class is refused. | deepset-ai/haystack, oven-sh/bun, Dicklesworthstone/franken_node, Dicklesworthstone/frankenredis, Dicklesworthstone/franken_overlap, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/rag-frameworks.md:26 Fuzz corpus for parsers |
Retrieval-augmented generation (RAG) frameworks, FrankenSuite | none | not applicableNo parsers of untrusted input. |
| RP-058Property-based tests with explicit invariants | Generate inputs with property strategies and assert named invariants (cross-version persistence, metamorphic identities) instead of example-only tests. | chroma-core/chroma, Dicklesworthstone/frankenjax, Dicklesworthstone/asupersyncSource: ecosystem/pickup/_evidence/vector-dbs.md:27 Property-based correctness testing (Hypothesis strategies + explicit invariants) |
Vector databases, FrankenSuite | none | not applicableNo library code. |
| RP-059UB and sanitizer lanes over the unsafe-dense code | Run sanitizers, coverage, and Miri (with the aliasing model and its rationale recorded) as separate lanes over the code with the most unsafe. | firecracker-microvm/firecracker, ggml-org/llama.cpp, oven-sh/bun, Dicklesworthstone/frankenfsSource: synthesis/vendor-port-learnings.md:680 G10. MIRI — UB detector over the unsafe-dense corner [Bun]. |
Sandboxed code execution, LLM inference engines, FrankenSuite | none | not applicableNo unsafe code. |
| RP-060Version and pin benchmark tasks and datasets; record the commit in every run | Never mutate a benchmark silently: version task configs, pin dataset versions or upstream commits in run manifests, bundle third-party datasets with provenance, and keep old configs reproducible. | EleutherAI/lm-evaluation-harness, harbor-framework/terminal-bench-1, xlang-ai/OSWorld, simular-ai/Agent-S, Skyvern-AI/skyvern, microsoft/PyRITSource: ecosystem/pickup/_evidence/eval-harnesses.md:25 Copy: never silently mutate a benchmark; bump a version. |
LLM eval harnesses, Computer-use agents, Browser-use stacks, Guardrails and red-teaming | Incumbent oracle pinned before implementation | partialdone: every packet is assessed at a pinned commit recorded in its header; RULEBOOK.md:68; packets/asupersync-assessment.md; missing: no benchmark task or dataset versioning and no per-run manifest recording the commit of each CI run |
| RP-061Separate benchmark data from harness code | Keep the harness repository code-only and the datasets in their own versioned home, referenced by id. | UKGovernmentBEIS/inspect_ai, UKGovernmentBEIS/inspect_evals, harbor-framework/terminal-bench-1, EleutherAI/lm-evaluation-harness, microsoft/graphragSource: ecosystem/pickup/_evidence/eval-harnesses.md:30 Copy: harness repo stays code; datasets live in their own versioned home. |
LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks | none | not applicableNo harness. |
| RP-062Registry as code enumerates the supported surface and drives the test matrix | Declare supported models, evals, chunkers, or attack plugins as typed registry entries so the test matrix is enumerable and a new entry is a new file. | qdrant/fastembed, openai/evals, HKUDS/LightRAG, promptfoo/promptfooSource: ecosystem/pickup/_evidence/embedding-serving.md:24 the supported-model contract and makes test matrices enumerable |
Embedding model serving, LLM eval harnesses, Retrieval-augmented generation (RAG) frameworks, Guardrails and red-teaming | none | candidateThe 21 stack slugs are fixed by ecosystem/pickup file names; a registry file that Gate K reads would make that contract explicit. |
| RP-063Named attack families and labelled pass/fail cases in code | Enumerate attack and evasion families as named modules with metadata, and keep false-positive and false-negative cases as parametrized test data, not a spreadsheet. | promptfoo/promptfoo, NVIDIA/garak, protectai/llm-guard, microsoft/PyRITSource: ecosystem/pickup/_evidence/guardrails.md:28 Copy: enumerate attack families as named modules with per-module metadata |
Guardrails and red-teaming | none | not applicableNo guard to attack. |
| RP-064Tool-call gating as a separate guard family | Treat guards on tool inputs and outputs as a category of their own, separate from prompt-text guards, with their own tests. | meta-llama/PurpleLlama, NVIDIA-NeMo/GuardrailsSource: ecosystem/pickup/_evidence/guardrails.md:34 Copy: guardrails for tool input/output are a separate category from prompt text rails. |
Guardrails and red-teaming | none | not applicableNo tools exposed. |
| RP-065Test tree mirrors the source tree, with a shared test-utils module | One test file or package per component, trainer, or encoder family, mirroring the source layout, with shared fixtures extracted into a test-utils package. | run-llama/llama_index, deepset-ai/haystack, huggingface/trl, huggingface/sentence-transformers, microsoft/autogen, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/rl-envs.md:19 Copy: per-algorithm test files + a shared fixtures module. |
Retrieval-augmented generation (RAG) frameworks, RL environments and RLHF training, Embedding model serving, Agent frameworks, Realtime voice agents | none | not applicableTests are the gate chain in one script. |
| RP-066Smallest real instance end to end: tiny-model smoke and golden-path examples | Run the full pipeline (config validation, load, train or quantize, save, evaluate or generate) on the smallest real model in single-digit steps, and run one golden-path example per format in CI. | axolotl-ai-cloud/axolotl, huggingface/trl, meta-pytorch/torchtune, Lightning-AI/litgpt, hiyouga/LlamaFactory, mit-han-lab/llm-awqSource: ecosystem/pickup/_evidence/fine-tuning.md:34 Tiny-model smoke trains as the standard unit of correctness (steps in the single digits) |
Fine-tuning infrastructure, Model quantization toolkits | none | not applicableNo training or model pipeline. |
| RP-067Config validation as its own tested surface | Expose config validation and normalization as an API and exercise it in every smoke test and per-recipe config test. | axolotl-ai-cloud/axolotl, meta-pytorch/torchtune, Lightning-AI/litgptSource: ecosystem/pickup/_evidence/fine-tuning.md:49 Config validation as its own testable surface |
Fine-tuning infrastructure | none | partialdone: Gate K1 inside bun run verify rejects missing front-matter keys, bad enums and missing sections; site/scripts/verify-site.sh; commit 68d9523; missing: validation is not exposed as a separate API, and Gate K has not yet run on pushed CI |
| RP-068Keep hardware CI tractable: cache priming and CPU legs for distributed logic | Prime model caches into every runner, cover distributed logic on CPU, and use free external hardware legs with canary-vs-control runs. | axolotl-ai-cloud/axolotl, hiyouga/LlamaFactory, unslothai/unslothSource: ecosystem/pickup/_evidence/fine-tuning.md:55 Keeping GPU CI tractable: caching + model-cache priming + free external GPU legs |
Fine-tuning infrastructure | none | not applicableNo hardware-bound tests. |
| RP-069Assert telemetry against semantic conventions, including streaming span lifecycles | Per instrumented provider, assert that spans follow the semantic conventions, that streaming spans close and carry token usage, and that the export path (masking, batching, propagation) is unit-tested. | traceloop/openllmetry, openlit/openlit, langfuse/langfuse-python, lmnr-ai/lmnr-python, langchain-ai/langsmith-sdkSource: ecosystem/pickup/_evidence/observability.md:26 Copy: semantic-convention assertions per instrumented provider, not just |
Agent observability and tracing | none | not applicableNo telemetry. |
| RP-070Scheduled correctness jobs for facts that change underneath you | Re-check externally changing facts on a schedule (model prices, upstream provider SDK changes) with their own job, instead of trusting the snapshot taken when the code was written. | langfuse/langfuse, comet-ml/opik, pipecat-ai/pipecatSource: ecosystem/pickup/_evidence/observability.md:33 Copy: cost tracking needs its own scheduled correctness job |
Agent observability and tracing, Realtime voice agents | none | candidateThe 21 evidence packs are 2026-09-23 GitHub API snapshots (stars, push dates, file paths) with no scheduled re-check. |
| RP-071Post-deploy checks against the shipped artifact | After every deploy, run the quality or smoke check against the live production artifact, labelled with the commit, not only against the branch build. | firecrawl/firecrawl, trycua/cua, letta-ai/letta-code, Dicklesworthstone/franken_markdown_website, Dicklesworthstone/frankentui_websiteSource: ecosystem/pickup/_evidence/web-search-apis.md:22 every merge to main gets an automatic quality benchmark run on the shipped artifact |
Web search API layers, Computer-use agents, Agent memory systems, FrankenSuite | none | candidateDeploys to fr.zeststream.ai are manual and no check compares the served pages with the repository. |
| RP-072Live-provider tests wired to secrets, skipped on forks | Test against the real endpoint in CI with repository secrets, path-triggered and skipped on forks, and keep a unit tier that intercepts transport for everything else. | tavily-ai/tavily-python, exa-labs/exa-py, serpapi/serpapi-python, livekit/agents, NVIDIA-NeMo/Guardrails, guidance-ai/guidanceSource: ecosystem/pickup/_evidence/web-search-apis.md:25 test against the real endpoint in CI, with unit-only separation for the rest |
Web search API layers, Realtime voice agents, Guardrails and red-teaming, Structured output and constrained decoding | none | not applicableNo live provider. |
| RP-073Automated checks a contribution must pass before merge | Document the automated checks a new benchmark or spec change must pass (policy gate script, spec lint, spelling, conventional commits) and run them before human review. | UKGovernmentBEIS/inspect_ai, UKGovernmentBEIS/inspect_evals, a2aproject/A2ASource: ecosystem/pickup/_evidence/multi-agent-protocols.md:25 Copy: spec changes go through machine-checked lint before human review. |
LLM eval harnesses, Agent-to-agent protocols (A2A class) | none | partialdone: every pull request runs the full gate chain; .github/workflows/verify.yml:10; CONTRIBUTING.md; missing: main has no branch protection (HTTP 404 on 2026-09-24), so a red check does not block a merge |
| RP-074Change record enforced: changelog or release note per change | Require a changelog entry when the version changes, or a release-note fragment per change, and automate the release record. | mem0ai/mem0, deepset-ai/haystack, microsoft/graphrag, exa-labs/exa-pySource: ecosystem/pickup/_evidence/agent-memory.md:24 Changelog enforcement keyed to version bump |
Agent memory systems, Retrieval-augmented generation (RAG) frameworks, Web search API layers | none | candidateCHANGELOG.md is kept by hand; no check ties it to a version or tag. |
| RP-075Agent contributor playbook in the repository | Keep AGENTS.md or CLAUDE.md with the build, test, and conduct rules agents must follow, next to the code. | lancedb/lancedb, milvus-io/milvus, weaviate/weaviate, chroma-core/chroma, Dicklesworthstone/franken_whisper, Dicklesworthstone/franken_markdown, Dicklesworthstone/frankenfsSource: ecosystem/pickup/_evidence/vector-dbs.md:37 how these projects onboard agents; worth mirroring |
Vector databases, FrankenSuite | Anti-reward-hacking law adopted | candidateCONTRIBUTING.md covers humans; there is no AGENTS.md stating the 12 forbidden patterns for agents working in this repository. |
| RP-076Claim-to-proof registry with machine gates | Bind every public claim to a machine-checkable key and a proof; the gate refuses wording above its evidence grade, demotes on digest mismatch, and fails CI on drift. | Dicklesworthstone/franken_engine, Dicklesworthstone/frankengit, Dicklesworthstone/franken_markdown, Dicklesworthstone/franken_node, Dicklesworthstone/frankengraphdb, Dicklesworthstone/frankensim, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:7 Every public claim bound to a machine-checkable key and a proof script |
FrankenSuite | Claim inventory registered before claims are made; Claim discipline enforced: README claims cross-checked against proofs | partialdone: Gate A2 checks method-page claims against their packet line, K2 checks verdict quotes on their cited lines, K3 resolves proof tokens; site/scripts/verify-site.sh; commit 1ef0f8e; commit 68d9523; missing: no gate compares wording with its evidence grade or demotes a claim on a digest mismatch |
| RP-077Freshness decay on evidence | Give every claim a maximum age since re-verification; the gate downgrades rows whose check has lapsed, so verified cannot silently mean verified once. | Dicklesworthstone/franken_engineSource: synthesis/cross-pollination.md:15 the gate auto-downgrades rows whose re-verification lags |
FrankenSuite | Demotion rules executed on schedule | candidatePackets and evidence packs carry assessment dates, but no gate demotes a claim when its date ages past a ceiling. |
| RP-078Negative-evidence ledger with required entry classes and retry conditions | Log rejections, retractions, and failed optimizations as first-class rows with mandatory classes and a retry condition, so a rejection is a paused hypothesis. | Dicklesworthstone/frankengit, Dicklesworthstone/franken_numpy, Dicklesworthstone/frankensqlite, Dicklesworthstone/franken_tts, Dicklesworthstone/franken_networkx, Dicklesworthstone/frankenlibc, Dicklesworthstone/frankentorchSource: synthesis/cross-pollination.md:23 A ledger where rejections, retractions, and failed optimizations are first-class rows |
FrankenSuite | Negative-evidence ledger initialized with its schema; Every falsified hypothesis banked with a retry predicate | candidateRejected approaches for this repository are recorded only in commit messages and synthesis prose; there is no ledger file with retry predicates. |
| RP-079Anti-extrapolation: an unrun baseline is recorded as incomplete | When the incumbent cannot be run within budget, record the comparison as incomplete or no admissible ratio; never extrapolate a ratio against a baseline that did not run. | Dicklesworthstone/franken_overlap, Dicklesworthstone/franken_ttsSource: synthesis/cross-pollination.md:31 A baseline that exceeds the compute budget is recorded as |
FrankenSuite | Incumbent oracle pinned before implementation | partialdone: the Rulebook forbids citing an ungated number; RULEBOOK.md:80; missing: no rule that a baseline which could not be run is recorded as incomplete with no ratio |
| RP-080Committed statistical baseline gate with environment tagging | Commit the benchmark baseline with per-row environment tags and captured hyperparameters, sign it off, and let a drift detector that demonstrably fires block regressions. | Dicklesworthstone/asupersyncSource: synthesis/cross-pollination.md:39 Benchmark regression as a signed, waiver-free gate |
FrankenSuite | none | not applicableNo benchmarks of its own. |
| RP-081Per-optimization proof bundles | Ship each accepted optimization as one reviewable bundle: SHA-pinned golden transcript, paired and reversed runs, and a keep or reject score gate. | Dicklesworthstone/frankenredis, Dicklesworthstone/frankenjaxSource: synthesis/cross-pollination.md:47 Every accepted optimization ships a bundle |
FrankenSuite | none | not applicableNo optimizations. |
| RP-082Measurement-integrity four-rule gate | Name the worker, name the harness, quote both estimators, and replicate before quoting; comparisons across different workers are invalid. | Dicklesworthstone/frankentorch, Dicklesworthstone/frankenpandas, Dicklesworthstone/frankengraphdbSource: synthesis/cross-pollination.md:55 Name the worker, name the harness, quote both estimators, replicate before quoting |
FrankenSuite | Receipts carry provenance; generation binding | not applicableNo measurements of its own. |
| RP-083Result-class doctrine for performance verdicts | Type every performance verdict: a self-speedup is maintenance; a campaign win needs the incumbent binary in the same invocation with A/A nulls in range; preregister the verdict rule. | Dicklesworthstone/franken_whisper, Dicklesworthstone/franken_networkx, Dicklesworthstone/franken_overlap, Dicklesworthstone/frankenmermaidSource: synthesis/cross-pollination.md:63 SELF-SPEEDUP is maintenance, never a campaign win |
FrankenSuite | Incumbent oracle pinned before implementation | not applicableNo performance verdicts of its own. |
| RP-084Structured skip honesty: an absent oracle or unrun gate fails | Emit a typed skip event instead of passing silently; make the absence of the oracle, the browser, or any scanned item a failure; refuse dormant lanes with a nonzero exit instead of a false green. | Dicklesworthstone/franken_snowflake, Dicklesworthstone/frankenscipy, Dicklesworthstone/franken_tts, Dicklesworthstone/frankengitSource: synthesis/cross-pollination.md:71 Runs that cannot execute honestly emit a typed skip event instead of silently passing |
FrankenSuite | none | adopted.github/workflows/verify.yml:84; site/scripts/verify-site.sh; commit a298c99; commit 9a99982; no browser found is a FAIL, and zero data-stat slots or zero resolved links fail Gates B and E |
| RP-085Maturity vocabulary and dated maturity registry | Label every capability on a ladder (source-present to production-admitted) where no lower label implies a higher one, and publish the dated distribution; partial never rounds up. | Dicklesworthstone/franken_drone_geometry_reconstruction, Dicklesworthstone/frankensim, Dicklesworthstone/franken_ocr, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:79 no lower label implies a higher one |
FrankenSuite | Proof taxonomy declared (provisional) | candidateApplies: site/self/index.html scores this repository with the same TRL and ring vocabulary it applies to others (commit a298c99), but there is no dated maturity registry and no rule that a lower label never implies a higher one. |
| RP-086Evidence color and no-laundering composition | Give evidence typed strengths and compose them so an aggregate can never carry a stronger color than its weakest component. | Dicklesworthstone/frankensim, Dicklesworthstone/frankengraphdbSource: synthesis/cross-pollination.md:87 composition rules prevent weak evidence from being laundered into strong claims |
FrankenSuite | Proof taxonomy declared (provisional) | adoptedRULEBOOK.md:19; stack/METHOD.md; the verdict itself is always tier Inference, whatever its citations |
| RP-087Receipt-bound remote execution with self-disqualification | Bind each remote batch to source hashes, job id, and raw logs; keep failed attempts; any code landing after the last executed gate opens a self-disqualification until the gate reruns. | Dicklesworthstone/franken_alignment, Dicklesworthstone/franken_node, Dicklesworthstone/franken_remoteSource: synthesis/cross-pollination.md:95 code landing after the last executed gate opens a self-disqualification ledger |
FrankenSuite | Receipts carry provenance; generation binding | not applicableAll gates run on public GitHub Actions; there is no private runner to receipt. |
| RP-088Full-funnel verdict aggregation with selection pressure stated | Aggregate every measured lane into typed verdicts, including dropped and undecidable ones, and quote the decidable-subset statistic with its selection pressure. | Dicklesworthstone/frankenpandas, Dicklesworthstone/frankensympySource: synthesis/cross-pollination.md:103 Aggregate every measured lane into typed verdicts |
FrankenSuite | none | not applicableNo measured lanes. |
| RP-089Adversarial self-audit: gates proven to trip on planted defects | Plant defects (a laundered claim, a weakened gate, a one-byte drift) and record that the gate fails; keep a permanent self-test that proves the gates still trip, and a mutation-score floor where it fits. | Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensympy, Dicklesworthstone/franken_code_browser, Dicklesworthstone/frankensim, Dicklesworthstone/franken_node, Dicklesworthstone/frankenredisSource: synthesis/cross-pollination.md:111 The evidence gate itself gets audited adversarially |
FrankenSuite | Pre-commit honesty gate runs and self-tests with a canary; Gate self-test: deliberately break a gate | partialdone: one-time planted defects recorded as failing: packet drift fails A and console.error fails I (9a99982), renamed data-stat and href fail B and E (a298c99), planted bad copies fail Gate K (68d9523); commit 9a99982; commit a298c99; commit 68d9523; missing: no permanent known-bad fixture that CI runs every time |
| RP-090Registry-agreement gate: parallel registries must agree exactly | When several tables describe the same set (registry, slug map, factory, data file), fail the build unless they agree exactly. | Dicklesworthstone/franken_agent_detectionSource: synthesis/uniqueness-catalog.md:28 the connector registry, tilde-expansion table, canonical-slug map, and factory registry must agree |
FrankenSuite | none | adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 1ef0f8e; Gate D requires briefs, packets and site/assets/data.js names to match one to one |
| RP-091Statistics computed from data, not typed | Compute every displayed count from the data at build or render time and fail when static fallback text differs from the computed value. | Dicklesworthstone/frankensim_website, Dicklesworthstone/frankentermSource: synthesis/uniqueness-catalog.md:331 Computed stats pipeline. |
FrankenSuite | Claim discipline enforced: README claims cross-checked against proofs | adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; commit 1ef0f8e; Gate B recomputes every aggregate from site/assets/data.js and fails on bare N of 44 literals without a STAT source note |
| RP-092Disclaim as data: unmeasured targets are machine-readable as unmeasured | Mark targets as targets and unmeasured values as unmeasured in the data itself (status fields, row-level basis labels), and count refusal sites, so absence of evidence is data rather than prose. | Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensearch, Dicklesworthstone/franken_manimSource: synthesis/uniqueness-catalog.md:38 the absence of measurement is machine-readable data, not prose |
FrankenSuite | Proof taxonomy declared (provisional) | candidateCandidate verdict rows carry no machine field distinguishing measured from asserted beyond the tier label. |
| RP-093Release structurally blocked until the gate passes | Make release a checked impossibility until the release gate passes (a release-check command that returns an error, a 100 percent declared-surface threshold with no downgrades). | Dicklesworthstone/franken_alignment, Dicklesworthstone/frankensqliteSource: synthesis/uniqueness-catalog.md:39 release is a checked impossibility, not an aspiration |
FrankenSuite | Release gate defined: what blocks publication | candidatev1.0.0 was tagged by hand after a green run; no workflow blocks a tag whose commit has not passed verify. |
| RP-094Upstream-anchored contracts | Anchor each contract rule to the exact upstream file and line at the pin with an expected token, and fail CI when upstream drifts. | Dicklesworthstone/franken_leanSource: synthesis/uniqueness-catalog.md:72 line naming the exact upstream file/line at the pin |
FrankenSuite | Incumbent oracle pinned before implementation | partialdone: K2 fails a verdict citation whose line no longer contains its quote, which is local citation integrity; site/scripts/verify-site.sh; commit 68d9523; missing: nothing re-fetches an upstream pin and fails CI when upstream moves |
| RP-095Reject fixture-only and synthetic artifacts as evidence | Refuse simulated or fixture-only artifacts as backing for a performance or capability claim. | Dicklesworthstone/franken_engine, Dicklesworthstone/franken_nodeSource: synthesis/uniqueness-catalog.md:65 ) are rejected by the gate as backing evidence |
FrankenSuite | Claim discipline enforced: README claims cross-checked against proofs | not applicableNo performance evidence of its own. |
| RP-096Publish rejected and failed artifacts instead of regenerating them | Keep failed receipts, rejected release artifacts, stale not-green suite results, and the reason CI is disabled, visible with identifiers, rather than regenerating into green. | Dicklesworthstone/franken_manim, Dicklesworthstone/frankensim, Dicklesworthstone/franken_markdown_website, Dicklesworthstone/franken_overlap, Dicklesworthstone/beads-for-frankentuiSource: synthesis/uniqueness-catalog.md:83 That artifact was rejected and is not attached |
FrankenSuite | none | partialdone: the repository and every CI run are public; .github/workflows/verify.yml:6; missing: no step uploads a failed run log as an artifact, and no red run has yet been observed to be retained |
| RP-097Machine-greppable honesty trailers on commits | End each commit with what was actually run (or its verification level) in a fixed, greppable form, so history records evidence, not just intent. | Dicklesworthstone/franken_surveillance_systemSource: synthesis/uniqueness-catalog.md:183 the pin's own trailer lists what was |
FrankenSuite | Close only on cited evidence; close_reason required | adoptedCONTRIBUTING.md:49; every commit subject ends with a verification level; convention only, not machine-checked in CI |
| RP-098Evidence tier on every statement | Label each statement in plans and findings with its epistemic status (fact, design, hypothesis, target; or evidence tier plus confidence) so readers never mistake a target for a result. | Dicklesworthstone/franken_surveillance_system, Dicklesworthstone/franken_nlpSource: synthesis/uniqueness-catalog.md:185 Every statement labeled FACT / DESIGN / HYPOTHESIS / TARGET / OPEN |
FrankenSuite | Proof taxonomy declared (provisional) | partialdone: the Rulebook requires a tier and confidence on every claim, and K2 enforces a tier on verdict bullets; RULEBOOK.md:19; site/scripts/verify-site.sh; commit 68d9523; missing: packet prose is not machine-checked statement by statement |
| RP-099Receipts carry provenance and generation binding | Every receipt or fixture records commit, tool versions, host, worker, oracle script hash, and generation command; a receipt from another generation is not evidence. | Dicklesworthstone/franken_remote, Dicklesworthstone/franken_snowflake, Dicklesworthstone/franken_manim, Dicklesworthstone/frankenpandas, Dicklesworthstone/frankenterm, Dicklesworthstone/frankengitSource: synthesis/uniqueness-catalog.md:279 All 1,387 conformance packets carry |
FrankenSuite | Receipts carry provenance; generation binding | partialdone: every packet header records repository, pinned commit, date and analyst method; RULEBOOK.md:68; missing: tool versions, host, worker, oracle script hash and generation command are not recorded |
| RP-100Machine-checked dependency admissibility | Check the dependency tree against a declared diet (banned crates, franken-only, zero-external) in CI, with the admission rationale recorded per dependency. | Dicklesworthstone/franken_snowflake, Dicklesworthstone/franken_alignment, Dicklesworthstone/franken_threed, Dicklesworthstone/frankensimSource: synthesis/uniqueness-catalog.md:175 Machine-checked dependency diet. |
FrankenSuite | none | candidateOne exact-pinned devDependency (esbuild) with no admission record or tree check. |
| RP-101Tolerance and floor ratchets: thresholds only tighten | Enforce tolerances and conformance floors with ratchet semantics in CI, so a threshold can tighten but never loosen without an explicit gate change. | Dicklesworthstone/frankenscipy, Dicklesworthstone/franken_markdownSource: synthesis/uniqueness-catalog.md:296 CI enforces the tolerance policy with ratchet semantics |
FrankenSuite | Never weaken a gate to land a change; gate changes need two-direction evidence | not applicableNo numeric thresholds in the gate chain. |
| RP-102Failure-mode taxonomy for honesty machinery | Name the ways evidence culture fails (demo hardcoding, proof laundering, refusal farming, commit pumping, gate self-weakening, follow-up laundering) so reviewers can call them by name. | Dicklesworthstone/franken_code_browser, Dicklesworthstone/franken_whisper, Dicklesworthstone/frankensimSource: synthesis/uniqueness-catalog.md:46 Coordinated-failure defect taxonomy. |
FrankenSuite | Anti-reward-hacking law adopted | candidateThe 12 forbidden patterns are in the starter kit, but this repository has no agent instructions file adopting them. |
| RP-103Bead closure requires evidence of minimum substance | Reject tracker closures without an evidence-bearing reason of minimum length, and sweep old thin closures back open. | Dicklesworthstone/frankentui, Dicklesworthstone/franken_nodeSource: synthesis/uniqueness-catalog.md:380 rejects bead closures without 80+ character evidence-bearing reasons |
FrankenSuite | Close only on cited evidence; close_reason required; False-closure repair sweep | not applicableThis repository keeps no bead graph of its own. |
| RP-104Typed proof grammar for claims | Give each claim proof a typed form (test, path, identifier, command, count, manual) so proof kinds can be checked and counted. | Dicklesworthstone/frankentuiSource: synthesis/uniqueness-catalog.md:379 The claims ledger uses a typed proof grammar |
FrankenSuite | Proof taxonomy declared (provisional) | partialdone: our_proof tokens have two resolvable forms, a path with optional line and a commit sha, which K3 resolves; site/scripts/verify-site.sh; commit 68d9523; missing: no typed kinds for test, identifier, command, count or manual proofs |
| RP-105SLO-as-code | Keep performance budgets in a machine-readable file that gates enforce or that changes runtime posture on breach. | Dicklesworthstone/frankentui, Dicklesworthstone/franken_drone_geometry_reconstruction, Dicklesworthstone/asupersyncSource: synthesis/uniqueness-catalog.md:382 frame budgets enforced by gates |
FrankenSuite | none | not applicableNo runtime performance budget. |
| RP-106In-place atomic swap per change, never big-bang or shadow A/B | Each change replaces the old implementation with a thin shim into the new one and deletes the old code in the same commit, so the new code is exercised in place and main stays shippable. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:42 Each pull request replaces the |
FrankenSuite | none | not applicableNo port. |
| RP-107Shipping pilots and a trial run before scale | Land the toolchain, CI, and rules first, prove the full pipeline on a few side-effect-free units with human-read drafts, and only then scale out. | github/copilot-agent-runtime, oven-sh/bunSource: synthesis/vendor-port-learnings.md:231 Full pipeline (1 implementer + 2 adversarial reviewers + |
FrankenSuite | none | not applicableNo bulk port or generation run. |
| RP-108Leaves-inward waves, with temporary interop counted down to zero | Order work from pure leaves inward to the most coupled core, move behaviour in waves (logic, state, orchestration, remove fallbacks, simplify), and count the temporary interop seam down to zero. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:67 T-C5. Temporary interop as a deliberately deletable seam. |
FrankenSuite | none | not applicableNo port. |
| RP-109Incremental rollout as last-mile validation | Ship the port incrementally to a pre-release audience so regressions correlate with recent changes and are cheap to root-cause. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:72 Doing the porting incrementally over |
FrankenSuite | none | not applicableNo staged audience. |
| RP-110Protect the oracle from the agent: frozen E2E tests and two-key waivers | Freeze the end-to-end suite as the contract; agents may not delete or weaken tests, update snapshots, raise baselines, or apply waiver labels without a second approver; zero tests skipped or deleted. | github/copilot-agent-runtime, oven-sh/bun, Dicklesworthstone/frankenredis, Dicklesworthstone/frankengitSource: synthesis/vendor-port-learnings.md:85 The agent changing an implementation cannot |
FrankenSuite | Never weaken a gate to land a change; gate changes need two-direction evidence; Waivers are public, time-bounded, and recorded | candidateGate A keeps the shipped packets byte-identical to the canonical ones, but nothing requires a second approver to change the canonical packets, RULEBOOK.md or the gate script. |
| RP-111Split-context adversarial review, default-refute, with attribution | Give the reviewer the diff and nothing else, tell it to assume the code is wrong, default verdicts to refuted unless it cites source and target lines, and attribute every caught bug in the fix commit. | oven-sh/bun, github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:247 The reviewer gets the diff and nothing else |
FrankenSuite | Plan independently reviewed before execution | partialdone: the author of a verdict never reviews it, and K1 fails when reviewed_by equals author; stack/METHOD.md:83; site/scripts/verify-site.sh; commit 68d9523; missing: nothing gives the reviewer only the diff, defaults findings to refuted, or attributes caught bugs in fix commits |
| RP-112Line-by-line old-versus-new review by several models | After each rebase, have one reviewer per model compare old and new code line by line for behavioural equality, flag leftover shims and changed E2E tests, and loop until every review is clean. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:98 a line-by-line comparison of the old TypeScript and the new Rust, confirming behavioral |
FrankenSuite | Plan independently reviewed before execution | not applicableNo port to compare. |
| RP-113Division of review labour | Agents do exhaustive old-versus-new comparison, tests and static analysis check what is mechanically enforceable, and humans concentrate on architecture, contracts, risk, and whatever the other layers flag. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:108 reviewers concentrated on architecture, API contracts, risk, and any suspicious places |
FrankenSuite | none | candidateThis batch splits drafting, gating and independent review across agents, but the split is not written down for the repository. |
| RP-114Rebase conflict as a feature of atomic delete | Delete old code in the same change that adds its replacement, so any incoming edit to already-ported code surfaces as a rebase conflict instead of being lost. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:114 T-C11. Rebase-conflict-as-a-feature from atomic delete. |
FrankenSuite | none | not applicableNo port. |
| RP-115Explorers fan out, one coordinator owns edits; child sessions own disjoint files | Use subagents for independent read-only questions, keep mutation with the coordinating agent, give parallel writers disjoint file ownership, and write kickoff prompts that name siblings and what to leave alone. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:121 T-C12. Subagents for exploration, child sessions for parallel diffs. |
FrankenSuite | none | candidateThis v1.1 batch assigned exclusive file ownership per agent in the lead prompt; the rule is not recorded in the repository. |
| RP-116A failure seen twice becomes standing machinery | When a failure mode recurs, turn it into a standing instruction, a reusable skill, an eval, a protected baseline, or a gate, not a one-off fix. | github/copilot-agent-runtime, Dicklesworthstone/frankenfsSource: synthesis/vendor-port-learnings.md:140 it belongs in standing instructions, a reusable skill, an eval |
FrankenSuite | none | partialdone: Gate A2 names the audit failure that created it; site/BUILD-GATES.md:64; missing: not every gate names a recurring failure, and no rule says a repeat failure must become a gate |
| RP-117Translate first, redesign second | Never change language and behaviour in the same change; land the faithful translation with its oracle green, then redesign in a separate change. | github/copilot-agent-runtime, Dicklesworthstone/frankensqliteSource: synthesis/vendor-port-learnings.md:143 Translate first, redesign second. |
FrankenSuite | none | not applicableNo port. |
| RP-118Minimal permanent ABI surface with a dispatch interior | Keep the permanent boundary tiny (a handful of exported functions) and route everything else through a dispatch table behind it. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:149 Adding, changing, or removing an API method touches the |
FrankenSuite | none | not applicableNo ABI. |
| RP-119Unsafe governance: counted, boundary-only, SAFETY-commented, compiler-enforced | Count every unsafe block and keep them at interop boundaries, require an accurate SAFETY comment above each, push the unsafe boundary away from each access, and encode the style guide as lint bans. | github/copilot-agent-runtime, oven-sh/bun, NVlabs/cuda-oxide, Dicklesworthstone/franken_native_capsule, Dicklesworthstone/frankentorchSource: synthesis/vendor-port-learnings.md:291 Use the compiler to enforce safety. Code comments do not |
FrankenSuite | none | not applicableNo unsafe code. |
| RP-120Classify every regression into named families | Sort each regression into a small fixed taxonomy (incomplete migration, state and lifetime, contract mismatch, host boundary, incorrect oracle) and use the counts to steer review. | github/copilot-agent-runtime, oven-sh/bunSource: synthesis/vendor-port-learnings.md:157 T-C18. The five regression families |
FrankenSuite | none | candidateSite defects found in audits are fixed and gated but not tallied by family. |
| RP-121Paired-operation completeness | Declare every paired operation (abort and cancel, persist and project, register and deregister) and fail a change that touches one half without the other. | github/copilot-agent-runtimeSource: synthesis/vendor-port-learnings.md:623 PAIR — Paired-operation completeness |
FrankenSuite | none | not applicableNo paired runtime operations. |
| RP-122Ambient and host-boundary inventory | Inventory every ambient input (timezone, cwd, PATH, env, host tools) with its capture point and refresh policy, and test that changing it after capture changes nothing. | github/copilot-agent-runtime, Dicklesworthstone/franken_nodeSource: synthesis/vendor-port-learnings.md:643 HOST — Ambient/host-boundary inventory |
FrankenSuite | none | candidateThe gate chain depends on the host Chrome, python3 and node versions; CI pins node 22 and bun 1.4.2 but Chrome floats with the runner image and nothing records the version used. |
| RP-123Porting rulebook adversarially reviewed before bulk work | Write the idiom and type rulebook before any bulk work, have it adversarially reviewed, and make uncertainty markers greppable and budgeted: flagging is better than wrong output. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:220 adversarially reviewed before use |
FrankenSuite | Plan independently reviewed before execution | partialdone: stack/METHOD.md was revised after a stress-test before any verdict shipped; commit 16deb8c; missing: the reviewer findings are not in the repository, and the cited commit is itself marked pending |
| RP-124Pre-computed ownership table as an orthogonal source of truth | Classify ownership of every pointer or handle in a cited table before the work, and have agents use the table verbatim instead of guessing locally. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:223 LIFETIMES.tsv — pre-computed ownership table as an orthogonal source of truth. |
FrankenSuite | none | not applicableNo ownership decisions. |
| RP-125Compiler errors as a work queue | Write the checker output to a file grouped by unit and drain it with bounded agent teams, rather than letting each agent rediscover errors. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:236 Compiler errors as a work queue. |
FrankenSuite | none | not applicableNo compile step. |
| RP-126No stubbing to satisfy the checker | Forbid stubs and justification-paragraph workarounds that make an error disappear; if the workaround needs a paragraph to defend, fix the code. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:243 comment to justify why the workaround is OK, the code is wrong |
FrankenSuite | Anti-reward-hacking law adopted | candidateCovered by the starter kit forbidden patterns, which this repository has not adopted in an agent instructions file. |
| RP-127Bounded loops with explicit IOUs, zero unresolved at the gate | Give every agent loop a round cap, turn unfixables into structured blocked-on markers consumed by a later phase, and fail the phase gate on any left. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:333 Bounded loops with explicit IOUs: every phase had a round cap |
FrankenSuite | Phase exit criteria defined before agents start | not applicableNo agent loops run inside this repository. |
| RP-128Class fix followed by instance re-audit against the new code | After fixing a bug class systemically, re-run the per-instance audit against the new code and record each instance as fixed or exempt. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:339 re-run the per-site audit against the |
FrankenSuite | none | candidateWhen a gate is added for a defect class, older pages are not separately re-audited beyond what the gate scans. |
| RP-129Compile-time layout assertions on serialized structures | Put size, alignment, and offset assertions next to every FFI, on-disk, or serialized struct so cross-language drift breaks the build. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:341 Compile-time layout asserts: 148 |
FrankenSuite | none | not applicableNo binary layouts. |
| RP-130Methodology preservation: never squash the rigor away | Keep rulebooks, ownership tables, and review-attribution commits in the main history or an archived appendix; a squash merge that deletes them erases the method. | oven-sh/bunSource: synthesis/vendor-port-learnings.md:727 Methodology-preservation rule [Bun]. |
FrankenSuite | none | partialdone: the Rulebook versions its amendments, v11-manifest.md records the lineage, and main history is unsquashed; RULEBOOK.md:197; v11-manifest.md; missing: no merge policy prevents a squash merge |
| RP-131Unrepresentable before checked before tested | Where a property is statically decidable, make the bug impossible to express (sealed types, borrow rules) and pin it with compile-fail tests; use runtime checks returning Option or Result for geometry- or device-dependent facts. | NVlabs/cuda-oxide, NVlabs/cutile-rsSource: synthesis/vendor-port-learnings.md:466 The runtime-check vs compile-reject decision rule. |
FrankenSuite | none | not applicableNo typed code. |
| RP-132Preconditions validated once into a branded proof token | Validate every precondition once against live conditions and return a token the safe action requires; anything without the token goes through an explicitly unsafe path. | NVlabs/cuda-oxide, NVlabs/cutile-rsSource: synthesis/vendor-port-learnings.md:424 kernels without a contract expose only raw |
FrankenSuite | none | not applicableNo runtime actions. |
| RP-133Fail-closed verify barriers between pipeline stages | Verify between each stage of a multi-stage pipeline and attach the failing stage and operation to the error, so a failure is attributable to the stage that broke it. | NVlabs/cuda-oxide, Dicklesworthstone/franken_ocrSource: synthesis/vendor-port-learnings.md:452 fail-closed verify barriers between IR stages, with |
FrankenSuite | none | adoptedsite/scripts/verify-site.sh; site/BUILD-GATES.md; each gate prints PASS or FAIL with its reason and the script exits nonzero if any gate fails |
| RP-134Headline numbers carry their scope caveat | Never quote a headline number without the caveat its author attached (workload-specific, attributed cost only, input versus diff size); quote a disavowal when the source gives one. | github/copilot-agent-runtime, oven-sh/bunSource: synthesis/external-validation-2026-09.md:21 That is a workload-specific result; the Rust runtime is not universally |
FrankenSuite | none | partialdone: the Rulebook requires quoting a source disavowal and forbids ungated numbers; RULEBOOK.md:80; missing: no rule that every headline number carries its author workload or cost caveat |
| RP-135Managed breaking migration: named opt-in features and advance warning | Move deprecated stacks behind named opt-in features, label breaking changes per component, warn once per process before a default flips, and give a stable spelling for teams that pin the old stack. | awslabs/aws-sdk-rustSource: synthesis/external-validation-2026-09.md:113 a once-per-process warning before a default flip |
FrankenSuite | none | not applicableNo public API to migrate. |
| RP-136Workflow files linted against a declared execution policy | State in the repository which events may start CI and that every workflow must call the repository's own verify entry point; enforce both, and full-SHA action pins, with a checker that fails on any workflow file that breaks them. | Dicklesworthstone/frankengitSource: updates/frankengit-2026-09-24.md:214 must delegate to repository-owned ./scripts/verify.sh |
FrankenSuite | none | candidateOur workflows run on push, pull_request and schedule by design; the transferable part is a gate that lints .github/workflows/*.yml for SHA pins, least-privilege permissions and delegation to a repository script, which no gate does today. |