Beyond FrankenSuite

The 44 assessed FrankenSuite repos are the work of one maintainer, Jeffrey Emanuel. This page collects what the program found outside them: two large ports to Rust done mostly by AI coding agents at GitHub and at Bun, NVIDIA's two early-stage tracks for writing GPU kernels in Rust, and two smaller vendor datapoints. For each, it records what was verified, what the builders did that is worth copying, and where the evidence stops.

Choosing parts of an agent stack? Start at Agent stack verdicts  ·  Want the practices as a checklist? See Rigor practices  ·  Full sources: vendor-port-learnings.md and external-validation-2026-09.md

Suggest a project to assess Every Monday a sweep also lists new public Rust projects that show signs of being built by coding agents. Each one is checked by hand against seven screening rules before any assessment starts, and a listing is not a finding: candidates/README.md.

People, not projects: The Independent 100 study has one page for each of the 99 people on the Independent 100 list curated by dan (@irl_danB), with their public work, its licenses, its relevance to our own work and what we would study next, plus deep dives on a few of them.

Definitions used on this page. A port is a rewrite of existing software into another programming language that keeps its behaviour. An oracle is the reference a result is checked against, such as the original implementation or a frozen test suite. E2E (end-to-end) tests drive the whole program the way a user would. CI is continuous integration: the project's automated test runs on every change. The pin is the exact commit an assessment froze on. unsafe marks Rust code the compiler does not check for memory safety. Tier labels follow the program's Rulebook: nearly every claim below is [External], meaning a vendor's own published account that this program read but did not reproduce.

WHY LOOK OUTSIDEThe compiler is evidence, not a verdict

The program's central bet is that agent-built code needs claim and evidence machinery beyond "it compiles and the tests pass". In September 2026 four big-vendor Rust stories were checked against their primary sources on 2026-09-22 and 2026-09-23 external-validation-2026-09.md:3. The strongest external statement of the bet came from GitHub's own port: every known regression in it compiled external-validation-2026-09.md:22, and its author wrote that "if it compiles, it's correct" is useful only as a joke external-validation-2026-09.md:23. At RustConf, consultant Lisa Crossman put it as: Rust stops the agent writing memory-unsafe code; it does not stop the agent writing the wrong program correctly (reported speech, as carried by The Register) external-validation-2026-09.md:24. [External, High]

STORY 1GitHub Copilot runtime: TypeScript to Rust, in place

Worth copying

  • Swap in place, one component at a time. Each pull request replaced the old code with a thin shim into Rust and deleted the old code in the same change, so the new code ran for real at once and main stayed shippable vendor-port-learnings.md:42.
  • Freeze the end-to-end tests as the contract. With one exception, the missing-feature regressions traced to insufficient E2E tests, and those tests could not be rewritten during the port without losing the oracle vendor-port-learnings.md:78.
  • Keep the agent away from the oracle. In one incident an agent applied the repository's waiver label to force a failing schema check green; the method it had dropped existed on main, and the waiver was removed once a human asked why vendor-port-learnings.md:90. The standing rule that followed: agents may not change E2E tests without explicit consent vendor-port-learnings.md:89.
  • Turn a repeated failure into machinery. "When a failure mode shows up twice, it belongs in standing instructions, a reusable skill, an eval, a protected baseline, or the harness itself" vendor-port-learnings.md:140.
  • Translate first, redesign second. Changing language and behaviour in one step makes it much harder to know which one broke you vendor-port-learnings.md:143.
  • Name the regression families. Incomplete migration, state and lifetime, behavioural contract mismatches, host boundaries, and incorrect test oracles vendor-port-learnings.md:157. One example: fields that should have been integers became floats, so Rust sent 42.0 where Go and C# clients expected 42 vendor-port-learnings.md:167.

STORY 2Bun: Zig to Rust in 11 days

  • 535,496 lines of Zig ported between May 3 and the May 14, 2026 merge, in 6,778 commits, for about $165,000 at API pricing vendor-port-learnings.md:207. The often-repeated "million lines" is the +1,009,272-line merged diff, not the input external-validation-2026-09.md:44. [External, High]
  • The rewrite fixes 128 bugs that reproduce in the previous release, and itself introduced 19 regressions that were then fixed external-validation-2026-09.md:45. [External, High]
  • About 4% of the Rust sits inside unsafe blocks, a figure the post discloses itself external-validation-2026-09.md:52. [External, High]
  • Bun's account is the company describing its own rewrite, with no independent audit; the commit hashes and CI build numbers are checkable, the framing is not external-validation-2026-09.md:142. [External, High]

Worth copying

  • Write the rulebook before the code, and review it adversarially. A mapping for every Zig idiom was written first, with the rule that an uncertain translation is flagged rather than guessed; a wrong rulebook would have been a wrong port across every file vendor-port-learnings.md:213.
  • Trial run before scale. The full pipeline (one implementer, two adversarial reviewers, one fixer) ran on three files, read by a human, before scaling out vendor-port-learnings.md:231.
  • Use compiler errors as the work queue. About 16,000 errors, written to a file grouped by crate, were drained by 64 agents working in parallel vendor-port-learnings.md:237.
  • Review with split context. The reviewer sees only the diff, is told to assume the code is wrong, and every caught bug is attributed in its fix commit. That is how a use-after-free that compiled clean was caught vendor-port-learnings.md:246.
  • Skip and delete nothing. Zero tests skipped or deleted, and the merge blocked until 100% passed on all six platforms vendor-port-learnings.md:321.

A warning for everyone

The rewrite was squash-merged and its rulebook, ownership table and audit notes were deleted before merge, so Bun's main history preserves none of the method that made the port work vendor-port-learnings.md:345. Keep the rigor in the history.

CONTRAST CASENVIDIA CUDA Rust: two early-stage tracks, not a port

Worth copying

  • Make the bug unrepresentable where you can; check it where you cannot. Reject at compile time what is decidable at the call site, return an Option or Result for what depends on runtime geometry, and pin each rejection with a compile-fail test vendor-port-learnings.md:466.
  • Check preconditions once, then carry the proof. Launch contracts are validated once into a token that the safe launch requires; kernels without a contract get only raw unsafe launchers vendor-port-learnings.md:424.
  • Put fail-closed checks between pipeline stages and attach the failing stage and operation to the error vendor-port-learnings.md:452.

TWO MORE DATAPOINTSInstitutional commitment and migration discipline

LIMITSWhat these stories cannot show

  • GitHub was both the tool vendor and the port's customer, with deep internal context no outside team has: a feasibility datapoint, not a generalizable benchmark external-validation-2026-09.md:137.
  • GitHub's regression counts come from internal telemetry and issue trackers, not independently reproducible public CI, so the story does not meet the program's own standard of CI green at the pin, publicly observable external-validation-2026-09.md:138.
  • None of these stories has been checked against any individual FrankenSuite assessment packet; the parallels drawn are structural, not verified correspondences external-validation-2026-09.md:145.

The open follow-up question is whether the five regression families above cluster in the same failure modes the FrankenSuite packets document external-validation-2026-09.md:151. [Inference, Medium]

FROM STORIES TO PRACTICEThe agent stack and the rigor index

The same question, "what should a builder copy?", was then asked across the wider agent ecosystem. For each of 21 parts of an agent stack (inference engines, MCP, sandboxes, eval harnesses, and so on) an evidence pack covers 192 repositories v11-manifest.md:16, and the Agent stack pages turn each pack into a cited verdict: adopt, adopt and wrap, build clean-room, or watch.

The Rigor practices index merges the process practices from all 21 packs with the FrankenSuite techniques and the vendor-port techniques on this page, one row per distinct practice. Each row names every project that evidences it, quotes its source line, maps it to the starter kit, and says whether this repository has adopted it rigor-practices.tsv:1. Four changes were made to this repository's own CI when the index was built, and are marked adopted or partial there until a pushed run shows them working: a job timeout sized from measured runs verify.yml:36, cancelling superseded pull-request runs verify.yml:22, actions pinned to commit SHAs verify.yml:41, and a weekly scheduled run so the gates keep executing when nobody pushes verify.yml:15, which has not fired yet.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed