The agent stack: adopt, copy, or build

The FrankenSuite assessment covered one developer’s repositories. This layer looks outward, at 21 parts of the stack an agent product is assembled from, from inference engines to voice agents. For each area it answers one question: adopt an existing project, copy its practices, or build your own. Every verdict is a judgment backed by cited evidence, checked by someone other than its author, and reflects the evidence as of 2026-09-23. Verdicts were written and reviewed by separate AI agent sessions; a human maintainer coordinated the work and ruled on disputes. No human outside the project has reviewed them yet.

TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

The four verdicts

  • Adopt
    A maintained incumbent covers the need and building would recreate it.
  • Adopt and wrap
    Adopt the incumbent, but the evidence shows a verification gap the adopter must close (conformance, nondeterminism, safety, supply chain).
  • Build clean-room
    No incumbent meets a named hard constraint (license, determinism, safety boundary, runtime, footprint).
  • Watch
    Evidence is too thin, or the field moves faster than a commitment would survive (spec churn, no conformance suite).

The full rules, including the evidence bar for each verdict, are in stack/METHOD.md. Verdicts are decided in a fixed order: Watch first, then Adopt and wrap, then Adopt, and Build clean-room only when a cited line shows that no incumbent meets a hard constraint. Practices worth copying across all areas are collected in Rigor practices.

20 Adopt and wrap · 1 Watch · 0 Adopt · 0 Build clean-room

In every area with an established project, adopt it; in none of the 21 did the evidence justify building from scratch, and in 20 the evidence shows a verification gap you must close yourself. One area, Computer-use agents, is on Watch: the evidence is too thin to commit to a project yet.

Model serving

Embedding model serving

Adopt and wrap

Inference, medium confidence: adopt and wrap an existing embedding server or library rather than writing one. Serve with Text Embeddings Inference (TEI) or vLLM's pooling endpoints (infinity too, but it was last pushed in March 2026, so confirm it is maintained first); run in-process with sentence-transformers, fastembed or llama-cpp-python. Wrap it in two ways the serving projects themselves do not: pin the model's Hugging Face revision and weight hashes (none of the surveyed serving repos pin revisions), and add a one-sided parity test that fails only when quality drops below a pinned score, as vLLM does with one MTEB task.

Confidence Medium · Reviewed by ReviewServing on 2026-09-23

LLM inference engines

Adopt and wrap

Inference, medium confidence: adopt and wrap an existing engine (vLLM or SGLang for GPU serving, llama.cpp for local and CPU use) and do not write your own. They are maintained and already cover batching, KV-cache management (the per-request attention memory) and multi-vendor hardware. Wrap the one you pick: pin its commit and the model weight revision, check its outputs against a reference (llama.cpp perplexity or Hugging Face transformers), and measure speed on your own GPU, because the shared serving benchmark is a script the engine authors fork and tune, not a neutral referee.

Confidence Medium · Reviewed by ReviewServing on 2026-09-23

Model quantization toolkits

Adopt and wrap

Inference, low confidence: adopt and wrap the quantizer your runtime already ships (llama.cpp's quantize tool for GGUF files, bitsandbytes through Hugging Face, torchao inside PyTorch, TensorRT-LLM or vLLM kernels for GPU serving) rather than writing a quantization toolkit. Wrap it with your own quality check: measure perplexity and at least one downstream task on your model before and after, against a like-for-like baseline, because the evidence confirms the ecosystem's measuring tools exist but did not verify any of their numbers or any threshold for negligible loss. Confidence is low because the evidence pack sampled rather than read most CI and eval results.

Confidence Low · Reviewed by ReviewServing on 2026-09-23

Structured output and constrained decoding

Adopt and wrap

Inference, medium confidence: adopt and wrap an existing constrained-decoding engine (a library that masks the model's next-token choices so output must match a JSON Schema, regex or grammar) instead of writing one or prompting for JSON and hoping. Use xgrammar or another engine through your serving stack (vLLM ships adapters for four), outlines as a front end, or llama.cpp grammars for local models. Wrap it with two tests of your own: feed the engine's accepted outputs through an independent validator such as Pydantic, and put a compile-time ceiling on pathological schemas, because no project in the evidence runs a fuzzer that checks the mask against the schema.

Confidence Medium · Reviewed by ReviewServing on 2026-09-23

Orchestration

Agent frameworks

Adopt and wrap

Inference, medium confidence: adopt an existing agent framework (LangGraph, the OpenAI Agents SDK, smolagents or DSPy) and do not write your own agent loop. Wrap it, because an agent run is nondeterministic and no surveyed framework ships a record-and-replay harness for model calls: script the model and the tools in your tests, make a test fail when the script and the run disagree, and declare any model-graded ("judge") test matrix before you run it.

Confidence Medium · Reviewed by control-plane pane 2 on 2026-09-23

Agent-to-agent protocols (A2A class)

Adopt and wrap

Inference, low confidence: if your agents must discover and delegate to agents from other teams, target A2A v1.x through one of its official SDKs, and do not design your own protocol. Wrap it: pin the shared compatibility test kit (the TCK) and run it against your build, test every transport the spec requires, and call any interop result "self-interop" until you have tested against an implementation from outside the A2A organization, because the evidence found none. Confidence is low because the evidence pack itself calls the category thin.

Confidence Low · Reviewed by control-plane pane 2 on 2026-09-23

Durable workflow orchestrators

Adopt and wrap

Inference, low confidence: run long or crash-prone agent work on an existing durable-execution engine (Temporal, Cadence or Conductor), and do not write your own event log and replay engine. Wrap it, because none of these engines checks your workflow code for determinism when you compile it: determinism is only caught when a recorded run is replayed, so your test suite must replay recorded histories, including one deliberately nondeterministic history that must fail, under a virtual clock. Confidence is low because the evidence pack calls itself thin and ran none of these suites.

Confidence Low · Reviewed by control-plane pane 2 on 2026-09-23

MCP servers and clients

Adopt and wrap

Inference, medium confidence: build MCP (Model Context Protocol) servers and clients on an official SDK, and do not hand-write the protocol. Wrap the SDK in the official conformance suite, pinned to a version and run once per spec revision you ship, with every known failure listed in a named file instead of skipped; the suite is still an alpha release that nobody in this program has run, so its pass is a floor, not a certificate.

Confidence Medium · Reviewed by control-plane pane 2 on 2026-09-23

Tools and environment

Browser-use stacks

Adopt and wrap

Inference, medium confidence: adopt an existing agent-browser library or tool server (Playwright MCP, browser-use, Stagehand) and a maintained browser farm, and wrap them with your own flake and determinism controls. The incumbents already separate real-browser tests from unit tests and ship evaluation harnesses, but the evidence found no flake-rate budget and no recorded model responses for replay in any of them. Keep unit tests off the live web, pin the benchmark datasets you score against, treat page content as untrusted input before letting an agent act on it, and check the farm's license first: browserless is SSPL or commercial.

Confidence Medium · Reviewed by ReviewTools on 2026-09-23

Computer-use agents

Watch

Inference, low confidence: watch computer-use agents (software that operates a desktop from screenshots, keyboard and mouse) and do not commit a product to one agent stack yet. The agent stacks you would adopt are evidenced by activity and by benchmark scores their own READMEs report, not by test, CI or conformance evidence, and the category is mostly research code. If you experiment now, use the existing benchmark harness, grounding evaluation and VM infrastructure rather than building your own, and score the machine's end state, never the agent's own account of what it did.

Confidence Low · Reviewed by ReviewTools on 2026-09-23

Sandboxed code execution

Adopt and wrap

Inference, medium confidence: adopt a maintained isolation substrate (Firecracker microVMs, gVisor, or Kata) or an agent SDK that delegates isolation to a substrate (E2B on Firecracker microVMs; Cloudflare Sandbox SDK on Cloudflare's own edge substrate), and wrap it with isolation tests of your own. Writing your own isolation layer would recreate years of adversarial test work, but no repository in the evidence publishes a sandbox-escape test suite, and the agent-facing SDKs test their API layer while leaving isolation to the substrate. Before you claim "contained", write the threat model, pin the substrate and guest image by hash, and run a red-then-green confinement suite against that pin.

Confidence Medium · Reviewed by ReviewTools on 2026-09-23

Web search API layers

Adopt and wrap

Inference, medium confidence: adopt an existing web search and extraction layer, either Firecrawl self-hosted (AGPL-3.0) or a commercial API (Tavily, Exa, SerpApi) through its official SDK, and wrap it with your own quality and latency checks. The ranking machinery in this category is closed, and no repository in the evidence publishes a relevance benchmark or a latency budget, so "the search is good" is something you have to measure yourself. Keep a frozen, hashed set of recorded queries and responses as your test oracle, and keep live vendor API tests in a separate lane behind secrets rather than in the required build.

Confidence Medium · Reviewed by ReviewTools on 2026-09-23

Memory and retrieval

Agent memory systems

Adopt and wrap

Inference, low confidence: adopt and wrap a maintained memory layer (Mem0's open-source SDK, or Graphiti for a temporal knowledge graph) behind a thin interface of your own, rather than designing a new one. Do not rely on any published memory benchmark score: the scores are self-reported, sometimes from a managed platform or an ensemble of runs, and graded by LLM judges. Re-run LoCoMo or LongMemEval yourself with the dataset pinned by hash and the judge model pinned by name and date before you trust a number. Confidence is low because the evidence pack itself flags adoption evidence as thin.

Confidence Low · Reviewed by ReviewServing on 2026-09-23

Retrieval-augmented generation (RAG) frameworks

Adopt and wrap

Inference, low confidence: adopt and wrap an existing RAG framework (LlamaIndex, LangChain or Haystack; LightRAG if you want graph-based retrieval, with GraphRAG a watch-level option whose CI and tests were not read) for loading, chunking and retrieving documents, rather than writing one. Wrap it with an evaluation you own: a frozen corpus with expected answers, a pinned judge model for any LLM-graded metric, and recorded LLM calls so CI runs without a live model, because published quality thresholds in this space do not name the judge that produced them. Confidence is low because the evidence pack read most of these repos by file path, not source.

Confidence Low · Reviewed by ReviewServing on 2026-09-23

Vector databases

Adopt and wrap

Inference, medium confidence: adopt and wrap an existing vector store rather than writing an index. Use pgvector if you already run Postgres, an embedded library (Chroma) for single-process apps, or a Qdrant server at scale, with faiss as the exact-search reference; Milvus, Weaviate, LanceDB and Vespa are watch-level options whose CI and tests were not read. Wrap it with your own recall check against exact search on your data, including your metadata filters, before trusting any number: the public benchmark harness most server vendors submitted to is no longer maintained, and no dedicated filtered-recall suite was verified in any vendor repo (only Qdrant's filter-relevant tests were checked).

Confidence Medium · Reviewed by ReviewServing on 2026-09-23

Eval and safety

Agent observability and tracing

Adopt and wrap

Inference, medium confidence: record agent runs with an existing tracing stack (Langfuse, Arize Phoenix or Comet Opik, instrumented through OpenTelemetry packages such as OpenLLMetry or OpenLIT), and do not build your own trace store. Check the license first if you will host tracing for others: Phoenix is under the Elastic License 2.0, which forbids offering it as a hosted service, and Langfuse keeps its enterprise directories under a separate enterprise license. Wrap it, because "OpenTelemetry-compatible" is mostly asserted rather than tested in this category: before you trust an integration, check that the spans it emits carry the attributes you depend on, that streamed calls close their spans, and that your cost numbers use an audited price table.

Confidence Medium · Reviewed by control-plane pane 2 on 2026-09-23

Guardrails and red-teaming

Adopt and wrap

Inference, low confidence: put an existing rail toolkit or scanner library in front of your model and its tools (NeMo Guardrails, Guardrails AI, or LlamaFirewall for tool calls), attack it with an existing red-team tool (garak, PyRIT, promptfoo), and do not write your own. Wrap both, because no project in the evidence publishes a false-positive or false-negative threshold you can inherit: keep your own labeled corpus of prompts that must pass and prompts that must be blocked, calibrate the model you use as a judge on its own set, and treat any "it blocks jailbreaks" claim as unproven until those two receipts exist. Confidence is low because the evidence is file trees only: no test was run and no CI result was checked.

Confidence Low · Reviewed by control-plane pane 2 on 2026-09-23

LLM eval harnesses

Adopt and wrap

Inference, medium confidence: run your evals on an existing harness (Inspect AI for agent and tool-use evals, lm-evaluation-harness for academic benchmarks, the SWE-bench harness for coding agents), and do not write your own runner. Wrap it with tests of your own graders: SWE-bench is the only harness whose adversarial grader tests (spoofed passes, truncated output) were read, the others were not checked for them, and no harness showed a checked pattern for model-graded ("judge") verdicts, so a score is only as honest as the grader tests you add.

Confidence Medium · Reviewed by control-plane pane 2 on 2026-09-23

Training and voice

Fine-tuning infrastructure

Adopt and wrap

Inference, medium confidence: adopt an existing fine-tuning stack (TRL as the reference trainer, or a launcher such as Unsloth, LlamaFactory, Axolotl or LitGPT) and wrap it with correctness checks for your own recipes. Several maintained launchers cover supervised, adapter and preference training, so writing a trainer would recreate them; but the evidence shows no conformance oracle for preference and RL trainers beyond "it ran and the loss is finite", and exact-loss regression goldens appear only in a project that has since been retired. Pin versions, run a tiny smoke train on every change, commit expected loss values for your recipe, and evaluate after training.

Confidence Medium · Reviewed by ReviewTools on 2026-09-23

Realtime voice agents

Adopt and wrap

Inference, medium confidence: adopt an existing realtime voice-agent framework (Pipecat or LiveKit Agents, on a maintained media transport) and wrap it with a latency regression gate of your own. The frameworks already handle speech-to-text, language model and text-to-speech stages, turn-taking and interruptions, and LiveKit Agents tests them with fake providers and a virtual clock; but no latency regression gate was found in either framework, and interruption thresholds have no published ground truth. Keep the fake-provider test surface, measure time-to-first-audio on your own harness with A/A baselines, and block releases on regressions.

Confidence Medium · Reviewed by ReviewTools on 2026-09-23

RL environments and RLHF training

Adopt and wrap

Inference, low confidence: adopt the standard environment interfaces (Gymnasium, PettingZoo) and an existing RLHF trainer (TRL or verl), and wrap them with reward-hacking and determinism checks of your own. The environment standards ship conformance checkers and the trainers ship per-algorithm tests, so rebuilding either would recreate that work; but no repository in the evidence ships tests named as reward-hacking checks (where a policy scores well by gaming the reward instead of doing the task), the closest being reward-shaping and reward unit tests, and the companion itself marks that finding contested. Run the conformance checker on every environment you ship, assert same-seed determinism, test the reward math directly, and add a negative control showing a degenerate policy cannot inflate your reward.

Confidence Low · Reviewed by ReviewTools on 2026-09-23

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed