← All agent-stack verdicts

LLM inference engines

Adopt and wrap Confidence Medium Group Model serving Evidence as of 2026-09-23

Written by VerdictsServing · Reviewed by ReviewServing on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. NODUS is the four-ring verdict scale (Invest, Pilot, Explore, Monitor) used for the FrankenSuite assessments; the agent-stack verdicts use their own scale (Adopt, Adopt and wrap, Build clean-room, Watch). TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Bottom line

Inference, medium confidence: adopt and wrap an existing engine (vLLM or SGLang for GPU serving, llama.cpp for local and CPU use) and do not write your own. They are maintained and already cover batching, KV-cache management (the per-request attention memory) and multi-vendor hardware. Wrap the one you pick: pin its commit and the model weight revision, check its outputs against a reference (llama.cpp perplexity or Hugging Face transformers), and measure speed on your own GPU, because the shared serving benchmark is a script the engine authors fork and tune, not a neutral referee.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

Copy these practices

Build only if

  • You need a memory-safe runtime for one fixed model with no ML-framework dependency and no GPU, and you accept a six-figure line count to get it: franken_ocr shipped that shape. [Verified] (packets/franken_ocr-assessment.md:72 Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR, no ML framework; packets/franken_ocr-assessment.md:72 105,330 Rust lines; no build.rs)
  • The one non-permissive license among the adopted engines, TensorRT-LLM's LTX-2 portion, covers a visual-generation model directory (visual_gen/models/ltx2/), not the serving path, so it does not justify building a serving engine; skip that directory or buy the commercial license. The other adopted engines are Apache-2.0 or MIT. [Verified] [Inference] (stack/licenses.tsv:69 The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory; stack/licenses.tsv:69 Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license; stack/licenses.tsv:99 vllm-project/vllm Apache-2.0 permissive; stack/licenses.tsv:85 sgl-project/sglang Apache-2.0 permissive; stack/licenses.tsv:30 ggml-org/llama.cpp MIT permissive; stack/licenses.tsv:40 InternLM/lmdeploy Apache-2.0 permissive)
  • Price in a moving target even then: franken_nlp's honest baseline is upstream llama.cpp, which improves while the rewrite has not executed a model. [Inference] (synthesis/briefs/franken_nlp.md:131 upstream llama.cpp improves monthly; packets/franken_nlp-assessment.md:20 nothing executes end to end)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • A maintained serving benchmark not owned by any engine appears with pinned configs and datasets; the wrap could then stop owning its own speed measurement.
  • llama.cpp re-enables its disabled bench.yml, or TGI either resumes pushes or is archived.
  • A FrankenSuite single-model runtime gains independent validation or a second maintainer, which would change the build-only-if case.

Sources

Starting a build here?

Begin from the pickup companion for LLM inference engines: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed