Bottom line
Inference, medium confidence: adopt and wrap an existing engine (vLLM or SGLang for GPU serving, llama.cpp for local and CPU use) and do not write your own. They are maintained and already cover batching, KV-cache management (the per-request attention memory) and multi-vendor hardware. Wrap the one you pick: pin its commit and the model weight revision, check its outputs against a reference (llama.cpp perplexity or Hugging Face transformers), and measure speed on your own GPU, because the shared serving benchmark is a script the engine authors fork and tune, not a neutral referee.
Adopt, do not rebuild
License checked on 2026-09-23; non-permissive licenses are named in the text.
- vllm-project/vllm Apache-2.0: the serving engine whose
benchmark_serving.pySGLang, LMDeploy and TGI mirror with serving benchmarks of their own; rebuilding its scheduler and KV-cache manager recreates the category's reference point. That others clone it is the pack author's characterization. [Verified] [Inference] (ecosystem/pickup/_evidence/inference-engines.md:7A high-throughput and memory-efficient inference and serving engine for LLMs
; ecosystem/pickup/_evidence/inference-engines.md:18Standardized serving benchmark harness
; ecosystem/pickup/_evidence/inference-engines.md:7is the de-facto throughput benchmark other projects clone
) - sgl-project/sglang Apache-2.0: vLLM-class serving framework with scheduled GPU CI per vendor; the NVIDIA workflow named nightly runs every second day. A fresh read on 2026-09-23 of https://raw.githubusercontent.com/sgl-project/sglang/main/.github/workflows/nightly-test-nvidia.yml showed a
cron: '0 14 */2 * *'schedule on a4-gpu-h100runner. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:8per-vendor nightly GPU fleets
; ecosystem/pickup/_evidence/inference-engines.md:21sgl-project/sglang/.github/workflows/nightly-test-nvidia.yml
) - ggml-org/llama.cpp MIT: C/C++ engine for local, CPU, Metal and Vulkan inference with a cross-platform build matrix; ships in-repo perplexity and llama-bench tools, which the pack author calls the baseline serving engines measure against (the companion rates the related claim contested). Its own process evidence in the pack is file names, plus byte-exact grammar tests read for the structured-output pack. [Verified] [Inference] (ecosystem/pickup/_evidence/inference-engines.md:26
Benchmark harness shipped inside the library, not just as scripts.
; ecosystem/pickup/_evidence/inference-engines.md:27Cross-platform backend build matrix.
; ecosystem/pickup/_evidence/inference-engines.md:12define the correctness/perf baseline the serving engines measure against
; ecosystem/pickup/pickup-inference-engines.md:127A field reference engine can carry the category's perf baseline without any enabled automated perf gate
; ecosystem/pickup/_evidence/structured-output.md:17asserts byte-exact expected grammar text for given JSON schemas
) - NVIDIA/TensorRT-LLM Apache-2.0 AND LicenseRef-LTX-2-Community (license: Apache-2.0 for the project, but the code in
tensorrt_llm/_torch/visual_gen/models/ltx2/is under the LTX-2 Community License, which requires entities with annual revenue of at least $10,000,000 to obtain a paid commercial license; read 2026-09-23 withgh api repos/NVIDIA/TensorRT-LLM/license, API spdx NOASSERTION): NVIDIA's vendor-tuned stack with its benchmark harness inside the library; use its numbers as a vendor reference, never as independent verification. [Verified] (stack/licenses.tsv:69The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory
; stack/licenses.tsv:69Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license
; ecosystem/pickup/_evidence/inference-engines.md:9the vendor-tuned reference implementation for this category
; ecosystem/pickup/pickup-inference-engines.md:226reference, never as independent verification of a third-party engine
) - InternLM/lmdeploy Apache-2.0: compress-deploy-serve toolkit with daily end-to-end tests pinned to named GPUs; evidence is workflow file names only, bodies not read. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:11
active daily E2E test matrix pinned to GPU SKUs
)
Copy these practices
- Unit-test the benchmark harness: vLLM tests its own benchmark CLIs (
tests/benchmarks/test_throughput_cli.py,test_latency_cli.py,test_serve_cli.py), so a broken benchmark fails CI instead of emitting numbers. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:19Benchmarks are unit-tested like code.
; ecosystem/pickup/_evidence/inference-engines.md:19vllm-project/vllm/tests/benchmarks/test_throughput_cli.py
) - Keep performance CI separate from correctness CI: vLLM's
.buildkite/performance-benchmarks/track; SGLang'spr-benchmark-rust.ymlandstress-test.yml. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:20Dedicated perf-regression CI track, separate from correctness CI.
) - Differential correctness against a pinned reference: vLLM
tests/basic_correctness/test_basic_correctness.py; a fresh read on 2026-09-23 of https://raw.githubusercontent.com/vllm-project/vllm/main/tests/basic_correctness/test_basic_correctness.py showed it running the same prompts throughhf_runnerand vLLM and comparing them withcheck_outputs_equal. Starter kit: Incumbent oracle pinned before implementation. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:22Correctness via differential tests against HF transformers
) - Accuracy gate with committed configs: vLLM
.buildkite/lm-eval-harness/configs/with a versioned baseline script; SGLangtest/lm_eval_configs/*.yaml. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:23First-party accuracy/eval harness wired to lm-eval-harness.
) - Pin the vendored core in a file: Ollama's
LLAMA_CPP_VERSIONandMLX_VERSION; TGI'srust-toolchain.tomlandcrate-hashes.json. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:29Dependency pinning for vendored inference cores.
) - Record precision baselines per quantized format: SGLang
python/sglang/test/precision_baseline_store.pyandtest_deterministic.py. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/inference-engines.md:25Recorded precision/quantization baselines.
; ecosystem/pickup/pickup-inference-engines.md:121per-format reference values, determinism tests
)
Build only if
- You need a memory-safe runtime for one fixed model with no ML-framework dependency and no GPU, and you accept a six-figure line count to get it: franken_ocr shipped that shape. [Verified] (packets/franken_ocr-assessment.md:72
Pure-Rust, CPU-only OCR engine for Baidu Unlimited-OCR, no ML framework
; packets/franken_ocr-assessment.md:72105,330 Rust lines; no build.rs
) - The one non-permissive license among the adopted engines, TensorRT-LLM's LTX-2 portion, covers a visual-generation model directory (
visual_gen/models/ltx2/), not the serving path, so it does not justify building a serving engine; skip that directory or buy the commercial license. The other adopted engines are Apache-2.0 or MIT. [Verified] [Inference] (stack/licenses.tsv:69The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory
; stack/licenses.tsv:69Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license
; stack/licenses.tsv:99vllm-project/vllm Apache-2.0 permissive
; stack/licenses.tsv:85sgl-project/sglang Apache-2.0 permissive
; stack/licenses.tsv:30ggml-org/llama.cpp MIT permissive
; stack/licenses.tsv:40InternLM/lmdeploy Apache-2.0 permissive
) - Price in a moving target even then: franken_nlp's honest baseline is upstream llama.cpp, which improves while the rewrite has not executed a model. [Inference] (synthesis/briefs/franken_nlp.md:131
upstream llama.cpp improves monthly
; packets/franken_nlp-assessment.md:20nothing executes end to end
)
Where FrankenSuite touches this
- franken_ocr: pure-Rust CPU inference for a 3B OCR vision-language model plus a five-model zoo, with int8 kernels checked against bit-identical scalar oracles; TRL 6, NODUS ring Pilot; its CI workflows were deleted from the tree at HEAD. [Verified] [Inference] (packets/franken_ocr-assessment.md:9
a DeepSeek-OCR-derived 3B MoE VLM
; packets/franken_ocr-assessment.md:9a five-model zoo (Unlimited-OCR, GOT-OCR2, SmolVLM2, OneChart, Polyphonic-TrOMR)
; packets/franken_ocr-assessment.md:9model-specific int8 kernels with bit-identical scalar oracles
; packets/franken_ocr-assessment.md:11TRL 6. NODUS ring: Pilot.
; packets/franken_ocr-assessment.md:11the CI workflows were deleted from the tree at HEAD
) - franken_tts: shipped pure-Rust CPU runtime for one text-to-speech model with 11 releases; TRL 7. [Verified] [Inference] (packets/franken_tts-assessment.md:9
pure-Rust CPU-only runtime for one model
; packets/franken_tts-assessment.md:911 GitHub releases (v0.1.0–v0.1.10)
; packets/franken_tts-assessment.md:181Technology readiness (TRL 1–9) | 7 | Real releases
) - franken_whisper: wraps whisper.cpp-class speech backends and ships an in-process Rust engine on FrankenTorch CPU kernels as the default in their place, plus a Rust Sortformer speaker-diarization port; TRL 5-6. [Verified] [Inference] (packets/franken_whisper-assessment.md:9
replaced the backends it wraps
; packets/franken_whisper-assessment.md:9that ships as the default
; packets/franken_whisper-assessment.md:9plus a Rust-native Sortformer 4-lane speaker diarization port
; packets/franken_whisper-assessment.md:9encoder/decoder transformer on FrankenTorch CPU kernels
; packets/franken_whisper-assessment.md:187Technology readiness (TRL 1–9) | 5–6 | Real engine, real releases
) - franken_nlp: plan-stage single-model engine that names upstream llama.cpp as its honest baseline; TRL 2-3, NODUS ring Monitor. [Verified] [Inference] (packets/franken_nlp-assessment.md:86
Official upstream llama.cpp supports Nanbeige4.2 (the honest baseline)
; packets/franken_nlp-assessment.md:18TRL: 2–3; NODUS ring: Monitor
) - frankentorch: Rust reimplementation of PyTorch CPU eager-mode semantics whose kernels franken_ocr, franken_whisper and frankensearch's reranker consume; TRL 4, NODUS ring Explore; CI red at the pin. [Verified] [Inference] (packets/frankentorch-assessment.md:20
A from-scratch Rust reimplementation of PyTorch's CPU eager-mode semantics
; packets/franken_ocr-assessment.md:72the sibling frankentorch kernel crate (path dep)
; packets/frankensearch-assessment.md:119pure-Rust frankentorch int8 BERT by default
; packets/franken_whisper-assessment.md:9encoder/decoder transformer on FrankenTorch CPU kernels
; packets/frankentorch-assessment.md:23TRL 4 — see §4.9). Substantive-but-unproven is the textbook Explore case
; packets/frankentorch-assessment.md:231CI red at the pin — the G1..G8 gate topology failed
) - The four runtimes above are fixed-model runtimes (each hard-codes the models it runs), not general multi-tenant serving engines; franken_ocr's own packet bounds its market to no-GPU OCR, so none contests the vLLM/SGLang batching lane. [Inference] (packets/franken_ocr-assessment.md:192
The engine's market is bounded (no-GPU OCR)
)
What we cannot say
- No engine was run and no throughput or latency number was reproduced for this verdict; the pack's star counts and push dates are point-in-time API values (ecosystem/pickup/_evidence/inference-engines.md:38
are point-in-time values from the GitHub API on 2026-09-23, not live guarantees
). They measure activity, not quality. - The shared serving benchmark is a convention engine authors tune, not an oracle (ecosystem/pickup/pickup-inference-engines.md:86-87
it is a convention, not an independent oracle — engine authors fork and tune it
). - llama.cpp has no enabled perf gate (ecosystem/pickup/_evidence/inference-engines.md:35
llama.cpp has no enabled benchmark CI.
), so its speed is measured in-repo without automated regression blocking. - Whether TensorRT-LLM's CI is green: it runs mostly in Jenkins (ecosystem/pickup/pickup-inference-engines.md:367
TensorRT-LLM CI greenness is not GitHub-verifiable
). - Whether TGI is still a live option, since the TGI pinning files cited above sit in a repo pushed last on 2026-03-21 (ecosystem/pickup/_evidence/inference-engines.md:34
TGI is going stale. Last push 2026-03-21
). - Ollama is cited only for its pin files; it is not a serving engine (ecosystem/pickup/_evidence/inference-engines.md:37
Ollama is a distribution layer, not a batching/serving engine.
). - Phones, browsers and ML-compilation deployment are not covered: mlc-llm, the pack's cross-device entry, has the thinnest evidence (ecosystem/pickup/_evidence/inference-engines.md:36
mlc-llm has the thinnest benchmark evidence.
).
Revisit when
- A maintained serving benchmark not owned by any engine appears with pinned configs and datasets; the wrap could then stop owning its own speed measurement.
- llama.cpp re-enables its disabled
bench.yml, or TGI either resumes pushes or is archived. - A FrankenSuite single-model runtime gains independent validation or a second maintainer, which would change the build-only-if case.
Sources
- Pickup companion: ecosystem/pickup/pickup-inference-engines.md, the charter seed, oracle candidates, and first claim registry for this area.
- Evidence pack: ecosystem/pickup/_evidence/inference-engines.md, the repositories checked against the GitHub API, with dates.
- Verdict source: stack/inference-engines.md, written under stack/METHOD.md.
Starting a build here?
Begin from the pickup companion for LLM inference engines: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.