Bottom line
Inference, low confidence: adopt and wrap the quantizer your runtime already ships (llama.cpp's quantize tool for GGUF files, bitsandbytes through Hugging Face, torchao inside PyTorch, TensorRT-LLM or vLLM kernels for GPU serving) rather than writing a quantization toolkit. Wrap it with your own quality check: measure perplexity and at least one downstream task on your model before and after, against a like-for-like baseline, because the evidence confirms the ecosystem's measuring tools exist but did not verify any of their numbers or any threshold for negligible loss. Confidence is low because the evidence pack sampled rather than read most CI and eval results.
Adopt, do not rebuild
License checked on 2026-09-23; non-permissive licenses are named in the text.
- ggml-org/llama.cpp MIT: GGUF quantized formats with a quantize tool and a perplexity tool in the same repo; the portable path for local and CPU inference. [Verified] (ecosystem/pickup/_evidence/quantization.md:8
GGUF quant formats, tools/quantize/, tools/perplexity/
) - bitsandbytes-foundation/bitsandbytes MIT: 8-bit and 4-bit quantization used through Hugging Face transformers, with a benchmarking directory and 4-bit/8-bit layer tests. A fresh read on 2026-09-23 of https://raw.githubusercontent.com/bitsandbytes-foundation/bitsandbytes/main/.github/workflows/tests-nightly.yml showed a nightly cron (
15 2 * * *) over a CUDA matrix of T4, A10 and L40S GPUs. [Verified] (ecosystem/pickup/_evidence/quantization.md:98-bit (LLM.int8()) and 4-bit (NF4) quantization widely adopted via Hugging Face
) - pytorch/ao BSD-3-Clause (torchao): quantization inside PyTorch, including a prototype GPTQ implementation (
torchao/prototype/gptq/) and per-GPU regression workflows; a fresh read on 2026-09-23 of https://raw.githubusercontent.com/pytorch/ao/main/.github/workflows/1xH100_tests.yml showed it triggered onpushandpull_requestand run on an H100 runner. [Verified] (ecosystem/pickup/_evidence/quantization.md:11PyTorch-native quantization (torchao)
; ecosystem/pickup/_evidence/quantization.md:11torchao/prototype/gptq/ (GPTQ reference in framework)
) - NVIDIA/TensorRT-LLM Apache-2.0 AND LicenseRef-LTX-2-Community and vllm-project/vllm Apache-2.0: TensorRT-LLM is Apache-2.0 except
tensorrt_llm/_torch/visual_gen/models/ltx2/, which is under the LTX-2 Community License requiring entities with annual revenue of at least $10,000,000 to obtain a paid commercial license (read 2026-09-23 withgh api repos/NVIDIA/TensorRT-LLM/license; the API reports NOASSERTION). TensorRT-LLM's LTX-2 terms cover a visual-generation model directory (visual_gen/models/ltx2/), not its quantization recipes. Both are the GPU serving stacks that consume GPTQ, AWQ and FP8 kernels; quantize with the recipes the serving engine you run already supports. [Verified] (stack/licenses.tsv:69The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory
; stack/licenses.tsv:69Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license
; ecosystem/pickup/_evidence/quantization.md:12quantization recipes (INT4/FP8 GPTQ/AWQ) feeding production inference kernels
; ecosystem/pickup/_evidence/quantization.md:10serving framework consuming quantized kernels (GPTQ/AWQ/FP8 Marlin etc. backends)
) - microsoft/onnxruntime MIT: static and dynamic quantization tooling for ONNX models. [Verified] (ecosystem/pickup/_evidence/quantization.md:15
static+dynamic quantization tooling
)
Copy these practices
- Golden-output tests per kernel with a diff tolerance: AutoGPTQ
tests/test_q4.pyasserts each backend kernel's output against pinned reference tensors instead of checking that it runs. Starter kit: Incumbent oracle pinned before implementation. [Verified] (ecosystem/pickup/_evidence/quantization.md:22AutoGPTQ tests/test_q4.py hard-codes CUDA_OLD_REFERENCE tensors and asserts max-diff/mean-relative-diff per backend kernel
) - Benchmark real layer shapes, not toy shapes: AutoGPTQ
tests/bench_autoawq_autogptq.pytimes GPTQ vs AWQ kernels on Yi-34Bdown_projshapes. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/quantization.md:21benchmarks Yi-34B down_proj K=20480/N=7168 shapes
) - Report perplexity and a downstream task together: llama.cpp
tools/perplexity/plus an lm-evaluation-harness task. A fresh read on 2026-09-23 of https://raw.githubusercontent.com/ggml-org/llama.cpp/master/tools/perplexity/README.md showed the tool is meant to judge the quality loss of quantized models against FP16 and also computes KL divergence against the full-precision model's logits. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/quantization.md:23Copy: repo-local perplexity tool + lm-eval-harness adapter for downstream zero-shot tasks; report both.
) - Build smoke on every PR, GPU numerics nightly: this is the pack's advice; bitsandbytes is the verified example, splitting
tests-pr.ymlfromtests-nightly.yml, whose nightly cron15 2 * * *runs on T4, A10 and L40S GPUs (fresh read above). torchao is the counter-example: it runs its H100 tests on every push and PR. Starter kit: none. [Inference] [Verified] (ecosystem/pickup/_evidence/quantization.md:24heavy numeric/correctness tests nightly on self-hosted GPUs
; ecosystem/pickup/_evidence/quantization.md:24bitsandbytes: tests-pr.yml, tests-nightly.yml, test-runner.yml split by GPU availability
) - Pair every quantization script with a quality delta and a speedup: the pack recommends one command that quantizes, reports the perplexity delta and reports kernel speedup. No repo in the pack is shown doing all three: torchao's
benchmarks/benchmark_gptq.pycovers only the first step (a fresh read on 2026-09-23 of https://raw.githubusercontent.com/pytorch/ao/main/benchmarks/benchmark_gptq.py showed it timinggptq_quantizeontorch.randnmatrices and printing the average time, with no quality or speedup output). Starter kit: none. [Inference] [Verified] (ecosystem/pickup/_evidence/quantization.md:25each method = script that (a) quantizes, (b) reports perplexity delta, (c) reports kernel speedup, all one command
; ecosystem/pickup/_evidence/quantization.md:25times GPTQ quantize + profiles
) - Stamp the source weights' hash into the quantized artifact: franken_ocr's converter writes the source safetensors SHA-256 into the
.focrqheader and verifies part hashes on pull. The companion did not observe this in any quantization repo, and marks the practice as imported from FrankenSuite and contested. Starter kit: Receipts carry provenance; generation binding. [Verified] [Inference] (packets/franken_ocr-assessment.md:41already stamps the source safetensors SHA256 into the artifact header
; ecosystem/pickup/pickup-quantization.md:119T3 | Low | CONTESTED
; ecosystem/pickup/pickup-quantization.md:123CONTESTED because no quantization repo in the evidence set demonstrates origin
)
Build only if
- You need quantized artifacts that carry their own provenance (source weight hash, recipe, converter version) verifiable offline. No quantization repo in the pack was shown doing this, so build that thin artifact layer, not a new quantizer; even franken_ocr's version is not yet tamper-evident end to end. [Inference] (ecosystem/pickup/pickup-quantization.md:119
pattern imported from sibling packets, not yet observed in a quantization repo
; packets/franken_ocr-assessment.md:41the write-time origin binding exists but is not tamper-evident end to end
) - License is not a reason to build a quantizer: the adopted tools are MIT, Apache-2.0 or BSD-3-Clause, and TensorRT-LLM's only non-permissive terms apply to a visual-generation model directory (
visual_gen/models/ltx2/). [Verified] [Inference] (stack/licenses.tsv:30ggml-org/llama.cpp MIT permissive
; stack/licenses.tsv:8bitsandbytes-foundation/bitsandbytes MIT permissive
; stack/licenses.tsv:78pytorch/ao BSD-3-Clause permissive
; stack/licenses.tsv:57microsoft/onnxruntime MIT permissive
; stack/licenses.tsv:99vllm-project/vllm Apache-2.0 permissive
; stack/licenses.tsv:69The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory
; stack/licenses.tsv:69Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license
) - A new quantization method is research, not tooling; the category has converged on GPTQ/AWQ-class 4-bit weight formats, so a builder who needs a method should use one of those. [Inference] (ecosystem/pickup/pickup-quantization.md:116
The category has converged on GPTQ/AWQ-class 4-bit weight formats as the reference methods
)
Where FrankenSuite touches this
- franken_ocr: a code-enforced int8 recipe (a test asserts exactly 2,148 FFN/expert tensors quantized, attention and
lm_headkept high precision); TRL 6, NODUS ring Pilot. [Verified] [Inference] (packets/franken_ocr-assessment.md:77Conservative int8 recipe: exactly 2,148 FFN/expert tensors int8
; packets/franken_ocr-assessment.md:7748 attention projections + lm_head high precision
; packets/franken_ocr-assessment.md:11TRL 6. NODUS ring: Pilot.
) - franken_ocr's measured dispatch: hand-written SDOT/SMMLA int8 kernels ship but are not the default, because interleaved measurements showed the compiler-vectorized scalar loop faster; int4 remains aspirational and honestly labeled, though a wasm-int4 browser artifact is released. [Maintainer claim] [Verified] (packets/franken_ocr-assessment.md:31
deliberately doesn't use them by default
; packets/franken_ocr-assessment.md:31LLVM-autovectorized scalar loop beating forced SDOT
; packets/franken_ocr-assessment.md:86int4 support | Aspirational (honestly labeled)
; packets/franken_ocr-assessment.md:205wasm-int4 artifact released, playground committed
) - frankentorch: AVX-512 VNNI dynamic int8 CPU kernels with a dedicated int8 exact-parity CI workflow, which itself carries a literal GH_TOKEN hygiene finding; TRL 4, NODUS ring Explore. [Verified] [Inference] (packets/frankentorch-assessment.md:122
dynamic_int8_x86.rs (8 sites: AVX-512 VNNI int8 paths, the pin's headline commit)
; packets/frankentorch-assessment.md:41Dynamic int8 exact-parity CI
; packets/frankentorch-assessment.md:174A literal GH_TOKEN in the CI workflow.
; packets/frankentorch-assessment.md:23TRL 4 — see §4.9). Substantive-but-unproven is the textbook Explore case
) - franken_tts and franken_nlp: franken_tts has int8 tiers behind a default-route admission gate; franken_nlp plans staged, parity-gated quantization recipes in its
.fnlpqformat but has executed none (TRL 2-3). [Verified] [Maintainer claim] [Inference] (synthesis/briefs/franken_ocr.md:53franken_tts has int8 tiers and a default-route admission gate
; packets/franken_nlp-assessment.md:111staged immutable quantization recipes
; packets/franken_nlp-assessment.md:18TRL: 2–3; NODUS ring: Monitor
)
What we cannot say
- Whether any quality-loss number in these repos is right: the pack confirmed the tools exist but ran no eval and read no result tables (ecosystem/pickup/_evidence/quantization.md:32
Perplexity/accuracy numbers were sampled, not exhaustively verified
). - Whether the named CI workflows test accuracy: most bodies were not read, and the one that was read turned out to be a format check (ecosystem/pickup/_evidence/quantization.md:35
workflow file names are real, but I did not read full workflow bodies
; ecosystem/pickup/_evidence/quantization.md:35turned out to be a code-quality/format check, not accuracy tests
). The two fresh workflow reads above confirm schedule and hardware, not what the tests assert. - AutoGPTQ is cited only for its test patterns; its standalone package looks unmaintained (ecosystem/pickup/_evidence/quantization.md:33
suggests the standalone GPTQ packaging is in maintenance mode
). - AWQ's reference repo is not a process model (ecosystem/pickup/_evidence/quantization.md:31
the canonical AWQ repo has no .github/workflows (API 404), no tests/ dir
), and which AWQ implementation to trust as an oracle is open (ecosystem/pickup/pickup-quantization.md:354Which exact AWQ commit (if any) is a trustworthy algorithm oracle
). - How active the category is: llama.cpp, vLLM and onnxruntime push daily, but quantization is one feature of those projects (ecosystem/pickup/_evidence/quantization.md:34
they are broader projects where quantization is a feature, not the core
). - What loss counts as negligible (ecosystem/pickup/pickup-quantization.md:377-378
no evidence-defined negligibility threshold was found
).
Revisit when
- Someone publishes a reproduced, per-format quality-loss table (eval set, hardware, tolerance named) for a GPTQ/AWQ implementation that has CI.
- torchao's GPTQ path leaves
prototype/, or AutoGPTQ resumes or archives. - franken_ocr certifies or kills its int4 route, or makes
.focrqprovenance tamper-evident.
Sources
- Pickup companion: ecosystem/pickup/pickup-quantization.md, the charter seed, oracle candidates, and first claim registry for this area.
- Evidence pack: ecosystem/pickup/_evidence/quantization.md, the repositories checked against the GitHub API, with dates.
- Verdict source: stack/quantization.md, written under stack/METHOD.md.
Starting a build here?
Begin from the pickup companion for Model quantization toolkits: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.