← All agent-stack verdicts

Model quantization toolkits

Adopt and wrap Confidence Low Group Model serving Evidence as of 2026-09-23

Written by VerdictsServing · Reviewed by ReviewServing on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. NODUS is the four-ring verdict scale (Invest, Pilot, Explore, Monitor) used for the FrankenSuite assessments; the agent-stack verdicts use their own scale (Adopt, Adopt and wrap, Build clean-room, Watch). TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Bottom line

Inference, low confidence: adopt and wrap the quantizer your runtime already ships (llama.cpp's quantize tool for GGUF files, bitsandbytes through Hugging Face, torchao inside PyTorch, TensorRT-LLM or vLLM kernels for GPU serving) rather than writing a quantization toolkit. Wrap it with your own quality check: measure perplexity and at least one downstream task on your model before and after, against a like-for-like baseline, because the evidence confirms the ecosystem's measuring tools exist but did not verify any of their numbers or any threshold for negligible loss. Confidence is low because the evidence pack sampled rather than read most CI and eval results.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • ggml-org/llama.cpp MIT: GGUF quantized formats with a quantize tool and a perplexity tool in the same repo; the portable path for local and CPU inference. [Verified] (ecosystem/pickup/_evidence/quantization.md:8 GGUF quant formats, tools/quantize/, tools/perplexity/)
  • bitsandbytes-foundation/bitsandbytes MIT: 8-bit and 4-bit quantization used through Hugging Face transformers, with a benchmarking directory and 4-bit/8-bit layer tests. A fresh read on 2026-09-23 of https://raw.githubusercontent.com/bitsandbytes-foundation/bitsandbytes/main/.github/workflows/tests-nightly.yml showed a nightly cron (15 2 * * *) over a CUDA matrix of T4, A10 and L40S GPUs. [Verified] (ecosystem/pickup/_evidence/quantization.md:9 8-bit (LLM.int8()) and 4-bit (NF4) quantization widely adopted via Hugging Face)
  • pytorch/ao BSD-3-Clause (torchao): quantization inside PyTorch, including a prototype GPTQ implementation (torchao/prototype/gptq/) and per-GPU regression workflows; a fresh read on 2026-09-23 of https://raw.githubusercontent.com/pytorch/ao/main/.github/workflows/1xH100_tests.yml showed it triggered on push and pull_request and run on an H100 runner. [Verified] (ecosystem/pickup/_evidence/quantization.md:11 PyTorch-native quantization (torchao); ecosystem/pickup/_evidence/quantization.md:11 torchao/prototype/gptq/ (GPTQ reference in framework))
  • NVIDIA/TensorRT-LLM Apache-2.0 AND LicenseRef-LTX-2-Community and vllm-project/vllm Apache-2.0: TensorRT-LLM is Apache-2.0 except tensorrt_llm/_torch/visual_gen/models/ltx2/, which is under the LTX-2 Community License requiring entities with annual revenue of at least $10,000,000 to obtain a paid commercial license (read 2026-09-23 with gh api repos/NVIDIA/TensorRT-LLM/license; the API reports NOASSERTION). TensorRT-LLM's LTX-2 terms cover a visual-generation model directory (visual_gen/models/ltx2/), not its quantization recipes. Both are the GPU serving stacks that consume GPTQ, AWQ and FP8 kernels; quantize with the recipes the serving engine you run already supports. [Verified] (stack/licenses.tsv:69 The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory; stack/licenses.tsv:69 Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license; ecosystem/pickup/_evidence/quantization.md:12 quantization recipes (INT4/FP8 GPTQ/AWQ) feeding production inference kernels; ecosystem/pickup/_evidence/quantization.md:10 serving framework consuming quantized kernels (GPTQ/AWQ/FP8 Marlin etc. backends))
  • microsoft/onnxruntime MIT: static and dynamic quantization tooling for ONNX models. [Verified] (ecosystem/pickup/_evidence/quantization.md:15 static+dynamic quantization tooling)

Copy these practices

Build only if

  • You need quantized artifacts that carry their own provenance (source weight hash, recipe, converter version) verifiable offline. No quantization repo in the pack was shown doing this, so build that thin artifact layer, not a new quantizer; even franken_ocr's version is not yet tamper-evident end to end. [Inference] (ecosystem/pickup/pickup-quantization.md:119 pattern imported from sibling packets, not yet observed in a quantization repo; packets/franken_ocr-assessment.md:41 the write-time origin binding exists but is not tamper-evident end to end)
  • License is not a reason to build a quantizer: the adopted tools are MIT, Apache-2.0 or BSD-3-Clause, and TensorRT-LLM's only non-permissive terms apply to a visual-generation model directory (visual_gen/models/ltx2/). [Verified] [Inference] (stack/licenses.tsv:30 ggml-org/llama.cpp MIT permissive; stack/licenses.tsv:8 bitsandbytes-foundation/bitsandbytes MIT permissive; stack/licenses.tsv:78 pytorch/ao BSD-3-Clause permissive; stack/licenses.tsv:57 microsoft/onnxruntime MIT permissive; stack/licenses.tsv:99 vllm-project/vllm Apache-2.0 permissive; stack/licenses.tsv:69 The LTX-2 Community License Agreement applies only to use of the code in the tensorrt_llm/_torch/visual_gen/models/ltx2/ directory; stack/licenses.tsv:69 Entities with annual revenues of at least $10,000,000 ... are required to obtain a paid commercial use license)
  • A new quantization method is research, not tooling; the category has converged on GPTQ/AWQ-class 4-bit weight formats, so a builder who needs a method should use one of those. [Inference] (ecosystem/pickup/pickup-quantization.md:116 The category has converged on GPTQ/AWQ-class 4-bit weight formats as the reference methods)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • Someone publishes a reproduced, per-format quality-loss table (eval set, hardware, tolerance named) for a GPTQ/AWQ implementation that has CI.
  • torchao's GPTQ path leaves prototype/, or AutoGPTQ resumes or archives.
  • franken_ocr certifies or kills its int4 route, or makes .focrq provenance tamper-evident.

Sources

Starting a build here?

Begin from the pickup companion for Model quantization toolkits: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed