← All agent-stack verdicts

Fine-tuning infrastructure

Adopt and wrap Confidence Medium Group Training and voice Evidence as of 2026-09-23

Written by VerdictsTools · Reviewed by ReviewTools on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked. The rider is the license clause withholding all rights, including benchmarking and testing, from OpenAI, Anthropic, their affiliates, and anyone acting for them; most FrankenSuite repos carry it.

Bottom line

Inference, medium confidence: adopt an existing fine-tuning stack (TRL as the reference trainer, or a launcher such as Unsloth, LlamaFactory, Axolotl or LitGPT) and wrap it with correctness checks for your own recipes. Several maintained launchers cover supervised, adapter and preference training, so writing a trainer would recreate them; but the evidence shows no conformance oracle for preference and RL trainers beyond "it ran and the loss is finite", and exact-loss regression goldens appear only in a project that has since been retired. Pin versions, run a tiny smoke train on every change, commit expected loss values for your recipe, and evaluate after training.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • huggingface/trl Apache-2.0: the post-training library (SFT, DPO, GRPO and reward trainers) that other stacks test against; building a trainer would recreate it. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:9 HF's canonical post-training library (SFTTrainer, DPOTrainer, GRPO, reward trainers); ecosystem/pickup/_evidence/fine-tuning.md:9 TRL is the reference the others test against)
  • unslothai/unsloth Apache-2.0 AND AGPL-3.0: LoRA, QLoRA and GRPO kernels with CI at scale, including free-GPU notebook regression legs. License: Apache-2.0 for the core package, but AGPL-3.0 for the Unsloth Studio UI and for the MoE kernels under unsloth/kernels/moe/; running a modified copy of those parts as a network service, or distributing them, triggers AGPL source obligations. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:7 LoRA/QLoRA/GRPO kernels); ~50 workflow files show CI at industrial scale; stack/licenses.tsv:96 unsloth/kernels/moe/LICENSE (MoE grouped-GEMM kernels inside the core package tree) is also AGPL-3.0)
  • hiyouga/LlamaFactory Apache-2.0: a CLI, GUI and YAML launcher tested on a three-OS, three-Python CPU matrix plus a self-hosted GPU runner. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:8 CLI/GUI/data-driven YAML launcher; ecosystem/pickup/_evidence/fine-tuning.md:23 3 OS × 3 Python × transformers-compat matrix on CPU)
  • axolotl-ai-cloud/axolotl Apache-2.0 AND LicenseRef-Axolotl-Community: a YAML-config launcher for SFT, DPO, LoRA and QLoRA with DeepSpeed and FSDP, CPU tests on every PR and GPU end-to-end tests behind a label. License: Apache-2.0 at the root, but src/axolotl/integrations/ is under the Axolotl Community License, which bars offering fine-tuning to third parties as a service. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:11 YAML-config SFT/DPO/LoRA/QLoRA launcher with DeepSpeed/FSDP; ecosystem/pickup/_evidence/fine-tuning.md:21 GPU e2e in Docker, gated behind run-gpu-tests PR label; stack/licenses.tsv:7 shall not use the Software for any activity that constitutes a commercial activity of offering for free or for sale any services)
  • Lightning-AI/litgpt Apache-2.0: hackable pretrain, finetune and eval code whose CPU tests use synthetic micro-configs and download no checkpoint. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:10 Hackable pretrain/finetune/eval (LoRA, adapters, full); ecosystem/pickup/_evidence/fine-tuning.md:40 synthetic micro-configs, no checkpoint download at all)

Copy these practices

  • A tiny smoke train on every change: validate the config, load data, train for five steps on a 135M-parameter checkpoint, and assert the checkpoint was written. The pack read one such Axolotl file; the per-family repeats are known by file name only. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:37 full validate_config → load_datasets → train round-trip on HuggingFaceTB/SmolLM2-135M with max_steps: 5; ecosystem/pickup/_evidence/fine-tuning.md:75 did not read every e2e test)
  • Exact expected losses at a fixed seed and dtype: commit per-step loss values for a tiny recipe and fail on any deviation; torchtune's version is the evidenced example, from a project no longer maintained. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:46 seed=9, dtype=fp32 overrides; compile=True/False both run)
  • Pinned versus latest, side by side: Unsloth runs a pinned small SFT leg next to one on the latest libraries, so an upstream break shows up as a divergence. Starter kit: Incumbent oracle pinned before implementation. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:60 canary-vs-control legs (tiny Qwen2.5-0.5B SFT pinned vs latest libs))
  • Retry only named transient failures: TRL reruns a failing test up to five times only when its message matches OSError, Timeout, HTTP 502/504 or out of memory; other failures block merges. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:61 --reruns 5 only for OSError/Timeout/HTTP 502/504/OOM messages)
  • Config validation as an API, exercised first: the one Axolotl end-to-end test file the pack read calls the config validator and normalizer before training; the pack did not read the others. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:52 calls validate_config(cfg) + normalize_config(cfg) before training; ecosystem/pickup/_evidence/fine-tuning.md:75 did not read every e2e test)
  • Evaluate or generate after training: assert the checkpoint actually materializes, then run an eval or generation round-trip. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:67 check_model_output_exists verifies checkpoints actually materialize)
  • Distributed logic tested without GPUs: Axolotl runs its DeepSpeed distributed tests on CPU; this covers logic, not numerics. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/fine-tuning.md:58 runs DeepSpeed distributed tests on CPU via)
  • A speed win needs the incumbent run side by side: franken_whisper's ledger counts a before/after comparison against yourself as maintenance, and a win only when the incumbent runs in the same invocation with its binary hash recorded; apply it to any launcher speed claim. Starter kit: Incumbent oracle pinned before implementation. [Verified] (packets/franken_whisper-assessment.md:114 requires the actual legacy incumbent running side-by-side in the same invocation, with the incumbent binary's SHA-256 recorded)

Build only if

  • Your deployment cannot carry a Python runtime. The FrankenSuite's Rust ports target that audience but are framework layers at TRL 4, not fine-tuning launchers, so even this constraint points to building on a framework, not to a new launcher. [Inference] (packets/frankenjax-assessment.md:63 Rust developers who need AD without a Python runtime; synthesis/00-overview.md:103 frankentorch | 4 | Explore | Rider)
  • The preference and RL correctness gap is a reason to write a differential test harness around an incumbent, not to write a trainer. [Inference] (ecosystem/pickup/pickup-fine-tuning.md:352 No verified conformance oracle for DPO/GRPO correctness beyond)
  • License does not justify building. The root-level API read (2026-09-23, gh api repos/<owner>/<repo> --jq .license.spdx_id) returned Apache-2.0 for all five, but the tree-level census found non-permissive parts: Unsloth's Studio UI and MoE kernels are AGPL-3.0, and Axolotl's integrations bar fine-tuning offered as a service. A fine-tuning-as-a-service product or an AGPL-averse one has a real constraint on those two parts; TRL, LlamaFactory and LitGPT read Apache-2.0 with no conditions noted, so the constraint points to choosing among incumbents, not to a new trainer. [Inference] (stack/licenses.tsv:96 unsloth/kernels/moe/LICENSE (MoE grouped-GEMM kernels inside the core package tree) is also AGPL-3.0; stack/licenses.tsv:7 src/axolotl/integrations/ is under the Axolotl Community License; stack/METHOD.md:79 Check the license of everything you tell a builder to adopt)

Where FrankenSuite touches this

  • frankentorch reimplements PyTorch's CPU eager semantics around a Deterministic Autograd Contract (replayable, seeded, provenance-logged gradients) and has an optimizer crate; the companion scopes general training frameworks out of this type, and the packet's reading of it as a reproducible-training base is inference. TRL 4, Explore; license: MIT with a rider barring OpenAI, Anthropic, their affiliates and anyone acting for them. [Verified] (packets/frankentorch-assessment.md:20 organized around the Deterministic Autograd Contract: replayable, seeded, provenance-logged gradient computation; packets/frankentorch-assessment.md:126 optimizers (SGD/Adam/AdamW per README; packets/frankentorch-assessment.md:299 is the reproducible-training workload: an agent team fine-tuning models; ecosystem/pickup/pickup-fine-tuning.md:8 NOT a general ML training framework (PyTorch, JAX); packets/frankentorch-assessment.md:102 License rider bars OpenAI, Anthropic, affiliates, and anyone acting for them; synthesis/00-overview.md:103 frankentorch | 4 | Explore | Rider)
  • frankenjax reimplements JAX's transform semantics and automatic differentiation with a differential conformance harness against JAX fixtures, not the ML framework around them. TRL 4, Explore; license: MIT with a rider barring OpenAI, Anthropic and anyone acting for them. [Verified] (packets/frankenjax-assessment.md:19 not JAX the ML framework, but its mathematical core; packets/frankenjax-assessment.md:21 the license rider bars OpenAI/Anthropic and their agents from even; synthesis/00-overview.md:90 frankenjax | 4 | Explore | Rider)

What we cannot say

Revisit when

  • A published conformance suite for DPO, GRPO or PPO trainers appears, or TRL adopts exact-loss regression tests.
  • An incumbent launcher loses maintenance the way torchtune did.
  • frankentorch ships a release and gains an outside consumer that fine-tunes on it.

Sources

Starting a build here?

Begin from the pickup companion for Fine-tuning infrastructure: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed