← All agent-stack verdicts

LLM eval harnesses

Adopt and wrap Confidence Medium Group Eval and safety Evidence as of 2026-09-23

Written by VerdictsOrchestration · Reviewed by control-plane pane 2 on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. NODUS is the four-ring verdict scale (Invest, Pilot, Explore, Monitor) used for the FrankenSuite assessments; the agent-stack verdicts use their own scale (Adopt, Adopt and wrap, Build clean-room, Watch). TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Bottom line

Inference, medium confidence: run your evals on an existing harness (Inspect AI for agent and tool-use evals, lm-evaluation-harness for academic benchmarks, the SWE-bench harness for coding agents), and do not write your own runner. Wrap it with tests of your own graders: SWE-bench is the only harness whose adversarial grader tests (spoofed passes, truncated output) were read, the others were not checked for them, and no harness showed a checked pattern for model-graded ("judge") verdicts, so a score is only as honest as the grader tests you add.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

Copy these practices

Build only if

  • No evidenced constraint rules out the incumbents. The evidenced gaps are grader tests and a judge-honesty discipline (recording the judge's identity and bounding its disagreement with a deterministic checker), which are wrappers you add around an adopted harness, not reasons to write a new one. [Inference] (ecosystem/pickup/pickup-eval-harnesses.md:343 SWE-bench shows a systematic adversarial battery; ecosystem/pickup/pickup-eval-harnesses.md:349 none of the evidence repos provided a verified judge-honesty)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • Inspect AI, lm-evaluation-harness or harbor ship an adversarial grader suite: the wrap shrinks to reusing it.
  • A harness publishes a judge-honesty pattern (judge identity pinned, measured run-to-run variance, agreement bounds with a deterministic checker).
  • franken_overlap checks in its first evidence run, which would show whether its claim gate works as a reusable benchmark-claim template.

Sources

Starting a build here?

Begin from the pickup companion for LLM eval harnesses: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed