← All agent-stack verdicts

Retrieval-augmented generation (RAG) frameworks

Adopt and wrap Confidence Low Group Memory and retrieval Evidence as of 2026-09-23

Written by VerdictsServing · Reviewed by ReviewServing on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. NODUS is the four-ring verdict scale (Invest, Pilot, Explore, Monitor) used for the FrankenSuite assessments; the agent-stack verdicts use their own scale (Adopt, Adopt and wrap, Build clean-room, Watch). TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Bottom line

Inference, low confidence: adopt and wrap an existing RAG framework (LlamaIndex, LangChain or Haystack; LightRAG if you want graph-based retrieval, with GraphRAG a watch-level option whose CI and tests were not read) for loading, chunking and retrieving documents, rather than writing one. Wrap it with an evaluation you own: a frozen corpus with expected answers, a pinned judge model for any LLM-graded metric, and recorded LLM calls so CI runs without a live model, because published quality thresholds in this space do not name the judge that produced them. Confidence is low because the evidence pack read most of these repos by file path, not source.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

Copy these practices

Build only if

  • The evidence names no hard constraint that these frameworks fail. The copyleft terms sit in optional LlamaIndex integration packages, not its MIT core, so they justify skipping those packages, not building a framework; the other adopted frameworks are MIT or Apache-2.0. What is missing is evaluation provenance, so build the evaluation harness (frozen corpus, oracle, pinned judge), not a framework. [Verified] [Inference] (stack/licenses.tsv:81 core (llama-index-core) MIT, but several separately published integration packages are copyleft; stack/licenses.tsv:44 langchain-ai/langchain MIT permissive; stack/licenses.tsv:18 deepset-ai/haystack Apache-2.0 permissive; stack/licenses.tsv:35 HKUDS/LightRAG MIT permissive; stack/licenses.tsv:55 microsoft/graphrag MIT permissive; stack/licenses.tsv:17 confident-ai/deepeval Apache-2.0 permissive; ecosystem/pickup/pickup-rag-frameworks.md:362-363 no evidence repo was found to freeze its LLM-judge backend)

Where FrankenSuite touches this

  • frankensearch is a retrieval primitive, not a RAG framework: BM25 plus static embeddings fused by reciprocal rank fusion, delivered as a fast first answer then a refined one. The maintainer states his relevance experiments do not certify its quality; TRL 5-6, NODUS ring Explore. [Verified] [Maintainer claim] [Inference] (packets/frankensearch-assessment.md:19 fused via Reciprocal Rank Fusion and delivered progressively; packets/frankensearch-assessment.md:102 These Python results do not certify Quill quality; packets/frankensearch-assessment.md:22 TRL 5–6 — see §4.9). Substantive and partially validated)
  • franken_nlp lists RAG faithfulness judging among the local jobs it targets, but no model path executes yet (TRL 2-3). [Maintainer claim] [Inference] (packets/franken_nlp-assessment.md:53 RAG faithfulness judging; packets/franken_nlp-assessment.md:18 TRL: 2–3; NODUS ring: Monitor)
  • frankenterm combines SQLite FTS5, Tantivy and fastembed with reciprocal rank fusion to search terminal output; the code exists and was not executed by its assessor. [Verified] (packets/frankenterm-assessment.md:79 semantic fastembed + RRF hybrid)

What we cannot say

Revisit when

  • A framework publishes quality numbers with the judge model, corpus revision and measured judge noise named.
  • A maintained successor to ragas appears, or ragas resumes regular releases.
  • frankensearch commits a labeled-corpus relevance receipt.

Sources

Starting a build here?

Begin from the pickup companion for Retrieval-augmented generation (RAG) frameworks: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed