← All agent-stack verdicts

Web search API layers

Adopt and wrap Confidence Medium Group Tools and environment Evidence as of 2026-09-23

Written by VerdictsTools · Reviewed by ReviewTools on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. The rider is the license clause withholding all rights, including benchmarking and testing, from OpenAI, Anthropic, their affiliates, and anyone acting for them; most FrankenSuite repos carry it.

Bottom line

Inference, medium confidence: adopt an existing web search and extraction layer, either Firecrawl self-hosted (AGPL-3.0) or a commercial API (Tavily, Exa, SerpApi) through its official SDK, and wrap it with your own quality and latency checks. The ranking machinery in this category is closed, and no repository in the evidence publishes a relevance benchmark or a latency budget, so "the search is good" is something you have to measure yourself. Keep a frozen, hashed set of recorded queries and responses as your test oracle, and keep live vendor API tests in a separate lane behind secrets rather than in the required build.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • firecrawl/firecrawl AGPL-3.0: an open-source web data API to search, scrape and interact, with a self-host path and a per-commit quality-evaluation pipeline; the only open incumbent in the set with real eval plumbing. License: AGPL-3.0, so running a modified Firecrawl as a network service obliges you to publish your changes. [Verified] (ecosystem/pickup/_evidence/web-search-apis.md:7 The web data API to search, scrape, and interact at scale; fully open-source monorepo with self-host path; ecosystem/pickup/_evidence/web-search-apis.md:37 Firecrawl is the exception and the only repo in this set with a real quality-eval pipeline; fresh read 2026-09-23, gh api repos/firecrawl/firecrawl --jq .license.spdx_id)
  • tavily-ai/tavily-python, exa-labs/exa-py, serpapi/serpapi-python MIT MIT MIT: official SDKs for the commercial search APIs, when a closed index is acceptable; each is a thin client you should not rewrite. [Verified] (ecosystem/pickup/_evidence/web-search-apis.md:10 Official Tavily Python SDK. Commercial vendor with open SDK; ecosystem/pickup/_evidence/web-search-apis.md:11 Official Exa Python package (neural search API); ecosystem/pickup/_evidence/web-search-apis.md:12 Official SerpApi Python wrapper.)
  • searxng/searxng AGPL-3.0: open metasearch aggregation to use as a backend under your own agent layer; it returns search-results JSON for humans and is not an agent API on its own. License: AGPL-3.0, with the same network-service source obligation for modified versions. [Verified] (ecosystem/pickup/_evidence/web-search-apis.md:8 Free metasearch engine; the open aggregation layer underneath agent search stacks; ecosystem/pickup/_evidence/web-search-apis.md:38 SearXNG is infrastructure, not an agent API.; fresh read 2026-09-23, gh api repos/searxng/searxng --jq .license.spdx_id)
  • jina-ai/reader Apache-2.0: URL-to-markdown extraction endpoint with a unit and end-to-end test split; the extraction half you would otherwise write. [Verified] (ecosystem/pickup/_evidence/web-search-apis.md:9 URL→LLM-friendly-markdown endpoint; ecosystem/pickup/_evidence/web-search-apis.md:28 Unit + e2e test tiers with a booted-server e2e harness)

Copy these practices

Build only if

  • You need control of the index and ranking: that machinery is closed in every vendor, so no incumbent offers it; but an internet-scale index is a different project type, not a web search API layer. [Verified] (ecosystem/pickup/_evidence/web-search-apis.md:36 lives inside Tavily/Exa/SerpApi/Brave/Perplexity and is not open; ecosystem/pickup/pickup-web-search-apis.md:25 Building a web index or crawler at internet scale)
  • A key-free, self-hosted requirement alone does not justify building, because Firecrawl ships a self-host path. Firecrawl and SearXNG are both AGPL-3.0 (fresh read 2026-09-23, gh api repos/<owner>/<repo> --jq .license.spdx_id); a builder who cannot accept AGPL source obligations for a modified network service still has AGPL-free adoption paths to try first: the commercial APIs through their MIT SDKs, Jina Reader (Apache-2.0) for extraction, Firecrawl's own SDKs, which are MIT, and running Firecrawl unmodified as a separate service. Only if none of those fits does the AGPL constraint justify building, and a clean-room verdict would still have to quote the clause. The other adopted repositories read MIT or Apache-2.0 in the same read. [Inference] (ecosystem/pickup/_evidence/web-search-apis.md:7 fully open-source monorepo with self-host path; stack/licenses.tsv:27 SDKs under apps/-sdk (python, js, ruby, elixir) are MIT; stack/METHOD.md:79 Check the license of everything you tell a builder to adopt)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • A vendor-independent relevance benchmark for agent search is published, or Firecrawl documents its scoring method.
  • Any incumbent publishes and gates a latency budget.
  • Firecrawl's self-host footprint turns out to block a real deployment, or Firecrawl or SearXNG changes license.

Sources

Starting a build here?

Begin from the pickup companion for Web search API layers: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed