← All agent-stack verdicts

Browser-use stacks

Adopt and wrap Confidence Medium Group Tools and environment Evidence as of 2026-09-23

Written by VerdictsTools · Reviewed by ReviewTools on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked. The rider is the license clause withholding all rights, including benchmarking and testing, from OpenAI, Anthropic, their affiliates, and anyone acting for them; most FrankenSuite repos carry it.

Bottom line

Inference, medium confidence: adopt an existing agent-browser library or tool server (Playwright MCP, browser-use, Stagehand) and a maintained browser farm, and wrap them with your own flake and determinism controls. The incumbents already separate real-browser tests from unit tests and ship evaluation harnesses, but the evidence found no flake-rate budget and no recorded model responses for replay in any of them. Keep unit tests off the live web, pin the benchmark datasets you score against, treat page content as untrusted input before letting an agent act on it, and check the farm's license first: browserless is SSPL or commercial.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • microsoft/playwright-mcp Apache-2.0: Microsoft's MCP server that gives agents a browser through an accessibility-tree-first observation model, tested against a local deterministic test server; building your own browser tool server would duplicate it. [Verified] (ecosystem/pickup/_evidence/browser-use.md:8 Microsoft's official Playwright MCP server; ecosystem/pickup/_evidence/browser-use.md:8 accessibility-tree-first observation model; ecosystem/pickup/_evidence/browser-use.md:20 Ship a local deterministic test server instead of hitting the live web in unit tests)
  • browser-use/browser-use MIT: agent library for using the browser, with declarative agent tasks scored by an LLM judge and a per-PR evaluation trigger. [Verified] (ecosystem/pickup/_evidence/browser-use.md:7 ships its own cloud eval platform triggered per-PR; ecosystem/pickup/_evidence/browser-use.md:21 Define agent tasks as declarative YAML with judge criteria + LLM-as-judge; community-contributable)
  • browserbase/stagehand MIT: SDK for extract, act and observe on any site, with an in-repo evaluation package over dataset-backed benchmark suites. [Verified] (ecosystem/pickup/_evidence/browser-use.md:9 ships a first-class in-repo eval harness (packages/evals) with dataset-backed benchmark suites)
  • browserless/browserless SSPL-1.0 OR LicenseRef-Browserless-Commercial: headless-browser farm with Docker matrices across Chrome, Firefox and WebKit and a memory-leak regression workflow for long-lived sessions; the infrastructure layer you would otherwise have to build. License: SSPL-1.0 or the browserless commercial license; closed-source commercial or CI use needs the commercial license. [Verified] (ecosystem/pickup/_evidence/browser-use.md:11 Headless-browser-as-a-service (Docker/Chrome/Firefox/WebKit matrices); ecosystem/pickup/_evidence/browser-use.md:28 leak-check.yml (memory-leak regression for long-lived headless sessions); fresh read 2026-09-23, gh api repos/browserless/browserless/license --jq '.content|@base64d')
  • ServiceNow/BrowserGym, web-arena-x/webarena, OSU-NLP-Group/Mind2Web Apache-2.0 Apache-2.0 MIT: the benchmark environments and datasets agents are scored on; a private benchmark would not be comparable with the datasets active projects vendor. [Verified] (ecosystem/pickup/_evidence/browser-use.md:25 Gym-style experiment loop + typed benchmark configs; ecosystem/pickup/_evidence/browser-use.md:14 realistic web environments + evaluation harness; ecosystem/pickup/_evidence/browser-use.md:15 The original LLM web-agent benchmark dataset/codebase; ecosystem/pickup/_evidence/browser-use.md:36 still current as benchmark standards (vendored into active projects' test data))

Copy these practices

  • Unit and real-browser suites in separate configs: Stagehand keeps specs that launch real Chrome out of the cacheable unit suite by giving them their own config. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:19 Separate unit vs. real-browser integration suites; never let them mix in one config)
  • A local deterministic test server for core specs: Playwright MCP runs its core specs against an in-repo test server on a three-OS matrix, with no external-network dependency. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:20 no external-network dependency for core specs)
  • Agent tasks as data, one process per task: tasks declare name, task, judge context and step budget; each runs in its own process so browser sessions cannot interfere. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:21 Each task gets its own Python process, preventing browser session interference)
  • Vendored benchmark datasets with a provenance file: Skyvern ships WebVoyager and Odysseys task files in-repo with an attribution file. Starter kit: Incumbent oracle pinned before implementation. [Verified] (ecosystem/pickup/_evidence/browser-use.md:24 Bundle third-party benchmark datasets in-repo and keep provenance)
  • Session recording as a tested feature: browser-use drives a CDP screencast to produce video of a run and tests it against a real headless browser, so a flaky run leaves a video to inspect (a recording, not a deterministic rerun). Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:26 Session recording/replay as a first-class tested feature)
  • Expensive evaluations only when relevant paths change: Stagehand filters CI jobs by changed paths and supports skip labels, with evaluation jobs split from the unit lane. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:27 Change-aware CI: only run expensive browser/eval jobs when relevant paths changed; label-based skip)

Build only if

  • No hard constraint that justifies a new browser stack is evidenced. The two gaps the pack names, a flake budget and deterministic replay of model responses, are harness work on top of an incumbent, not reasons to write a browser driver. [Inference] (ecosystem/pickup/_evidence/browser-use.md:32 No repo I checked documents an explicit flake-rate budget or quarantined-flake labeling scheme; ecosystem/pickup/_evidence/browser-use.md:33 Nobody I verified ships recorded LLM responses (no VCR-cassette-style fixtures found))
  • Treating page content as hostile input is a requirement the companion adds for this type; it is a policy layer between page text and tool calls, not a separate browser. [Inference] (ecosystem/pickup/pickup-browser-use.md:48 Trust boundary: hostile page content. Page content is untrusted)
  • browserless's license is a real constraint for closed-source commercial products and closed-source CI: SSPL-1.0 or a paid commercial license. It justifies buying that license or running browsers directly under the Playwright library (Apache-2.0: fresh read 2026-09-23, gh api repos/microsoft/playwright --jq .license.spdx_id), not writing a browser farm or driver. The other adopted repositories, Playwright MCP included, read Apache-2.0 or MIT (fresh read 2026-09-23, gh api repos/<owner>/<repo> --jq .license.spdx_id; LICENSE file for BrowserGym, where the API said NOASSERTION). [Inference] (stack/METHOD.md:79 Check the license of everything you tell a builder to adopt; ecosystem/pickup/_evidence/browser-use.md:8 Microsoft's official Playwright MCP server)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • An incumbent publishes a flake-rate budget or recorded-response fixtures for deterministic agent-task reruns.
  • A maintained successor to WebArena or Mind2Web becomes the dataset the active projects vendor.
  • A FrankenSuite website wires its browser suites into CI, which would make it a working example of the lane separation described here.

Sources

Starting a build here?

Begin from the pickup companion for Browser-use stacks: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed