Bottom line
Inference, medium confidence: adopt an existing agent-browser library or tool server (Playwright MCP, browser-use, Stagehand) and a maintained browser farm, and wrap them with your own flake and determinism controls. The incumbents already separate real-browser tests from unit tests and ship evaluation harnesses, but the evidence found no flake-rate budget and no recorded model responses for replay in any of them. Keep unit tests off the live web, pin the benchmark datasets you score against, treat page content as untrusted input before letting an agent act on it, and check the farm's license first: browserless is SSPL or commercial.
Adopt, do not rebuild
License checked on 2026-09-23; non-permissive licenses are named in the text.
- microsoft/playwright-mcp Apache-2.0: Microsoft's MCP server that gives agents a browser through an accessibility-tree-first observation model, tested against a local deterministic test server; building your own browser tool server would duplicate it. [Verified] (ecosystem/pickup/_evidence/browser-use.md:8
Microsoft's official Playwright MCP server
; ecosystem/pickup/_evidence/browser-use.md:8accessibility-tree-first observation model
; ecosystem/pickup/_evidence/browser-use.md:20Ship a local deterministic test server instead of hitting the live web in unit tests
) - browser-use/browser-use MIT: agent library for using the browser, with declarative agent tasks scored by an LLM judge and a per-PR evaluation trigger. [Verified] (ecosystem/pickup/_evidence/browser-use.md:7
ships its own cloud eval platform triggered per-PR
; ecosystem/pickup/_evidence/browser-use.md:21Define agent tasks as declarative YAML with judge criteria + LLM-as-judge; community-contributable
) - browserbase/stagehand MIT: SDK for extract, act and observe on any site, with an in-repo evaluation package over dataset-backed benchmark suites. [Verified] (ecosystem/pickup/_evidence/browser-use.md:9
ships a first-class in-repo eval harness (packages/evals) with dataset-backed benchmark suites
) - browserless/browserless SSPL-1.0 OR LicenseRef-Browserless-Commercial: headless-browser farm with Docker matrices across Chrome, Firefox and WebKit and a memory-leak regression workflow for long-lived sessions; the infrastructure layer you would otherwise have to build. License: SSPL-1.0 or the browserless commercial license; closed-source commercial or CI use needs the commercial license. [Verified] (ecosystem/pickup/_evidence/browser-use.md:11
Headless-browser-as-a-service (Docker/Chrome/Firefox/WebKit matrices)
; ecosystem/pickup/_evidence/browser-use.md:28leak-check.yml (memory-leak regression for long-lived headless sessions)
; fresh read 2026-09-23,gh api repos/browserless/browserless/license --jq '.content|@base64d') - ServiceNow/BrowserGym, web-arena-x/webarena, OSU-NLP-Group/Mind2Web Apache-2.0 Apache-2.0 MIT: the benchmark environments and datasets agents are scored on; a private benchmark would not be comparable with the datasets active projects vendor. [Verified] (ecosystem/pickup/_evidence/browser-use.md:25
Gym-style experiment loop + typed benchmark configs
; ecosystem/pickup/_evidence/browser-use.md:14realistic web environments + evaluation harness
; ecosystem/pickup/_evidence/browser-use.md:15The original LLM web-agent benchmark dataset/codebase
; ecosystem/pickup/_evidence/browser-use.md:36still current as benchmark standards (vendored into active projects' test data)
)
Copy these practices
- Unit and real-browser suites in separate configs: Stagehand keeps specs that launch real Chrome out of the cacheable unit suite by giving them their own config. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:19
Separate unit vs. real-browser integration suites; never let them mix in one config
) - A local deterministic test server for core specs: Playwright MCP runs its core specs against an in-repo test server on a three-OS matrix, with no external-network dependency. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:20
no external-network dependency for core specs
) - Agent tasks as data, one process per task: tasks declare name, task, judge context and step budget; each runs in its own process so browser sessions cannot interfere. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:21
Each task gets its own Python process, preventing browser session interference
) - Vendored benchmark datasets with a provenance file: Skyvern ships WebVoyager and Odysseys task files in-repo with an attribution file. Starter kit: Incumbent oracle pinned before implementation. [Verified] (ecosystem/pickup/_evidence/browser-use.md:24
Bundle third-party benchmark datasets in-repo and keep provenance
) - Session recording as a tested feature: browser-use drives a CDP screencast to produce video of a run and tests it against a real headless browser, so a flaky run leaves a video to inspect (a recording, not a deterministic rerun). Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:26
Session recording/replay as a first-class tested feature
) - Expensive evaluations only when relevant paths change: Stagehand filters CI jobs by changed paths and supports skip labels, with evaluation jobs split from the unit lane. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/browser-use.md:27
Change-aware CI: only run expensive browser/eval jobs when relevant paths changed; label-based skip
)
Build only if
- No hard constraint that justifies a new browser stack is evidenced. The two gaps the pack names, a flake budget and deterministic replay of model responses, are harness work on top of an incumbent, not reasons to write a browser driver. [Inference] (ecosystem/pickup/_evidence/browser-use.md:32
No repo I checked documents an explicit flake-rate budget or quarantined-flake labeling scheme
; ecosystem/pickup/_evidence/browser-use.md:33Nobody I verified ships recorded LLM responses (no VCR-cassette-style fixtures found)
) - Treating page content as hostile input is a requirement the companion adds for this type; it is a policy layer between page text and tool calls, not a separate browser. [Inference] (ecosystem/pickup/pickup-browser-use.md:48
Trust boundary: hostile page content. Page content is untrusted
) - browserless's license is a real constraint for closed-source commercial products and closed-source CI: SSPL-1.0 or a paid commercial license. It justifies buying that license or running browsers directly under the Playwright library (Apache-2.0: fresh read 2026-09-23,
gh api repos/microsoft/playwright --jq .license.spdx_id), not writing a browser farm or driver. The other adopted repositories, Playwright MCP included, read Apache-2.0 or MIT (fresh read 2026-09-23,gh api repos/<owner>/<repo> --jq .license.spdx_id; LICENSE file for BrowserGym, where the API said NOASSERTION). [Inference] (stack/METHOD.md:79Check the license of everything you tell a builder to adopt
; ecosystem/pickup/_evidence/browser-use.md:8Microsoft's official Playwright MCP server
)
Where FrankenSuite touches this
- No FrankenSuite packet builds an agent browser stack. The nearest contact is test practice: franken_markdown_website ships a 30-check headless-Chromium end-to-end suite that nothing runs on push. Website, TRL 8, Monitor. [Verified] (packets/franken_markdown_website-assessment.md:126
30-check e2e (bun run test, Chromium headless vs localhost:8899)
; packets/franken_markdown_website-assessment.md:126But it was not run in this assessment, and nothing runs it on push
; synthesis/00-overview.md:73franken_markdown_website | 8 | Monitor | Rider
) - frankentui_website has the same shape: Playwright end-to-end and performance suites with no CI enforcing them, the ungated-suite gap the lane-separation practice above addresses. Website, TRL 9, Monitor. [Verified] (packets/frankentui_website-assessment.md:158
The Playwright E2E + perf suites exist (20 files) but nothing enforces them on push
; synthesis/00-overview.md:105frankentui_website | 9 | Monitor | Rider
) - franken_engine is a JS/TS execution substrate for adversarial extension workloads, not a web browser or a browser-automation layer. TRL 4–5, Explore. [Verified] (packets/franken_engine-assessment.md:19
A from-scratch Rust execution substrate for adversarial JS/TS extension workloads
; synthesis/00-overview.md:69franken_engine | 4–5 | Explore | Rider
)
What we cannot say
- How flaky these stacks are: flakiness evidence was thin, and the clearest mechanisms seen were per-test retries, evaluation lanes kept off the unit path, weekly browser-binary caching and run diffing (ecosystem/pickup/_evidence/browser-use.md:32
Flakiness practices were thin in what I could verify
; ecosystem/pickup/_evidence/browser-use.md:32weekly browser-binary caching (browser-use), and --preview/Braintrust run-diffing (stagehand)
). - Whether any agent-task result can be rerun deterministically: determinism comes from replaying the benchmark, not the model (ecosystem/pickup/_evidence/browser-use.md:33
Determinism for LLM-driven tests is mostly
; ecosystem/pickup/_evidence/browser-use.md:33Nobody I verified ships recorded LLM responses (no VCR-cassette-style fixtures found)
); how to keep an LLM judge comparable across model retirements is open (ecosystem/pickup/pickup-browser-use.md:398How to pin an LLM-as-judge across model deprecations
). - Skyvern's evaluation scripts post to its commercial task store, so only its datasets and attribution are self-contained (ecosystem/pickup/_evidence/browser-use.md:37
Skyvern's evaluation scripts reference their commercial task store
). - The benchmark repositories move slowly (ecosystem/pickup/_evidence/browser-use.md:36
webarena and Mind2Web last pushed Nov 2025
). Steel was verified active but its tests were not found (ecosystem/pickup/_evidence/browser-use.md:34Steel evidence is shallow.
). - Who uses these stacks in production: adopter claims were not verified (ecosystem/pickup/_evidence/browser-use.md:35
Adopter claims avoided.
). Star counts measure popularity. Licenses were not in the pack; they come from the author's fresh read (browserless SSPL-or-commercial, the rest Apache-2.0 or MIT). A workflow file that exists does not show its runs are green.
Revisit when
- An incumbent publishes a flake-rate budget or recorded-response fixtures for deterministic agent-task reruns.
- A maintained successor to WebArena or Mind2Web becomes the dataset the active projects vendor.
- A FrankenSuite website wires its browser suites into CI, which would make it a working example of the lane separation described here.
Sources
- Pickup companion: ecosystem/pickup/pickup-browser-use.md, the charter seed, oracle candidates, and first claim registry for this area.
- Evidence pack: ecosystem/pickup/_evidence/browser-use.md, the repositories checked against the GitHub API, with dates.
- Verdict source: stack/browser-use.md, written under stack/METHOD.md.
Starting a build here?
Begin from the pickup companion for Browser-use stacks: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.