Bottom line
Inference, low confidence: watch computer-use agents (software that operates a desktop from screenshots, keyboard and mouse) and do not commit a product to one agent stack yet. The agent stacks you would adopt are evidenced by activity and by benchmark scores their own READMEs report, not by test, CI or conformance evidence, and the category is mostly research code. If you experiment now, use the existing benchmark harness, grounding evaluation and VM infrastructure rather than building your own, and score the machine's end state, never the agent's own account of what it did.
Adopt, do not rebuild
License checked on 2026-09-23; non-permissive licenses are named in the text.
- xlang-ai/OSWorld Apache-2.0: the desktop-agent benchmark of 369 tasks in real Ubuntu and Windows VMs, scoring VM state rather than chat; a private benchmark would cut you off from the numbers the field reports. [Verified] (ecosystem/pickup/_evidence/computer-use.md:12
NeurIPS 2024 benchmark of 369 tasks in a real Ubuntu/Windows desktop VM
; ecosystem/pickup/_evidence/computer-use.md:12Anthropic's Computer Use, OpenAI CUA, academic papers all report on it
; ecosystem/pickup/_evidence/computer-use.md:22State-based evaluation (score VM state, not chat)
) - likaixin2000/ScreenSpot-Pro-GUI-Grounding MIT: the grounding-accuracy harness (does the predicted click point fall inside the ground-truth box); use it to measure grounding instead of inventing a metric. [Verified] (ecosystem/pickup/_evidence/computer-use.md:28
Grounding accuracy eval script (ScreenSpot protocol)
; ecosystem/pickup/_evidence/computer-use.md:42is the verified grounding-accuracy benchmark harness
) - microsoft/OmniParser CC-BY-4.0: the grounding layer that turns a screenshot into located UI elements. License: CC-BY-4.0, a content license with an attribution requirement, unusual for code (the README badge says MIT); the older Ultralytics-based icon-detector weights keep their AGPL license, so pick the current weights and read the terms before shipping it inside a product. [Verified] (ecosystem/pickup/_evidence/computer-use.md:9
parse screenshot into structured UI elements
; stack/licenses.tsv:56Earlier Ultralytics-based icon detectors retain their original AGPL license
; fresh read 2026-09-23,gh api repos/microsoft/OmniParser --jq .license.spdx_id) - trycua/cua MIT AND AGPL-3.0: cross-OS VM infrastructure for computer-use training, evaluation and data generation, with a benchmark library built and tested in CI; the VM fleet you would otherwise stand up yourself. License: MIT at the root, but
libs/python/somis AGPL-3.0, so running a modified copy of that library in a network service triggers source release. [Verified] (ecosystem/pickup/_evidence/computer-use.md:8Cross-OS (macOS/Windows/Linux/Android) VM-based computer-use infra
; ecosystem/pickup/_evidence/computer-use.md:26Benchmark harness as a first-class library with CI
; stack/licenses.tsv:94libs/python/som/LICENSE is GNU AFFERO GENERAL PUBLIC LICENSE v3
)
Copy these practices
- Score the machine, not the transcript: OSWorld evaluates the VM's end state through getter functions that read state and separate metric functions that score it. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/computer-use.md:22
State-based evaluation (score VM state, not chat)
) - Infeasible tasks that must be refused explicitly: an infeasible task scores 1.0 only if the agent's last action is an explicit FAIL, so running out the clock on an impossible task earns nothing. Starter kit: Acceptance shape: positive observable + planted negative + no-claim line (provisional). [Verified] (ecosystem/pickup/_evidence/computer-use.md:24
prevents reward-hacking by refusal
; ecosystem/pickup/pickup-computer-use.md:56refusal-by-timeout scores 0, which blocks reward-hacking by refusal
) - Pin the benchmark commit in every run manifest: a run recorded without the exact harness commit cannot be compared after the harness moves. Starter kit: Incumbent oracle pinned before implementation. [External] (ecosystem/pickup/_evidence/computer-use.md:33
record the exact benchmark commit in every run manifest — prevents silent harness drift
) - One directory per agent generation, each with its own eval config: Agent-S keeps S1 through S3 and their OSWorld setups side by side so older results stay reproducible. Starter kit: Receipts carry provenance; generation binding. [Verified] (ecosystem/pickup/_evidence/computer-use.md:29
Multi-generation agent structure with versioned eval configs
) - Per-task JSON configs: each task carries its instruction, setup and evaluator, so tasks are portable across harnesses. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/computer-use.md:23
Per-task JSON configs (instruction + setup + evaluator)
) - Scheduled live smoke tests of the VM fleet: cua runs periodic live-environment smokes to catch infrastructure drift that unit tests miss. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/computer-use.md:32
Live infra smoke tests (fleets)
)
Build only if
- A Windows-first project may have to invest in harness work itself: the Windows benchmark shows thinner maintenance than OSWorld and one of its container pointers failed verification. That is harness investment, not a reason to build an agent stack. [Verified] (ecosystem/pickup/_evidence/computer-use.md:39
WindowsAgentArena is the thinner half.
; ecosystem/pickup/_evidence/computer-use.md:41WindowsAgentArena tree listing failed
) - Otherwise no hard constraint is evidenced; a new desktop benchmark or VM fleet would duplicate OSWorld and cua. [Inference] (ecosystem/pickup/_evidence/computer-use.md:12
NeurIPS 2024 benchmark of 369 tasks in a real Ubuntu/Windows desktop VM
; ecosystem/pickup/_evidence/computer-use.md:8Cross-OS (macOS/Windows/Linux/Android) VM-based computer-use infra
) - License gives no reason to build an agent stack: a fresh read on 2026-09-23 (
gh api repos/<owner>/<repo> --jq .license.spdx_id) returned Apache-2.0 for OSWorld and MIT for ScreenSpot-Pro and cua. cua'ssomlibrary is AGPL-3.0, which a closed network service must avoid or replace, and OmniParser's CC-BY-4.0 asks for attribution; both are reasons to read terms or swap one component, not to rebuild the VM fleet or the parser. [Inference] (stack/licenses.tsv:94libs/python/som/LICENSE is GNU AFFERO GENERAL PUBLIC LICENSE v3
; stack/METHOD.md:79Check the license of everything you tell a builder to adopt
)
Where FrankenSuite touches this
- No FrankenSuite packet builds a screenshot-driven desktop agent. frankenterm controls fleets of coding agents through terminal panes (Robot Mode API, MCP surface, policy-gated actions), a text channel rather than pixels, for terminal work. TRL 7, Explore; license: MIT with a rider barring OpenAI, Anthropic and anyone acting for them. [Verified] (packets/frankenterm-assessment.md:9
a WezTerm-derived terminal runtime (absorbed in-tree, September 2026) plus an agent-swarm control plane
; packets/frankenterm-assessment.md:17License (MIT + OpenAI/Anthropic rider, read verbatim)
; synthesis/00-overview.md:102frankenterm | 7 | Explore | Rider
) - franken_remote is a remote-workstation daemon and client with a screen capture and media pipeline, the kind of substrate a desktop agent would drive; it has no complete application and no hardware-qualified capture path yet. TRL 3–4, Explore; license: MIT with a rider barring OpenAI, Anthropic and anyone acting for them. [Maintainer claim] (packets/franken_remote-assessment.md:19
A Tailscale-identity-based remote workstation
; packets/franken_remote-assessment.md:19no hardware-qualified capture/codec path
; packets/franken_remote-assessment.md:21the license rider names OpenAI and Anthropic as Restricted Parties
; synthesis/00-overview.md:81franken_remote | 3–4 | Explore | Rider
)
What we cannot say
- Whether any agent stack works outside benchmarks: the category is research-heavy and adoption is unproven (ecosystem/pickup/_evidence/computer-use.md:37
adoption of any single agent stack is unproven beyond benchmarks
). UI-TARS-desktop was checked for activity only; Agent-S for activity, README scores and its directory and eval-config layout; neither for tests or CI (ecosystem/pickup/_evidence/computer-use.md:7A native GUI-agent desktop app + agent stack (Agent TARS)
; ecosystem/pickup/_evidence/computer-use.md:10README reports OSWorld/WindowsAgentArena scores
; ecosystem/pickup/_evidence/computer-use.md:29per-generation dirs keep old configs reproducible
). - Benchmark percentages on READMEs are self-reported (ecosystem/pickup/_evidence/computer-use.md:38
self-reported and deserve discounting
), and a small eval set drawn from the full set is a cherry-picking risk (ecosystem/pickup/pickup-computer-use.md:196a small eval set drawn from the full set is a cherry-picking risk surface
). - OSWorld's own quality control is not visible as CI (ecosystem/pickup/_evidence/computer-use.md:40
OSWorld itself has no GitHub CI visible
); cua's workflows were read as a listing, which shows they exist, not that they pass. - The Windows half is thinner and partly unverified (ecosystem/pickup/_evidence/computer-use.md:39
no visible CI workflows in its .github/ listing beyond default
; ecosystem/pickup/_evidence/computer-use.md:41treat that single pointer as unverified
). - The benchmark-pinning practice is a third-party observation not re-verified by the pack (ecosystem/pickup/_evidence/computer-use.md:33
community practice visible in the wild
; ecosystem/pickup/pickup-computer-use.md:194is a third-party observation not independently re-verified this pass
). - Only the ScreenSpot-Pro harness was verified; the original ScreenSpot repositories were dropped (ecosystem/pickup/_evidence/computer-use.md:42
the original ScreenSpot paper repo name variants are low-star forks and were dropped
). - Licenses were not in the pack; they come from the author's fresh read. Stars measure popularity.
Revisit when
- An independent party reproduces an agent stack's OSWorld or OSWorld-V2 score from a pinned harness commit.
- A computer-use agent product publishes its test and CI evidence, or is deployed by someone other than its maker.
- WindowsAgentArena resumes maintenance or is archived; revisit six months after 2026-09-23 either way.
Sources
- Pickup companion: ecosystem/pickup/pickup-computer-use.md, the charter seed, oracle candidates, and first claim registry for this area.
- Evidence pack: ecosystem/pickup/_evidence/computer-use.md, the repositories checked against the GitHub API, with dates.
- Verdict source: stack/computer-use.md, written under stack/METHOD.md.
Starting a build here?
Begin from the pickup companion for Computer-use agents: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.