← All agent-stack verdicts

Realtime voice agents

Adopt and wrap Confidence Medium Group Training and voice Evidence as of 2026-09-23

Written by VerdictsTools · Reviewed by ReviewTools on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked. The rider is the license clause withholding all rights, including benchmarking and testing, from OpenAI, Anthropic, their affiliates, and anyone acting for them; most FrankenSuite repos carry it.

Bottom line

Inference, medium confidence: adopt an existing realtime voice-agent framework (Pipecat or LiveKit Agents, on a maintained media transport) and wrap it with a latency regression gate of your own. The frameworks already handle speech-to-text, language model and text-to-speech stages, turn-taking and interruptions, and LiveKit Agents tests them with fake providers and a virtual clock; but no latency regression gate was found in either framework, and interruption thresholds have no published ground truth. Keep the fake-provider test surface, measure time-to-first-audio on your own harness with A/A baselines, and block releases on regressions.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • pipecat-ai/pipecat BSD-2-Clause: an open-source voice-agent pipeline framework maintained by Daily and the community, with latency metric classes and their own unit tests. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:7 maintained by Daily and the community; pushed same-day as verification; ecosystem/pickup/_evidence/voice-agents.md:19 Latency metric classes with their own unit tests)
  • livekit/agents Apache-2.0: LiveKit's realtime voice-agent framework with a plugin tree of about forty speech and model providers and fake doubles for every pipeline stage. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:8 LiveKit's framework for building realtime voice AI agents; ecosystem/pickup/_evidence/voice-agents.md:8 plugin tree covers ~40 STT/LLM/TTS providers; ecosystem/pickup/_evidence/voice-agents.md:18 Fake/mock service doubles for every pipeline stage)
  • livekit/livekit Apache-2.0: the realtime media transport (SFU, a server that forwards audio and video streams) the agent frameworks ride on; the companion keeps transport internals out of scope for this type. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:12 Underlying realtime media transport (SFU) the agent frameworks ride on; ecosystem/pickup/pickup-voice-agents.md:27 the media transport/SFU internals (use a pinned incumbent as the transport,)
  • TEN-framework/ten-framework LicenseRef-TEN-Open-Source-License: an alternative framework with a multi-OS CI matrix. License: Apache-2.0 with added conditions: no hosting on end-user devices (including mobile) and no deployment that competes with Agora's offerings. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:9 heavy multi-OS CI matrix (linux/mac/win × arm64/x64); fresh read 2026-09-23, gh api repos/TEN-framework/ten-framework/license --jq '.content|@base64d')
  • openai/openai-agents-js MIT: a first-party vendor framework for voice agents in JavaScript runtimes. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:10 signal that a first-party vendor ships voice-agent orchestration)

Copy these practices

  • A fake for every stage: livekit/agents has fake speech-to-text, model, text-to-speech, VAD, turn-detector and session doubles, so full agent sessions run in tests without API keys or network. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:18 Tests run deterministic agent sessions without API keys or network.)
  • Latency metrics as tested classes: Pipecat detects time-to-first-audio from speech onset in PCM buffers and unit-tests that metric, alongside time-to-first-byte. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:19 TTFA (time-to-first-audio) detected via RMS speech-onset detector on PCM buffers)
  • Turn-taking on a virtual clock: a fake clock keeps turn-taking and latency assertions from flaking on real-time scheduling. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:22 fake clock so turn-taking/latency assertions don't flake on real-time scheduling)
  • A dedicated interruption test surface: false-interruption resume, protected speech, hold windows and overlapping speech each get tests. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:21 Dedicated interruption / turn-taking test surface)
  • Live-provider tests behind secrets, off the unit path: a separate workflow runs against real realtime APIs, triggered only by relevant path changes and skipped on forks. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:23 path-triggered only by realtime-code changes, skipped on forks)
  • Network faults injected in tests: livekit/agents drives a toxiproxy container to degrade connections during tests. Starter kit: Acceptance shape: positive observable + planted negative + no-claim line (provisional). [Verified] (ecosystem/pickup/_evidence/voice-agents.md:29 Fault injection / network degradation in tests)
  • Simulate the caller: ServiceNow/eva evaluates voice agents with bot-to-bot calls, a simulated user and perturbations such as noise and degraded connections. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:26 copy the pattern of simulating the caller, not just unit-testing the pipeline)

Build only if

  • No hard constraint that justifies a new orchestration framework is evidenced; the companion keeps models and transport out of scope. A requirement for on-device or offline speech is a stage choice, plugged into Pipecat's or LiveKit Agents' provider interface, not a framework rewrite (not TEN's: its license bars hosting on end-user devices). [Inference] (ecosystem/pickup/pickup-voice-agents.md:26 The models themselves (no clean-room STT/TTS/LLM retraining); ecosystem/pickup/_evidence/voice-agents.md:8 plugin tree covers ~40 STT/LLM/TTS providers)
  • License: a fresh read on 2026-09-23 (gh api repos/<owner>/<repo> --jq .license.spdx_id, LICENSE file for TEN, where the API said NOASSERTION) returned BSD-2-Clause for Pipecat, Apache-2.0 for LiveKit Agents and livekit, and MIT for openai-agents-js. TEN's added conditions rule TEN out for on-device or Agora-competing products; they justify choosing another incumbent, not building. [Inference] (stack/METHOD.md:79 Check the license of everything you tell a builder to adopt)

Where FrankenSuite touches this

What we cannot say

Revisit when

  • Pipecat or LiveKit Agents adds a latency regression gate or publishes a time-to-first-audio benchmark.
  • ServiceNow/eva's scenarios are hash-pinned and adopted by a framework's CI.
  • franken_whisper certifies end-to-end listen latency, or franken_tts or franken_whisper appears as a provider plugin in an incumbent framework.

Sources

Starting a build here?

Begin from the pickup companion for Realtime voice agents: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed