Bottom line
Inference, medium confidence: adopt an existing realtime voice-agent framework (Pipecat or LiveKit Agents, on a maintained media transport) and wrap it with a latency regression gate of your own. The frameworks already handle speech-to-text, language model and text-to-speech stages, turn-taking and interruptions, and LiveKit Agents tests them with fake providers and a virtual clock; but no latency regression gate was found in either framework, and interruption thresholds have no published ground truth. Keep the fake-provider test surface, measure time-to-first-audio on your own harness with A/A baselines, and block releases on regressions.
Adopt, do not rebuild
License checked on 2026-09-23; non-permissive licenses are named in the text.
- pipecat-ai/pipecat BSD-2-Clause: an open-source voice-agent pipeline framework maintained by Daily and the community, with latency metric classes and their own unit tests. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:7
maintained by Daily and the community; pushed same-day as verification
; ecosystem/pickup/_evidence/voice-agents.md:19Latency metric classes with their own unit tests
) - livekit/agents Apache-2.0: LiveKit's realtime voice-agent framework with a plugin tree of about forty speech and model providers and fake doubles for every pipeline stage. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:8
LiveKit's framework for building realtime voice AI agents
; ecosystem/pickup/_evidence/voice-agents.md:8plugin tree covers ~40 STT/LLM/TTS providers
; ecosystem/pickup/_evidence/voice-agents.md:18Fake/mock service doubles for every pipeline stage
) - livekit/livekit Apache-2.0: the realtime media transport (SFU, a server that forwards audio and video streams) the agent frameworks ride on; the companion keeps transport internals out of scope for this type. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:12
Underlying realtime media transport (SFU) the agent frameworks ride on
; ecosystem/pickup/pickup-voice-agents.md:27the media transport/SFU internals (use a pinned incumbent as the transport,
) - TEN-framework/ten-framework LicenseRef-TEN-Open-Source-License: an alternative framework with a multi-OS CI matrix. License: Apache-2.0 with added conditions: no hosting on end-user devices (including mobile) and no deployment that competes with Agora's offerings. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:9
heavy multi-OS CI matrix (linux/mac/win × arm64/x64)
; fresh read 2026-09-23,gh api repos/TEN-framework/ten-framework/license --jq '.content|@base64d') - openai/openai-agents-js MIT: a first-party vendor framework for voice agents in JavaScript runtimes. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:10
signal that a first-party vendor ships voice-agent orchestration
)
Copy these practices
- A fake for every stage: livekit/agents has fake speech-to-text, model, text-to-speech, VAD, turn-detector and session doubles, so full agent sessions run in tests without API keys or network. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:18
Tests run deterministic agent sessions without API keys or network.
) - Latency metrics as tested classes: Pipecat detects time-to-first-audio from speech onset in PCM buffers and unit-tests that metric, alongside time-to-first-byte. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:19
TTFA (time-to-first-audio) detected via RMS speech-onset detector on PCM buffers
) - Turn-taking on a virtual clock: a fake clock keeps turn-taking and latency assertions from flaking on real-time scheduling. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:22
fake clock so turn-taking/latency assertions don't flake on real-time scheduling
) - A dedicated interruption test surface: false-interruption resume, protected speech, hold windows and overlapping speech each get tests. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:21
Dedicated interruption / turn-taking test surface
) - Live-provider tests behind secrets, off the unit path: a separate workflow runs against real realtime APIs, triggered only by relevant path changes and skipped on forks. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:23
path-triggered only by realtime-code changes, skipped on forks
) - Network faults injected in tests: livekit/agents drives a toxiproxy container to degrade connections during tests. Starter kit: Acceptance shape: positive observable + planted negative + no-claim line (provisional). [Verified] (ecosystem/pickup/_evidence/voice-agents.md:29
Fault injection / network degradation in tests
) - Simulate the caller: ServiceNow/eva evaluates voice agents with bot-to-bot calls, a simulated user and perturbations such as noise and degraded connections. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/voice-agents.md:26
copy the pattern of simulating the caller, not just unit-testing the pipeline
)
Build only if
- No hard constraint that justifies a new orchestration framework is evidenced; the companion keeps models and transport out of scope. A requirement for on-device or offline speech is a stage choice, plugged into Pipecat's or LiveKit Agents' provider interface, not a framework rewrite (not TEN's: its license bars hosting on end-user devices). [Inference] (ecosystem/pickup/pickup-voice-agents.md:26
The models themselves (no clean-room STT/TTS/LLM retraining)
; ecosystem/pickup/_evidence/voice-agents.md:8plugin tree covers ~40 STT/LLM/TTS providers
) - License: a fresh read on 2026-09-23 (
gh api repos/<owner>/<repo> --jq .license.spdx_id, LICENSE file for TEN, where the API said NOASSERTION) returned BSD-2-Clause for Pipecat, Apache-2.0 for LiveKit Agents and livekit, and MIT for openai-agents-js. TEN's added conditions rule TEN out for on-device or Agora-competing products; they justify choosing another incumbent, not building. [Inference] (stack/METHOD.md:79Check the license of everything you tell a builder to adopt
)
Where FrankenSuite touches this
- franken_tts is a shipped pure-Rust CPU-only runtime for one text-to-speech model, with a streaming conversation layer. It is a stage an orchestration framework would call, not a framework; its faster-than-real-time figure is a maintainer claim that drops below real time under load. TRL 7, Explore; license: MIT with a rider barring OpenAI, Anthropic and anyone acting for them. [Verified] (packets/franken_tts-assessment.md:9
a genuine, shipped, pure-Rust CPU-only runtime for one model
; packets/franken_tts-assessment.md:9a streaming conversation layer
; packets/franken_tts-assessment.md:72heavily-loaded runs measured 0.66–1.05×, i.e. sub-real-time under load
; packets/franken_tts-assessment.md:16License (MIT + OpenAI/Anthropic rider, read verbatim)
; synthesis/00-overview.md:85franken_tts | 7 | Explore | Rider
) - franken_whisper is an agent-first Rust speech-to-text stack with a streaming listen mode whose end-to-end latency certification is still open. Its first listen latency campaign published the numbers but refused a cross-arm comparison because the A/A controls fell outside their band, which is the comparison discipline the voice frameworks above lack; it is applied in the project's ledger, not in CI. TRL 5–6, Explore; license: MIT with a rider barring OpenAI, Anthropic and anyone acting for them. [Verified] (packets/franken_whisper-assessment.md:9
an agent-first Rust speech-to-text orchestration stack
; packets/franken_whisper-assessment.md:80end-to-end latency certification still open
; packets/franken_whisper-assessment.md:127the numbers (TTFT, commit lag, WER per fixture) are published, the comparison is refused
; synthesis/00-overview.md:58C4 no test CI / deploy-only
; packets/franken_whisper-assessment.md:15License (MIT + OpenAI/Anthropic rider, read verbatim)
; synthesis/00-overview.md:86franken_whisper | 5–6 | Explore | Rider | 1 | no | C4
)
What we cannot say
- How fast any of these frameworks is: latency is tested, not gated, and no time-to-first-audio benchmark dashboard or regression gate was found (ecosystem/pickup/_evidence/voice-agents.md:36
Latency measurement is thinner than the category deserves
; ecosystem/pickup/_evidence/voice-agents.md:36I did not find a published time-to-first-audio benchmark dashboard or regression gate in either repo
; ecosystem/pickup/_evidence/voice-agents.md:36Latency is tested, not gated.
). - What good interruption behavior is: thresholds have no incumbent-published ground truth (ecosystem/pickup/pickup-voice-agents.md:428
(false-interruption rate, resume latency bounds) have no incumbent-published
). - ServiceNow/eva is a research-grade methodology reference, and its dataset revision was not hash-verified (ecosystem/pickup/_evidence/voice-agents.md:37
ServiceNow/eva is research-grade
; ecosystem/pickup/pickup-voice-agents.md:417ServiceNow/eva's HuggingFace dataset revision has not been
). - Whether a pinned incumbent can be run side by side for comparative claims, given license, API cost and keys (ecosystem/pickup/pickup-voice-agents.md:421
Whether the pinned incumbent (livekit/agents or pipecat) can
). - The live-provider workflow was read from its file, not observed running; provider pricing was not in the pack and licenses come from the author's fresh read; stars measure popularity. vocode was dropped because its repository 404s (ecosystem/pickup/_evidence/voice-agents.md:35
vocode-ai/vocode 404s and was dropped
).
Revisit when
- Pipecat or LiveKit Agents adds a latency regression gate or publishes a time-to-first-audio benchmark.
- ServiceNow/eva's scenarios are hash-pinned and adopted by a framework's CI.
- franken_whisper certifies end-to-end listen latency, or franken_tts or franken_whisper appears as a provider plugin in an incumbent framework.
Sources
- Pickup companion: ecosystem/pickup/pickup-voice-agents.md, the charter seed, oracle candidates, and first claim registry for this area.
- Evidence pack: ecosystem/pickup/_evidence/voice-agents.md, the repositories checked against the GitHub API, with dates.
- Verdict source: stack/voice-agents.md, written under stack/METHOD.md.
Starting a build here?
Begin from the pickup companion for Realtime voice agents: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.