← All agent-stack verdicts

Agent observability and tracing

Adopt and wrap Confidence Medium Group Eval and safety Evidence as of 2026-09-23

Written by VerdictsOrchestration · Reviewed by control-plane pane 2 on 2026-09-23 (separate AI agent sessions; review record)

Bracketed tags are evidence tiers from RULEBOOK.md §1: [Verified] means checked directly, [CI-observed] read from a project’s CI pages, [Maintainer claim] asserted by the project, [External] a dated third-party source, [Inference] our judgment. A gold path:line link opens the cited line. TRL is technology readiness level, scored 1 to 9. CI is continuous integration: automated checks that run on every push. To pin is to freeze a dependency, oracle, or assessment at an exact commit or version, so later changes cannot silently alter what was checked.

Bottom line

Inference, medium confidence: record agent runs with an existing tracing stack (Langfuse, Arize Phoenix or Comet Opik, instrumented through OpenTelemetry packages such as OpenLLMetry or OpenLIT), and do not build your own trace store. Check the license first if you will host tracing for others: Phoenix is under the Elastic License 2.0, which forbids offering it as a hosted service, and Langfuse keeps its enterprise directories under a separate enterprise license. Wrap it, because "OpenTelemetry-compatible" is mostly asserted rather than tested in this category: before you trust an integration, check that the spans it emits carry the attributes you depend on, that streamed calls close their spans, and that your cost numbers use an audited price table.

Adopt, do not rebuild

License checked on 2026-09-23; non-permissive licenses are named in the text.

  • langfuse/langfuse MIT AND LicenseRef-Langfuse-Enterprise: a self-hostable platform for traces, nested spans, prompt and response logging, cost tracking, datasets and evals. License: MIT with conditions: the core is MIT, but code under the ee/, web/src/ee/ and worker/src/ee/ directories needs a Langfuse Enterprise License, so stay out of those paths unless you hold one. Its CI diffs the server-generated API spec against the SDK and audits model prices daily. [Verified] (ecosystem/pickup/_evidence/observability.md:7 trace capture, span hierarchies, prompt/response logging, cost tracking, datasets + evals in one OSS platform (MIT); ecosystem/pickup/_evidence/observability.md:32 sdk-api-spec.yml, openapi-export-check.yml; stack/licenses.tsv:46 have a valid Langfuse Enterprise License; stack/licenses.tsv:46 and/or 'worker/src/ee/' directories)
  • Arize-ai/phoenix Elastic-2.0: observability and evaluation built on OpenTelemetry, with its trace view checked by browser tests. License: source-available under the Elastic License 2.0 (ELv2), which bars providing the software to third parties as a hosted or managed service; its phoenix-otel and phoenix-client packages are Apache-2.0, and phoenix-evals is ELv2. The quoted clause restricts providing it to third parties as a service; it does not mention running it for your own team. [Verified] (ecosystem/pickup/_evidence/observability.md:9 packages include phoenix-otel (OTel-native) and phoenix-evals; ecosystem/pickup/_evidence/observability.md:34 Phoenix runs playwright.yaml for UI trace-view verification; stack/licenses.tsv:6 You may not provide the software to third parties as a hosted or managed service; stack/licenses.tsv:6 packages/phoenix-otel and packages/phoenix-client are Apache-2.0, packages/phoenix-evals is ELv2)
  • comet-ml/opik Apache-2.0: open-source tracing and eval for LLM and agent workflows, with the only scheduled ingestion-load suite found in the category and a separate CI workflow per framework integration. [Verified] (ecosystem/pickup/_evidence/observability.md:8 Comet's OSS tracing/eval platform for LLM and agent workflows; ecosystem/pickup/_evidence/observability.md:25 Scheduled ingestion-load benchmarks against a live local stack; ecosystem/pickup/_evidence/observability.md:40 is the only ingestion-load benchmark found among these repos)
  • traceloop/openllmetry and openlit/openlit Apache-2.0 Apache-2.0: the instrumentation layer, per-provider OpenTelemetry packages that emit spans for model and tool calls; OpenLLMetry tests span attributes against recorded provider responses, and OpenLIT tests that streamed spans close. [Verified] (ecosystem/pickup/_evidence/observability.md:10 per-provider OpenTelemetry instrumentation packages; ecosystem/pickup/_evidence/observability.md:16 OTel-native auto-instrumentation SDK (Python/TS/Go))

Copy these practices

  • Assert span attributes per provider, against recordings: for each instrumented provider, test the semantic-convention attributes of its spans against recorded (cassette) responses, not only that spans exist. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:26 Per-provider OTel semantic-convention compliance tests with VCR cassettes)
  • Streamed calls must close their spans: test that a streamed response ends its span and records token usage. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:27 explicit tests that streaming spans close correctly and carry token-usage attributes)
  • Test the export path, including masking: unit-test batching, context propagation and secret or personal-data masking on the OpenTelemetry export path. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:28 test the OTel export path as a first-class unit surface (masking, batching, context propagation)
  • Named ingestion load shapes on a schedule: a weekly job pushes fixed shapes (100k traces of one span, 250k spans, 1 GB payloads, bursts) through a local stack. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:25 100k traces × 1 span, 250k spans, 1 GB heavy payloads, burst + concurrency shapes)
  • Audit the price table on a schedule: a daily job checks the default model prices that cost numbers are computed from. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:33 cost tracking needs its own scheduled correctness job — prices change under you)
  • Demote stale cost claims: the companion proposes that a cost claim whose price-table audit is more than seven days old drops to inference. Starter kit: Demotion rules executed on schedule. [Inference] (ecosystem/pickup/pickup-observability.md:167 whose price-table audit is older than 7 days is demoted to T3)
  • Check traces in the UI, not only the API: end-to-end tests confirm ingested spans render as complete trees in the product, not merely that ingestion returned 200. Starter kit: none. [Verified] (ecosystem/pickup/_evidence/observability.md:34 verify end-to-end that ingested spans actually render as complete traces in the UI)

Build only if

  • No evidenced constraint rules out the open-source incumbents. The evidenced gap is a neutral, cross-vendor conformance suite for agent span conventions, which the companion proposes building against the pinned OpenTelemetry conventions as a test beside an adopted stack, not as a new tracing platform. [Inference] (ecosystem/pickup/pickup-observability.md:98 No neutral, cross-vendor LLM-agent OTel semconv conformance suite was)
  • The licenses do not justify building either. If you will run tracing as a service for customers, Phoenix's Elastic License 2.0 rules out its server, but Opik (Apache-2.0), Langfuse's MIT core outside its ee/ directories, and Phoenix's own Apache-2.0 phoenix-otel instrumentation package remain available, so switch incumbents rather than write a trace store. [Verified] (stack/licenses.tsv:15 comet-ml/opik Apache-2.0 permissive; stack/licenses.tsv:46 All content that resides under the 'ee/'; stack/licenses.tsv:6 packages/phoenix-otel and packages/phoenix-client are Apache-2.0)

Where FrankenSuite touches this

What we cannot say

  • Whether any platform's OpenTelemetry ingestion interoperates: only OpenLLMetry was seen with explicit convention-compliance tests, and the pack treats the rest as marketing until a test file is seen (ecosystem/pickup/_evidence/observability.md:39 is asserted more than proven. Only OpenLLMetry has explicit semconv-compliance tests).
  • What the compliance and lifecycle tests assert inside: file names and workflow contents were read, individual test assertions were not (ecosystem/pickup/_evidence/observability.md:42 per-symbol claims inside test files were not verified).
  • Whether load testing is a norm: Opik's suite is the only one found, so copying it copies one example (ecosystem/pickup/_evidence/observability.md:40 is the only ingestion-load benchmark found among these repos).
  • The hosted commercial core (LangSmith and similar) could not be studied; only client SDKs are open (ecosystem/pickup/_evidence/observability.md:38 is closed-source hosted; only its client SDKs (langchain-ai/langsmith-sdk) are open).
  • Langtrace is stale and is not a model to copy (ecosystem/pickup/_evidence/observability.md:41 Langtrace is stale (last push 2025-11-17)).
  • Stars show which projects are used, not which capture spans correctly.

Revisit when

  • A neutral OpenTelemetry conformance suite for agent spans appears, or a platform publishes one: "OTel-compatible" claims can then be checked instead of trusted.
  • A second project publishes a scheduled ingestion-load benchmark, or a proxy-versus-SDK loss comparison.
  • The test bodies behind the compliance and lifecycle files are read at a pinned commit, which would move those practices from file existence to checked behavior.

Sources

Starting a build here?

Begin from the pickup companion for Agent observability and tracing: it carries the charter seed, the oracle candidates and how to verify them, and the first claims to register, so a new project starts from evidence instead of from scratch.

Help fix it

Each opens a GitHub issue form with this page’s address filled in. You need a GitHub account.

Made by Joshua Nowak (ZestStream) with AI coding agents. Not affiliated with or funded by Jeffrey Emanuel; Joshua uses his public tools and is a paying subscriber to jeffreys-skills.md.

Source on GitHub MIT licensed