Instrumenting a GenAI framework for OpenTelemetry means locating usable extension points, mapping the framework's operations onto semantic conventions, verifying the telemetry it produces, and keeping all of that working as the framework evolves. Because every new framework repeats the cycle, the engineering cost accumulates: the post notes that even an experienced contributor can spend days on a single framework, and independent implementations tend to diverge on attribute choices, units, and span boundaries, which makes the resulting telemetry harder to query uniformly.

The authors have been evaluating whether a team of agents can take over much of that labor. In their setup, agents examine a framework, draft an instrumentation plan, implement it against the OpenTelemetry Semantic Conventions for GenAI, and check the emitted telemetry with OpenTelemetry Weaver, while engineers review the plan and the final pull request.

The components

Three pieces make up the system: the GenAI semantic conventions, Weaver, and an orchestrator.

  • The GenAI semantic conventions act as the contract, specifying span and attribute names, units, and which fields carry the model, the tokens, the messages, and the tool calls. Generated plugins are expected to follow those definitions.
  • Weaver is the validator. Its live-check capability opens an OTLP listener, feeds the telemetry emitted by a running instrumentation through a set of advisors, and compares it against the resolved semantic-convention registry. Built-in advisors cover fundamentals such as missing_attribute and type_mismatch; the default OTel Rego policies add naming and formatting rules, and custom ones can be supplied with --advice-policies. Every finding carries a level — violation, improvement, or information — and Weaver exits non-zero when the report contains a violation.
  • The orchestrator schedules the agents and posts the resulting pull request. The team runs a self-hosted multi-agent task platform, but notes that an orchestrator only needs to check out a repository, run the agents, and open a pull request.

Architecture

Pointed at a framework repository — LangChain is given as an example — the system produces a pull request containing generated instrumentation.

flowchart LR
    repo[("Target framework<br/>repository")] --> team
    subgraph team["Agent team"]
        direction TB
        lead["Team Lead"]
        research["Research"]
        plan["Plan Review"]
        code["Coding"]
        review["Code Review<br/>& Tests"]
        e2e["E2E & Deploy"]
        observe["Observation"]
        lead --- research
        lead --- plan
        lead --- code
        lead --- review
        lead --- e2e
        lead --- observe
    end
    team --> pr["Pull request with<br/>generated instrumentation"]
    semconv["GenAI semantic<br/>conventions"] -.->|"target spec"| team
    weaver["Weaver<br/>live-check"] -.->|"conformance signal"| team

The orchestrator divides the work among specialized agents:

  • Team Lead Agent initializes the run, sequences the other agents, and owns the loop's state.
  • Instrumentation Research Agent clones the target framework, studies its architecture, and looks first for public extension points such as hooks and callbacks. After surveying existing implementations, it produces a research report listing candidate instrumentation strategies, considering method wrapping only when those extension points cannot observe the required lifecycle.
  • Plan Review Agent scores the candidates on complexity, maintainability, and coverage, selects the best one, and writes a fine-grained execution-plan.md specifying which extension points to use and where each attribute should come from. The authors describe this as essentially a spec-generation step and the natural point for a human to review.
  • Coding Agent implements execution-plan.md against the GenAI semantic conventions, building on the shared GenAI utilities.
  • Code Review & Test Validation Agent reviews the generated code, writes unit tests, and iterates with the Coding Agent until reviews are addressed and tests pass.
  • E2E Generation & Deployment Agent runs in parallel with coding, preparing the integration-test harness so it is ready when the plugin is.
  • Observation Agent runs the harness in Kubernetes, captures OTLP output, hands it to Weaver for semantic validation, and watches the workload over time for memory leaks and other runtime regressions.

Loop engineering

The Team Lead Agent schedules each step, and other agents return control to it whenever the plan needs to change.

flowchart TB
    research["1. Instrumentation research<br/><i>clone, study, propose strategies</i>"]
    plan["2. Plan review & scoring<br/><i>produces execution-plan.md</i>"]
    code["3. Coding<br/><i>implements against semconv</i>"]
    tests["4. Code review & unit tests"]
    e2e["5. E2E integration test<br/>on Kubernetes"]
    check["6. Weaver live-check<br/><i>semantic validation</i>"]
    soak["7. Soak & runtime watch<br/><i>memory, leaks, regressions</i>"]
    pr["Pull request"]
    research --> plan --> code --> tests --> e2e --> check
    check -->|"violation"| code
    check -->|"clean"| soak
    soak -->|"regression"| code
    soak -->|"stable"| pr

After code review and unit tests, the framework's examples are packaged into container images, instrumented, and run in a test Kubernetes cluster while Weaver inspects the emitted OTLP data. A violation routes the work back to the Coding Agent; a clean run proceeds to a soak test for memory leaks and other runtime regressions. An engineer can interrupt the Team Lead to change the plan or repeat a step.

What the generated instrumentation looks like

Two choices shape the generated code.

First, execution-plan.md fixes the instrumentation points before any code is written, and the Coding Agent is not told to wrap methods — the research stage prefers public hooks and callbacks when they cover the required lifecycle. The OpenAI Agents SDK, for instance, exposes lifecycle hooks, so that strategy takes priority there. The CrewAI plugin covered in the post supports versions back to 0.80, whose public callbacks did not expose the complete start/end lifecycle needed for these spans, so its plan selected method wrapping; the authors note that newer CrewAI releases provide a broader event-listener API that should be evaluated first for those versions. The wrapping plan appears as an explicit, reviewable list of targets:

_CREWAI_UNINSTRUMENT_TARGETS = (
    ("crewai.crew", "Crew", "kickoff"),
    ("crewai.crew", "Crew", "kickoff_async"),
    ("crewai.flow.flow", "Flow", "kickoff"),
    ("crewai.flow.flow", "Flow", "kickoff_async"),
    ("crewai.agent", "Agent", "execute_task"),
    ("crewai.task", "Task", "execute_sync"),
    ("crewai.tools.tool_usage", "ToolUsage", "_use"),
)

Reviewers can check those targets against the framework's API.

Second, the agents map framework objects onto shared GenAI utilities instead of writing attribute names themselves. opentelemetry-util-genai now lives in the OpenTelemetry Python GenAI repository under util/opentelemetry-util-genai, and its handler.invoke_local_agent() method creates an AgentInvocation for an agent running in the same process.

A minimal wrapper adapting the CrewAI example to the upstream API — and not an excerpt from the original generated plugin — looks like this, where wrapped is the original bound Agent.execute_task method, instance is the CrewAI agent, and args and kwargs are the method's arguments (wrapper registration and SDK configuration are omitted):

from opentelemetry.util.genai.handler import get_telemetry_handler
def wrap_execute_task(wrapped, instance, args, kwargs):
    handler = get_telemetry_handler()
    with handler.invoke_local_agent(agent_name=instance.role) as invocation:
        if invocation.should_capture_content:
            invocation.agent_description = instance.goal
        return wrapped(*args, **kwargs)

The utility starts an INTERNAL span, makes it current during the call, and ends it when the context manager exits. If the wrapped method raises an exception, the utility records the failure and lets the exception propagate. The wrapper checks should_capture_content before copying the agent's goal, which may contain user-provided instructions.

Shared utilities handle attribute names and telemetry lifecycle, but the instrumentation still has to map the framework's data correctly; the team validates those mappings with tests and human review alongside Weaver's registry checks.

The original generated plugins used the team's extensions to the earlier utility package in alibaba/loongsuite-python, including the fork-specific InvokeAgentInvocation type. The example above targets the upstream API so readers can start with OpenTelemetry's utilities, and migrating the plugins to those APIs is part of the upstreaming work described below.

Verifying the output with Weaver

Weaver compares emitted OTLP data with the resolved semantic-convention registry, but the Observation Agent does not call it directly. It drives the run through a conformance runner, which starts the scenario, points it at an OTLP endpoint, and hands what it emits to weaver registry live-check. A public example, run from the root of the semantic-conventions-conformance repository, is:

otel-conformance scenarios/gen-ai/python/crewai/openinference

Its conformance.yaml names the runner and the instrumentation:

runner: genai-conformance
instrumented_library: crewai
instrumentation_library: openinference-instrumentation-crewai

The post stresses that this public scenario demonstrates the runner with OpenInference, not the team's LoongSuite plugin, and that the genai-conformance wrapper supplies the semantic-convention registry pin and advice policies so repeated runs use the same definitions.

Each scenario then runs under its own live-check, and a violation fails the run. That is the default; --report-only is the opt-out, downgrading semantic findings to warnings while still failing on a scenario that crashed or produced nothing to measure.

In a separate run with the team's LoongSuite plugin, a sample entity came back with this finding:

{
  "live_check_result": {
    "all_advice": [
      {
        "id": "span_status_ok_set_by_instrumentation",
        "context": { "status_code": "ok" },
        "message": "Span 'execute_tool ping' has status.code='ok'; instrumentations must leave status UNSET on success (OK is reserved for application code).",
        "level": "violation",
        "signal_type": "span",
        "signal_name": "execute_tool ping"
      }
    ],
    "highest_advice_level": "violation"
  }
}

The runner's domain-agnostic advice policy flags an instrumentation that sets span status to OK — the Coding Agent had done that on a tool span — and Weaver's non-zero exit code sent the finding back for a fix.

The same runner also produces a coverage matrix showing which registry-defined attributes a plugin emits and which it misses. Coverage is measurement rather than gating, so those runs use --report-only: a missing attribute is recorded, not a build break. The matrix shows coverage for 29 attributes across 14 LoongSuite Python instrumentations, and the community live conformance matrix presents the same signal across public scenarios, though it does not currently include the LoongSuite plugins.

Separately, Weaver live-check reports attributes a plugin emitted that the registry does not define as missing_attribute violations. When the team temporarily accepts a known violation, it lists the finding id and the reason in expected_violations; the run also fails if Weaver stops reporting that violation, signaling that the entry is stale and should be removed. Coverage counts only registry-defined attributes, so custom extensions do not raise a plugin's score, and if several frameworks need the same attribute that the registry does not define, the team proposes adding it to the GenAI semantic conventions.

The Coding Agent had added gen_ai.crewai.* attributes to the CrewAI plugin for details absent from the GenAI semantic conventions; because OpenTelemetry's naming guidance warns that this prefix could clash with future standard attributes, the authors say future prompts should require agents to emit only registry-defined GenAI attributes.

What has shipped so far

The system has driven end-to-end instrumentation development for a range of GenAI frameworks, agent SDKs, and agent benchmarks. The generated plugins live in the instrumentation-loongsuite packages in Loongsuite Python, Alibaba's Apache-2.0 distribution of the OpenTelemetry Python instrumentation, and each was reviewed by a human at the pull-request stage. They currently cover frameworks and SDKs including CrewAI, LangChain, LangGraph, LiteLLM, AutoGen, AgentScope, Dify, Google ADK, the Claude Agent SDK, the Microsoft Agent Framework, MCP, and mem0, alongside agent benchmarks such as BFCL-v4, WebArena, and MiniSWEAgent. Conformance checks help keep their telemetry consistent across frameworks.

The team is also working to bring this instrumentation to the OpenTelemetry Python GenAI project; the discussion is open in opentelemetry-python-genai#185, with the first instrumentation pull requests now under review.

What did not work

The first version had no Weaver check: a reviewing model judged whether the telemetry followed the spec, but it sometimes approved output that violated the conventions. The team replaced that judgment with a registry comparison and a non-zero exit code for violations.

The Claude Agent SDK posed a different limit. Its Python package wraps the closed-source Claude Code CLI, where the agent loop, tool dispatch, and model calls happen. The plugin can report what crosses the process boundary but cannot capture details inside the CLI, and research and planning cannot recover data that the framework does not expose.

Planning still needs framework expertise as well: the first execution-plan.md often misses details of the framework's lifecycle, so an engineer familiar with it reviews the plan before implementation. According to the authors, the agents are more reliable at implementing a reviewed plan than at choosing the instrumentation strategy alone.

Where human review matters

Semantic conventions define attribute names and span kinds, but mapping a framework's operations to them still requires judgment. Some values have a direct source: gen_ai.tool.name and gen_ai.tool.call.id come from the tool-call object, while gen_ai.usage.input_tokens and gen_ai.usage.output_tokens come from the provider's usage data. Other mappings need more judgment — CrewAI exposes crews, tasks, agents, and flows, and the instrumentation must decide which of their operations represent an invoke_agent operation and which need a different span, a decision that requires human review.

The post closes with three invitations for framework maintainers and contributors: file an issue on the GenAI semantic conventions repository when spotting a gap the agents are likely to hit, join the GenAI Semantic Conventions and Instrumentation SIG discussions to help shape what gets standardized next, and contribute to OpenTelemetry Python GenAI as the generated instrumentations move upstream.