An enterprise should be able to change agent frameworks without rebuilding its entire quality system.
Agent evaluation has had an awkward coupling problem. The team selects an SDK or orchestration framework, then adopts tracing and evaluation tools that understand that framework’s event model. Six months later a second team chooses a different stack, and quality evidence fragments.
AWS’s August 2026 AgentCore Evaluations announcement is interesting less because of the evaluation service itself than because of its interoperability claim: if an agent emits OpenTelemetry telemetry, the service can evaluate agents built with multiple frameworks, including LangGraph, LlamaIndex, OpenAI Agents SDK, Google ADK, Claude Agent SDK and Strands.
That points to a useful enterprise principle:
make the trace the contract, not the framework.
Evaluation needs the full trajectory
An agent cannot be evaluated reliably from final text alone.
A good-looking answer may have:
- used the wrong source;
- called an unauthorised tool;
- spent ten times the expected budget;
- retried a dangerous action;
- ignored an approval requirement;
- taken a path that only happened to succeed.
Conversely, a failed final task can reveal a correctly controlled system that escalated when evidence was insufficient.
Evaluation needs the trajectory:
goal → context → reasoning events → tool calls → observations → decisions → action → verification
Not every element needs to expose private chain-of-thought. Operational events, inputs, outputs, policy decisions and timing are enough to evaluate much of the system behaviour.
A common telemetry layer reduces switching cost
If agent traces use common semantic conventions, an organisation can send them to:
- operational monitoring;
- security analytics;
- cost analytics;
- evaluation pipelines;
- regression tests;
- incident review;
- compliance evidence stores.
The orchestration framework can change while the downstream controls remain stable.
This is the same architectural advantage enterprises already gained by separating application instrumentation from individual observability vendors.
The hard part is semantic consistency
OpenTelemetry provides plumbing, not a universal definition of “good agent behaviour”.
Teams still need conventions for:
- agent and workflow identity;
- task outcome;
- tool name and business capability;
- policy decision;
- human approval;
- model destination;
- evidence source;
- cost;
- latency;
- retry;
- error class;
- accepted/rejected outcome.
Without consistent semantics, traces become technically interoperable and analytically useless.
A platform team should define an internal agent telemetry profile.
Evaluation should run at three timescales
Pre-release
Golden tasks, adversarial cases, policy tests and regression suites.
Continuous production
Sampled scoring, drift monitoring, latency, cost, tool failures and safety signals.
Incident-driven
Deep trajectory replay after a bad outcome or near miss.
All three should use the same fundamental trace IDs so evidence can be connected.
Separate evaluation from the model judge
LLM-as-judge is useful, but not sufficient.
Agent quality also includes deterministic checks:
- did the correct tool execute?
- was approval present?
- did the write match the approved amount?
- was the source fresh?
- did the agent stay under cost?
- did a transaction actually commit?
- was the customer case reopened later?
The strongest evaluation combines model-based scoring with system facts and eventual business outcomes.
Portability is strategic
Frameworks and models will keep changing quickly. Enterprise evaluation needs a slower-moving foundation.
If an organisation can preserve:
- a task corpus;
- expected controls;
- trace semantics;
- acceptance thresholds;
- historical benchmark runs;
then it can compare new architectures against the same definition of success.
That turns model experimentation into governed engineering.
The Lexdatum view
OpenTelemetry has a plausible path to becoming the neutral transport layer for agent evidence.
The standard alone will not solve evaluation. But it can give enterprises something valuable: a stable contract between rapidly changing agent runtimes and the slower systems that decide whether those agents are reliable enough to operate.
Own the evaluation criteria. Standardise the trace. Let frameworks compete underneath.
Sources
Research note: This analysis is based on public information available on 1 September 2026. Product capabilities, prices and regulatory interpretations can change. Vendor-reported claims are identified as such. This is not legal, security, financial or investment advice.