Research · Area 05 of 07
Evals, Observability & AgentOps
The hardest question in agentic AI is not “can we build it” but “did it work.” Step-level tracing is largely solved; outcome scoring is not, and the gap between “we have traces” and “we know it worked” is where the open problems sit.
Stoa Labs studies outcome-level evaluation, judge reliability and calibration, adversarial cases, injection resistance in practice, and the operational signals that tell you when trust should contract.
This area is the connective tissue of the lab: every other research area’s hardest problem eventually lands here as a verification question.
01 · Definition
Four segments, one data substrate
Measuring whether agents actually work, watching them run, and operating them safely in production. The area covers four segments sharing one data substrate (traces) but with different maturity.
- Agent evaluation. Measuring whether an agent accomplished a goal, not just whether a model returned a good string. The field cleanly splits this into step-level tracing (largely solved and tooling-supported) and outcome scoring (largely unsolved, requiring domain judgment). Eval-driven development, running the same evaluator in development, as a pre-release gate, and on live traffic, is now standard vocabulary.
- Observability and tracing. Capturing and inspecting runs: spans, tool calls, tokens, latency, cost. The most mature and most commoditized segment.
- AgentOps and infrastructure. The operational lifecycle: sandboxed execution, deployment, reliability, and cost governance. “AgentOps” is now an established term for the DevOps-analog discipline for production agents.
- Security and access control. Prompt-injection defense, agent identity and auth, egress and policy enforcement, sandbox isolation. The fastest-heating, least-settled segment.
02 · Why it matters
Deployment is outrunning verification
Enterprises are deploying agents faster than they can tell whether the agents work.
- Spend has become a board-level line item. Agentic tasks consume 5 to 30 times the tokens of chatbots, and only about 39 percent of organizations can tie AI spend to business outcomes.
- Prompt injection remains the dominant production security failure mode, with no general solution.
- The incumbent tooling was built to monitor LLM calls, not to judge goal-level outcomes.
The gap between “we have traces” and “we know it worked” is where both the research problems and the commercial demand sit.
03 · Open questions
What we are trying to find out
Exploration-mode questions: pursued to learn something, allowed to not pan out.
- Outcome scoring. Whether the agent achieved the goal cannot be recovered by replaying traces; it needs domain-informed success criteria. Can outcome evaluation be systematized per domain without collapsing into bespoke one-offs?
- Judge reliability without gold references. LLM-as-judge degrades sharply on open-ended tasks and carries family, length, and confidence biases. What do calibrated, bias-corrected judges look like, and how cheap can they get (distilled evaluators already score full production traffic at a fraction of frontier-judge cost)?
- Injection resistance in practice. Defenses are mitigations, not guarantees. Which combinations (untrusted-content labeling, tool-policy constraints, trace monitoring) measurably reduce risk on real agent stacks?
- Cost-to-value attribution. Can token spend be attributed to business outcomes well enough to steer agent fleets?
04 · Directions
What this work could become
Candidate development directions, not commitments or year-one programs.
- Outcome-level evaluation methodology (the flagship candidate). Develop success-criteria patterns, reference construction methods, and judge-calibration techniques using Stoa-owned, public, licensed, commissioned, or independently synthetic inputs. Client-specific gold references, expert calibration, and delivered evaluation suites remain client work and do not graduate into Stoa methodology or toolkits.
- Eval-driven development enablement. Standing up the development, gate, and production loop on top of existing platforms rather than competing with them: the methodology and wiring, not another dashboard.
- Judge calibration and anti-bias as a named specialty, including distilled evaluators for full-traffic scoring.
- Agent security assessment. Injection and tool-policy red-teaming, which pairs naturally with eval work since both live in the trace.
05 · Boundaries
What we deliberately avoid
Building another observability dashboard (commoditized, with well-capitalized incumbents and analytics-scale data infrastructure requirements) or a sandbox provider (entrenched and capital-intensive). Differentiation is judgment-heavy eval methodology and security craft, not platform infrastructure.
06 · Landscape · 2026-07
The landscape, July 2026
Benchmark turnover: Terminal-Bench 2.0 and the tau-bench family moved to the frontline, static accuracy is giving way to interactive and policy-adherence evaluation, and METR-style time-horizon framing (frontier agents handle roughly two-hour software tasks at 50 percent success) is the most decision-useful lens for buyers.
Tooling consolidated: Langfuse was acquired by ClickHouse in January 2026, with Braintrust, LangSmith, and Arize rounding out a crowded field converging on feature parity, all built on a data model that treats agents as call sequences rather than goal-level sessions. OpenTelemetry graduated from CNCF in May 2026 but its GenAI semantic conventions remain experimental with no stabilization timeline.
Security went tier-one: OWASP holds prompt injection as the top LLM threat, NIST published agent-specific taxonomy, MCP mandated OAuth 2.1, and a real supply-chain incident (a malicious LiteLLM package) demonstrated the blast radius. AI FinOps went mainstream, with nearly all FinOps teams now managing AI spend.
Landscape notes are dated on purpose. This layer moves monthly, and we treat any claim older than a quarter as due for reverification.
07 · Sources
Selected sources
- Terminal-Bench 2.0 (ICLR 2026); tau-bench family; METR time-horizon framing
- OpenTelemetry GenAI semantic conventions (status: experimental)
- OWASP LLM Top 10; NIST adversarial-ML taxonomy (agent vectors)
- IBM and Red Hat AgentOps definitions; 2026 eval-platform comparisons
Where this work goes
These methods anchor our evaluation and verification services.