OBSERVABILITY
Understand evaluation metrics
Interpret automated checks, missing scores, and per-agent comparisons.
Overview
Metrics evaluate available spans and show check results in Observe. Request recording must be enabled to evaluate captured inference content or import traces from your instrumentation. Metrics help locate behavior to review. They do not establish that an entire application is safe or correct, and a missing score does not mean a pass.
| Category | Included checks |
|---|---|
| Tool use | Tool-call accuracy, repeated tool calls, tool errors, silent tool errors, and tool-call grounding. |
| Reliability | Failed output and repeating output. |
| Leakage | System-prompt, thinking-markup, raw tool-call, credential, and personal-data leakage checks. |
| Safety | Prompt-injection heuristic and jailbreak judgment. |
| Instruction following | Whether output follows the supplied task, constraints, and format. |
| Context accuracy | Whether claims are supported by the supplied long context. |
The catalog contains 11 system checks and 5 LLM-judged checks. System checks use rules over available data; they can produce false positives and miss problems outside their patterns. LLM checks depend on configured evaluation providers/judge models and the evidence present in spans. A judge result is a model assessment, not a guarantee.
Scores use a 0–100 scale, with higher scores representing better outcomes. Binary checks usually produce 0 or 100; some judgments produce a partial score of 50. For completed (ok) metrics, the detail chip shows scores below 50 as Fail, 50 as Partial, and above 50 as Pass. The compact pass count includes completed scores of 50 or greater, so open the details to distinguish partial results from full passes. Category averages summarize completed scores, so compare their sample counts as well as the averages.
Check each metric's status before interpreting its score. Work can be pending, unavailable, skipped, or errored when it lacks required evidence or evaluation configuration. The long-context accuracy judgment applies only to sufficiently large prompts (at least 20,000 tokens for this check). A metric without suitable input cannot establish that an issue is absent.
Use Account → Metrics to review enabled checks and evaluation-provider configuration where your role permits it. LLM-based evaluation can involve an external provider and associated processing/costs. Repeated judge failures can disable an affected metric; inspect errors and settings before assuming evaluation is still active.
To investigate a flagged score, find the affected trace/span and compare the check's evidence against the actual task. Fix the application or instrumentation, then compare representative new runs. The observation MCP reports metric keys, statuses, scores, and safe metadata; detailed rationales and span content remain outside its response surface. Metric results are scheduled for deletion after 30 days.