Does the evidence support the diagnosis?
Check evidence completeness, taxonomy coverage, confidence, and conflicts.
Project note · 2025–26
Agent Failure Diagnostics
TB2 Diagnostics
Turning long agent-failure traces into evidence-backed error types and repair priorities, reducing the cost of manual review.
Why I started
A team that only sees “task failed” cannot tell whether to improve planning, tool use, environment understanding, or execution. Manual trace review averaged 770 seconds and reviewers could disagree, making evaluation an unstable guide for iteration.
The real question
How can traces become reviewable, aggregatable diagnoses that guide repair—without letting a model freely invent causes?
Choices I made
Every action should have a reason.
Manually analyze 46 failed traces before building an L1/L2 taxonomy.
Categories need to come from observed failures; model-invented labels drift and make batches difficult to compare.
Require every judgment to cite logs, commands, errors, or timing evidence.
An evidence chain keeps diagnoses reviewable and prevents explanations that sound plausible but are unsupported by the trace.
Return low-confidence and conflicting cases to human arbitration.
Automation is best used for triage and consistent formatting; human fallback controls misclassification risk and expands taxonomy boundaries.
How the system moves
How an agent failure becomes an actionable repair priority
Start / inputProcessDecisionOutcome
- Runtime
Capture the failure trace
Collect commands, logs, errors, timing, and model decisions.
- Pipeline
Extract evidence and turning points
Separate the final symptom, upstream anomaly, and root cause.
- LLM judge
Attribute with a fixed taxonomy
Return L1/L2 label, cited evidence, explanation, and confidence.
Store a structured diagnosis
Use it for batch statistics, issue clustering, and repair ranking.
Send to human arbitration
Review the original trace and confirm or correct the root cause.
Expand taxonomy boundaries
Add uncovered cases to the next taxonomy iteration.
Engineering loop
Turn diagnosis into repair and regression tests
Prioritize by frequency and impact, rerun the task after repair, and feed the new trace back into diagnosis.
Business meaningThe business value of diagnosis is removing guesswork from long traces: confident cases move quickly, disputed cases go to people, and every result points to the next repair cycle.
What I made
From judgment to something that runs.
Attribute
Locate turning points in traces and separate visible errors from root causes.
Judge
Require a fixed taxonomy and evidence format with cause, citation, and confidence.
Arbitrate
Sample high-confidence outputs and manually review conflicts or uncovered cases.
What the work left behind
- 92% agreement with human review
- 91.7% label coverage
- 0.89 average confidence
- Diagnosis time reduced from 770s to 354s
Results come from a specific agent and task set. The taxonomy still needs testing across new models, toolchains, and out-of-distribution failures.
Connect diagnoses to testable interventions, then measure whether accurate diagnosis actually improves task success.