Building an Evidence Model

Problem

Raw telemetry is ambiguous. A spike in error rate is evidence of something — but not yet evidence of a specific root cause or customer impact.

Current Practice

Observability tools store and query signals. Investigation conclusions are assembled outside the platform in runbooks, chat threads, and engineer notebooks.

Why It Breaks Down

Without a shared evidence model, teams cannot replay reasoning, compare hypotheses, or trust automated investigation output.

Architecture

An evidence model qualifies signals with context, links related observations, and preserves provenance so every conclusion traces back to source data.

Tradeoffs

Structured evidence improves trust and automation but adds ingestion and normalization cost. Start with high-signal evidence types before expanding taxonomy.

Implementation

Define evidence types (signal, change, relationship, impact), attach resource identity and timestamps, and expose inspectable evidence cards in investigation workflows.

Further Reading

• OpenTelemetry data model
• SRE workbook — incident response documentation
• Observability engineering (O'Reilly)