Observability
Production monitoring should show user-facing health, service dependencies, AI and retrieval behavior, tool execution, background processing, security events, and cost or quota pressure.
Signals
| Signal | Examples |
|---|---|
| Availability | Ingress and API health, synthetic checks, successful authentication |
| Performance | Request latency, model latency, retrieval latency, tool duration |
| Reliability | Error rate, timeouts, retries, queue depth, failed ingestion, dead letters |
| Capacity | CPU, memory, replicas, database load, search and model quota |
| AI operations | Tokens, model deployment, safety blocks, evaluation failures |
| Security | Failed access, policy changes, unusual tool use, secret-store access |
| Business operations | Tool success, approval backlog, workflow completion, active users |
Correlation
Propagate a correlation identifier through ingress, API, workflow, retrieval, tool execution, and external provider calls where possible. Keep timestamps synchronized and record the environment, service, deployment version, agent, conversation, workflow, and tool execution identifiers needed for investigation.
Alert Design
Every alert needs an owner, severity, actionable threshold, notification route, runbook, and escalation path. Start with failures that affect users or data, then add capacity and anomaly alerts. Avoid alerts that cannot lead to a clear action.
Define dashboards and alerts for:
- availability and elevated error rate,
- database, storage, search, and secret-store failures,
- ingestion backlog and repeated processing errors,
- model throttling and quota exhaustion,
- failed or unusually long tool executions,
- approval queues that stop business workflows,
- unexpected token or cost growth,
- backup and recovery-job failures.
Pair infrastructure telemetry with AI Auditability.