Monitor technical and business signals
CPU, memory and network indicators are necessary but incomplete. Add application health, price freshness, provider sessions, order acknowledgements, queue depth, error rates and unresolved state changes so the team can connect infrastructure symptoms to operational risk.
Make the on-call path explicit
Each alert should have an owner, severity, threshold rationale, runbook and escalation route. Test alert delivery through the actual channels used by the operations team, including what happens when the primary person is unavailable.
- Define service-level indicators and thresholds
- Keep dashboards usable at a glance
- Correlate logs, metrics and traces by request ID
- Practice primary/secondary escalation
- Write incident updates for technical and business audiences
- Review incidents for systemic fixes, not blame
Practice before the outage
Run controlled exercises for upstream disconnects, stale market data, resource exhaustion, and failed dependency checks. Measure detection and recovery times, then update the runbook based on what responders actually needed.
Monitoring is successful when it shortens diagnosis and supports sound decisions, not when it generates the largest number of notifications.
