Do you feel that your monitoring system is a complicated ball of contrived tools tenuously strung together, and ignored by everyone except the team holding the strings? If you are building or designing your next monitoring system, you should read about the seven habits exhibited by the most successful monitoring systems in the world today.
1. It’s About the Data
Awesome monitoring tools treat metrics and telemetry data as first class citizens, and go out of their way to make them easy to export.
2. Use Monitoring for Feedback
Great monitoring systems are driven by purpose. They are designed to provide operational feedback about production systems to people who understand how those systems work.
3. Alert on what you Draw
To avoid confusion and safeguard productivity, your telemetry data must always agree with your alerting.
4. Standardize Processing, but Liberate Collection
Awesome monitoring systems standardize on a single means of metrics processing, storage, analysis, and visualization, but they declare open season on data collectors. Every engineer should be free to implement whatever means she deems appropriate to monitor the services she’s responsible for.
5. Let the Users Define Their Own Interactions
Forget the “single pane of glass”. It is better to have people who know the systems curating meaningful collections of metrics, than it is to have an externally imposed uber-view.
6. Include Monitoring in the Software Development Life-cycle
Monitoring is unit testing for operations. It is the best if not the only way to verify that your design and engineering assumptions bear out in production.
7. Evolve Rather Than Transform
Healthy monitoring systems stay relevant because they’re constantly being iterated by the engineers that rely on them to solve their everyday problems.
