Recently, I ran into an issue where a misconfiguration lead to data not being pushed to InfluxDb for an extended period of time. It sucked and I wish I would’ve found out about it earlier. Here, I’ll cover how you can use Kapacitor to receive an alert if InfluxDb has not received data for an extended period of time.
In order to receive an alert when InfluxDb is not receiving data, you should be using the
deadman method. Here’s a description from the Kapacitor documentation
Helper function for creating an alert on low throughput, a.k.a. deadman’s switch.
Typically, you would call
deadman on a
The TICKScript to get an alert if there was no data for the
redis measurement for more than 5 minutes would look like this.
var data = stream |from() .database('telegraf') .retentionPolicy('autogen') .measurement('redis') data |deadman(0.0, 5m) .stateChangesOnly() .log('/tmp/alerts.log')
deadman takes two arguments, “threshold” and “interval”. They are documented as follows…
- Threshold – trigger alert if throughput drops below threshold in points/interval.
- Interval – how often to check the throughput.
In the above we pass it 0.0 for threshold. This translates to alert me if NO data has been received. Then, we pass it 5m as the interval, meaning alert me if NO data has been received for 5 minutes.
Chronograf provides a really nice UI for creating deadman alerts. Simply select the “Deadman” alert type and use the dropdown to configure the threshold at which you want to send an alert if no data has been received.
I hope that some of you found this post helpful. If you have any questions or comments, feel free to drop a note below, or, as always, you can reach me on Twitter as well.
Did you enjoy this blog post?
If so, please consider checking out my side project Domain Clamp. It's a SaaS which monitors domains and SSL certificates and sends notifications before anything expires. If you work at an agency, then you're probably not the registrant for your client's domains or the SSL certificate owner. This means you won't get expiration notifications. You don't want a client's domain or SSL certificate to expire under your watch. Believe me, I've been there.
Domain Clamp solves this problem by letting you monitor the SSL certificate and registration for any domain you'd damn please. Free accounts are available so please head on over »