Recently, I ran into an issue where a misconfiguration lead to data not being pushed to InfluxDb for an extended period of time. It sucked and I wish I would’ve found out about it earlier. Here, I’ll cover how you can use Kapacitor to receive an alert if InfluxDb has not received data for an extended period of time.
In order to receive an alert when InfluxDb is not receiving data, you should be using the
deadman method. Here’s a description from the Kapacitor documentation
Helper function for creating an alert on low throughput, a.k.a. deadman’s switch.
Typically, you would call
deadman on a
The TICKScript to get an alert if there was no data for the
redis measurement for more than 5 minutes would look like this.
var data = stream |from() .database('telegraf') .retentionPolicy('autogen') .measurement('redis') data |deadman(0.0, 5m) .stateChangesOnly() .log('/tmp/alerts.log')
deadman takes two arguments, “threshold” and “interval”. They are documented as follows…
- Threshold – trigger alert if throughput drops below threshold in points/interval.
- Interval – how often to check the throughput.
In the above we pass it 0.0 for threshold. This translates to alert me if NO data has been received. Then, we pass it 5m as the interval, meaning alert me if NO data has been received for 5 minutes.
Chronograf provides a really nice UI for creating deadman alerts. Simply select the “Deadman” alert type and use the dropdown to configure the threshold at which you want to send an alert if no data has been received.
I hope that some of you found this post helpful. If you have any questions or comments, feel free to drop a note below, or, as always, you can reach me on Twitter as well.
Hi, I'm Max!
I'm a software developer who mainly works in PHP, but also dabbles in Ruby and Go. Technical topics that interest me are monitoring, security and performance.
During the day I solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I also blog about tech, work on open source and hunt for bugs.
If you'd like to get in touch with me the best way is on Twitter.