Send A Kapacitor Alert If No Data Is Received

Published: May 23, 2017

Tags:

Recently, I ran into an issue where a misconfiguration lead to data not being pushed to InfluxDb for an extended period of time. It sucked and I wish I would’ve found out about it earlier. Here, I’ll cover how you can use Kapacitor to receive an alert if InfluxDb has not received data for an extended period of time.

The deadman method

In order to receive an alert when InfluxDb is not receiving data, you should be using the deadman method. Here’s a description from the Kapacitor documentation

Helper function for creating an alert on low throughput, a.k.a. deadman’s switch.

Typically, you would call deadman on a StreamNode.

The TICKScript to get an alert if there was no data for the redis measurement for more than 5 minutes would look like this.

var data = stream
    |from()
        .database('telegraf')
        .retentionPolicy('autogen')
        .measurement('redis')

data
    |deadman(0.0, 5m)
        .stateChangesOnly()
        .log('/tmp/alerts.log')

deadman takes two arguments, “threshold” and “interval”. They are documented as follows…

In the above we pass it 0.0 for threshold. This translates to alert me if NO data has been received. Then, we pass it 5m as the interval, meaning alert me if NO data has been received for 5 minutes.

Chronograf

Chronograf provides a really nice UI for creating deadman alerts. Simply select the “Deadman” alert type and use the dropdown to configure the threshold at which you want to send an alert if no data has been received.

A screenshot showing how deadman alerts are configured in Chronograf

Conclusion

I hope that some of you found this post helpful. If you have any questions or comments, feel free to drop a note below, or, as always, you can reach me on Twitter as well.

Max ChadwickHi, I'm Max!

I'm a software developer who mainly works in PHP, but also dabbles in Ruby and Go. Technical topics that interest me are monitoring, security and performance.

During the day I solve challenging technical problems at Something Digital where I mainly work with the Magento platform. I also blog about tech, work on open source and hunt for bugs.

If you'd like to get in touch with me the best way is on Twitter.