Live data from Hacker News

Ask HN: How to do simple heartbeat monitoring?

news.ycombinator.com

11–20 of 86 posts

Re: Ask HN: How to do simple heartbeat monitoring?

#12
> What's the easiest way to have a cron-like query hit our API and integrate with our existing stack?

Write a cron job that greps the logs and pulls on your api with curl?

With another 20 minutes of work you can remember the log size on each run and grep only what's new on the next run.

Re: Ask HN: How to do simple heartbeat monitoring?

#14
This is one of those things that _seems_ really simple but the details make it pretty complicated.

Not sure about Signoz and PagerDuty, but there are plenty freemium services like UpTimeRobot that work fine for basics.

And then something like AWS CloudWatch has a lot more advanced options about how to treat missing data.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

With a lot of knobs to tune around what intervals you need and what counts as 'down', which you'll want to think about pretty carefully given what you need.

Re: Ask HN: How to do simple heartbeat monitoring?

#17
Writing a heartbeat process using UDP for a Fluentd logger is as one of the first things I learned in Elixir! In my case, basically when then there was no heartbeat, the TCP forwarder /w a connection pool would be taken offline. When the heartbeat came back, the forwarder was free to send messages again.

The heartbeat process itself basically just sent pings every second on a UDP port and waited for a reply. If we didn’t get a reply in one second, it’d assume the connection was bad until a ping came again.

Re: Ask HN: How to do simple heartbeat monitoring?

#18
You need to implement a deadman switch. For example if using Prometheus you can configure it to access an HTTP endpoint of a deadman switch service every X seconds. When that service detects you have not accessed it in some time it will alert you.

For example: https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-pr...

Re: Ask HN: How to do simple heartbeat monitoring?

#19
I clicked to say something about Instrumentation Amplifiers or interfacing with chest straps ;)

You might want to use a different service for monitoring your stack just to make sure you have some redundancy there. Seems like you got an answer for how to do this with Signoz but if that is down then you won't know.

Re: Ask HN: How to do simple heartbeat monitoring?

#20
post #18

You need to implement a deadman switch. For example if using Prometheus you can configure it to access an HTTP endpoint of a deadman switch service every X seconds. When that service detects you have not accessed it in some time it will alert you. For example: https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-pr...

And now you need to monitor the deadman switch service!
Post reply on HN