Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: How to do simple heartbeat monitoring?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 86 posts

Re: Ask HN: How to do simple heartbeat monitoring?

#51

Post by Spivak »

fullstop wrote: post #20
what2 wrote: post #18

You need to implement a deadman switch. For example if using Prometheus you can configure it to access an HTTP endpoint of a deadman switch service every X seconds. When that service detects you have not accessed it in some time it will alert you. For example: https://blog.ediri.io/how-to-set-up-a-dead-mans-switch-in-pr...

And now you need to monitor the deadman switch service!

Typically that's why people use dead man's switch as a service. You don't assume that they'll never go down, but you're paying for someone who's failures are very likely uncorrelated with your own.

Re: Ask HN: How to do simple heartbeat monitoring?

#52

Post by jarofgreen »

I had a similar concern about a custom monitoring script that was regularly writing out a metrics file in Prometheus format to be picked up by the Prometheus node monitor. What if the script breaks?

So I set the script to also write out a metric that was just the time stamp the metrics were last updated. Then it was simple to set up an alert in Prometheus - I can't access the config now so you'll have to look it up yourself, but it was basically "alert if metric less than now minus a time gap"

Re: Ask HN: How to do simple heartbeat monitoring?

#53

Post by Temporary_31337 »

hinkley wrote: post #37
Joker_vD wrote: post #26

Earlier quoted context omitted.

Oh, you just run two instances of those and point them at each other.

Even number so that everything is symmetrical.

You want odd numbers (n+1) so you can have a quorum in case of network partition.

Re: Ask HN: How to do simple heartbeat monitoring?

#55

Post by ravenstine »

cweagans wrote: post #54

You're looking for a dead man's switch. https://deadmanssnitch.com is a good hosted service or Uptime Kuma ( https://github.com/louislam/uptime-kuma ) can be configured to do the same thing.

Haven't needed to use it in a long time, but many years ago I used Dead Man's Snitch and it's a solid service. :thumbsup:

Re: Ask HN: How to do simple heartbeat monitoring?

#56

Post by ravenstine »

nsguy wrote: post #19

I clicked to say something about Instrumentation Amplifiers or interfacing with chest straps ;) You might want to use a different service for monitoring your stack just to make sure you have some redundancy there. Seems like you got an answer for how to do this with Signoz but if that is down then you won't know.

Same, honestly, though not specifically about chest straps. Extremely primitive but effective heartbeat monitoring can be done with a cheap SpO2 monitor from Amazon and reverse engineering the Bluetooth LE signal. You won't get a true beat, but one that's more of an approximation. I know I have some C++ code lying around somewhere for this.

Re: Ask HN: How to do simple heartbeat monitoring?

#57

Post by silverquiet »

nsguy wrote: post #19

I clicked to say something about Instrumentation Amplifiers or interfacing with chest straps ;) You might want to use a different service for monitoring your stack just to make sure you have some redundancy there. Seems like you got an answer for how to do this with Signoz but if that is down then you won't know.

Yeah, honestly, I'd like a smartwatch type monitor without any smart features; just something that sits on my wrist and displays my heart rate so I can watch it myself. I've looked a bit, but such a thing hasn't been easy to find.

Re: Ask HN: How to do simple heartbeat monitoring?

#58

Post by hatsix »

divbzero wrote: post #45

If this is a cloud deployment, there might be a recommended method native to your cloud if you search for synthetic monitoring or canary . For example, AWS recommends using CloudWatch Synthetics to create a canary: https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitori...

Before resorting to Synthetics, you can use Cloudwatch to setup anomaly reporting on the load balancer. If your requests drop to 0 in 5/10/15 minutes, alert... If you have more than N 500s in 5/10/15 minutes, alert.

Synthetics are there to tell you when something outside of your provider is breaking your website, like routing errors, geo-related issues, CDN issues (assuming you're not using your provider's CDN).

Re: Ask HN: How to do simple heartbeat monitoring?

#59

Post by kraig911 »

Back in the day I had a script that would telnet once every so often to a configured time frame and listen on each one. Then another script monitoring the number of each and if the average over 10 mins was less than 90% each node send a message in your manner of choice (email or something)
Post reply on HN