Ask HN: How to do simple heartbeat monitoring?

Hey there!

My team is hosting an API that sends open telemetry data to Signoz & manages on-call via PagerDuty. We've configured Signoz to hit Pagerduty when there's a series of 500 errors.

However, our server went down last night and NO opentel data was sent to Signoz. We weren't notified that the server went down as there weren't 500 responses to report. What's the easiest way to have a cron-like query hit our API and integrate with our existing stack? Is this feasible with our existing vendors? Should I have a serverless function running on a timer that uses Pagerduty's API? Should I be migrating to another monitoring service?

Any advice would be appreciated!

Believe you can do this in Signoz: https://signoz.io/docs/monitor-http-endpoints/

For my applications, monitored by prometheus + grafana, we have alerts when no data is reported for certain metrics in the past 5 minutes, indicating a malfunction in the subsystem.

With a metric, you can use a monotonic counter to serve as a heartbeat. A timestamp would work. In your monitoring system, when the heartbeat value has not increased in X minutes, you alert.

thanks! this has been recommended by others. I'm not feeling excited about migrating to another monitoring service but it might be the right move.

we use updown.io for this and are happy with it.

there's probably functionality built in to your other monitoring tools, or you could write a little serverless function to to it, but for me i really like to have it isolated. i wanted a tool that's not part of the rest of our monitoring stack, not part of our own codebase, and not something we manage.