Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana OnCall: an easy-to-use on-call management tool

grafana.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Grafana OnCall: an easy-to-use on-call management tool

#41

Post by CameronNemo »

CSDude wrote: post #34
CameronNemo wrote: post #31

Earlier quoted context omitted.

That's why other products have deduplication keys / aliases so you don't miss important ones. Care to link to the docs? I'm interested.

https://support.atlassian.com/opsgenie/docs/what-is-alert-de... https://support.pagerduty.com/docs/event-management

Thanks for the links.

From the article:

With Grafana OnCall’s automatic grouping of alerts within Slack, you can avoid alert storms and reduce the noise your teams are exposed to during an incident.

Seems like the same feature described using different terminology.

Re: Grafana OnCall: an easy-to-use on-call management tool

#42

Post by kevindong »

abhishekjha wrote: post #3
halfmatthalfcat wrote: post #2

Is there really anybody else in the "Pager" category of SaaS products other than PagerDuty that have any traction?

Also what happens if pagerduty goes down?

In the year I used it, I never personally noticed it going down. Although that being said, their SLA is only 99.9% delivery in any calendar month within 5 minutes. The penalty for missing that SLA is only 10% of that month's bill.

> Once an Incident is triggered, PagerDuty will deliver the First Responder Alert within the Notification Delivery Period for 99.9% of the notifications sent by PagerDuty for the Customer during any calendar month. The “Notification Delivery Period” is five (5) minutes and it is measured as the time it takes PagerDuty to deliver a First Responder Alert to telecommunication providers in accordance with the Service configuration and Contact Information.

> ...

> If PagerDuty fails to meet the SLA set forth herein, Customer may receive a service credit. Customer will be eligible for a credit toward future fees owed to PagerDuty for the PagerDuty Service. The Service Credit is calculated as ten percent (10%) of the fees paid for or attributable to the month when the alleged SLA breach occurred.

https://www.pagerduty.com/standard-service-level-agreement/

Re: Grafana OnCall: an easy-to-use on-call management tool

#43

Post by rmetzler »

deeblering4 wrote: post #23
CSDude wrote: post #17

> Alerts from each integration 300 5 minutes > Alerts from the whole team 500 5 minutes > API requests per API key 300 5 minutes Product looks great but those API request limits are too low, because alerts rain when you are having incidents and rate limiting all of them is harmful. That's why other products have deduplication keys / aliases so you don't miss important ones. https://grafana.com/docs/grafana-cloud/onca…

I'd think that receiving even 1/5th the rate limit in a 5 minute window would be disorienting enough to render alerting effectively useless. I'd question the configuration which fires that many alerts in that time frame, and suggest improving alert aggregations and dependencies to get the number down to one or a handful of meaningful alerts.

It would be great to have a dependency graph or labels in the alerts, so they are easily mapped to the things that can break and are important enough to be monitored.

We just had a short outage where an editor removed the index page in the cms which is central to the site. It's stupid that this is possible but we just operate the cms while we build and operate everything around it for our customer.

I think a large part of our alerts where triggered all at once but the one thing they had in common was that the alerts all pointed to the index page in the cms. E.g. the public www alert for index, the public api alert for index, the preview www alert for index, the preview api alert for index....

Re: Grafana OnCall: an easy-to-use on-call management tool

#44

Post by steveBK123 »

For a product that's been around 12 years, I've been surprised at how minimally featured PagerDuty is.

Stuff like national holiday awareness, integration to vacation calendars, a better UI for swapping days/overrides, etc.

PD schedule checking and trade negotiation becomes yet another thing in the long list of things I need to do when taking a day off. HR system request off, Department Outlook calendar update, PagerDuty coverage check, Outlook out-of-office status & auto-replies, Slack set away, update status AND pause notifications.

I suppose that's because as an on-call developer I am not the user. The user, management who bought the product, gets KPIs & pretty graphs, so they are happy.

Re: Grafana OnCall: an easy-to-use on-call management tool

#45

Post by EwanToo »

CameronNemo wrote: post #41
CSDude wrote: post #34

Earlier quoted context omitted.

https://support.atlassian.com/opsgenie/docs/what-is-alert-de... https://support.pagerduty.com/docs/event-management

Thanks for the links. From the article: With Grafana OnCall’s automatic grouping of alerts within Slack, you can avoid alert storms and reduce the noise your teams are exposed to during an incident. Seems like the same feature described using different terminology.

The output alerts feature looks largely the same, but the input API limits are the part in question.

What happens if you get 1000 API calls about "Alert 1" and 1 API call about "Alert 2".

You want both on call's to trigger once, but will alert 2 get though?

Re: Grafana OnCall: an easy-to-use on-call management tool

#47

Post by ethbr0 »

steveBK123 wrote: post #44

For a product that's been around 12 years, I've been surprised at how minimally featured PagerDuty is. Stuff like national holiday awareness, integration to vacation calendars, a better UI for swapping days/overrides, etc. PD schedule checking and trade negotiation becomes yet another thing in the long list of things I need to do when taking a day off. HR system request off, Department Outlook calendar update, PagerD…

Every delightful, successful developer product is eventually doomed to become JIRA.

Re: Grafana OnCall: an easy-to-use on-call management tool

#48

Post by rvnx »

ethbr0 wrote: post #47
steveBK123 wrote: post #44

For a product that's been around 12 years, I've been surprised at how minimally featured PagerDuty is. Stuff like national holiday awareness, integration to vacation calendars, a better UI for swapping days/overrides, etc. PD schedule checking and trade negotiation becomes yet another thing in the long list of things I need to do when taking a day off. HR system request off, Department Outlook calendar update, PagerD…

Every delightful, successful developer product is eventually doomed to become JIRA.

A multi-billion USD success story ?

Re: Grafana OnCall: an easy-to-use on-call management tool

#50

Post by dharmab »

CSDude wrote: post #17

> Alerts from each integration 300 5 minutes > Alerts from the whole team 500 5 minutes > API requests per API key 300 5 minutes Product looks great but those API request limits are too low, because alerts rain when you are having incidents and rate limiting all of them is harmful. That's why other products have deduplication keys / aliases so you don't miss important ones. https://grafana.com/docs/grafana-cloud/onca…

I was once in a job where I was solo on call for tens of thousands of cores globally and at worst we had like 2000 alerts in a week. These limits seem quite high to me.
Post reply on HN