Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana Labs launches free incident management tool in Grafana Cloud

grafana.com

Post reply on HN
Print view View original

51–56 of 56 posts

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#51

Post by matryer »

donavanm wrote: post #28

How do you/users programmatically quantify MTTR (and related metrics) per incident, or in aggregate? Although it shades towards problem management this would seem necessary to achieve the claim of “reduces mean time to repair (MTTR).” Bonus questions, are you tracking or driving improvement in the related times for detection/response/mitigate/recover? Disclosure: Principal at AWS currently in a similar apace. Though…

Hey, thanks for your question. The tool keeps track of declaration and resolution times by watching when the status is changed. It also lets you manually specify when the incident really started, and when it actually ended. We can use this data to measure a few things, and watching how this changes over time helps us figure out if we're getting better or worse, on average. We want to be careful what we incentivise by default, and we're actively working on this area. The data is going to be available for people to build their own visualisations (in Grafana).

I'd be very interested to hear your thoughts too?

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#52

Post by hztar »

gaffneyc wrote: post #29
divygoel wrote: post #25

Earlier quoted context omitted.

Hey @nikolay! I work at Grafana Labs & focus on pricing - would you be up for a 15 minute chat to discuss this further? If so, feel free to either drop me a note at divy.goel@grafana.com or let me know how best to reach you :)

We recently swapped our metrics to Grafana Cloud and were really surprised (despite being documented) that pricing is based on samples per minute not metrics series. So, for example, if we send a metric every 15s (the Prometheus default) then we get charged as if that were four separate metrics. Support was very helpful explaining everything and they reversed the charge but it still feels weird.

We experienced the same thing. Pulled the plug on using the cloud version

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#53

Post by RichiH »

oxfordmale wrote: post #50
divygoel wrote: post #44

Earlier quoted context omitted.

Hey @oxfordmale - Sorry to hear about your experience. Would you be open to a live discussion so I can understand your issue further? Surprised to hear this as none of our Enterprise plugins had a change in licensing (i.e. going from free to paid, or shifting within paid tiers) as far as I am aware. Would love to dig into this further. If you're up for it, feel free to send me a note at divy.goel@grafana.com or let m…

It concerns this plugin. We had a previous version of this working for years. It suddenly was deactivated one day and after us reaching out the support team, we were advised we had to upgrade to Grafana Enterprise to be able to get this working again. This is fair enough, however, what shows utter lack of respect is that you couldn't give us a grace period and this resulted in our Production Monitoring to be broken f…

NB: I work for Grafana Labs as Directory of Community.

As this is not how we (try to) operate, I also had a look.

From what I could find, it seems the account you are referring to is a very early Cloud account. For reasons I don't know and which might be lost to history, your account had an old and non-standard license attached to. From the viewpoint of today, the license itself seems broken.

To be clear, that is neither your fault nor do I believe you could have caught it even if you looked. While there was no change in the licensing requirements on the plugin, an upgrade to a significant rewrite of the plugin "fixed" the problem of accepting a broken license. That "fix" meant the plugin stopped working.

Again, this is not your fault. But it was not a deliberate action by Grafana Labs nor caught by our testing, either.

I believe your company is in contact with David Dorman, our Head of Self-Service, about this. If you'd like me to ask him to follow up with you directly as well, please let me know how to best contact you.

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#54

Post by RichiH »

oxfordmale wrote: post #39
drc wrote: post #36

Earlier quoted context omitted.

Thanks for feedback re: robot emoji. You can also use a backslash command if you want to add a new piece of text to the timeline from Slack. re: postmortem workflow. The timeline view is built to help postmortems, one of the ways we're doing this is making it easy to paste the info from the timeline as rich text or markdown from the timeline into your post mortem workflow. You'll see that on the top right of the time…

Are you going to upgrade this feature to an Enterprise license one day and then revoke access without a grace period? This happened to one of our Grafana plugins and resulted in a several day outage while we scrambled to sort out the payment. As your company has shown zero respect for its customers, I will not be using any of your systems.To be absolutely clear it is fair to charge for any of your products, however,…

For anyone reading this, there's a reply and relevant context at https://news.ycombinator.com/item?id=32837295

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#55

Post by donavanm »

matryer wrote: post #51
donavanm wrote: post #28

How do you/users programmatically quantify MTTR (and related metrics) per incident, or in aggregate? Although it shades towards problem management this would seem necessary to achieve the claim of “reduces mean time to repair (MTTR).” Bonus questions, are you tracking or driving improvement in the related times for detection/response/mitigate/recover? Disclosure: Principal at AWS currently in a similar apace. Though…

Hey, thanks for your question. The tool keeps track of declaration and resolution times by watching when the status is changed. It also lets you manually specify when the incident really started, and when it actually ended. We can use this data to measure a few things, and watching how this changes over time helps us figure out if we're getting better or worse, on average. We want to be careful what we incentivise by…

Sorry for the delay, was traveling on holiday.

In short yeah, using the incident status as the implied times makes sense for the bulk of cases. Totally agree on picking out signal from the users inherent actions, but allowing them to provide more specific data when they know better.

Digging in a little further Im personally interested in moving past the incident data and inspecting the incoming alert(s) and related telemetry/metric/alarm data. For example think of the alarm definitions like “five 1m datapoints with a value above 0.1.” There’s a good argument to count impact (and incident duration) from that first datapoint > 0.1. Then theres the delta from metric processing to alert to incident creation. On the backend theres frequently a delta between mitigating impact and actual incident resolution, again I think getting back to the source alarm/alert/metric data would get us a more accurate view of operations and customer impact.

Re: Grafana Labs launches free incident management tool in Grafana Cloud

#56

Post by farhan0410 »

CSMastermind wrote: post #49
drc wrote: post #36

Earlier quoted context omitted.

Thanks for feedback re: robot emoji. You can also use a backslash command if you want to add a new piece of text to the timeline from Slack. re: postmortem workflow. The timeline view is built to help postmortems, one of the ways we're doing this is making it easy to paste the info from the timeline as rich text or markdown from the timeline into your post mortem workflow. You'll see that on the top right of the time…

For postmortems the basis is having document templates that are linked to the incident. From there the ability to automatically add information to the individual postmortem documents with things like timeline and MTTA/MTTR. Add the ability to comment on, share, export, and collaborate on these documents. The second piece of functionality is around action items. Almost all postmortems generate action items. I need a w…

This is exactly how our post-mortem functions work -- we pre-generate a google doc with a template and we have the ability to copy (in rich text / markdown) the critical timeline.

On the action item section today actions are tied to Github issues and we are working to extend those - namely Jira/ServiceNow. We do already have a few that gives you a quick look at current status, how long the incident took, and open action items which you can filter by labels or severity (e.g. all severe production issues)

The postmortem doc, meeting link and slack channel are all automatically created for every incident

Post reply on HN