Grafana releases OnCall open source project • Hacker News Board

Re: Grafana releases OnCall open source project

#41

Post by contravariant » Tue, Jun 14, 2022, 5:15 PM UTC

skullone wrote: post #35

pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

That seems like a perfectly reasonable architecture. If only all of us could work on battle tested components like those during our job!

For something that is supposed to add some more features to the basic email/HTTP message alert like grafana generates, I do wonder what extra features require an additional 2 databases, a message queue and a separate task queue.

Re: Grafana releases OnCall open source project

#42

Post by matsemann » Tue, Jun 14, 2022, 5:16 PM UTC

Equiet wrote: post #16

It's surprising how seemingly difficult it is to build a good on-call scheduling system. Everything I tried so far (not naming the companies here) felt like the UX was the last thing on the developers' minds. Which is tolerable during business hours but really annoying at 2am. Is there some hidden complexity or is it just a consequence of engineers building a product for other engineers? Also, any tips what worked fo…

Have had lots of bad experiences with that from Pagerduty at least. Want to generate a schedule far in advance, so people know when they will be oncall and can plan/switch.

Of course, in a few months we may have some new people having joined, some quit, or other circumstances. A single misclick when fixing that can invalidate the whole schedule and generate another. Infuriating.

Or the UI itself, might have become better tha last two years, but having to click "next week" tens of times to see when I was scheduled (since I wasnt just interested in my next scheduled time but all of them) were annoying.

Re: Grafana releases OnCall open source project

#43

Post by MarquesMa » Tue, Jun 14, 2022, 5:17 PM UTC

pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

This. I find open source projects written in Go or Rust are usually more pleasant to work with than Java, Django or Rails, etc. They have less clunky dependencies, are less resource-hungry, and can ship with single executables which make people's life much easier.

Just think about Gitea vs GitLab.

Re: Grafana releases OnCall open source project

#44

Post by matsemann » Tue, Jun 14, 2022, 5:18 PM UTC

nojito wrote: post #15

ucosty wrote: post #9

Earlier quoted context omitted.

Why is that unfortunate? Unless you're looking to make proprietary changes to Grafana Oncall and host it as a SAAS, it's the same as running any other GPL software.

GPL and its variants are a no go where I work.

Running a service with a GPL license is different than including their code in your projects, though. So while it may be a blanket ban, it may be worth it to clarify the scope of that ban.

Re: Grafana releases OnCall open source project

#45

Post by ildari » Tue, Jun 14, 2022, 5:22 PM UTC

Hey HN, Ildar here, one of the co-founders of Amixr and one of the software engineers behind Grafana OnCall. Finally we open-sourced the product I'm really excited about that. Please try it out and leave your feedback

Re: Grafana releases OnCall open source project

#46

Post by matsemann » Tue, Jun 14, 2022, 5:22 PM UTC

MarquesMa wrote: post #43

pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

This. I find open source projects written in Go or Rust are usually more pleasant to work with than Java, Django or Rails, etc. They have less clunky dependencies, are less resource-hungry, and can ship with single executables which make people's life much easier. Just think about Gitea vs GitLab.

Not sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment.

As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes.

Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.

Re: Grafana releases OnCall open source project

#47

Post by arccy » Tue, Jun 14, 2022, 5:27 PM UTC

CSMastermind wrote: post #24

juliennakache wrote: post #5

Looking forward to trying this out. I've always felt that PagerDuty was absurdly expensive for the feature set they were offering. It costs something at least $250 per user for organization larger than 5 person - even if you're not an engineer who is ever directly on call. At my previous company, IT had to regularly send surveys to employees to assess if they really needed to have a PagerDuty account. Alerts are a ke…

> I've always felt that PagerDuty was absurdly expensive for the feature set they were offering For anyone out there in the same spot, I'll say that I switched my last company to Atlassian's OpsGenie and it was a 10x cost savings for the same feature set.

the opsgenie api is really bad though if you want to manage it as code/declaratively

Re: Grafana releases OnCall open source project

#48

Post by gjulianm » Tue, Jun 14, 2022, 5:27 PM UTC

theptip wrote: post #36

lazyant wrote: post #31

Earlier quoted context omitted.

Curious as to what architecture you would have preferred or why this pretty standard stack (that can be deployed to k8s) is not giving you.

For a simple low-scale app you can often do without Redis and Celery/RMQ if you just push everything into Postgres. Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.

I bet quite a lot, probably at least 10-50 per second without doing anything special for performance, i.e. multiple queries per alert, calling different APIs, things like that. I don't know of many places that are dealing with alerts measured in "per second" as a unit.

Not to mention that having multiple components doesn't mean it's "scalable" by default, it could happen that some part of the pipeline doesn't like multiple instances of something.

Re: Grafana releases OnCall open source project

#49

Post by pachico » Tue, Jun 14, 2022, 5:30 PM UTC

I love Grafana, don't get me wrong, but I have the sensation they are now in that position where, companies that got a massive capital injection and, therefore, a massive increase of work power, release too much and too soon.

It doesn't have anything to do, of course, with the fact that this morning we suddenly found that all our dashboards stopped working because we were upgraded to Grafana v9, for which there is not a stable release nor documentation for breaking changes.

Luckily they rolled back our account.

Re: Grafana releases OnCall open source project

#50

Post by gjulianm » Tue, Jun 14, 2022, 5:33 PM UTC

lazyant wrote: post #31

pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

Curious as to what architecture you would have preferred or why this pretty standard stack (that can be deployed to k8s) is not giving you.

Installation in a regular system without Kubernetes? Right now I can install Grafana, Prometheus and Alertmanager in a regular Linux system using distribution packages, and just worry about those programs themselves. If I want to install OnCall, I need not only OnCall plus four other non-trivial dependencies that will still need configuration, management and troubleshooting. All for something that is going to deal with far less load than any of Grafana/Prometheus/Alertmanager. I honestly do not understand it.