Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grafana releases OnCall open source project

grafana.com

Post reply on HN
Print view View original

51–60 of 134 posts

Re: Grafana releases OnCall open source project

#51

Post by netingle »

sandstrom wrote: post #28

I think it would be great if it was easier to mix and match Grafana SaaS and self-hosted products. For example, we need to run Loki ourselves, for security / privacy reasons, but wouldn't mind using hosted versions of Tempo, Prometheus and OnCall. Right now it isn't super-easy to link e.g. self-hosted loki search queries with SaaS-Prometheus.

Its very much our aim to make this mix of self-hosted and cloud services as easy as going all-cloud; but I agree we're not quite there yet.

Do you mind if I ask what isn't super-easy about linking self-hosted loki search queries with SaaS-Prometheus? You should be e.g. able to add a Prometheus data source to your local Grafana (or securely expose your Loki to the internet and add a Loki data source to your Cloud Grafana)

Re: Grafana releases OnCall open source project

#52

Post by picozeta »

theptip wrote: post #36
lazyant wrote: post #31

Earlier quoted context omitted.

Curious as to what architecture you would have preferred or why this pretty standard stack (that can be deployed to k8s) is not giving you.

For a simple low-scale app you can often do without Redis and Celery/RMQ if you just push everything into Postgres. Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.

How does a message queue work via Postgres? Many people (including me) use Redis to run background jobs.

Re: Grafana releases OnCall open source project

#53

Post by motakuk »

pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

I agree that multi-component architecture is harder to deploy. We did our best and prepared tooling to make deployment an easy thing.

Helm (https://github.com/grafana/oncall/tree/dev/helm/oncall), docker-composes for hobby and dev environments.

Besides deployment, there are two main priorities for OnCall architecture: 1) It should be as "default" as possible. No fancy tech, no hacking around 2) It should deliver notifications no matter what.

We chose the most "boring" (no offense Django community, that's a great quality for a framework) stack we know well: Django, Rabbit, Celery, MySQL, Redis. It's mature, reliable, and allows us to build a message bus-based pipeline with reliable and predictable migrations.

It's important for such a tool to be based on message bus because it should have no single point of failure. If worker will die, the other will pick up the task and deliver alert. If Slack will go down, you won't loose your data. It will continue delivering to other destinations and will deliver to Slack once it's up.

The architecture you see in the repo was live for 3+ years now. We were able to perform a few hundreds of data migrations without downtimes, had no major downtimes or data loss. So I'm pretty happy with this choice.

Re: Grafana releases OnCall open source project

#54

Post by jlg23 »

juliennakache wrote: post #5

Looking forward to trying this out. I've always felt that PagerDuty was absurdly expensive for the feature set they were offering. It costs something at least $250 per user for organization larger than 5 person - even if you're not an engineer who is ever directly on call. At my previous company, IT had to regularly send surveys to employees to assess if they really needed to have a PagerDuty account. Alerts are a ke…

Thanks, I think I finally understand why some friends of mine, who can implement this for any company in half a day, take $2000/day...

Re: Grafana releases OnCall open source project

#55

Post by infogulch »

picozeta wrote: post #52
theptip wrote: post #36

Earlier quoted context omitted.

For a simple low-scale app you can often do without Redis and Celery/RMQ if you just push everything into Postgres. Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.

How does a message queue work via Postgres? Many people (including me) use Redis to run background jobs.

lmgtfy https://www.crunchydata.com/blog/message-queuing-using-nativ...

Re: Grafana releases OnCall open source project

#56

Post by FridgeSeal »

matsemann wrote: post #46
MarquesMa wrote: post #43

Earlier quoted context omitted.

This. I find open source projects written in Go or Rust are usually more pleasant to work with than Java, Django or Rails, etc. They have less clunky dependencies, are less resource-hungry, and can ship with single executables which make people's life much easier. Just think about Gitea vs GitLab.

Not sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment. As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes. Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.

Python application deployments are all fun and games until suddenly the documentation starts unironically suggesting that you should “write your configuration as a Python script” that should get mounted to some random specific directory within the app as if that could ever be a sane and rational idea.

Re: Grafana releases OnCall open source project

#57

Post by pphysch »

vhold wrote: post #33
pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

AlertManager is one component of a more complicated infrastructure. https://prometheus.io/docs/introduction/overview/#architectu... https://kubernetes.io/docs/concepts/overview/components/

OnCall also does nothing unless you have something external firing alerts for you. They both fill similar niches in a larger monitoring system; this does not excuse OnCall having a drastically more complex internal architecture.

Re: Grafana releases OnCall open source project

#58

Post by motakuk »

juliennakache wrote: post #5

Looking forward to trying this out. I've always felt that PagerDuty was absurdly expensive for the feature set they were offering. It costs something at least $250 per user for organization larger than 5 person - even if you're not an engineer who is ever directly on call. At my previous company, IT had to regularly send surveys to employees to assess if they really needed to have a PagerDuty account. Alerts are a ke…

Check this ;) https://github.com/grafana/oncall/tree/dev/tools/pagerduty-m...

Re: Grafana releases OnCall open source project

#59

Post by pphysch »

chrisandchris wrote: post #34
lazyant wrote: post #31

Earlier quoted context omitted.

Curious as to what architecture you would have preferred or why this pretty standard stack (that can be deployed to k8s) is not giving you.

Not OP, but one may interpret your response as "I don't understand why you prefer a single binary over this architecture that requires 6 different services and prefers k8s". IMHO, OP just stated that one could solve this with less dependencies and have the same (if not a better) result.

Yes, thank you. I would be surprised if this same product couldn't be delivered with just Python(Django) + SQLite + Redis (assuming writing everything in Go is unrealistic). Spinning up a venv and launching a local Redis instance is significantly more reasonable than having to configure MySQL, RabbitMQ, and Celery.

Re: Grafana releases OnCall open source project

#60

Post by Deritio »

motakuk wrote: post #53
pphysch wrote: post #23

A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc. I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my curren…

I agree that multi-component architecture is harder to deploy. We did our best and prepared tooling to make deployment an easy thing. Helm ( https://github.com/grafana/oncall/tree/dev/helm/oncall ), docker-composes for hobby and dev environments. Besides deployment, there are two main priorities for OnCall architecture: 1) It should be as "default" as possible. No fancy tech, no hacking around 2) It should deliver no…

Hearing your message bus assumption sounds like one of the most ridiculous claims I heard.

Sorry but why is rabbitmq really necessary?

Post reply on HN