Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't recommend serious use of an all-in-one local Grafana Loki setup

utcc.utoronto.ca

Post reply on HN
Print view View original

51–60 of 100 posts

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#51

Post by marcosdumay »

danwee wrote: post #24
Spivak wrote: post #11

It really is amazing at just how bad the various log shipping systems are for the simple use case of "I have logs on some servers and I want them to be over here." We somehow peaked at rsyslog and have been struggling ever since. If you don't follow the one-true-architecture you will get bitten in a million ways. * Log ingestion on the host pulls logs from the application/system/whatever, timestamps the logs itself (…

> * Log ingestor ships the logs to a durable queue > * Log processor reads from the queue and ships the logs off to persistent storage Why do we need the durable queue in between? Why not let the Log Ingestor ship the logs off to persistent storage?

Because this was created for 400lb gorillas.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#52

Post by tomrod »

tiwarinitish86 wrote: post #34

Parseable is an open source Loki alternative. - Single binary - Written in Rust (lightweight, fast and stable) - Use S3 bucket or Mount point - Visualize with Grafana https://github.com/parseablehq/parseable (founder here)

Post a Show HN?

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#53

Post by detaro »

x86a wrote: post #42
detaro wrote: post #28

Earlier quoted context omitted.

And even the dashboard system kinda feels like there should be something better, but somehow there isn't (or I don't know it).

Is Grafana (the product) really that bad? we've been using it for years with hundreds of dashboards and have never had any complaints* *until recently... the new "time series" panel is a disaster

no, it's not "that bad", but it also constantly annoys me with things that don't feel right, even though they generally can be made to do what I want in the end. It just feels odd to me that this is the best there is given how big a deal has been made about monitoring in the past decade.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#54

Post by guhcampos »

dpkirchner wrote: post #13
gganley wrote: post #7

Earlier quoted context omitted.

Not that I don't believe that's entirely possible, but do you have a link to something demonstrating this behavior? A _very_ cursory google search didn't come up with anything immediately. Like this article might just be for show but its the first thing that came up ¯\_(ツ)_/¯ https://grafana.com/docs/grafana/latest/developers/contribut...

Grafana requires you to sign a CLA before they will accept any work, which can be really expensive (unless you have in-house lawyers or don't care about understanding the real ramifications of a contract): https://grafana.com/docs/grafana/latest/developers/cla/

Do you also run every single EULA through your lawyers?

CLAs are becoming common because litigation is becoming common. It's a product of our times and mostly a safeguard for companies in case a person is able to slip in some malevolent code or write hate speech in docs, or if someone tries to claim copyright on docs/code.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#55

Post by francoismassot »

Quickwit is an open source Loki alternative too.

Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :)

https://github.com/quickwit-oss/quickwit

(disclaimer: I'm one of the cofounders)

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#56

Post by ochoseis »

> An attempt to upgrade our Loki 2.7.4 to 2.8.1 failed badly and could not be reverted, forcing us to delete our entire accumulated log data for the second time in a few months (after the first time).

This sucks, but it’s also why you take filesystem snapshots or perform a backup before upgrades.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#57

Post by spicyusername »

medellin wrote: post #36
eriksjolund wrote: post #30

It seems Red Hat believes in Loki Red Hat logging product manager says: "We made the decision to move to Loki and Vector" https://www.youtube.com/watch?v=QZ4Hv85lEJ0&t=938s

They also push systemd…

Systemd is great.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#58

Post by r3trohack3r »

dpkirchner wrote: post #37
r3trohack3r wrote: post #27

Earlier quoted context omitted.

This appears to be a 7 clause contract written in good faith to ensure Grafana Labs can continue building a product and service offering around their open source project after accepting your contribution. Am I missing something? Do you have any specific problems with the CLA? Is there an alternative option to a CLA that ensures the original copyright owner can continue offering the code base under multiple licenses a…

Perhaps my contributions are also made in good faith with full awareness that they are subject to the licenses? We did OSS for decades without CLAs and most projects still do not require CLAs.

Subject to which licenses? The current license? Is Grafana obligated to keep the licenses they distribute the current software under unmodified forever after you contribute? Do they loose their right to modify the license of the codebase with your contribution? If they decide to move it to GPL, do they need to get your permission first? If they decide to take the project closed source, do they need your permission? Can you sue for damages if they don’t get your permission first?

Your good faith doesn’t hold up in court and I understand why they’d want to clarify ownership of the contributions. Just because we’ve always done it this way doesn’t mean people aren’t open to liability. Just because another project accepts the risk of a random contributor winning a lawsuit against them doesn’t mean Grafana should. I’m surprised CLAs aren’t more common.

I was personally surprised at how generous their CLA was with ownership rights for you and your contribution to the project. You retain a lot when contributing.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#59

Post by Spivak »

ilyt wrote: post #46
Spivak wrote: post #11

It really is amazing at just how bad the various log shipping systems are for the simple use case of "I have logs on some servers and I want them to be over here." We somehow peaked at rsyslog and have been struggling ever since. If you don't follow the one-true-architecture you will get bitten in a million ways. * Log ingestion on the host pulls logs from the application/system/whatever, timestamps the logs itself (…

The whole stuff is so much worse than "old unix" architecture of "you give logger an address to push stuff, and it pushes stuff there". We have DNS, we don't need log sender to have a service discovery mechanism on top of that. Set it to log server address and be done, scale at that point if you need to, we know how to do it. Log processor doesn't need a fucking queue. Log sender does, for network reliability one. An…

> We have DNS, we don't need log sender to have a service discovery mechanism on top of that.

Having service discovery solves some issues.

* DNS TTL and applications holding on to DNS names indefinitely (prometheus, haproxy, nginx, and I bet your app somewhere all do this).

* Applications that don't support DNS record priorities.

* Serving different results to different clients based on their identity that isn't isn't random.

> Log processor doesn't need a fucking queue. Log sender does, for network reliability one

Yes. That's what the queue is for. The log sender also has a queue but as it lives on the host itself minimizing its use is how you don't lose logs on server crashes. If your architecture is the log processor accepts logs, and stores them in a queue for buffering then you've implemented the same architecture. But if that queue lives on the log processor itself then you risk data loss if that server dies. Having a shared queue in front of the pool of log processors, is simpler, has better throughput, easier to shard, and more reliable. Logs can't get stuck on a particular processor anymore because its lease will end and another worker will pick it up.

Post reply on HN