It seems Red Hat believes in Loki Red Hat logging product manager says: "We made the decision to move to Loki and Vector" https://www.youtube.com/watch?v=QZ4Hv85lEJ0&t=938s
Seems like a 'low-key' bad decision from the sounds of it.
I can't recommend serious use of an all-in-one local Grafana Loki setup
41–50 of 100 posts
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#42Grafana is a terribly ran company that once made a pretty decent OSS dashboard system. Stay away from them for anything but some charts.
And even the dashboard system kinda feels like there should be something better, but somehow there isn't (or I don't know it).
Is Grafana (the product) really that bad? we've been using it for years with hundreds of dashboards and have never had any complaints*
*until recently... the new "time series" panel is a disaster
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#43While it's not a tool I love, we have been running it for over a year, without issue.
We store maybe 250GB worth of logs in each instance, and ingest an estimated 1-2k lines a second.
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#44So far, no problems using Vector to ship logs to Grafana Cloud (Loki).
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#45Earlier quoted context omitted.
Grafana requires you to sign a CLA before they will accept any work, which can be really expensive (unless you have in-house lawyers or don't care about understanding the real ramifications of a contract): https://grafana.com/docs/grafana/latest/developers/cla/
Is there anything nonstandard or suspicious about this CLA?
I don't know. Do you really know what you're giving up by signing these? I'd have to study CLAs and hope that I am interpreting them correctly within our respective jurisdictions, or ask a lawyer.
All of this stands in the way of contributing. And this is their decision to make, of course, but it is hostile to would-be contributors.
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#46It really is amazing at just how bad the various log shipping systems are for the simple use case of "I have logs on some servers and I want them to be over here." We somehow peaked at rsyslog and have been struggling ever since. If you don't follow the one-true-architecture you will get bitten in a million ways. * Log ingestion on the host pulls logs from the application/system/whatever, timestamps the logs itself (…
The whole stuff is so much worse than "old unix" architecture of "you give logger an address to push stuff, and it pushes stuff there".
We have DNS, we don't need log sender to have a service discovery mechanism on top of that. Set it to log server address and be done, scale at that point if you need to, we know how to do it.
Log processor doesn't need a fucking queue. Log sender does, for network reliability one. And that gives you ability to restart log processor quickly (only need to process current message in transit and close) and with zero impact (as long as you're down shorter than the logger's queue)
Only reason to add queue is if you have multiple readers for logs. That also conveniently gives you a form of QoS on log processor, if you read with equal rate from all sources the most spammy ones will hit their own internal queue limit first and wont cause other servers to miss the logs. Even then you might just op for the loggers sending things into 2 places at once.
The "shit logs" (whether by volume or needing messaging decoding) is a problem that's complex but IMO most of that should be within log processor, as it should be. That's also a good place to resolve any geoip or DNS if needed.
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#47So far, no problems using Vector to ship logs to Grafana Cloud (Loki).
That's kind of his point. They don't seem to care about local.
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#48Grafana is a terribly ran company that once made a pretty decent OSS dashboard system. Stay away from them for anything but some charts.
And even the dashboard system kinda feels like there should be something better, but somehow there isn't (or I don't know it).
You only think that because you haven't seen the alternatives...
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#49Earlier quoted context omitted.
And even the dashboard system kinda feels like there should be something better, but somehow there isn't (or I don't know it).
Is Grafana (the product) really that bad? we've been using it for years with hundreds of dashboards and have never had any complaints* *until recently... the new "time series" panel is a disaster
It's not bad. You can question some decisions about it but currently and for last few years it's absolute peak with no competition in industry
Re: I can't recommend serious use of an all-in-one local Grafana Loki setup
#50I’ve been using quickwit.io for some local data processing job logs and it seems to be very easy to run, not very IO intensive and running fine on a single node with modest hardware with >2 billion log rows. It has a really cool dynamic schema feature too. I found it easier to setup and configure than Loki. The UI is very basic for now but I’m excited to see what the future holds for this project!
Thanks a lot for your kind comment!
To complete your description of Quickwit, It is a distributed search engine for logs and traces. It's written in Rust, ingest at speed, horizontally scalable, and separates compute from storage.
Last but not least, Grafana integration is planned for next month :)