Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I can't recommend serious use of an all-in-one local Grafana Loki setup

utcc.utoronto.ca

Post reply on HN
Print view View original

71–80 of 100 posts

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#71

Post by disrupthq »

While this post does raise some valid points, it seems to underscore an underlying issue with many open source software: the lack of comprehensive documentation and the assumption of a high level of technical know-how. The issues with Loki seem largely symptomatic of this trend. The good news is, it's a fixable problem. The bad news is, it's often not prioritized by the developers.

Moreover, the post speaks volumes about the potential pitfalls of "all-in-one" solutions. These can seem appealing due to their simplicity, but as this author has experienced, they often come with their own challenges, especially when things go wrong.

The friction with the Loki setup underscores the importance of having a good contingency plan in place for log management systems, as loss of log data can be a severe issue.

As for the author's point about the devs wanting you to use their cloud service, it's a common business model. Free or cheap software that's difficult to set up on your own, but with a paid service that makes it easy. The question is whether this trade-off is worth it for your specific use case.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#72

Post by nickstinemates »

j0ner wrote: post #39

If you use Loki as the default logging driver with Docker and the Loki container shuts down, the rest of your containers will freeze up. This has been an issue for almost 3 years. https://github.com/grafana/loki/issues/2361

Was a pain to get setup initially with all of the moving parts and the docker plugin for it, but has been working well for me ever since in my homelab. Smooth sailing.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#73

Post by [deleted] »

SEJeff wrote: post #61
francoismassot wrote: post #55

Quickwit is an open source Loki alternative too. Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :) https://github.com/quickwit-oss/quickwit (disclaimer: I'm one of the cofounders)

Care to give the Quickwit pitch? Especially on why it is better than Loki.

[deleted]

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#74

Post by pnathan »

I've run Loki in production for over a year... as far as I know, Loki is not designed to be done all-in-one, that's kind of a dev/local design.

N.b., I've run this in both AWS/GCP in a k8s scenario, against S3/GCS respectively as a long-term store.

I've also run Mimir, and they both work _fantastic_ in this deployment scenario (as described).

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#75

Post by edapted »

Ed here, (@slim-bean on github), one of the original authors and lead of the Loki project.

The criticisms brought by Chris are valid, and it's good feedback. I just wish it was done so in a more constructive manner.

I personally know what Loki is capable of, I've watched it grow from something tiny to something I'm incredibly proud of and in awe of.

I run Loki on several Raspberry Pi's ingesting 20-100GB a day, I also run Loki clusters on thousands of cores ingesting hundreds of TB's a day. It's an amazingly flexible and capable project built by an amazing team by a company I'm incredibly proud to work for.

That being said....

> Loki doesn't feel like it's been built to be operable by people who don't know its code and its internal details

How true this is... while I know for a fact there are hundreds to thousands of folks out there who are successfully running Loki (and thank you to those who share your success stories, it means the world to us), it can be very rough around the edges...

But we are working hard to improve this, we are doing the best we can.

All I really ask is for folks who want to give feedback about Loki, PLEASE DO! but please be patient.

The author of this article says they tried to work with us to improve Loki and that it didn't work out. I'm truly sorry for that, these posts are full of constructive feedback and I would really love to see this put into issues and pull requests to make the project better.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#76

Post by evanxg852000 »

SEJeff wrote: post #61
francoismassot wrote: post #55

Quickwit is an open source Loki alternative too. Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :) https://github.com/quickwit-oss/quickwit (disclaimer: I'm one of the cofounders)

Care to give the Quickwit pitch? Especially on why it is better than Loki.

From my little knowledge of Loki's internals. I think contrary to Loki, Quickwit uses a fully feature search engine library underneath called Tantivy (https://github.com/quickwit-oss/tantivy). Quickwit offers different services (indexer, searcher, ..) that can be ran and scaled independently. It also supports indexing from various sources including file, Rest API, Kafka, Pulsar, Kinesis and more are planned based on community interest. Last but not least, Elasticsearch query API support is being worked on.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#77

Post by pachico »

Twirrim wrote: post #47
pachico wrote: post #44

So far, no problems using Vector to ship logs to Grafana Cloud (Loki).

That's kind of his point. They don't seem to care about local.

I understand. For us, Loki locally is just to provide parity for test environments.

However, I would have used vector anyway locally as I prefer to have a centralised collector.

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#78

Post by lazyant »

Spivak wrote: post #11

It really is amazing at just how bad the various log shipping systems are for the simple use case of "I have logs on some servers and I want them to be over here." We somehow peaked at rsyslog and have been struggling ever since. If you don't follow the one-true-architecture you will get bitten in a million ways. * Log ingestion on the host pulls logs from the application/system/whatever, timestamps the logs itself (…

> it becomes nigh-impossible to lose data

I imagine if you implement it "correctly" you don't lose data but my experience with Elastic Search has been horrible.

I've lost data many times, for things like logs reaching an artificial maximum number of indices, and ES shutting down, or just not being able to support the simple case of a log coming both as a json and as a plain-text; there's no setting to say "just cast to text if there's a conflict", it drops the log and the workaround is to find among the many outdated ES posts out there, a piece of Ruby code to fix that one case. Many other issues (I compiled a list of like 20 stupid things about ES and the many ways I've lost data and gave up adding stuff).

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#79

Post by ilyt »

danwee wrote: post #62
ilyt wrote: post #46

Earlier quoted context omitted.

The whole stuff is so much worse than "old unix" architecture of "you give logger an address to push stuff, and it pushes stuff there". We have DNS, we don't need log sender to have a service discovery mechanism on top of that. Set it to log server address and be done, scale at that point if you need to, we know how to do it. Log processor doesn't need a fucking queue. Log sender does, for network reliability one. An…

Whenever I have to deal with logs, it's either: a) simplicity of rsyslog b) monstruosity of ELK | Grafana | etc. Somehow I like Prometheus (I think it's "simple"), but it's not enough to display and search for logs. Somehow, none of the companies I have worked for, have used "simple tools" like rsyslog to handle logs. They all used cloud (Datadog, New relic) or self hosted (ELK, Prometheus + Grafana). I wonder why (I…

We just use rsyslog to send to ELK instance but it's less than perfect and it doesn't log everything we want to coz not every app have very good login.

The problem I have encountered that even "simple" (just my home NAS + few devices) setups require some log mungling to get useful info into whatever system uses it. Many apps don't have "log in JSON" option in the first place, and near-always there is no real standard in fields of that message either.

And also near-always I want to filter out or rate-limit some particularly spammy message or service just because I don't even want to look at it when browsing logs as it is just noise

> Easy, no? My use case is not thousands of machines nor Terabytes of data logs per second. I just have a few machines and I don't want to deal with multi-clustered solutions or anything like that. Just 2 binaries! Does that exist?

...graylog I guess ? I looked at it and it is apparently pretty integrated, but price on higher volumes made us do ELK on "actual big stuff"

Re: I can't recommend serious use of an all-in-one local Grafana Loki setup

#80

Post by francoismassot »

SEJeff wrote: post #61
francoismassot wrote: post #55

Quickwit is an open source Loki alternative too. Like said in one comment here, it works well on billions of logs on one modest instance. And Grafana integration is on the way :) https://github.com/quickwit-oss/quickwit (disclaimer: I'm one of the cofounders)

Care to give the Quickwit pitch? Especially on why it is better than Loki.

Good point.

Unlike index-free solutions like Loki or Parseable, Quickwit is built on top of a modern full-text search index (tantivy). At query time, Quickwit produces much faster results (all other things being equal: CPU, memory, etc.), especially when the volume of data to analyze is large or queries are complex (high cardinality values, aggregations). Quickwit also stores data in a columnar format, so it's also good at OLAP-style queries (no joins though).

This comes with a cost during ingestion; Quickwit is more resource hungry than Loki but can still ingest at 20MB/s to 40 MB/s on a commodity instance with 4CPU. Similarly, regarding storage footprint, Loki compresses logs better because it does not maintain those extra data structures. Still, a Quickwit index tends to be much smaller than an Elasticsearch index.

The next release of Quickwit (may) will be shortly followed by the publication of a benchmark against Elasticsearch/OpenSearch, and by another one later, against Loki. You'll be able to see for yourself.

Post reply on HN