I can't recommend serious use of an all-in-one local Grafana Loki setup

Whenever I have to deal with logs, it's either:

a) simplicity of rsyslog

b) monstruosity of ELK | Grafana | etc.

Somehow I like Prometheus (I think it's "simple"), but it's not enough to display and search for logs. Somehow, none of the companies I have worked for, have used "simple tools" like rsyslog to handle logs. They all used cloud (Datadog, New relic) or self hosted (ELK, Prometheus + Grafana). I wonder why (I guess it's because "money buys you simplicity")

I just want the following:

- on each machine I want to get logs from: install the agent (a simple binary) + simple /etc/myagent.conf. The agent forwards logs to my "main log server"

- on my "main log server": install the "log processor" (again, just a binary please!) + simple /etc/mylogprocessor.conf. The "log processor" shows me a nice localhost:9090/ web interface in which I can search for logs (indexed by any field I want).

Easy, no? My use case is not thousands of machines nor Terabytes of data logs per second. I just have a few machines and I don't want to deal with multi-clustered solutions or anything like that. Just 2 binaries! Does that exist?

I installed promtail a few weeks back and I ran into this bug, that had been outstanding for months: https://github.com/grafana/loki/issues/8663 (e.g. a fix had been written but had not been released):

Due to a buffering issue, Loki would exit in case of configuration error without printing any error message or anything at all.

There is definitely something weird about how the project is run.

Vector is wonderful, and while I have no experience with Loki, if it's as bad as this post and thread suggest, perhaps Vector makes it more manageable by normalising and buffering everything coming into it.

For example, another comment here talks about Loki locking up Docker if it's the logging backend and the container crashes. I suspect that wouldn't be possible, or would be less likely and more manageable with Vector in the middle because it will buffer. I've also dealt with normalising logs from different sources and it can be a pain, but Vector will do some or all of that already, reducing the requirements put on Loki.

I’ve always viewed all in one mode for Loki as a demo or very small install version. It’s meant to be run in its component architecture as a scaling ingestion and query engine.

That said: 1) I feel like Loki is languishing and not reaching its full potential. User experience needs a lot of work.

2) Grafana is a for profit company

3) Grafana sees its future in the margin rich Saas offering

4) open source is still supported, but only to a certain point and the rest is commercial. I wouldn’t expect material support if you’re not paying for it.

Here is a question, I mean it honestly, I'm relatively old school and have built many apps using syslog. When it comes to doing log mining, I've got a fairly old school utility belt, I poke around with less, I cat through grep (really ripgrep), I cat through grep and pipe to awk and extract things. Sometimes I fire up cut. I get a ton of milage from sort and uniq. Obviously, I fire up zcat in place of cat when needed. I also generously apply find when needed. It feels like I find what I need pretty quickly. Admittedly, I generally don't have terabytes of logs with these tools but it handles 10s of GB shockingly well.

With Splunk, ELK, Greylog, it feels insanely pokey. I know they have the parsers and such. At times I've kind of boned up on their search syntax but I've never gone "all in" with any of them, maybe because they all don't seem like a really solid long term solution. They seems to have a different kind of model than what I want, the time range is kind of nice but often times I won't have a time range until later. My model involves winnowing down the the data I want and then extracting pieces and viewing the data different ways. Am I just using all these tools the wrong way? Is my mental model off? Maybe it's a log consistency thing, it's always sort of a great day when you get "Error: abc failed because xyz and def." and that's the answer to everything. Many times I'll be spending time looking at logs and I'll notice an increase in a certain behavior happened before the outage happened and that's the give away.. Then a new grafana dashboard is created with a new metric to try and identify that before it happens again.

Loki kind of looks like it supports my method but again, I'm back to that "I haven't gone all in" with it problem. As I'm rambling, I've seen these sexy dashboards with like red/yellow/green lights and some latency graphs and cool looking stuff and then a little table of the last 20 "log messages" and maybe I'm used to looking at logs that you don't show in your dashboard or something like that.

They all feel like a square hole to my round peg. Maybe it's just me.