After having used Datadog for several years, going back to Grafana / Loki / Prometheus felt like regressing by two decades. As much as I appreciate free solutions, I feel like Grafana has really fallen behind when it comes to developer experience
Preview of Explore Logs, a new way to browse your logs without writing LogQL
61–70 of 95 posts
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#62Earlier quoted context omitted.
I'm an old fart so I use things like "cat" and "grep", and maybe "sed" and "cut" if the lines are particularly long. I have one log file per day per host on my syslog server and can use "sort" to order across multiple files. Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.
> seems a very high barrier to entry compared with running cat It really isn't. It's a single binary with a relatively simple configuration file, you throw logs at it via an API (which a bunch of logging agents support, and syslogs can be sent to it). Then the actual queries aren't all that complex, it's just a difference of cd-ing to the correct folder for the date/server to be able to cat and grep vs writing a quer…
"to be able to cat and grep"
Admitably I learned how to use basic tools 25 years ago, but that's an investment that can be used for decades.
cat *web*log | "grep 34.5.22.4" | sort -n | less
Will whatever today's favoured log query/filter/etc be around in 25 years? Last time I looked at this people were going on about logstash and elasticsearch. Nobody could show me how to to the above command without touching the mouse.
Now sure, cat and grep can be sluggish on millions of lines (which is the main reason I'm tempted by loki or similar), and there's always some twat that comes along with "useless use of cat" [0], but the kind of pipeline processing serves me well and it seems a very different way to think about things when you need to access things from a database. Maybe I'm in a local maximum, but it's good-enough for me to find out what's going on.
[0] https://stackoverflow.com/questions/11710552/useless-use-of-...
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#63Earlier quoted context omitted.
It fairly risky to download 500MB of log and analyse it locally in the machine. I know People do it anyways. Just saying.
Risky how exactly? If it has data in it that it shouldn't it's a problem no matter where it resides.
In theory a logfile could contain privileged information (indeed it almost certainly will - IP addresses etc), putting that on a laptop increases risk of losing it.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#64Earlier quoted context omitted.
If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.
Yup and the reason no one markets something like "tail the logs for server X" is because, if you're talking in the context of an individual server, you're too small for anyone to care about.
I've got logs from hundreds of servers that I use standard tools to look at, and that's a small system. Centralising logs has been a thing for decades.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#65After having used Datadog for several years, going back to Grafana / Loki / Prometheus felt like regressing by two decades. As much as I appreciate free solutions, I feel like Grafana has really fallen behind when it comes to developer experience
Could you provide more details? Although I've never had the opportunity to use Datadog at any of my previous positions, I am quite familiar with Grafana and I'm generally pretty happy with it.
What's the TL;DR for why Datadog is better?
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#66Earlier quoted context omitted.
It fairly risky to download 500MB of log and analyse it locally in the machine. I know People do it anyways. Just saying.
Risky how exactly? If it has data in it that it shouldn't it's a problem no matter where it resides.
It is not about what data is in the log. It is about the fact that once the data is download, it most likely going to stay in the machine.
Logs often contain privileged info, if not reveal a bit about how the application behaves. It is risky to do that.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#67Earlier quoted context omitted.
Dump the logs into elastic, loki or whatever, along with pod name as a label. Usually I use Kibana, so I don't want to speak for Loki, but seems pretty straight forward.
You missed the key criterion, which is being able to see the logs from that process "as a text file", or the way I'd rephrase it "with the same ease of a text file." Kibana is ok (definitely beats grep) when you want to look across a fleet and determine if a specific thing is happening. But when you have a specific symptom that happens on a particular instance, what you want to do is see logs in the order they happen…
Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened.
Best to configure view for this to limit columns and maybe pre-configure some filters. Plus annotate your logs with timestamp, so you rely on time of event and not time of ingestion.
But these both are one-time configuration thing and then you can simply scroll.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#68Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#69Earlier quoted context omitted.
You missed the key criterion, which is being able to see the logs from that process "as a text file", or the way I'd rephrase it "with the same ease of a text file." Kibana is ok (definitely beats grep) when you want to look across a fleet and determine if a specific thing is happening. But when you have a specific symptom that happens on a particular instance, what you want to do is see logs in the order they happen…
Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened. Best to configure view for this to limit columns and maybe pre-configure some filters. Plus annotate your logs with timestamp, so you rely on time of event and not time of ingestion. But these both are one-time configuration thing and then you can simply scroll.
> Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened.
What's missing is that I don't want to learn and use some clunky web UI in order to do this. I want the UI to be "download this text file" and then use the tools I already know and understand (local text processing utilities and text editors)
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#70Earlier quoted context omitted.
> seems a very high barrier to entry compared with running cat It really isn't. It's a single binary with a relatively simple configuration file, you throw logs at it via an API (which a bunch of logging agents support, and syslogs can be sent to it). Then the actual queries aren't all that complex, it's just a difference of cd-ing to the correct folder for the date/server to be able to cat and grep vs writing a quer…
"to be able to cat and grep" Admitably I learned how to use basic tools 25 years ago, but that's an investment that can be used for decades. cat *web*log | "grep 34.5.22.4" | sort -n | less is hardly a complex thing to learn. Sure you can then build on that pipeline -- "cut -b -10|uniq -c" and if you want something really complex then you can use awk, or perl, or python, and do all sorts of things with the data. Will…
"it's just a difference of cd-ing to the correct folder for the date/server" to be able to cat/grep.
You have to connect to your server, get to the correct folder, and then run the cats and greps which are easy (if you have to do some more advanced filtering with awk it gets more complicated.)
Connecting to Grafana and running a simple label query is practically the same in terms of complexity and time, but with vastly more features available.
> Will whatever today's favoured log query/filter/etc be around in 25 years? Last time I looked at this people were going on about logstash and elasticsearch. Nobody could show me how to to the above command without touching the mouse.
You can run ElasticSearch queries via the API, and can still do it today. I don't know about, but Loki is a statically compiled binary with only optional external dependencies. You'd still be able to run it in 25 year just fine.