Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preview of Explore Logs, a new way to browse your logs without writing LogQL

grafana.com

Post reply on HN
Print view View original

21–30 of 95 posts

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#21

Post by andres »

School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

If you get a chance, please check out kubetail (https://github.com/kubetail-org/kubetail). It's an open source log viewer for Kubernetes. Currently you can use it to look at pod logs from beginning to end, grouped together by workload (e.g. Deployment, CronJob) with basic filtering available (e.g. node-id, AZ). It doesn't let you look at historical logs yet but that's where we're headed. We just launched so we're eager for feedback and we like to build out new features quickly.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#22

Post by pluies »

skrtskrt wrote: post #19
candiddevmike wrote: post #2

Why have explore logs as a separate app instead of bundled with Loki? It would be nice if Loki had the same kind of barebones querying/debugging functionality as Prometheus...

Loki is just the backend just like Prometheus is just the backend

Yeah but Prometheus has a web ui where you can run PromQL queries and it'll give you basic graphs back, which is handy for throwing a quick query at it before putting it into something more long-term like a Grafana dashboard or an alerting rule.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#23

Post by pluies »

School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

Fwiw this is how I use Loki most of the time. Pick an app label, pick a time period, look at raw logs. The LogQL for this ends up something like `{app="workload-foo"}`. Loki is excellent at that.

Then if I know which pod I'll filter down to a specific pod with `{pod="workload-foo-1234"}`, sometimes I'll search for a specific term (error message etc) with `{pod="workload-foo-1234"} |= "error message"` then look at the logs around that. There's really no point writing complicated queries unless you need to.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#24

Post by School-Cotton »

tkone wrote: post #20
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.

When I formed these opinions I was working on Materialize, which is basically the polar opposite of "simple and non-distributed". However it was still quite common that I knew exactly which process was doing something weird and unexpected.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#25

Post by jdoss »

NortySpock wrote: post #14
jdoss wrote: post #3

I recently setup Victoria Metrics + https://github.com/prometheus/snmp_exporter + Grafana to get start tracking bandwidth on my top of rack switches in my Datacenter rack which has been a pretty awesome setup. The way you can auto generate a config for your SNMP MIBs with SNMP Exporter was unexpectedly not a terrible experience. My next task is to get centralized logging going with Victoria Logs + Vector, I'll have t…

I've been eyeing a VictoriaLogs setup for my docker container fleet, but I haven't quite spotted where docker's remote logging export options overlap with VictoriaLogs ingestion options. Wrinkle: two docker remote logging plugins I tried (e.g. loki, elastic) didn't seem to work on ARM processors out of the box.

Check out Vector for shipping logs from Docker. It might work out for you https://vector.dev/docs/reference/configuration/sources/dock...

I use Podman for all of my container stuff and there are issues with how Podman produces JSON logs

https://github.com/vectordotdev/vector/issues/6807 https://github.com/containers/podman/issues/16317

which needs to get fixed before I can use it for my workloads.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#26

Post by skrtskrt »

pluies wrote: post #22
skrtskrt wrote: post #19

Earlier quoted context omitted.

Loki is just the backend just like Prometheus is just the backend

Yeah but Prometheus has a web ui where you can run PromQL queries and it'll give you basic graphs back, which is handy for throwing a quick query at it before putting it into something more long-term like a Grafana dashboard or an alerting rule.

wow I always thought that Prom UI was just a tacked-on part of alertmanager or something because it’s so rudimentary. In my experience, everyone just uses Grafana Explore since that’s what Grafana was originally purpose-built for and it’s crazy easy to set up. Just pull down a container or helm chart or whatever.

Since Grafana built Loki, it doesn't make any sense why Grafana would create a separate querying UI app for Loki when they already have Grafana Explore. Prometheus (and I assume its UI) was created by Google [edit: sorry, created by SoundCloud, inspired by a Google Borg tool] before Grafana became the de facto Prometheus query UI, so it’s not really analogous.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#27

Post by School-Cotton »

pluies wrote: post #23
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

Fwiw this is how I use Loki most of the time. Pick an app label, pick a time period, look at raw logs. The LogQL for this ends up something like `{app="workload-foo"}`. Loki is excellent at that. Then if I know which pod I'll filter down to a specific pod with `{pod="workload-foo-1234"}`, sometimes I'll search for a specific term (error message etc) with `{pod="workload-foo-1234"} |= "error message"` then look at the…

That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#28

Post by mason55 »

tkone wrote: post #20
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.

Yup and the reason no one markets something like "tail the logs for server X" is because, if you're talking in the context of an individual server, you're too small for anyone to care about.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#29

Post by richardwhiuk »

skrtskrt wrote: post #26
pluies wrote: post #22

Earlier quoted context omitted.

Yeah but Prometheus has a web ui where you can run PromQL queries and it'll give you basic graphs back, which is handy for throwing a quick query at it before putting it into something more long-term like a Grafana dashboard or an alerting rule.

wow I always thought that Prom UI was just a tacked-on part of alertmanager or something because it’s so rudimentary. In my experience, everyone just uses Grafana Explore since that’s what Grafana was originally purpose-built for and it’s crazy easy to set up. Just pull down a container or helm chart or whatever. Since Grafana built Loki, it doesn't make any sense why Grafana would create a separate querying UI app f…

Prometheus was created by SoundCloud, not Google, but was inspired by the Google Borgmon tool.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#30

Post by ta1243 »

School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

I'm an old fart so I use things like "cat" and "grep", and maybe "sed" and "cut" if the lines are particularly long.

I have one log file per day per host on my syslog server and can use "sort" to order across multiple files.

Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.

Post reply on HN