Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preview of Explore Logs, a new way to browse your logs without writing LogQL

grafana.com

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#41

Post by skrtskrt »

nine_k wrote: post #11
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

Could LogQL do.something like select * from stdout, stderr where session_id = 123456 ? If not, why?

yep, with the caveat that you probably don't want to have the backend of whatever log system you use (not exactly sure how Loki does it) to have an index on something as high-cardinality as session id so that query could get slow.

But these log query systems can also optimize these queries for instance by by sampling, using distributed trace ids to ensure you get shown corresponding, allowing you to get only logs where at least one step in the trace errored, etc.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#42

Post by nklmilojevic »

jdoss wrote: post #3

I recently setup Victoria Metrics + https://github.com/prometheus/snmp_exporter + Grafana to get start tracking bandwidth on my top of rack switches in my Datacenter rack which has been a pretty awesome setup. The way you can auto generate a config for your SNMP MIBs with SNMP Exporter was unexpectedly not a terrible experience. My next task is to get centralized logging going with Victoria Logs + Vector, I'll have t…

This is what I've been doing on my cluster:

https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...

https://github.com/nklmilojevic/home/tree/main/kubernetes/ap...

Here you have Vector in aggregator + agent mode and several sources. VictoriaLogs also recently added Grafana datasource so it is fairly easy to set it up:

https://github.com/nklmilojevic/home/blob/main/kubernetes/ap...

I'm a big fan of VictoriaMetrics as well and we use it extensively in my company at high scale.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#43

Post by blue_cookeh »

wsatb wrote: post #9

I'll preface this with the fact I haven't look at Loki in a bit, so maybe this has changed. But I found the documentation needing a lot of work and the configuration for promtail to be obtuse and not very user friendly. I haven't used it for those reasons, not because of the query language.

It's also incredibly easy to shoot yourself in the foot and rack up huge cloud bills - something we recently hit: https://github.com/grafana/loki/issues/8756

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#44

Post by skrtskrt »

liampulles wrote: post #39
wsatb wrote: post #9

I'll preface this with the fact I haven't look at Loki in a bit, so maybe this has changed. But I found the documentation needing a lot of work and the configuration for promtail to be obtuse and not very user friendly. I haven't used it for those reasons, not because of the query language.

Our team uses loki and I have to say I think their collected helm charts are pretty easy to use - my problem is more that it seems to be quite slow to run on-prem. Very often my loki query times out and I have to do more work filtering down the log lines or selecting a narrower time range. I'm kind of amazed the UI doesn't select small time ranges iteratively to build up the response, especially since I believe this…

Loki UI in Grafana Explore seems to only select 1000 lines by default for me?

Also Loki on the backend splits/parallelizes requests if it can.

The Grafana backend Mimir / Loki / Tempo products all appear to be architected pretty similar, and I'm more experienced operating Mimir, but the answer to read load often has to just do with right-sizing the deployment scale, and using caches aggressively.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#46

Post by debarshri »

fellerts wrote: post #17
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

Just want to chip in and say that I wholeheartedly agree with you. I'm not a cloud developer either, but I'm regularly forced into what's apparently called "Google Cloud's operations suite" to grovel through logs. Compared to working with Linux journals using the tried and true text manipulation tools, it feels like looking through a straw with oven mitts on. I'd happily download a 500 MB text file instead, but there…

It fairly risky to download 500MB of log and analyse it locally in the machine. I know People do it anyways. Just saying.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#48

Post by bennine »

andres wrote: post #21
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

If you get a chance, please check out kubetail ( https://github.com/kubetail-org/kubetail ). It's an open source log viewer for Kubernetes. Currently you can use it to look at pod logs from beginning to end, grouped together by workload (e.g. Deployment, CronJob) with basic filtering available (e.g. node-id, AZ). It doesn't let you look at historical logs yet but that's where we're headed. We just launched so we're e…

Interesting. Will be following this tool.

There is a CLI tool with the same name that does something similar - https://github.com/johanhaleby/kubetail

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#49

Post by remram »

liampulles wrote: post #38
School-Cotton wrote: post #27

Earlier quoted context omitted.

That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.

Even more than that, if you are running multiple instances of the app in multiple pods concurrently, then all of those logs will be joined together.

That's definitely false

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#50

Post by jhrmnn »

School-Cotton wrote: post #24
tkone wrote: post #20

Earlier quoted context omitted.

If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.

When I formed these opinions I was working on Materialize, which is basically the polar opposite of "simple and non-distributed". However it was still quite common that I knew exactly which process was doing something weird and unexpected.

Maybe it’s the difference between tracking a bug (abnormal operation) vs understanding behavior of a complex system (normal operation)?
Post reply on HN