While this is a step in the right direction, just let me write something closer to SQL. Influx did this correctly.
That’s what LogQL is, which is already in Loki. This is a new feature.
Preview of Explore Logs, a new way to browse your logs without writing LogQL
31–40 of 95 posts
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#32I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
Could LogQL do.something like select * from stdout, stderr where session_id = 123456 ? If not, why?
yes it can, if you tag your log stream correctly - either by having the stream externally tagged via attributes, or internally by following certain conventions in the log line.
You can also do something like
select client_ip from requests where elapsed_ms > 10000
which is incredibly powerful
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#33I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
I'm an old fart so I use things like "cat" and "grep", and maybe "sed" and "cut" if the lines are particularly long. I have one log file per day per host on my syslog server and can use "sort" to order across multiple files. Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.
You may be amazed at how hard these tools are to get started with relative to that. I have been thoroughly unimpressed with and unable to really get started with any of these tools because of the overemphasis on cloud. Not sure what people were doing before, but sshing to the prod box kinda sucks.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#34Earlier quoted context omitted.
Fwiw this is how I use Loki most of the time. Pick an app label, pick a time period, look at raw logs. The LogQL for this ends up something like `{app="workload-foo"}`. Loki is excellent at that. Then if I know which pod I'll filter down to a specific pod with `{pod="workload-foo-1234"}`, sometimes I'll search for a specific term (error message etc) with `{pod="workload-foo-1234"} |= "error message"` then look at the…
That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.
You'd have the label shown in the output that indicates the log line in question is from a different process/pod/container/host/whatever.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#35I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
I'm an old fart so I use things like "cat" and "grep", and maybe "sed" and "cut" if the lines are particularly long. I have one log file per day per host on my syslog server and can use "sort" to order across multiple files. Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.
For analysing text logs lnav is pretty good, if you need to work with a live updated view of the log in response to commands.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#36I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
I'm an old fart so I use things like "cat" and "grep", and maybe "sed" and "cut" if the lines are particularly long. I have one log file per day per host on my syslog server and can use "sort" to order across multiple files. Loki was sold to me at fosdem a couple of years ago as this, but I still haven't got round to working it out, seems a very high barrier to entry compared with running cat.
> seems a very high barrier to entry compared with running cat
It really isn't. It's a single binary with a relatively simple configuration file, you throw logs at it via an API (which a bunch of logging agents support, and syslogs can be sent to it).
Then the actual queries aren't all that complex, it's just a difference of cd-ing to the correct folder for the date/server to be able to cat and grep vs writing a query that selects by server name and filters by date.
The learning curve and maintenance of Loki are quite minimal, but the value add is quite significant in most cases. Being able to do cross-host queries, metrics from logs (how many times did error X occur in the logs), as well as easy visualisations is pretty useful.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#37I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.
Sorry, did plenty of "distributed" tracing back in the day and this is just not the case. I can't help but feel like you're after-the-fact rationalizing as if you need this for diagnosing anything "distributed" or "complicated".
Distributed anything is actually easier in most cases because you will always have input and output. Sure, if you're debugging a complicated and coordinated "dance" between two concurrent threads/processes then yeah fully agreed, but then you're deep in uncharted territory and you need all the help you can get.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#38Earlier quoted context omitted.
Fwiw this is how I use Loki most of the time. Pick an app label, pick a time period, look at raw logs. The LogQL for this ends up something like `{app="workload-foo"}`. Loki is excellent at that. Then if I know which pod I'll filter down to a specific pod with `{pod="workload-foo-1234"}`, sometimes I'll search for a specific term (error message etc) with `{pod="workload-foo-1234"} |= "error message"` then look at the…
That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.
Even more than that, if you are running multiple instances of the app in multiple pods concurrently, then all of those logs will be joined together.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#39I'll preface this with the fact I haven't look at Loki in a bit, so maybe this has changed. But I found the documentation needing a lot of work and the configuration for promtail to be obtuse and not very user friendly. I haven't used it for those reasons, not because of the query language.
Our team uses loki and I have to say I think their collected helm charts are pretty easy to use - my problem is more that it seems to be quite slow to run on-prem. Very often my loki query times out and I have to do more work filtering down the log lines or selecting a narrower time range.
I'm kind of amazed the UI doesn't select small time ranges iteratively to build up the response, especially since I believe this is what the CLI does. Perhaps this is also part of their cloud offering provides and it is part of their marketing strategy. Not a good one because if we came down to the decision I would start by looking for something else from being p'ed off by Loki.
But I guess it still works pretty well considering it is free.
Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL
#40I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…
Dump the logs into elastic, loki or whatever, along with pod name as a label. Usually I use Kibana, so I don't want to speak for Loki, but seems pretty straight forward.
You missed the key criterion, which is being able to see the logs from that process "as a text file", or the way I'd rephrase it "with the same ease of a text file."
Kibana is ok (definitely beats grep) when you want to look across a fleet and determine if a specific thing is happening. But when you have a specific symptom that happens on a particular instance, what you want to do is see logs in the order they happened, and Kibana isn't close. Querying and viewing logs are just slow and cumbersome relative to less/grep.