Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Preview of Explore Logs, a new way to browse your logs without writing LogQL

grafana.com

Post reply on HN
Print view View original

81–90 of 95 posts

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#81

Post by kiitos »

debarshri wrote: post #66
dstroot wrote: post #51

Earlier quoted context omitted.

Risky how exactly? If it has data in it that it shouldn't it's a problem no matter where it resides.

It is not about what data is in the log. It is about the fact that once the data is download, it most likely going to stay in the machine. Logs often contain privileged info, if not reveal a bit about how the application behaves. It is risky to do that.

If logs contain privileged info, then the damage is done the moment they're published. Whether or not they're downloaded to a laptop is irrelevant, the risk impact is the same.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#83

Post by [deleted] »

School-Cotton wrote: post #24
tkone wrote: post #20

Earlier quoted context omitted.

If you’re debugging something simple or non-distributed, this product isn’t for you. If you’re working on anything distributed, log aggregation becomes a must. But, also, if you’re working on anything distributed and you’re looking at logs, you’re desperate. Distributed traces are so much higher quality.

When I formed these opinions I was working on Materialize, which is basically the polar opposite of "simple and non-distributed". However it was still quite common that I knew exactly which process was doing something weird and unexpected.

[deleted]

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#84

Post by dotancohen »

jldugger wrote: post #52
School-Cotton wrote: post #6

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

> maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. This is still a valid use case but pretend for a minute you have thousands or millions of log lines to inspect. Even after filtering for ERROR level only, you still have too many "those are normal" errors,…

  > then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal
I would love to hear more about this process.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#85

Post by nebster »

liampulles wrote: post #38
School-Cotton wrote: post #27

Earlier quoted context omitted.

That will, if I understand correctly, get the logs for one pod, not for one process. For example if the pod restarted 10 times you will not get 10 separate files from that query.

Even more than that, if you are running multiple instances of the app in multiple pods concurrently, then all of those logs will be joined together.

I'm not sure I really understand this.

If you mean one instance in each pod, then each should be labelled differently and you can filter down to one instance.

If you mean running multiple instances in each pod (and container?), then the standard kubectl log output will also have them all joined together. For both of those, you would need to add another unique identifier to each line, or run each instance in a separate container so you can submit the logs with the pod name and container name combined being the unique identifier.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#86

Post by dzikimarian »

School-Cotton wrote: post #69
dzikimarian wrote: post #67

Earlier quoted context omitted.

Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened. Best to configure view for this to limit columns and maybe pre-configure some filters. Plus annotate your logs with timestamp, so you rely on time of event and not time of ingestion. But these both are one-time configuration thing and then you can simply scroll.

> Well, honestly I don't understand what's missing - you just pick time window, instance and have logs displayed line-by-line as they happened. What's missing is that I don't want to learn and use some clunky web UI in order to do this. I want the UI to be "download this text file" and then use the tools I already know and understand (local text processing utilities and text editors)

If GUI is main issue, you can use CLI client to extract data from elastic :-)

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#87

Post by eddyfromtheblok »

ta1243 wrote: post #62
sofixa wrote: post #36

Earlier quoted context omitted.

> seems a very high barrier to entry compared with running cat It really isn't. It's a single binary with a relatively simple configuration file, you throw logs at it via an API (which a bunch of logging agents support, and syslogs can be sent to it). Then the actual queries aren't all that complex, it's just a difference of cd-ing to the correct folder for the date/server to be able to cat and grep vs writing a quer…

"to be able to cat and grep" Admitably I learned how to use basic tools 25 years ago, but that's an investment that can be used for decades. cat *web*log | "grep 34.5.22.4" | sort -n | less is hardly a complex thing to learn. Sure you can then build on that pipeline -- "cut -b -10|uniq -c" and if you want something really complex then you can use awk, or perl, or python, and do all sorts of things with the data. Will…

Loki has a cli tool, called LogCLI. It's passable for needle-in-haystack searches, and the label browser is handy. But Loki doesn't handle multiline searches well. I'm with you on the ease of grep sort uniq, pretty easy to fashion up a quick report, sorted numerically - No enterprise data analysis suute needed.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#88

Post by willseth »

hamandcheese wrote: post #57
willseth wrote: post #31

Earlier quoted context omitted.

That’s what LogQL is, which is already in Loki. This is a new feature.

LogQL is nothing like SQL. Try aggregating and quickly you'll be asking yourself wtf an instant query is and how is that different.

I mean, time series data is different from generic tabular data, so obviously there are impedance differences that are reflected in the query languages. I can see how some people might feel more at home using something even more like SQL, but there are a lot of common use cases where SQL is awkward and/or more verbose.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#89

Post by jldugger »

dotancohen wrote: post #84
jldugger wrote: post #52

Earlier quoted context omitted.

> maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. This is still a valid use case but pretend for a minute you have thousands or millions of log lines to inspect. Even after filtering for ERROR level only, you still have too many "those are normal" errors,…

> then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal I would love to hear more about this process.

The simplest technique, and the one I currently use, is just "(n+bad)/(n+good)" where n is basically the strength of a prior belief that bad/good = 1. At some level I think this might replicate TF-IDF[1] but I haven't sat down to prove it or find where they diverge.

[1]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#90

Post by dotancohen »

jldugger wrote: post #89
dotancohen wrote: post #84

Earlier quoted context omitted.

> then apply some Laplace smoothing or other bayesian techniques to score fingerprints by strength of association with abnormal I would love to hear more about this process.

The simplest technique, and the one I currently use, is just "(n+bad)/(n+good)" where n is basically the strength of a prior belief that bad/good = 1. At some level I think this might replicate TF-IDF[1] but I haven't sat down to prove it or find where they diverge. [1]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf

But this still requires you to classify each line manually to determine bad or good, no?
Post reply on HN