Live data from Hacker News

Preview of Explore Logs, a new way to browse your logs without writing LogQL

grafana.com

91–95 of 95 posts

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#91

Earlier quoted context omitted.

The simplest technique, and the one I currently use, is just "(n+bad)/(n+good)" where n is basically the strength of a prior belief that bad/good = 1. At some level I think this might replicate TF-IDF[1] but I haven't sat down to prove it or find where they diverge. [1]: https://en.wikipedia.org/wiki/Tf%E2%80%93idf

But this still requires you to classify each line manually to determine bad or good, no?

Not manually, it just requires you to be able to group them along a dimension of interest. For example, if I get a page from us-east-1a, I can compare all the logs from that against us-east-1b. Or, you can group all the logs from the hour after the incident started to the hour a day ago (or a week ago).

I pulled this technique from canary analysis and applied it to production outage analysis. In canary, you have a guaranteed random stable population that lets you perform accurate comparisons. Elsewhere, we can try to make that assumption but it might break down. For example, regional holidays can radically alter customer behavior over time or between regions. So it's not perfect but it's often good enough to provide me insights while on call.

And, it requires advanced log queries to perform all these filtering, grouping, counting and scoring functions.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#92

Earlier quoted context omitted.

But this still requires you to classify each line manually to determine bad or good, no?

Not manually, it just requires you to be able to group them along a dimension of interest. For example, if I get a page from us-east-1a, I can compare all the logs from that against us-east-1b. Or, you can group all the logs from the hour after the incident started to the hour a day ago (or a week ago). I pulled this technique from canary analysis and applied it to production outage analysis. In canary, you have a gu…

OK, I'm starting to see where you're going with this - I also compare incident-affected logs with pre-incident logs, or HTTP requests that I want to debug with similar requests that are known good.

What tools are you using? For me it's often just grep and awk with temp files, maybe a touch of python occasionally.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#93

Earlier quoted context omitted.

Not manually, it just requires you to be able to group them along a dimension of interest. For example, if I get a page from us-east-1a, I can compare all the logs from that against us-east-1b. Or, you can group all the logs from the hour after the incident started to the hour a day ago (or a week ago). I pulled this technique from canary analysis and applied it to production outage analysis. In canary, you have a gu…

OK, I'm starting to see where you're going with this - I also compare incident-affected logs with pre-incident logs, or HTTP requests that I want to debug with similar requests that are known good. What tools are you using? For me it's often just grep and awk with temp files, maybe a touch of python occasionally.

Currently, splunk. But that was expensive before the acquisition, so im sure someone will come along and suggest i replace it with opensearch or something.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#94

Earlier quoted context omitted.

LogQL is nothing like SQL. Try aggregating and quickly you'll be asking yourself wtf an instant query is and how is that different.

I mean, time series data is different from generic tabular data, so obviously there are impedance differences that are reflected in the query languages. I can see how some people might feel more at home using something even more like SQL, but there are a lot of common use cases where SQL is awkward and/or more verbose.

Are logs time series data? That seems to be the thesis behind LogQL. But way more often than not I'm searching for a needle in a haystack, not charting trends over time.

Re: Preview of Explore Logs, a new way to browse your logs without writing LogQL

#95

I'm not really a cloud expert so maybe I'm fundamentally missing something about how I'm "supposed to work", but honestly all I have ever wanted to do, when looking at logs, is see the log from one process, from beginning to end, as a text file. You can of course do this using kubectl but only for the most recent two instances of a given pod which isn't helpful when investigating an incident that happened a while ago…

I use LogQL a fair amount. Often times even just negative filtering is quite useful. I do a fair amount of tracking down of issues with LogQL. Looking for logs specific to a customer support ticket. Filtering for logs by a traceId for distributed traces. I have serious doubts this new UI is something I will care about at all. The explore ui for setting labels is atrocious and painful, and I'd rather just give me the…

> Please FFS someone fix the Ctrl+f creating a vscode like find dialog that only finds inside the text input. I never want to do a find specifically isolated to my LogQL

Quick update: that has been fixed and will be available in Grafana 11.1.

Post reply on HN