Live data from Hacker News

Tips for analyzing logs

jvns.ca

51–60 of 136 posts

Re: Tips for analyzing logs

#51

Jeez, this stuff is frontpage on HN? Sounds… pretty basic. I’m sure our AI overlords could produce deeper content.

I'm interested in this idea as well. Seems like it would be useful for detecting unusual issues assuming you have a large enough data set of "normal" runs.

I've been mulling this idea over in my head as well. I have a fleet of PCs out in the wild, all running the same software. It would be nice to have an easy way to detect strange behavior like processes that are continually respawning / segfaults / crashes / etc, without explicitly writing a bunch of search terms.

Re: Tips for analyzing logs

#52
post #8
post #7

> Often log lines will include a request ID. Yes, always include a request id in every request structure you create and include it also in the response and print it. It would seem something obvious that everyone does by default but instead, no, it's not so obvious it seems.

Not so obvious. How to implement it without passing request id to all the functions, when they are unrelated to request/http ? Especially in languages without thread-locals such as javascript?

Nodejs now has async local storage built in, which makes this fairly easy.

Re: Tips for analyzing logs

#53
I wrote https://github.com/ljw1004/seaoflogs - an interactive filtering tool, for similar ends to what's described here. I wrote it because my team was struggling to analyze LSP logs (that's the protocol used by VSCode to communicate with language servers). But I made it general-purpose able to analyze more log formats too - for instance, we want to correlate LSP logs with server logs and other traffic logs.

(1) I wanted something where colleagues could easily share links in workplace chat with each other, so we could cooperatively investigate bugs.

(2) For LSP we're often concerned with responsiveness, and I thought the best way to indicate times when viewing a log is with whitespace gaps between log messages in proportion to their time gap.

(3) For LSP we have lots of interleaved activity going on, and I wanted to have visual "threads" connecting related logs.

(4) As the post and lnav say, interactivity is everything. I tried to take it a step further with (1) javascript, (2) playground-style updates as you type, (3) autocomplete which "learns" what fields are available from structured logs.

My tool runs all in the browser. (I spent effort figuring out how people can distribute it safely and use it for their own confidential logs too). It's fast enough up to about 10k lines of logs.

Re: Tips for analyzing logs

#54
post #53

I wrote https://github.com/ljw1004/seaoflogs - an interactive filtering tool, for similar ends to what's described here. I wrote it because my team was struggling to analyze LSP logs (that's the protocol used by VSCode to communicate with language servers). But I made it general-purpose able to analyze more log formats too - for instance, we want to correlate LSP logs with server logs and other traffic logs. (1) I wa…

That sounds really interesting, although too limited for my current work focus which deals with Really Big logs.

I tried the demo link, it rendered a lonely @ at the top left of the page, the rest was blank. The console said:

    Uncaught TypeError: [].filter(...).reverse()[0] is undefined
        onload https://ljw1004.github.io/seaoflogs/demo.html:14
This was in Firefox 98.0 in Windows.

Re: Tips for analyzing logs

#56
post #25

My tips for analyzing logs: cd ~/logs cat * | grep

I guess it depends, but would you not encounter issues like permission blocks, or there being so much content in the logs that it is slow to find what you need?

Re: Tips for analyzing logs

#57
post #19

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Huh, I almost posted a duplicate recommendation. My only complaint with lnav was that it had to be built from source on Linux and the build was frigging huge. Apparently they have a pre-compiled linux-musl binary now.

> that it had to be built from source on Linux

Seems to be packaged quite widely https://repology.org/project/lnav/versions

Re: Tips for analyzing logs

#58
One of my pet peeves is "The Useless Use of cat Award". Someone awarded it to me as a teenager in the late 90s and I've been sore ever since.

Yup, it's often a waste of resources to run an extra 'cat'. It really demonstrates that you don't have the usage of the command receiving the output completely memorized. You know, the thousand or so commands you might be piping it into.

But, if you're doing a 'useless' use of cat, you're probably just doing it in an interactive session. You're not writing a script. (Or maybe you are, but even still, I bet that script isn't running thousands of times per second. And if it is, ok, time to question it).

So you're wasting a few clock cycles. The computer is doing a few billion of these per second? By the time you explain the 'useless' use of cat to someone, the time you wasted explaining to them why they are wrong, is greater than the total time that their lifetime usage of cat was going to waste.

There's a set of people who correct the same three pairs of homophones that get used incorrectly, but don't know what the word 'homophone' is. (Har har, they're/their/there). I always liken the people who are so quick to chew someone out for using cat, in the same batch of people who do this: what if I just want to use cat because it makes my command easier to edit? I can click up, warp to the front of the line, and change it real quick.

Sorry. I did say, it is a pet peeve.

Re: Tips for analyzing logs

#59

Has anyone tried passing logs into chatGPT? I had thought it would be especially effective here.

It may work for a few lines, but you can't really pipe a GB of logs into it, which is what we typically deal with.

It works for simple explanations though.

Re: Tips for analyzing logs

#60

My biggest quality of life improvement for understanding logs has been lnav ( https://lnav.org/ ) -- does everything mentioned in this post in a single tool with interactive filtering and quick logical and time based navigation.

Our Easy Data Transform software is intended for data wrangling and desktop ETL. But it has lots of features useful for browsing and manipulating log files, including:

* powerful filtering (including with regex)

* smooth scrolling of millions of rows of data

* support for csv, text, xml, json, Excel and other formats

* available for Window and Mac

Post reply on HN