Grepping logs is still terrible • Hacker News Board

Re: Grepping logs is still terrible

#91

Post by Frondo » Thu, May 07, 2015, 3:25 PM UTC

What you lose when you move away from text logs is not any real benefit; what you lose is the illusion of control you have with text logs.

Text logs can be corrupted, text logs can be made unusable, you need a ton of domain-specific knowledge to even begin to make sense of text logs, etc.

But there's always a sense that, if you had the time, you could still personally extract meaning from them. With binary logs, you couldn't personally sit there and read them out line by line.

The issue is psychology, not pragmatism, and that's why text logs have been so sticky for so long.

Re: Grepping logs is still terrible

#92

Post by arenaninja » Thu, May 07, 2015, 3:39 PM UTC

Great to see some effort in this area. I've been using New Relic and it's pretty great for errors because we've setup Slack/email notifications. However, there's nothing for general log (e.g.: access log) parsing. I'm installing an ELK stack on my machine right now and hope that it's enough

Re: Grepping logs is still terrible

#93

Post by dragonwriter » Thu, May 07, 2015, 3:41 PM UTC

felixgallo wrote: post #90

dragonwriter wrote: post #89

Earlier quoted context omitted.

It seems to me that a simple transitional tool for a binary logging system would be for the implementer of the binary logging system to also include a tool that consumed a binary log file on stdin and produced a stream on stdout in one (or more, selecting which by command line arguments) common text log formats. That lets you develop an ecosystem of supporting tools that take advantage of any strengths of the binary…

what is the point of such a 'transition' if there never arrives any point at which there is net added value to a binary format?

If there is some (not initially necessarily net for all users -- benefit being, after all, something that varies from user to users, but significant for some subset of users) benefit, the point is to mitigate the cost of moving out of a native text format, and increase the number of users for whom there is initially a net benefit, which also increases the initial use of the binary format and the effort likely devoted to building auxiliary tools which leverage it to some advantage, increasing the speed at which the net benefit of the format for a wider range of users is increased.

This may or may not ever make it a net benefit for every user, but that's okay. There's a whole lot of space between "this technology is the best choice for everyone" and "this technology is the best choice for no one".

Re: Grepping logs is still terrible

#94

Post by KaiserPro » Thu, May 07, 2015, 4:01 PM UTC

ownagefool wrote: post #37

KaiserPro wrote: post #21

I think there are a number of issues that are getting mushed into one. * Journal is just terrible. * some text logs are perfectly fine. * when you are in rescue mode, you want text logs * some people use text logs as a way to compile metrics I think the most annoying thing for me about journald is that it forces you to do something their way. However its optional, and in centos7 its turned off, or its beaten into suc…

Not that I think it matters all that much but journald will be running on your centos box, but it'll be configured to spew text into /var/log. Just type journalctl and you should see the data there.

I suspected as much, so long as I don't have to use its tools I couldn't care less. Unless it eats resources.

Re: Grepping logs is still terrible

#95

Post by hartator » Thu, May 07, 2015, 4:18 PM UTC

Isn't everything will be solved by a some kind of grep that's date/timespan aware?

Re: Grepping logs is still terrible

#96

Post by deathanatos » Thu, May 07, 2015, 4:21 PM UTC

robinhouston wrote: post #3

> For example: find all logs between 2013-12-24 and 2015-04-11, valid dates only. That’s a straw man. If you’re grepping logs, you don’t need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates. But I suppose 2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01])) doesn’t look so bad. The whole thing is similarly exaggerated.

> If you’re grepping logs, you don’t need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates.

Even _if_ I agreed with your assumption[1], are you actually suggesting that

    2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01]))

is a serious solution? I admit that it is shorter than the author's solution, _but it still proves his point_.

And then what about multi-line log lines? `grep` can't tell where the next line is; sure, I can -A, but there's no number I can plug in that's going to just work: I need to guess, and if I get a truncated result or too much output, adjust. Worse, I get too much output _and_ a truncated record where I need it…

    log-cat --from 2013-12-24 --to 2015-04-11 | grep

[1] most log file formats I've run across do not guarantee the date to appear in a given location.

Re: Grepping logs is still terrible

#97

Post by deathanatos » Thu, May 07, 2015, 4:40 PM UTC

It seems to me that most of the worry about a binary log file being "opaque" could be solved with a single utility:

    log-cat

… that just outputs it in text. Then you can attack the problem with whatever text-based tools you want.

But to me, having a utility that I could do things like, get a range of log lines — in sorted order —, or, grep on just the message, would be amazing. These are all things that proponents of grep I'm sure will say "you can!" do with grep… but you can't.

The dates example was a good one. I'd much rather:

    log-cat  --from 2014-12-14 --to 2015-01-27

Also, my log files are not "sorted". They are, but they're sorted _per-process_, and I might have multiple instances of some daemon running (perhaps on this VM, perhaps across many VMs), and it's really useful to see their logs merged together[2]. For this, you need to understand the notion of where a record starts and ends, because you need to re-order whole records. (And log records' messages are _going_ to contain newlines. I'm not logging a backtrace on one line.) grep doesn't sort. |sort doesn't know enough about a text log to adequately sort, but

    $ log-cat logs/*.log --from 2014-12-14 --to 2015-01-27

Binary files offer the opportunity for structured data. It's really annoying to try to find all 5xx's in a log, and your grep matches the process ID, the line number, the time of day…

I've seen some well-meaning attempts at trying to do JSON logs, s.t. each line is a JSON object[1]. (I've also seen it attempted were all that is available is a rudimentary format string, and the first " breaks everything.)

Lastly, log files sometimes go into metrics (I don't really think this is a good idea, personally, but we need better libraries here too…). Is your log format even parseable? I've yet to run across one that had an unambiguous grammar: a newline in the middle of a log message, with the right text on the second line, can easily get picked up as a date, and suddenly, it's a new record. Every log file "parser" I've seen was a heuristic matcher, and I've seem most all of them make mistakes. With the simple "log-cat" above, you can instantly turn a binary log into a text one. The reverse — if possible — is likely to be a "best-effort" transformation.

[1]: the log writer is forbidden to output a newline inside the object. This doesn't diminish what you can output in JSON, and allows newline to be the record separator.

[2]: I get requests from mobile developers tell me that the server isn't acting correctly all the time. In order to debug the situation, I first need to _find_ their request in the log. I don't know what process on what VM handled their request, but I often have a _very_ narrow time-range that it occurred in.

Re: Grepping logs is still terrible

#98

Post by ghshephard » Thu, May 07, 2015, 4:42 PM UTC

nrr wrote: post #72

ghshephard wrote: post #44

Earlier quoted context omitted.

grep definitely breaks down on large systems. I have one environment with approx 5 million nodes - (1e6), and the only way to coherently manage the log updates from them is in binary format. But even still - I like to have the text files as journals of original entry - so I can occasionally do a tail -f incoming.log| egrep -i "somedevice". And having the original files in text format is zero impediment to getting the…

I hate arguing semantics, but 1e6 is not just large but very large indeed. (: That said, I'd be curious to know some more of the details of that system actually! If you're aggregating all of those devices together, using something binary in that context definitely makes sense. In fact, if I were in your shoes and tasked with designing some means of solving that problem, I would probably use something like protobuf or…

The remote nodes all log to central DNS servers, and Trap Servers. The DNS servers have a nice update.log file that provides their IP address information, and some nice text configs. The trap data, goes into a binary file (database actually) and requires analysis through a web interface.

As a result - the DNS updates are used by me approximately 20x more often than the trap data, when doing diagnostics, even though, in theory, the trap data is incredibly richer, and, of course, has the 15 mandatory fields that are functions of the binary logging. (Time, Date, Event ID, Trap Type, etc, etc...)

Re: Grepping logs is still terrible

#99

Post by sobkas » Thu, May 07, 2015, 4:52 PM UTC

ghshephard wrote: post #2

It's beyond me how he doesn't understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job). Here is the thing he doesn't seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into data…

> text logs are a universal format

Then punch in the face is a universal form of communication. Also EBCDIC is the only encoding future will recognize!

Re: Grepping logs is still terrible

#100

Post by kbenson » Thu, May 07, 2015, 4:54 PM UTC

benjaminjackman wrote: post #66

ghshephard wrote: post #54

Earlier quoted context omitted.

Well - to be clear, if I I run into a log file with it's data on a single line, 95% of the time it will take 95% of the time I just give up on the multi-line .json files - unless it's really, really critical, I probably don't want to spend 30 minutes writing code to re-assemble the data. Text Log files, wherever possible, should capture their data on a single line. If they need to go multi-line, then having a transac…

Maybe I am misunderstanding but it sounds like you are encountering bad json log file practices because json entries are spanning multiple lines. Which implies they are being printed in non compact form aka prettified. Thats a problem in the pure text world too. And hurts worse when it happens there. Its kind of an apples to oranges comparison. Json log files should ideally print using compact form (which will never…

I have no problem with json output in log files, but I would greatly prefer it be constrained to the message portion of a logline. At a minimum I generally want three things per line, a timestamp (in ISO 8601 or something close), a message type (info, warning, error, etc) or log entry source, and the message itself. I don't want to be looking into the JSON structure itself for a timestamp, especially when the field encoding the timestamp may be called something slightly different based on what generated the log...

In that respect, whether the message is JSON, or YAML, or XML doesn't matter, that can easily be worked on later, but the first thing I want to be able to do is filter by time and type.