Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is still terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

31–40 of 124 posts

Re: Grepping logs is still terrible

#31

Post by laumars »

Oh jeez. Yes there are better and more performant tools for parsing optimised binary databases; nobody disputes that. And yes, tools like Splunk are more user friendly than grep; nobody disputes that either. But to advocate a binary only system for logs is short sighted because logs are the goto when everything else fails and thus need to be readable when every other tool dies. There's quite a few scenarios that could cause this too:

  * log file corruption - text parsing would still work,

  * tooling gets deleted - there's a million ways you
    can still render plain text even when you've lost
    half your POSIX/GNU userland,

  * network connection problems, breaking push to a
    centralised database - local text copies would still
    be readable.
In his previous blog post he commented that there's no point running both a local text version and a binary version, but since the entirety of his rant is really about tooling rather than log file format, I'm yet to see a convincing argument against running the two paradigms in parallel.

Re: Grepping logs is still terrible

#32

Post by pjc50 »

mdekkers wrote: post #12
pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

Power-law distribution and economies of scale?

There are a lot of hobbyists, a vast number of people with a Linux box in the corner of the office or a few cloud instances, a smaller number of people running IT for multinationals and one or two people who have whole datacenters to themselves. The larger the system, the lower the computer/human ratio.

Re: Grepping logs is still terrible

#33

Post by crdoconnor »

scrollaway wrote: post #8
ghshephard wrote: post #2

It's beyond me how he doesn't understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job). Here is the thing he doesn't seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into data…

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

I've been producing a few services recently which output a chunk of JSON for each log message followed by a newline.

I think it actually solves most of the problems text logs have that binary don't (inability to easily present structured data, etc.) yet keeps the advantages of a text log (human readable, resistant to file corruption, future-proof).

Re: Grepping logs is still terrible

#34

Post by erikb »

After reading the article I wonder if there are lots of tools that do all the binary advantages in indexes but leave the logs as text files, why that is not fine. To get the binary advantage the log does not have to be binary.

The example with the timestamps is also strange. No matter how you store the timestamps, parsing a humanly reasonable query like "give me 10 hours starting from last Friday 2am" to an actual filter is a complex problem. The problem is complex no matter how you store your timestamp. You can choose to do the complexity before and create complex index structures. You can choose to have complex algorithms to parse simple timestamps in binary or text form, you can build complex regexes. But something needs to be complex, because the problem space is. Just being binary doesn't help you.

And that's really the point here, isn't it? Just being binary in itself is not an advantage. It doesn't even mean by itself that it will save disk space. But text in itself is an advantage, always, because text can be read by humans without help (and in some instances without any training or IT education), binary not.

Yesterday I was thinking there might be something about binary logs. Now I'm convinced there isn't. The only disadvantage seems to be that you also lose disk space if you store it in clear text. But disk space isn't an issue in most situations (and in many situations where it is an issue you might have resources and tools at hand to handle that as well) It is added complexity for no real advantage. Thanks for clearing that up.

Re: Grepping logs is still terrible

#35

Post by icebraining »

mrweasel wrote: post #18
pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

>For small systems, grep works fine That really the key for me. My go to example is searching for IP numbers across different logs. If I have just one machine, and I want to find an IP in the SSH, web and mail logs I shouldn't have to use multiple tools for getting that data. Logstash, Splunk and other tools store stuff binary, as he writes, and that's perfectly valid, the only solution in fact. But I don't want to b…

In the setup the author presents, using syslog-ng and elasticsearch, it seems the logs are serialized as json for the transmission.

Re: Grepping logs is still terrible

#36

Post by ghshephard »

crdoconnor wrote: post #33
scrollaway wrote: post #8

Earlier quoted context omitted.

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

I've been producing a few services recently which output a chunk of JSON for each log message followed by a newline. I think it actually solves most of the problems text logs have that binary don't (inability to easily present structured data, etc.) yet keeps the advantages of a text log (human readable, resistant to file corruption, future-proof).

Speaking for myself - multiline .json output is problematic, as most of the parsing tools work best when the data is on a single line, and it's a cognitive struggle to deal with multi-line output, even if you are clever with your tools. I usually have to end up writing a json parser in python to get the data into a format that I can manipulate it. (Thankfully, python does 95% of the work for you when reading a json file)

But - here is the thing, even though the .json format isn't convenient for me, I can, with about 20-30 minutes effort, write a parser that can get the data into a convenient format, because it started out as a text file.

Re: Grepping logs is still terrible

#37

Post by ownagefool »

KaiserPro wrote: post #21

I think there are a number of issues that are getting mushed into one. * Journal is just terrible. * some text logs are perfectly fine. * when you are in rescue mode, you want text logs * some people use text logs as a way to compile metrics I think the most annoying thing for me about journald is that it forces you to do something their way. However its optional, and in centos7 its turned off, or its beaten into suc…

Not that I think it matters all that much but journald will be running on your centos box, but it'll be configured to spew text into /var/log.

Just type journalctl and you should see the data there.

Re: Grepping logs is still terrible

#38

Post by ghshephard »

erikb wrote: post #34

After reading the article I wonder if there are lots of tools that do all the binary advantages in indexes but leave the logs as text files, why that is not fine. To get the binary advantage the log does not have to be binary. The example with the timestamps is also strange. No matter how you store the timestamps, parsing a humanly reasonable query like "give me 10 hours starting from last Friday 2am" to an actual fi…

With regards to disk space - compressed text logs are pretty common. The frequency with which they are compressed is adjustable, and, gzcat is a pretty well known mechanism for opening them.

Re: Grepping logs is still terrible

#39

Post by rubenv »

ghshephard wrote: post #13
scrollaway wrote: post #8

Earlier quoted context omitted.

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

But at least you have a fighting chance. What if that exact same data was dumped into a binary file, that you did not know how to decode? Originally, you had a problem - the data wasn't formatted in a manner that you could parse cleanly. Now, you have a new problem - not only is the data not formatted properly, it's now in some opaque binary file. Saying that there are poorly formatted text files isn't a hit against…

Sure, an opaque binary file is pointless.

But that's not what most logging systems that log to binary files offer. They give you specs (example: http://www.freedesktop.org/wiki/Software/systemd/journal-fil...) and tools.

Binary doesn't have to mean closed/opaque.

Re: Grepping logs is still terrible

#40

Post by nailer »

robinhouston wrote: post #3

> For example: find all logs between 2013-12-24 and 2015-04-11, valid dates only. That’s a straw man. If you’re grepping logs, you don’t need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates. But I suppose 2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01])) doesn’t look so bad. The whole thing is similarly exaggerated.

Using regexs for time is like using regexs for HTML: it's possible-ish, but most people are probably doing it wrong and storing things using their correct data structures is a much simpler solution.
Post reply on HN