Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is still terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

71–80 of 124 posts

Re: Grepping logs is still terrible

#71

Post by EmanueleAina »

aidos wrote: post #15
scrollaway wrote: post #8

Earlier quoted context omitted.

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

Like you, I also deal with the kinda weird uwsgi logs. I feel like "universal format" probably didn't mean the format of all the lines in all the logs is the same - though your definition is probably more accurate. Despite that, I can be pretty sure when I walk in to a foreign system there will be nginx logs, just where I expect them, almost certainly in the format I'm used to. And even if the format differs, it's no…

Sure, on a site that uses ElasticSearch for its logs I would have no idea where to look at. I'd be more at ease with SQL, but first you need to locate the DB, figure out the schema, get the SQL dialect right.

That said, I'd be far more at ease writing a SQL query to extract analytics from logs than cooking up some regexes and doing complex stuff with awk.

And I find the --since/--until parameters to journalctl far easier than matching dates by regex. Or even the --boot parameter to restrict logs to a specific boot, which with would be probably doable with awk but definitely not as trivial.

I think that binary logs give you some compelling features, without taking away any: you can always just dump the logs on stdout and use grep as much as you want. :)

Re: Grepping logs is still terrible

#72

Post by nrr »

ghshephard wrote: post #44
nrr wrote: post #28

Earlier quoted context omitted.

Oddly enough, even for large (>=1e5 physical machines) systems, grep works fine. Better yet, if the logs are important, you're shunting them off for some sort of longer-term storage for post-processing and indexing _anyway_, irrespective of the underlying disk format. Some folks continue to use plain text even then, just with some distributed systems magic wrapped around the traditional Unix tools. (If you're shuntin…

grep definitely breaks down on large systems. I have one environment with approx 5 million nodes - (1e6), and the only way to coherently manage the log updates from them is in binary format. But even still - I like to have the text files as journals of original entry - so I can occasionally do a tail -f incoming.log| egrep -i "somedevice". And having the original files in text format is zero impediment to getting the…

I hate arguing semantics, but 1e6 is not just large but very large indeed. (:

That said, I'd be curious to know some more of the details of that system actually! If you're aggregating all of those devices together, using something binary in that context definitely makes sense. In fact, if I were in your shoes and tasked with designing some means of solving that problem, I would probably use something like protobuf or capnp to emit those messages since they're well-known and well-understood serialization mechanisms.

Now, that's the integration and aggregation side of this exercise.

On a local node-by-node basis, though, I absolutely agree; having the raw text as journals of original entry for inspection in real time with `tail -f` (or, if you're using multilog, `tail -F`…) would still be incredibly useful.

Going back to Mr. Nagy's article, the space of problems that `tail -f` solves is barely overlapped by the space of problems solved by aggregation. I think he's conflated the two spaces in his article here (and especially in the one previous) whereby he's applied a one-size-fits-all solution to both where it demonstrably does not fit all.

Re: Grepping logs is still terrible

#73

Post by EmanueleAina »

seren wrote: post #59
kaolinite wrote: post #50

Earlier quoted context omitted.

Does that really matter? Log files are often unimportant when they get over a month or two old, what is it in your log files that has to be kept for 40 years? Longevity of log files hardly seems like a reason to pick an otherwise inferior format.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

There's a good chance that you'd be reading EBCDIC logs. :)

40 years from now, you will probably be able to invoke journalctl on the system and parse the dumped output as plain text. Or call gunzip on the compressed logs, $DEITY knows if we will be still using gzip by then. And if the system does not boot, you won't be able to connect the peripherals anywhere else... :)

Re: Grepping logs is still terrible

#74

Post by scrollaway »

seren wrote: post #59
kaolinite wrote: post #50

Earlier quoted context omitted.

Does that really matter? Log files are often unimportant when they get over a month or two old, what is it in your log files that has to be kept for 40 years? Longevity of log files hardly seems like a reason to pick an otherwise inferior format.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up.

There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation.

However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of are read+write.

Stepping back though this entire argument is absurd. Thinking about "whatever will those people do 40 years from now with the tools of today" is fairly braindead once you understand that the quality of the tools will affect their longevity. So if the logging system becomes an actual, factual problem over time, the tools will die off by naturally-artificial selection.

Re: Grepping logs is still terrible

#75

Post by EmanueleAina »

pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

For sure the storage format should not hinder you from using grep if you want. Even with systemd you can pipe journalctl's output and use the same old regexes as its default behaviour is to be a glorified `cat` (but being able to use the --since and --until flags instead of matching date ranges by regexes makes it much better than `cat` for me).

Re: Grepping logs is still terrible

#76

Post by laumars »

geographomics wrote: post #55
laumars wrote: post #31

Oh jeez. Yes there are better and more performant tools for parsing optimised binary databases; nobody disputes that. And yes, tools like Splunk are more user friendly than grep; nobody disputes that either. But to advocate a binary only system for logs is short sighted because logs are the goto when everything else fails and thus need to be readable when every other tool dies. There's quite a few scenarios that coul…

The ease of recovering data from a corrupted log file depends on whether the logged events have been written as sequential records. This is true for text-based logs (the record delimited being a newline), and is also true of the most popular binary (i.e. structured) log formats, namely Windows event logs, and systemd's journals. Probably not if you're storing them in a more general purpose database though. So this re…

But if you're not storing them in a database then your primary advantage of using a binary format (namely performance) evaporates.

Re: Grepping logs is still terrible

#77

Post by ptaipale »

scrollaway wrote: post #74
seren wrote: post #59

Earlier quoted context omitted.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up. There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation. However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of…

> There's no tool out there that generates log files it cant itself read.

There are plenty of tools that don't read their logs - more precisely, computer units where you don't log in, units that you don't operate on console. Embedded devices that perform some function and also keep some log, but which cannot be used for reading that log. You will need to read that log using something else. Plain text (ASCII, and now ISO Latin and UTF-8) is a fairly stable format for everything, and will be for the next 50 years.

Re: Grepping logs is still terrible

#78

Post by jkyle »

kuschku wrote: post #45
rubenv wrote: post #39

Earlier quoted context omitted.

Sure, an opaque binary file is pointless. But that's not what most logging systems that log to binary files offer. They give you specs (example: http://www.freedesktop.org/wiki/Software/systemd/journal-fil... ) and tools. Binary doesn't have to mean closed/opaque.

And the specs will be gone in 40 years. While ASCII will stick around.

    And the specs will be gone in 40 years. While ASCII will stick around.
Why would they be gone? You realize ASCII is a 'spec' too?

If a binary format has an open specification, it's as future proof as ASCII. ASCII's durability is due to a clear and open specification that's easily implemented. Not some magic sauce that makes it instantly human readable.

That text you see? It's not what's actually in the file. That's just 1's and 0's like every other format. There's literally no difference between ASCII and any other "binary" format.

Re: Grepping logs is still terrible

#79

Post by seren »

scrollaway wrote: post #74
seren wrote: post #59

Earlier quoted context omitted.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up. There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation. However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of…

I have already worked on very basic embedded system where you only way of getting logs is connecting to the device using a serial line, and after fiddling a bit with the baud rate, you can get some readable output.

In this case, you can't really do anything from the device itself.

Arguably, this is not the use case for binary logger but I was originally addressing the "40 years old logs" argument, that do exist in the real world.

Re: Grepping logs is still terrible

#80

Post by lmm »

scrollaway wrote: post #74
seren wrote: post #59

Earlier quoted context omitted.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up. There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation. However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of…

> There's no tool out there that generates log files it cant itself read.

If only. I've used two such systems myself.

When the tool for your binary data file says "data corrupt" when you try to open it, what do you do?

Post reply on HN