Grepping logs is terrible
asylum.madhouse-project.org
Grepping logs is terrible
1–10 of 105 posts
Re: Grepping logs is terrible
#2Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files.
I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be better suited for anyway, on proven db systems?
Maybe I'm missing something.
Re: Grepping logs is terrible
#3The author's use of logs is sophisticated and proactive. Sadly, most Linux installations I've dealt with are lazy and reactive, where logs are kept around "just in case" for future forensics (hah!).
Re: Grepping logs is terrible
#4I guess that much of the resistance against the binary logs of systemd is the unfamiliarity and to some extent lack of well known tools for dealing with them. Sysadmins that have years of experience with traditional Unix tools now suddenly have to start almost from scratch when it comes to everyday tools for examining the system. Not only that, programmers are also most familiar with text based formats, and libraries for handling these formats have to become more available in the most popular programming languages and become familiar for programmers that develop tools for analysing systems. Until that happens, sysadmins feel that they are set back by the introduction of binary logs, even if binary logs are technically superior.
Re: Grepping logs is terrible
#5People don't want it because it's binary, not because you can't grep it.
* you need to use a new proprietary tool to interact with them
* all scripts relating to logs are now broken
* binary logs are easy to corrupt, e.g. if they didn't get closed properly.
>You can have a binary index and text logs too! / You can. But what's the point?
The point is having human-readable logs without having to use a proprietary piece of crap to read them. A binary index would actually be a perfect solution - if you're worried about the extra space readable logs take, just .gz/.bz2 them; on decent hardware, the performance penalty for reading is almost nonexistent.
If you generate 100GB/day, you should be feeding them into logstash and using elasticsearch to go through them (or use splunk if $money > $sense), not keeping them as files. Grepping logs can't do all the stuff the author wants anyway, but existing tools can, that are compatible with rsyslog, meaning there is no need for the monstrosity that is systemd.
Re: Grepping logs is terrible
#6Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…
I think I'm missing the same thing. He keeps going on about structure, but it wasn't obvious to me where the solution (?) actually introduces query-able structure.
He needs a log database, clearly. And when you put it that way, it's obvious why grepping logs is a nice, quick solution in many cases when you aren't getting "100Gb of logs a day".
Re: Grepping logs is terrible
#7People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating to logs are now broken * binary logs are easy to corrupt, e.g. if they didn't get closed properly. >You can have a binary index and text logs too! / You can. But what's the point? The point is having human-readable logs without having to use a proprietary piece…
What's wrong with Splunk? Honest question.
Re: Grepping logs is terrible
#8If your logs aren't text, and it's a small system, I'm not going to look at them. Therefore they don't exist. That's one reason why people don't like binary logs - they are effectively useless.
On the flip side, if the system is huge - then we can use tools like splunk.
grep/tail/awk are the first three tools I use on any system - if you create logs that I can't manipulate with those three tools, then you haven't created logs for your system that I can use.
Re: Grepping logs is terrible
#9This applies more generally than just to logs. I love Unix, but "everything is text" is not actually great. It's better that Unix utils output arbitrary ASCII than that they output arbitrary binary data, but it's obvious why people don't do serious IPC 'the Unix way.' Imagine if instead of exchanging JSON, or ProtoBufs, or whatever, your programs all exchanged text you had to regex into some sort of adhoc structure. So why do we manage our logs and our pipelines that way? There's no actual reason that the terminal couldn't interpret structured data into text for us so that, in the world of intercommunicating processes on the other side of the TTY, everything is well-structured, semantically comprehensible data.
Re: Grepping logs is terrible
#10I don't have experience with binary logs. I think the fragility of binary logs is not baseless though. AFAIK there was (is?) a problem in systemd's journal where a local corruption of the log could cause a global unavailability of the logged data.
People like text logs because local corruptions remain local. Some lines could be gibberish, but that's all. I'm not suggesting that this couldn't be done with binary logs, but you have to carefully design your binary logging format to keep this property.
Otherwise I agree with the author that we shouldn't be afraid of binary formats in general, we need much more general formats and tools though (grep, less equivalents).
I'm not fond of "human readable" tree formats like XML or JSON either. bencode could be equally "human readable" as an utf-8 text if one has a less equivalent for bencode.