Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

21–30 of 105 posts

Re: Grepping logs is terrible

#21

Post by jeady »

I think the author is conflating several problems here. There are several ways logs can be used, and efficiency is a scale. For example, if I receive a bug report, I like to be able to locate the textual logs from when the incident occurred and actually just sit and read what was happening at the time. On the other hand, if I'm doing higher-level analysis such as what features do users use most, clearly it's more efficient to have some sort of structure format because you're interested in the logs in aggregate. The author makes it sound like they're advocating optimizing for the aggregate use case at the expense of other use cases. I think that the declaration that textual logs are terrible is an oversimplification of the considerations in play.

Also, if the author has a 5-node cluster producing 100Gbs of logs a day, the logs may also be too verbose or poorly organized. I work on a system that produces 100s of Gbs of logs a day but with proper organization they're perfectly manageable.

I think that a more nuanced solution is to log things that are useful to manual examination in text form, but high-frequency events that are not particularly useful could reasonably be logged elsewhere (e.g. a database or binary log that is asynchronously fed into a database).

In conclusion, as is frequently the case with engineering, I think the author oversimplifies the problem here and tries to present a one-size-fits-all solution instead of taking a more pragmatic solution. Textual logs are useful when meant for human consumption (debugging) and when they can be organized such that the logs of interest at any time are limited in size, and some other binary-based format is useful for aggregate higher-level analysis.

Re: Grepping logs is terrible

#22

Post by chakrit »

lazzlazzlazz wrote: post #6
moonshinefe wrote: post #2

Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…

I think I'm missing the same thing. He keeps going on about structure, but it wasn't obvious to me where the solution (?) actually introduces query-able structure. He needs a log database, clearly. And when you put it that way, it's obvious why grepping logs is a nice, quick solution in many cases when you aren't getting "100Gb of logs a day".

I think his point was that querying a structured data is better than grepping unstructured text. SQL vs Regex, for example. I get the impression that he didn't state what solution to use but simply that binary/structured > text/unstructured. He even says that Journal isn't his ideal solution and never will be.

Re: Grepping logs is terrible

#23

Post by 616c »

leni536 wrote: post #10

I don't have experience with binary logs. I think the fragility of binary logs is not baseless though. AFAIK there was (is?) a problem in systemd's journal where a local corruption of the log could cause a global unavailability of the logged data. People like text logs because local corruptions remain local. Some lines could be gibberish, but that's all. I'm not suggesting that this couldn't be done with binary logs,…

> I don't have experience with binary logs. I think the fragility of binary logs is not baseless though. AFAIK there was (is?) a problem in systemd's journal where a local corruption of the log could cause a global unavailability of the logged data.

From my experience (I do not want to troll and presume you have not tried it), systemd starts off where it picked up when an old log is corrupted and stars a new one. There is a command line utility to verify the integrity of these files (on my Windows laptop at work, cannot check). Now, I am not sure the state of log file repair. I was told it is not possible. However, it seems this means the file is corrupted in a way it is not easily indexed. It is likely it is still readable. I wish I had seen this last time.

https://www.reddit.com/r/linux/comments/1y6q0l/systemds_bina...

Granted, I use Arch Linux on an old laptop. I had these corruptions routinely happen when I had disabled ACPI controls (I do not use the fancy WMs, I am back to Ratpoision) and completely, and I mean completely drained the battery until it came crashing to a halt). So, I am not surprised about these corruptions.

Anyone using systemd boxes in production who can comment on this? Flamewar or not, I would like to know more. I do not really care for it one way or the other. Parts I like, parts I do not.

Re: Grepping logs is terrible

#24

Post by 616c »

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin?

I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting I am not sure where to start.

Re: Grepping logs is terrible

#25

Post by nosir33 »

onion2k wrote: post #14

Binary logs are opaque! Just as much as text logs. I don't agree with the second assertion there. Text logs are only opaque as far as the format is concerned, but not so much as far as the content goes. Using the example in the article; 127.0.0.1 - - [04/May/2015:16:02:53 +0200] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0" You can read a lot of information without knowing the format, the application that generated it, or…

> That's a lot more information than you could get from a binary log without any tools.

The environment I work in I am frequently looking at logs that other teams generate. If I needed to ramp up on their custom logging toolset just to perform simple queries I am going to give up and waste the the teams time by getting them to perform the queries for me.

Re: Grepping logs is terrible

#26

Post by nosir33 »

616c wrote: post #24

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin? I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting…

I don't know of any systems that do this.

As an software developer, I generally use log levels to indicate severity in my logs. So grepping for ERROR should catch anything I had the foresight to log at the ERROR level.

Simple heuristics like the number of WARN level logs a minute may be useful.

Beyond that it sounds interesting. It may be hard to do in a general way, so focusing on Apache logs or something common may be a simpler task.

Re: Grepping logs is terrible

#27

Post by tljr »

leni536 wrote: post #11
blueskin_ wrote: post #5

People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating to logs are now broken * binary logs are easy to corrupt, e.g. if they didn't get closed properly. >You can have a binary index and text logs too! / You can. But what's the point? The point is having human-readable logs without having to use a proprietary piece…

Why does it have to be proprietary?

Doesn't have to be but usually is.

Re: Grepping logs is terrible

#28

Post by ars »

616c wrote: post #24

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin? I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting…

I use the logwatch program for that. There is no machine learning, it's entirely manual with a large list of things it filters out, but the defaults are quite good.

It emails me any log entires it doesn't know about. I did have to add a large number of ssh lines that it should not bother me about, but other than that it works very well and I find it very useful.

Re: Grepping logs is terrible

#29

Post by nosir33 »

moonshinefe wrote: post #2

Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…

One of the initial challenges I see from an OPS perspective is that the most recent logs are often the most interesting. The latency of the logs being ingested into a DB would prevent me from using the DB. Generally, I find my self grepping logs on the prod servers.

Re: Grepping logs is terrible

#30

Post by ars »

moonshinefe wrote: post #2

Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…

"you have 100Gb of logs a day"

Logs have lots of redundancy, so they compress quite nicely. So it is actually practical to grep those files since on disk they are not so large, and 100Gb of memory data is not a problem to grep.

Post reply on HN