Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

61–70 of 105 posts

Re: Grepping logs is terrible

#61

Post by madhouse »

regularfry wrote: post #32
leni536 wrote: post #11

Earlier quoted context omitted.

Why does it have to be proprietary?

It doesn't, but nothing is universal like `grep`. If you find a machine that's logging stuff which doesn't have `grep`, you're already having a bad day. You just can't say that about binary log formats. Text is a lowest common denominator; and yes, that cuts both ways, but the advantages of universality can't be trivially thrown away.

The machines I'm administering will all log the same way, therefore, within that context, they are universal. We have well documented tools and workflows, so anyone new to the system can catch up and start working with the logs within minutes.

We don't unexpectedly find machines that don't conform to our policies. We control the machines, we know where and how to find the logs. If we found any where we had to grep, we'd be having a very bad day.

Our lowest common denominator is not text, because we control the environment, and we can raise the bar. Being able to do that is - I believe - important for any admin.

Re: Grepping logs is terrible

#62

Post by chronid »

leni536 wrote: post #51
chronid wrote: post #44

Earlier quoted context omitted.

You assume the other system is a linux box with systemd installed. That may be true, or may not be true at all. :)

It's not like specific part's of journalctl can't be ported to other systems and packaged separately. It's a distribution issue, not a fundamentaly flaw in binary logs.

Sure, the question is why fix something that is not broken? Why should someone write a systemd journal format reader (but journald is just an example here) when text files work already? Why rewrite tools to support it?

If you get to the point when standard unix tools are useless, well, it's time to use a _real_ database and/or log management system. Not the time to write your own. No one is (should be?) going around grepping 100 GBs/day worth of logs.

Re: Grepping logs is terrible

#63

Post by bad_user »

leni536 wrote: post #50
cthalupa wrote: post #31

Earlier quoted context omitted.

It doesn't have to be - but let's look at reality here. NIH syndrome is everywhere, we have millions of competing protocols and formats, everyone thinks they can build a better solution than someone else, etc. I suppose that if there was a large push to universally log things in binary the possibility exists that sanity would prevail and we'd get one format that everyone agreed upon, but I don't see any reason that t…

It's not like text formats are universal. Thankfully we have settled on utf-8. The same could happen to a slightly more structured universal binary format that would be more suitable for many applications (like logging) and would have an established toolset just like 'text' now.

That doesn't make sense. There can be no such thing as a universal binary format, unless you reinvent text/plain. Analyze a little on why we refer to some formats as being text-based and you'll see it.

Re: Grepping logs is terrible

#64

Post by 616c »

ars wrote: post #28
616c wrote: post #24

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin? I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting…

I use the logwatch program for that. There is no machine learning, it's entirely manual with a large list of things it filters out, but the defaults are quite good. It emails me any log entires it doesn't know about. I did have to add a large number of ssh lines that it should not bother me about, but other than that it works very well and I find it very useful.

Cool tip. I remember hearing the name but knowing it had these features. I will definitely check it out.

Re: Grepping logs is terrible

#65

Post by madhouse »

halayli wrote: post #16

logs should be in text. The last thing you want is to find out that your binary format cannot be decoded due to a bug in the logging or because file got corrupted. Not to mention that you won't be able to integrate with a lot of log systems like Splunk and friends. On the other hand, if you have logs, you need to store them in a centralized place and have an aging policy, etc... Grepping is definitely not the answer.…

Please don't confuse log storage with log transport. We can transform the stored format into any other, if so need be.

(For example, I use Kibana at home. Works great, though I have no text logs stored.)

Re: Grepping logs is terrible

#66

Post by malka »

616c wrote: post #24

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin? I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting…

you can use fail2ban for this. It is used to automatically ban IP that, for instance, tries to bruteforce your SSH, but it really is an engine that match regexp log file lines, and fires an action if the regexp match.

So you can use it for other usages (such as sending an admin a mail if suddenly your server sends 500 errors, or a unusual amount of 404 errors for instance)

Re: Grepping logs is terrible

#67

Post by madhouse »

jeady wrote: post #21

I think the author is conflating several problems here. There are several ways logs can be used, and efficiency is a scale. For example, if I receive a bug report, I like to be able to locate the textual logs from when the incident occurred and actually just sit and read what was happening at the time. On the other hand, if I'm doing higher-level analysis such as what features do users use most, clearly it's more eff…

With a binary log storage system, nothing stops you from browsing all logs that happened around the time of the incident. Instead of locating the files, you just tell the engine to show you the logs from that time onwards (or from a little bit before).

As for our logs being too verbose: nope, read the article.

Also, it's not an one-size-fits-all solution: I have no problem with people using text. All the article wants to show, is that binary logs are not evil, bad, useless, etc, and that there are actually very good reasons to use them.

For example, storing logs in a database is one kind of binary log storage: most databases don't store the data as text.

Re: Grepping logs is terrible

#68

Post by erikb »

Look at a first year computer science student. He will already put prints in his programs and if he is smart and has a bigger assignment he might already start to write other programs to parse that output. You can't beat that, because it is nearly impossible for a newbie to even know that there might be a problem with text logging and that binary logging might be a solution. In fact he might not even know that what he does is called logging. But he is already doing it!

So even if binary logging is way better (I can't say, not enough experience) you simply can't beat text logging, because text logging is natural. It just happens.

print("Hello World!")

Re: Grepping logs is terrible

#69

Post by madhouse »

regularfry wrote: post #34

This is all well and good if you want to, and can, spend time up front figuring out how to parse each and every log line format which might appear in syslog so you can drop it in your structured store. The alternative is to leave everything unstructured, and understand the formats minimally and lazily. Laziness is a virtue, right?

Why would I need to be able to parse everything up front? Taking the syslog example, that has a commonly understood format. As a default case, I can just split the parts and have structured data (esp. with RFC5424, where structured data is part of the protocol to begin with).

Then, I can add further parsers for the MESSAGE part whenever I feel like it, or whenever there is need. I don't need that up front.

Re: Grepping logs is terrible

#70

Post by madhouse »

babuskov wrote: post #38

If you need to grep logs on regular basis, you're doing it wrong. Store important data in the database so that you can query it efficiently. Keep logs for random searches when something unexpected happens. I log gigabytes per day, but only grep maybe once-twice a year.

Agreed. But if I keep my logs in a database, I may aswell use the database to query my logs, instead of grepping in them.

(And voila, you have binary log storage.)

Post reply on HN