Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is still terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

21–30 of 124 posts

Re: Grepping logs is still terrible

#21

Post by KaiserPro »

I think there are a number of issues that are getting mushed into one.

* Journal is just terrible.

* some text logs are perfectly fine.

* when you are in rescue mode, you want text logs

* some people use text logs as a way to compile metrics

I think the most annoying thing for me about journald is that it forces you to do something their way. However its optional, and in centos7 its turned off, or its beaten into such a way that I haven't noticed its there.... (if that is the case, I've not really bothered to look, I poked about to see if logs still live in /var/log/ they did, and that was the end of it. Yes, I know that if this is the case, I've just undermined my case. Shhhhh.)

/var/log/messages for kernel oopes, auth for login, and all the traditional systemy type things are good for text logs. Mainly because 99.9% of the time you get less than 10 lines a minute.

being able to sed, grep, tee and pipe text files are brilliant on a slow connection with limited time/mental capacity. ie. a rescue situation. I'm sure there will be a multitude of stable tools that'll popup to deal with a standardised binary log format, in about ten years.

The last point is the big kicker here. This is where, quite correctly its time to question the use of grep. Regex is terrible. Its a force/problem amplfier. If you get it correct, well done. Wrong? you might not even know.

Unless you don't have a choice, you need to make sure that your app kicks out metrics directly. Or as close to directly as possible. Failing that you need to use something like elastic search. However because you're getting the metrics as an afterthought, you have to do much more work to make sure that they are correct. (although forcing metrics into an app is often non trivial)

If you're starting from scratch, writing custom software, and think that log diving is a great way to collect metrics, you've failed.

if you are using off the shelf parts, its worth Spending the time and interrogating the API to gather stats directly. you never know, collectd might have already done the hard work for you.

The basic argument he puts forth is this: text logs are a terrible way to interchange and store metrics. And yes, he is correct.

Re: Grepping logs is still terrible

#22

Post by danieltillett »

mdekkers wrote: post #12
pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

I would tend to agree with the OP but with a caveat - most of the people who administer system work on small systems, while most people who's full time job is administration work on large systems. Basically there are an awful lot of people in the world who's job description includes part time system administration.

Re: Grepping logs is still terrible

#23

Post by onli »

mdekkers wrote: post #12
pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

You don't need evidence for the obvious. There are a few million personal desktop pcs with linux on them, then there are single servers used by exactly one person. Count that against the people working as a professional sysadmin on a big system.

Re: Grepping logs is still terrible

#24

Post by 4ydx »

mdekkers wrote: post #12
pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

http://www.internetlivestats.com/total-number-of-websites/

If I read it correctly there are about 250 million active sites (roughly). It seems unlikely that they are all massive corporate sites.

As an aside, the idea that systemd is a good thing is hilarious to me at the least because it is so brash about making an important change to a huge chunk of the system. Yes the bugs will eventually get ironed out, but in the meantime? Count me out! I have work to do and am not interested in being a free tester for Redhat on my live systems.

Re: Grepping logs is still terrible

#26

Post by AlfaWolph »

scrollaway wrote: post #8
ghshephard wrote: post #2

It's beyond me how he doesn't understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job). Here is the thing he doesn't seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into data…

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

He's referring to the universal format being text itself.

Re: Grepping logs is still terrible

#27

Post by 4ydx »

Sounds like somebody in the systemd camp. I really dislike added complexity when it is totally unnecessary. If people want to transform their logs into a different storage format, that is up to them. Text files, however, are a fantastically simple way of storing... (drumroll please) text. Surprising /s

Re: Grepping logs is still terrible

#28

Post by nrr »

pjc50 wrote: post #6

Binary logs may be fine for you, but don't force it on us! This is really the important point here. For small systems, grep works fine. The number of people administering small systems is much greater than the number of people administering large systems. The systemd controversy has caused people to fear that change they don't want will be imposed on them and their objections insultingly dismissed: a consequence of i…

Oddly enough, even for large (>=1e5 physical machines) systems, grep works fine. Better yet, if the logs are important, you're shunting them off for some sort of longer-term storage for post-processing and indexing _anyway_, irrespective of the underlying disk format. Some folks continue to use plain text even then, just with some distributed systems magic wrapped around the traditional Unix tools.

(If you're shunting _all_ of your log data off at that scale, you're crazy, and you'll melt your switches if you aren't careful.)

The name of the game is to think of the problems that you're solving and how they relate to the business bottom line. No sooner, no later. Additionally, what's most troubling is that we've turned this exercise into an emotional one, not one with any sort of scientific-oriented perspective.

I can personally say with conviction that I'd like to sit down and actually collect data on, e.g., how many instructions it takes to store logs to disk in plain text versus a binary format, how many it takes to retrieve logs from disk in both situations, and how much search latency I incur when trying to retrieve said logs from disk in the same. At scale, which is where most of my attention lies these days, that's the kind of thing that matters because those effects get amplified automatically—often to operators' and capacity planners' horrors—by the number of machines you have.

If you're dealing with smaller systems, it won't matter as much, but at that point, you're probably dealing with the other side of this, which is having information on how many requests you get for historical log data and what sort of criteria were used in that search. If you're getting requests less frequently than, say, once per quarter, it likely wouldn't be worth your time to invest in what Mr. Nagy is evangelizing.

tl;dr: Continue using your ad hoc grep-fu, but be mindful of how much time it takes you to get the data you're looking for. That alone will be your decision criterion for adopting something like this.

Re: Grepping logs is still terrible

#29

Post by 4ydx »

robinhouston wrote: post #3

> For example: find all logs between 2013-12-24 and 2015-04-11, valid dates only. That’s a straw man. If you’re grepping logs, you don’t need a regular expression that matches only valid dates because you can assume that the timestamps on the log records are valid dates. But I suppose 2013-12-(2[4-9]|3.)|2014-..-..|2015-0([123]-..|4-(0.|1[01])) doesn’t look so bad. The whole thing is similarly exaggerated.

Agreed. It makes one wonder just how much administration this person has actually done in their life.

Re: Grepping logs is still terrible

#30

Post by Spooky23 »

Windows has had binary logging forever. Is windows administration some wonderland of awesome capability for getting intelligence out of logs? Hell no.

For administering Unix like systems, the ability to use a variety of tools to process streams of text is an advantage and valuable capability.

That said, your needs do change when you're talking about managing 10 vs 10,000 vs 100,000 hosts. I think what you're really seeing here is a movement to "industrialize" the operations of these systems and push capabilities from paid management tools into the OS.

Post reply on HN