Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is still terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Grepping logs is still terrible

#101

Post by nrr »

ghshephard wrote: post #98
nrr wrote: post #72

Earlier quoted context omitted.

I hate arguing semantics, but 1e6 is not just large but very large indeed. (: That said, I'd be curious to know some more of the details of that system actually! If you're aggregating all of those devices together, using something binary in that context definitely makes sense. In fact, if I were in your shoes and tasked with designing some means of solving that problem, I would probably use something like protobuf or…

The remote nodes all log to central DNS servers, and Trap Servers. The DNS servers have a nice update.log file that provides their IP address information, and some nice text configs. The trap data, goes into a binary file (database actually) and requires analysis through a web interface. As a result - the DNS updates are used by me approximately 20x more often than the trap data, when doing diagnostics, even though,…

Memories of supporting subscriber CPEs and having to go through Drum to analyze the data coming out of logged SNMP traps/notifications are flashing back. Thanks for that. (:

But, yeah, assuming that the nodes in discussion here are not amd64 machines but are instead subscriber CPEs, that's a totally workable (and, frankly, agreeable) solution.

Re: Grepping logs is still terrible

#102

Post by scrollaway »

deathanatos wrote: post #97

It seems to me that most of the worry about a binary log file being "opaque" could be solved with a single utility: log-cat … that just outputs it in text. Then you can attack the problem with whatever text-based tools you want. But to me, having a utility that I could do things like, get a range of log lines — in sorted order —, or, grep on just the message, would be amazing. These are all things that proponents of…

What you're describing is journalctl.

Re: Grepping logs is still terrible

#103

Post by geographomics »

felixgallo wrote: post #87
geographomics wrote: post #64

Earlier quoted context omitted.

The system should provide a standard API for writing and reading logs. The precise format of the underlying log files is thus rather unimportant at this level of abstraction. Other than the logging subsystem and recovery tools, there's no need for any software to be accessing such log files directly (outside of the API functions). This is how Windows has done it for years.

You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case. And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forev…

This isn't really true as the Windows event logs contain text as well as the other structured data, which you can search for using tools on the system. For example to search for some specific text in the system log using Powershell:

    Get-EventLog -LogName System | Where {$_.Message -Match "something"}
To process text as fields, as with awk, one would use the Split method (at least to start off with):

    Get-EventLog -Log System | Where {$_.Message -Match "something"} | %{ $_.Message.Split()[5,4] }
But as message text is often parameterised, it may be easier to take advantage of this data to get what you need. For instance, this command would extract the latest machine sleep and wake times from the system log, and calculate the duration:

    Get-EventLog -Log System -Source Microsoft-Windows-Power-Troubleshooter -InstanceID 1 | Select-Object @{n="SleepTime";e={$_.ReplacementStrings[0]}}, @{n="WakeTime";e={$_.ReplacementStrings[1]}}, @{n="SleepDuration";e={([DateTime]$_.ReplacementStrings[1])-([DateTime]$_.ReplacementStrings[0])}}
One can also sort and get unique values, just as in Unix-type systems - this command lists all drives defragmented in the past 30 days:

   Get-EventLog -Log Application -Source Microsoft-Windows-Defrag -InstanceID 258 -After (Get-Date).AddDays(-30) | Select @{n="Drive";e={$_.ReplacementStrings[1]}} | Sort Drive | Unique -AsString
So all the same capabilities are there, and then some. You just need to know your tools well enough to take advantage of it.

Re: Grepping logs is still terrible

#104

Post by geographomics »

laumars wrote: post #76
geographomics wrote: post #55

Earlier quoted context omitted.

The ease of recovering data from a corrupted log file depends on whether the logged events have been written as sequential records. This is true for text-based logs (the record delimited being a newline), and is also true of the most popular binary (i.e. structured) log formats, namely Windows event logs, and systemd's journals. Probably not if you're storing them in a more general purpose database though. So this re…

But if you're not storing them in a database then your primary advantage of using a binary format (namely performance) evaporates.

The difference is that a general purpose database typically organises data by fixed-size pages, so new data could be anywhere in the file as there is no guarantee of page ordering with regard to inserts. Whereas a specialised file format for logging would add new records at the end of the file (or in a circular fashion, depending on the design). But will have features similar to a database like a defined schema, and some form of indexing. This is true of systemd journals and Windows event logs anyway.

Re: Grepping logs is still terrible

#105

Post by geographomics »

erikb wrote: post #61
geographomics wrote: post #46

Earlier quoted context omitted.

Another advantage of using structured data rather than free-form text is that you can more precisely encode the essence of the event, with fields for timestamp, event source, type of event, its severity, any important parameters, and so on. This permits logging to be independent of the language of the system operator. Rather than grepping for what is almost always English text, one can query a language-independent se…

About your first point: Independence. You are less independent of English and more dependent on the binary format and the tools who can handle it. It's a trade-off. And it might be just an opinion, but for me it's not a good trade-off. Learning English was a one-time endeavour for me. But binary formats and tools have to be learned separately. About what you put in the log message: You can also put different fields i…

This is also just anecdotal, but I used to work with a Chinese sysadmin (I was in the UK, he was in China) who found it much more preferable to work on the localised Windows servers we had installed over there, as all the UI and messages were in his native tongue. I'm sure it was easier for him to gain expertise in the tools he needed for his work, than to become proficient enough in English to understand every obscure log or error message that the system might throw at him.

Re: Grepping logs is still terrible

#106

Post by geographomics »

dfox wrote: post #82
Spooky23 wrote: post #30

Windows has had binary logging forever. Is windows administration some wonderland of awesome capability for getting intelligence out of logs? Hell no. For administering Unix like systems, the ability to use a variety of tools to process streams of text is an advantage and valuable capability. That said, your needs do change when you're talking about managing 10 vs 10,000 vs 100,000 hosts. I think what you're really s…

I think that largest problem with Event Log is overreliance on structure. Often you have one particular log record that you know is the problem, but no idea what it means because you have some generic event code and bunch of meaningless structured data. Freeform text logs usually contain more detail as to what exactly happened.

That's not a limitation of the Event Log system itself, as you can easily write freeform text in the message rather than building it up with localised strings and structured data, e.g. https://msdn.microsoft.com/en-us/library/6w20x90k(v=vs.110)....

Re: Grepping logs is still terrible

#107

Post by peterwwillis »

scrollaway wrote: post #8
ghshephard wrote: post #2

It's beyond me how he doesn't understand that text logs are a universal format, easily accessible, that can be instantly turned into whatever binary format you desire with a highly efficient insertion process (Splunk is just one of those that does a great job). Here is the thing he doesn't seem to understand - all of us who are sysadmins absolutely understand the value of placing complex and large log files into data…

> text logs are a universal format Uh, I don't know what world you live in but I'd like the address because mine sucks in comparison. Text logs are definitely not a "universal format". Easily accessible, sure. Human readable most of the time? Okay. Universal? Ten times nope. Give you an example: uwsgi logs don't even have timestamps, and contain whatever crap the program's stdout outputs, so you often end up with thr…

"Text logs" is not a format at all, so it can't really be a universal format, either. But if there were such a thing as a "universal" format, it would probably by definition encompass everything in time and space. You think timestamps are a problem? Just wait until your logs get trapped in a quantum state. Talk about a heisenbug...

Re: Grepping logs is still terrible

#109

Post by rquirk »

erikb wrote: post #63
rquirk wrote: post #62

Earlier quoted context omitted.

> "give me 10 hours starting from last Friday 2am" journalctl --since="$(date -d'last friday 2am' '+%F %X')" --until="$(date -d'last friday 2am + 10 hours' '+%F %X')" Now I'm no systemd apologist but maybe some of the hate towards systemd, journald and pals is unwarranted. If one gives these newer tools a chance, they actually have some nice features. Despite the Internet's opinion, seems like they were not actually…

It's not a counter argument to journalctl's usefulness being independent of binary logs, right? In fact using that nice tool for querying I don't really care how it stores logs under the hood.

Right, I'm sure you could do the same thing with grep and a classic /var/log/messages - in fact there's probably something to do this already. Or you'd find a gz from the day in question and read all of it. Just happened that I'd recently read the man page for journald and that was something I recalled.

Re: Grepping logs is still terrible

#110

Post by jack9 »

> Does database store the data in text files? No? That's my point.

This guy is a first class idiot who knows enough to reformulate a decided issue into yet another troll article. "a database (which then goes and stores the data in a binary format)". How about a text file IS a database. It's encoded 1s and 0s in a universal format instead of the binary DB format which can be corrupted with the slightest modification or hardware failure.

Post reply on HN