Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is still terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Grepping logs is still terrible

#81

Post by TazeTSchnitzel »

jkyle wrote: post #78
kuschku wrote: post #45

Earlier quoted context omitted.

And the specs will be gone in 40 years. While ASCII will stick around.

And the specs will be gone in 40 years. While ASCII will stick around. Why would they be gone? You realize ASCII is a 'spec' too? If a binary format has an open specification, it's as future proof as ASCII. ASCII's durability is due to a clear and open specification that's easily implemented. Not some magic sauce that makes it instantly human readable. That text you see? It's not what's actually in the file. That's j…

Text encodings have come and gone before, too. We don't use the Baudot code on modern computers still, and EBCDIC is confined to IBM mainframes.

Re: Grepping logs is still terrible

#82

Post by dfox »

Spooky23 wrote: post #30

Windows has had binary logging forever. Is windows administration some wonderland of awesome capability for getting intelligence out of logs? Hell no. For administering Unix like systems, the ability to use a variety of tools to process streams of text is an advantage and valuable capability. That said, your needs do change when you're talking about managing 10 vs 10,000 vs 100,000 hosts. I think what you're really s…

I think that largest problem with Event Log is overreliance on structure. Often you have one particular log record that you know is the problem, but no idea what it means because you have some generic event code and bunch of meaningless structured data.

Freeform text logs usually contain more detail as to what exactly happened.

Re: Grepping logs is still terrible

#83

Post by nieve »

geographomics wrote: post #64
thaumaturgy wrote: post #48

Cool, so which standard binary log storage format should we all switch to? Should I submit patches to jawstats so that it'll support google-log-format 1.0 beta, or the newer Amazon Cloud Storage 5 format? Or both? Or just go with the older Microsoft Log Storage Format? Or wait until Gruber releases Fireball Format? Has he decided yet whether to store dates as little-endian Unix 64 bit int timestamps, or is he still t…

The system should provide a standard API for writing and reading logs. The precise format of the underlying log files is thus rather unimportant at this level of abstraction. Other than the logging subsystem and recovery tools, there's no need for any software to be accessing such log files directly (outside of the API functions). This is how Windows has done it for years.

Even if you can manage one per OS, it's not good enough. Have you ever worked in a non-monoculture or dealt with recovery of a system severely damaged by malice or accident?

I doubt my Linux (including webOS & Android), FreeBSD, and OS X boxes are going to settle on a single binary format in the next couple of decades or even a single API & toolset. In your brave new world the very first thing I'm going to need to do if I have to combine logs across them is to extract data from at least three formats and the most convenient format is often going to be text - i.e. right back where we started, but with extra work for each OS. More likely you'll get a mix of things using the system APIs, custom binary formats, custom text formats, and syslog. Adding more steps to get at the same data doesn't help.

More importantly, binary logs are unreliable when you're dealing with a system that's completely trashed. You can often get usable text logs off a disk that's throwing I/O errors every few dozen bytes or even from a corrupted raw disk image. They may not be cryptographically "sealed", but I'd rather have them than an error message about the binary format being corrupt. That should be an implementation detail, but I haven't seen much interest from the binary logs camp in making the file formats resilient.

Re: Grepping logs is still terrible

#84

Post by Spooky23 »

scrollaway wrote: post #74
seren wrote: post #59

Earlier quoted context omitted.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up. There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation. However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of…

It isn't a strawman at all. It's a real problem that exists today for folks who need to access old documents from the 80s, for example.

Re: Grepping logs is still terrible

#85

Post by nieve »

danieltillett wrote: post #22
mdekkers wrote: post #12

Earlier quoted context omitted.

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

I would tend to agree with the OP but with a caveat - most of the people who administer system work on small systems, while most people who's full time job is administration work on large systems. Basically there are an awful lot of people in the world who's job description includes part time system administration.

By definition just about everyone who's actually working in DevOps counts.

Re: Grepping logs is still terrible

#86

Post by shit_parade2 »

scrollaway wrote: post #74
seren wrote: post #59

Earlier quoted context omitted.

It is not about reading 40 years old logs, but rather reading logs from today generated by 40 years old system. For example, many nuclear power plant in the west were built 40 years ago. Amongst the myriad of sensors, devices in a power plant, I think that most of them are outputting ASCII logs. There are still readable today. (Same can be said about avionics, space probes, etc.) Now imagine yourself 40 years from no…

This is a strawman that keeps being brought up. There's no tool out there that generates log files it cant itself read. So there's not going to be any "oh gee I have these files being generated and nothing can read them" situation. However, there is just about near-zero system out there that generates text logs that it can itself read. Text logs are write-only for most logging systems, while all binary logs I know of…

People usually read log files because something went wrong, like a system crash, why do you assume the OS that generated the log file will be readily available?

Re: Grepping logs is still terrible

#87

Post by felixgallo »

geographomics wrote: post #64
thaumaturgy wrote: post #48

Cool, so which standard binary log storage format should we all switch to? Should I submit patches to jawstats so that it'll support google-log-format 1.0 beta, or the newer Amazon Cloud Storage 5 format? Or both? Or just go with the older Microsoft Log Storage Format? Or wait until Gruber releases Fireball Format? Has he decided yet whether to store dates as little-endian Unix 64 bit int timestamps, or is he still t…

The system should provide a standard API for writing and reading logs. The precise format of the underlying log files is thus rather unimportant at this level of abstraction. Other than the logging subsystem and recovery tools, there's no need for any software to be accessing such log files directly (outside of the API functions). This is how Windows has done it for years.

You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case.

And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forever, and you have a thousand excellent tools at your disposal that you would like to be able to continue to use.

Recent log spelunking for me included 'cat log.? | grep fail | sed 's/^.worker_id$//g' | awk '{ print $5, $4 }' | sort -n -r | sed 30q'.

There's no analogue in any binary logging system I've ever found.

Re: Grepping logs is still terrible

#88

Post by nailer »

gambiter wrote: post #57
nailer wrote: post #42

Earlier quoted context omitted.

If 'tooling gets deleted' is a problem you probably have much bigger concerns than log files.

'tooling gets deleted' could easily happen after changing logging systems... while it would be shortsighted to uninstall your old logging system entirely (if you have logs laying around in that format) it's not unheard of. The more likely situation would be that the logs are stored on a shared storage server, and the machine you are using to look at the logs doesn't have the logging system installed.

> The more likely situation would be that the logs are stored on a shared storage server, and the machine you are using to look at the logs doesn't have the logging system installed.

So expose the shared storage to a system running any current mainstream Linux distribution. I understand what you're saying, but this still doesn't seems like a huge concern.

Re: Grepping logs is still terrible

#89

Post by dragonwriter »

felixgallo wrote: post #87
geographomics wrote: post #64

Earlier quoted context omitted.

The system should provide a standard API for writing and reading logs. The precise format of the underlying log files is thus rather unimportant at this level of abstraction. Other than the logging subsystem and recovery tools, there's no need for any software to be accessing such log files directly (outside of the API functions). This is how Windows has done it for years.

You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case. And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forev…

It seems to me that a simple transitional tool for a binary logging system would be for the implementer of the binary logging system to also include a tool that consumed a binary log file on stdin and produced a stream on stdout in one (or more, selecting which by command line arguments) common text log formats.

That lets you develop an ecosystem of supporting tools that take advantage of any strengths of the binary format, while still allowing the freedom of using the (initially, at least, probably far more capable) set of tools available for the text formats.

Re: Grepping logs is still terrible

#90

Post by felixgallo »

dragonwriter wrote: post #89
felixgallo wrote: post #87

Earlier quoted context omitted.

You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case. And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forev…

It seems to me that a simple transitional tool for a binary logging system would be for the implementer of the binary logging system to also include a tool that consumed a binary log file on stdin and produced a stream on stdout in one (or more, selecting which by command line arguments) common text log formats. That lets you develop an ecosystem of supporting tools that take advantage of any strengths of the binary…

what is the point of such a 'transition' if there never arrives any point at which there is net added value to a binary format?
Post reply on HN