Grepping logs is still terrible • Hacker News Board

Re: Grepping logs is still terrible

#111

Post by thaumaturgy » Thu, May 07, 2015, 9:03 PM UTC

geographomics wrote: post #103

felixgallo wrote: post #87

Earlier quoted context omitted.

You missed the joke at the end where he correctly pointed out that Windows' logging is a total joke, and that discovering information from Windows logs is essentially impossible unless the tool writer specifically predicted your use case. And that's the nub of it: text logs are for when you may have many varied, complex reader use-cases, and you don't understand all those cases well enough yet to lock them down forev…

This isn't really true as the Windows event logs contain text as well as the other structured data, which you can search for using tools on the system. For example to search for some specific text in the system log using Powershell: Get-EventLog -LogName System | Where {$_.Message -Match "something"} To process text as fields, as with awk, one would use the Split method (at least to start off with): Get-EventLog -Log…

Most binary formats contain text; that isn't what distinguishes them from text formats.

One of the objections though is that with binary formats you're limited to the capabilities of the tools that have been built to handle that particular format, which you're illustrating nicely. In a binary format world, I would have to know the capabilities and limitation of dozens, maybe hundreds of different tools for extracting useful information from logs, instead of the small handful of tools I use to do the same job now, which can be applied to any log file formatted as plain text.

And that's assuming that all these other tools will be as powerful as Powershell, which isn't a bet I'd want to make.

madhouse has some fair points about the limitations of text logs, but "everything should be stored in binary formats" is a not a great idea. Actually, "a terrifying new hell" is probably closer to how I feel about it.

Re: Grepping logs is still terrible

#112

Post by kasabali » Thu, May 07, 2015, 10:39 PM UTC

KaiserPro wrote: post #94

ownagefool wrote: post #37

Earlier quoted context omitted.

Not that I think it matters all that much but journald will be running on your centos box, but it'll be configured to spew text into /var/log. Just type journalctl and you should see the data there.

I suspected as much, so long as I don't have to use its tools I couldn't care less. Unless it eats resources.

> Unless it eats resources

I have no idea on its effect in practice, but theoretically it should have negative effect because it soaks in all the logs, and then it forwards them to the logging daemon, even when journal storage is turned off.

Re: Grepping logs is still terrible

#113

Post by deathanatos » Fri, May 08, 2015, 1:05 AM UTC

scrollaway wrote: post #102

deathanatos wrote: post #97

It seems to me that most of the worry about a binary log file being "opaque" could be solved with a single utility: log-cat … that just outputs it in text. Then you can attack the problem with whatever text-based tools you want. But to me, having a utility that I could do things like, get a range of log lines — in sorted order —, or, grep on just the message, would be amazing. These are all things that proponents of…

What you're describing is journalctl.

I've not yet had the opportunity to try systemd. :-) Someday.

Re: Grepping logs is still terrible

#114

Post by 4ydx » Fri, May 08, 2015, 1:06 AM UTC

nl wrote: post #58

4ydx wrote: post #24

Earlier quoted context omitted.

http://www.internetlivestats.com/total-number-of-websites/ If I read it correctly there are about 250 million active sites (roughly). It seems unlikely that they are all massive corporate sites. As an aside, the idea that systemd is a good thing is hilarious to me at the least because it is so brash about making an important change to a huge chunk of the system. Yes the bugs will eventually get ironed out, but in the…

I'm pretty sure that counts (eg) each wordpress.com subdomain as a separate website. [1] counts like that and gives a roughly comparable number. That gives a lot of economy scales. [1] http://news.netcraft.com/archives/category/web-server-survey...

The link I included states that those are unique hostnames. Perhaps they are including subdomains on the same ip address, but you might note that rather than quoting the 1 billion sites, I reduced that by their estimated 25% being actually active. Additionally they state that there are on average 3 users per site in 2014. Maybe that doesn't mean anything, but as a rough estimate that all implies far more small sites than large ones.

Re: Grepping logs is still terrible

#115

Post by 4ydx » Fri, May 08, 2015, 1:24 AM UTC

Frondo wrote: post #91

What you lose when you move away from text logs is not any real benefit; what you lose is the illusion of control you have with text logs. Text logs can be corrupted, text logs can be made unusable, you need a ton of domain-specific knowledge to even begin to make sense of text logs, etc. But there's always a sense that, if you had the time, you could still personally extract meaning from them. With binary logs, you…

A substring of text may or may not be a date and based on the excellent tools available in linux you can decide how to extract that "data point". If binary logging is little more than a stream of text, then that is fine, but I seriously doubt that is the push happening. Personally I prefer having a raw stream of data that I have to work with as best as I can rather than having to use some flag defined by somebody else to range across dates. That is the fundamental difference it seems: do you want a collection of tools that can be applied in a variety of ways or do you want the "one way" (with potential versioning... have fun!).

Again if the binary log is simply better compressed data, well we have ways of compressing text already as an afterthought. This really, fundamentally, seems to be a conflict in how people want to administer their systems and, for the most part, this seems to be about creating a "tool" that people then have to pay money for to better understand.

Re: Grepping logs is still terrible

#116

Post by 4ydx » Fri, May 08, 2015, 1:25 AM UTC

My main problem with this is that ascii is not something that will ever change over time. The data format is wonderfully static. Forever. Introduce a binary format? You get versioning. It is a major downside.

Re: Grepping logs is still terrible

#117

Post by geographomics » Fri, May 08, 2015, 1:28 AM UTC

thaumaturgy wrote: post #111

geographomics wrote: post #103

Earlier quoted context omitted.

This isn't really true as the Windows event logs contain text as well as the other structured data, which you can search for using tools on the system. For example to search for some specific text in the system log using Powershell: Get-EventLog -LogName System | Where {$_.Message -Match "something"} To process text as fields, as with awk, one would use the Split method (at least to start off with): Get-EventLog -Log…

Most binary formats contain text; that isn't what distinguishes them from text formats. One of the objections though is that with binary formats you're limited to the capabilities of the tools that have been built to handle that particular format, which you're illustrating nicely. In a binary format world, I would have to know the capabilities and limitation of dozens, maybe hundreds of different tools for extracting…

In the case of wanting to stick to a text-only workflow, rather than taking advantage of the structured data features, then you only need a tool that converts the binary log format to your preferred text format. Which isn't too arduous. In systemd that would be journalctl, in Windows anything that can use the event log API such as Powershell or many other utilities.

The examples I posted above were just to show the equivalent capabilities in Powershell but really it's all flexible enough to use whatever you like.

Re: Grepping logs is still terrible

#118

Post by mdekkers » Fri, May 08, 2015, 2:36 AM UTC

4ydx wrote: post #24

mdekkers wrote: post #12

Earlier quoted context omitted.

"The number of people administering small systems is much greater than the number of people administering large systems" Do you have any evidence for this statement? Because it sounds all kinds of wrong.

http://www.internetlivestats.com/total-number-of-websites/ If I read it correctly there are about 250 million active sites (roughly). It seems unlikely that they are all massive corporate sites. As an aside, the idea that systemd is a good thing is hilarious to me at the least because it is so brash about making an important change to a huge chunk of the system. Yes the bugs will eventually get ironed out, but in the…

...systemd?

Re: Grepping logs is still terrible

#119

Post by laumars » Fri, May 08, 2015, 6:48 AM UTC

geographomics wrote: post #104

laumars wrote: post #76

Earlier quoted context omitted.

But if you're not storing them in a database then your primary advantage of using a binary format (namely performance) evaporates.

The difference is that a general purpose database typically organises data by fixed-size pages, so new data could be anywhere in the file as there is no guarantee of page ordering with regard to inserts. Whereas a specialised file format for logging would add new records at the end of the file (or in a circular fashion, depending on the design). But will have features similar to a database like a defined schema, and…

You can do the same kind of indexing with text files too, eg you see this with dictionary and thesaurus databases.

Thus if you're going to sacrifice the "read anyway" ability of a log file then you really need to go for a fully optimised database to really take advantage of a binary format - rather than this half-and-half approach that has none of the real benefits of either but all of the same drawbacks of both.

Re: Grepping logs is still terrible

#120

Post by crdoconnor » Fri, May 08, 2015, 10:10 AM UTC

kbenson wrote: post #100

benjaminjackman wrote: post #66

Earlier quoted context omitted.

Maybe I am misunderstanding but it sounds like you are encountering bad json log file practices because json entries are spanning multiple lines. Which implies they are being printed in non compact form aka prettified. Thats a problem in the pure text world too. And hurts worse when it happens there. Its kind of an apples to oranges comparison. Json log files should ideally print using compact form (which will never…

I have no problem with json output in log files, but I would greatly prefer it be constrained to the message portion of a logline. At a minimum I generally want three things per line, a timestamp (in ISO 8601 or something close), a message type (info, warning, error, etc) or log entry source, and the message itself. I don't want to be looking into the JSON structure itself for a timestamp, especially when the field e…

>I don't want to be looking into the JSON structure itself for a timestamp

A) JSON parsers are relatively common and reliable.

B) The timestamp would be human readable even without the parser.

>especially when the field encoding the timestamp may be called something slightly different based on what generated the log...

I often come across logs that put timestamps in different places on the line and encode them differently (or don't output a timestamp at all, sometimes). This is no different to having to deal with a differently named JSON property.