Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Grepping logs is terrible

asylum.madhouse-project.org

Post reply on HN
Print view View original

91–100 of 105 posts

Re: Grepping logs is terrible

#91

Post by regularfry »

madhouse wrote: post #69
regularfry wrote: post #34

This is all well and good if you want to, and can, spend time up front figuring out how to parse each and every log line format which might appear in syslog so you can drop it in your structured store. The alternative is to leave everything unstructured, and understand the formats minimally and lazily. Laziness is a virtue, right?

Why would I need to be able to parse everything up front? Taking the syslog example, that has a commonly understood format. As a default case, I can just split the parts and have structured data (esp. with RFC5424, where structured data is part of the protocol to begin with). Then, I can add further parsers for the MESSAGE part whenever I feel like it, or whenever there is need. I don't need that up front.

Because in my experience, the interesting stuff isn't in the syslog metadata. It's in the message part. Until you add that further parser, you're grepping.

Re: Grepping logs is terrible

#92

Post by gambiter »

madhouse wrote: post #54
moonshinefe wrote: post #2

Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…

You're missing the point. I'm not using a custom logging format. I'm using binary log storage , with emphasis on the storage. There is a database and a search engine behind it. Logging format and log storage format are two very different things. Also, I'm not shocked people prefer text files. I'm shocked why they're so much against binary log storage. There's an important distinction between the two: you can prefer t…

> There's an important distinction between the two: you can prefer text, if that fits your case better, without hating on binary storage.

Except according to the article (which you posted and are defending all over this thread, so I'm guessing you actually wrote it?) the author has NO intention of honoring those who prefer text logs, in fact using the phrase "so vigilantly against text based log storage". To use your own reply, you can prefer binary, if it fits your case better, BUT DON'T HATE ON TEXT STORAGE.

Re: Grepping logs is terrible

#94

Post by linuxydave »

oblio wrote: post #93
linuxydave wrote: post #90

Earlier quoted context omitted.

ELK stack - Elasticsearch, Logstash, and Kibana. The whole stack is opensource :)

Interesting, but it's not a SaaS. It doesn't look like a direct rival to Splunk.

It is a direct rival to Splunk :) They do very similar things however IMHO Splunk is the better solution right now. There are LaaS companies that use ELK if you need a cloud solution - Loggly is the first one that springs to mind and I think another is LogSene.

Re: Grepping logs is terrible

#95

Post by datenwolf »

TheLoneWolfling wrote: post #84
datenwolf wrote: post #72

Earlier quoted context omitted.

> You basically have a very fast laser that > can do volumetric scans at a high framerate, > did I get this right? Sort of. The laser itself is constantly sweeping its wavelength (over a bandwidth of >100nm). Using it as a light source in a interferometer where one leg is reflected by a fixed mirror and the other leg goes into the sample something interesting happens: The interferometric fringes produced for a certai…

Alright, gotta say, that's cool. Frequency-sweeping... How are you doing that? Is the laser itself able to frequency sweep? Or are you chirping pulses?

    > Frequency-sweeping... How are you doing that?
The basic principle is called FDML; there's a short description of how it works on our company website:

http://optores.com/index.php/technology/5-fourier-domain-mod...

A much more thorough description is found in the paper that introduced FDML for the first time:

https://www.osapublishing.org/oe/abstract.cfm?URI=oe-14-8-32...

    > Is the laser itself able to frequency sweep?
The laser itself is doing the sweeps.

    > Or are you chirping pulses?
No. In fact one of the PhDs that came out of our group was generating pulses by compressing the sweeps:

http://www.nature.com/ncomms/journal/v4/n5/full/ncomms2870.h...

Re: Grepping logs is terrible

#96

Post by 616c »

malka wrote: post #66
616c wrote: post #24

On a slightly unrelated note, as a largely amateur Linux user: have people made systems that instead of grepping for info, use machine learning do detect normal patterns of a log file (like what type of events, similar, at different intervals) and report the anomalous output via email or report to an admin? I was thinking this would be a cool area of research for me to try programming again, but it seems so daunting…

you can use fail2ban for this. It is used to automatically ban IP that, for instance, tries to bruteforce your SSH, but it really is an engine that match regexp log file lines, and fires an action if the regexp match. So you can use it for other usages (such as sending an admin a mail if suddenly your server sends 500 errors, or a unusual amount of 404 errors for instance)

Of course. Not to be dismissive, but I am familiar with fail2ban. I was wondering if anyone had this idea that did not require manual or pre-set rules, like that the program would go passive for a few days, reading log files and learning certain log entries will be indentical minus timestamp, then some change with a small amount of text in addition, and others have never been seen (or will not match in the next stage). Next stage turns active, and the machine filters down and sends you anything it has not seen over time and knows must be something anomolous.

I like fail2ban, a lot, and alternatives in that field, but when I looked at the Arch Linux package last time there were dozens of commented-out, but heavily commented nonetheless regexp template files like you describe. I think this would be a neat machine learning thing.

What I am going for: use AI to train a passive entry-level sysadmin to warn you.

Re: Grepping logs is terrible

#97

Post by 616c »

falcolas wrote: post #88
nosir33 wrote: post #26

Earlier quoted context omitted.

I don't know of any systems that do this. As an software developer, I generally use log levels to indicate severity in my logs. So grepping for ERROR should catch anything I had the foresight to log at the ERROR level. Simple heuristics like the number of WARN level logs a minute may be useful. Beyond that it sounds interesting. It may be hard to do in a general way, so focusing on Apache logs or something common may…

In addition to logging, you can send out a statsd[0] message, graph it, and use something like Skyline[1] for alerting based on trend issues. You can also use logstash to generate metrics on logs when sending them up to Elasticsearch. [0] https://github.com/etsy/statsd [1] https://github.com/etsy/skyline

Excellent sample projects, especially Skyline. This seems the closest thing to what I had envisioned sofar.

Very cool stuff. Do you use it?

Re: Grepping logs is terrible

#98

Post by TheLoneWolfling »

datenwolf wrote: post #95
TheLoneWolfling wrote: post #84

Earlier quoted context omitted.

Alright, gotta say, that's cool. Frequency-sweeping... How are you doing that? Is the laser itself able to frequency sweep? Or are you chirping pulses?

> Frequency-sweeping... How are you doing that? The basic principle is called FDML; there's a short description of how it works on our company website: http://optores.com/index.php/technology/5-fourier-domain-mod... A much more thorough description is found in the paper that introduced FDML for the first time: https://www.osapublishing.org/oe/abstract.cfm?URI=oe-14-8-32... > Is the laser itself able to frequency swee…

Interesting.

What you're doing sounds a lot like time-domain spectroscopy in an odd sort of way.

What are the advantages of this versus just chirping a pulsed supercontinuum source?

Re: Grepping logs is terrible

#99

Post by tatterdemalion »

njharman wrote: post #82
tatterdemalion wrote: post #9

This applies more generally than just to logs. I love Unix, but "everything is text" is not actually great. It's better that Unix utils output arbitrary ASCII than that they output arbitrary binary data, but it's obvious why people don't do serious IPC 'the Unix way.' Imagine if instead of exchanging JSON, or ProtoBufs, or whatever, your programs all exchanged text you had to regex into some sort of adhoc structure.…

JSON is text! Text is not synonymous with unstructured.

Of course JSON is encoded in Unicode, making it "text," but when it is said that text is the universal protocol of Unix, it means that the only guarantee a well-behaving Unix utility can make is that it will output ASCII. You cannot leverage the further structure of JSON or any other protocol because utilities that interpret JSON do not compose with those many Unix utilities which emit non-JSON data.

Only entropic bits are truly "unstructured data." The question is one of how much semantic structure you can rely on in the data you are processing, which is a continuum.

Re: Grepping logs is terrible

#100

Post by wang_li »

alephnil wrote: post #4

I guess that much of the resistance against the binary logs of systemd is the unfamiliarity and to some extent lack of well known tools for dealing with them. Sysadmins that have years of experience with traditional Unix tools now suddenly have to start almost from scratch when it comes to everyday tools for examining the system. Not only that, programmers are also most familiar with text based formats, and libraries…

It's like no one remembers the reasons we switched away from fixed format records. The biggest of which is that text based logging is a lot more future proof. Sure I might have to change a regex when time stamps improve their resolution to milliseconds, but at least I won't have to rebuild my entire suite and deal with two incompatible binary files on disk.
Post reply on HN