11–20 of 105 posts
People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating to logs are now broken * binary logs are easy to corrupt, e.g. if they didn't get closed properly. >You can have a binary index and text logs too! / You can. But what's the point? The point is having human-readable logs without having to use a proprietary piece…
Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…
People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating to logs are now broken * binary logs are easy to corrupt, e.g. if they didn't get closed properly. >You can have a binary index and text logs too! / You can. But what's the point? The point is having human-readable logs without having to use a proprietary piece…
What's wrong with Splunk? Honest question.
I don't agree with the second assertion there. Text logs are only opaque as far as the format is concerned, but not so much as far as the content goes. Using the example in the article;
127.0.0.1 - - [04/May/2015:16:02:53 +0200] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0"
That isn't necessarily an argument against binary logging, but the notion that text log files are opaque in the same way as binary logs isn't really true.
People don't want it because it's binary, not because you can't grep it. * you need to use a new proprietary tool to interact with them * all scripts relating to logs are now broken * binary logs are easy to corrupt, e.g. if they didn't get closed properly. >You can have a binary index and text logs too! / You can. But what's the point? The point is having human-readable logs without having to use a proprietary piece…
What's wrong with Splunk? Honest question.
On the other hand, if you have logs, you need to store them in a centralized place and have an aging policy, etc... Grepping is definitely not the answer. Systems like Splunk exist for a reason.
Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…
As the author himself points out: "I'm sorry, but deciding how much and what we log is not your job. Its ours, and this is the amount we have to deal with."
That goes both ways. If I only have one or two servers, having to run a centralized logging services doesn't scale either, the overhead is not worth the trouble.
If I want to look for an IP in logs from multiple service, text files are perfect. Doing the same across multiple servers, yes, then you want centralized logging. Binary logging ruins the first case, while text based works in both (sort of).
I don't really see the point of binary logs. Either you're small enough that text files won't be an issue, or you're large enough to have centralized logging.
It seems that there's a push towards "scaleable solution" for everything, but people keep forgetting that you need to scale down as well. Most of us will never have to run more than a handful of servers, and in these cases the Twitter/Google/Facebook-like infrastructure just isn't worth the hassle.
Yes, grepping logs is terrible if "you have 100Gb of logs a day". I'm not sure why the author is thinking his use case is anything near the norm or why he's shocked in most use cases people prefer text files. I'm also not getting why he just doesn't use scripts to parse the logs and insert them into a database at that point. Why use some ad-hoc logging binary format if you're doing complex queries that SQL would be b…
This obviously only works when you are trouble shooting a specific issue, not when you need to investigate something that happened in the past (where the logging for the session wasn't enabled). However, it has proven to be an excellent tool for troubleshooting issues in the system.
I have used session-based logging both when I worked at Ericsson (the AXE system), and at Symsoft (the Nobill system), and both were excellent. However, I get a feeling that they are not in widespread use (may be wrong on that though), so that's why I wrote a description of them: http://henrikwarne.com/2014/01/21/session-based-logging/
Grep them, tail them, copy and paste, search, transform them, look at them in less, open them in any editor. I love two write little bash oneliners that answer questions about logs. I can use these onliners everywhere anytime.
I dont have any of the efficiency problems the author talks about.