Earlier quoted context omitted.
I prefer sending the logs to some other computer over UDP.
This only works if you can tolerate loss of some log items.
[deleted]
Another Redis case: Centralized logging
21–30 of 47 posts
Re: Another Redis case: Centralized logging
#22Re: Another Redis case: Centralized logging
#23Earlier quoted context omitted.
This only works if you can tolerate loss of some log items.
Very true. If you can, then it's a great way to log things.
Yikes, losing messages? If anyone designs a system that knowingly makes itself harder to diagnose under stress by destroying evidence, I only hope it's themselves and not their successors who are there to endure the pain.
Re: Another Redis case: Centralized logging
#24Earlier quoted context omitted.
Very true. If you can, then it's a great way to log things.
Yikes, losing messages? If anyone designs a system that knowingly makes itself harder to diagnose under stress by destroying evidence, I only hope it's themselves and not their successors who are there to endure the pain.
As I said, it depends. If you're logging cycle completion times or web page visits, you don't care about losing a few.
Re: Another Redis case: Centralized logging
#25Earlier quoted context omitted.
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a gr…
There's syslog the protocol, syslog the API, and syslog the daemon. There are syslog daemon and protocol replacements that are much better, but retain the API for wide-spread compatibility.
I think my contention would be that if you shake off the early-90s-Unixisms and adopt any modern store (forget Redis even and just "use sqlite instead of fopen"), the machinery required to do 98% of what the best syslog daemons do is trivial. The syslog infrastructure just doesn't add much... except compatibility with a very dubious ecosystem of tools.
Re: Another Redis case: Centralized logging
#26Earlier quoted context omitted.
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a gr…
We use syslog where I work, and I've always felt the same way, but never heard any suggestions for better options with as wide adoption, support, and background as syslog. Out of pure curiosity, what do you see as the tool most likely to displace syslog in the future? Is there any alternative available that fixes most of these problems without rolling your own from pieces and parts?
I think we're stuck with syslog, but for app logging, ad-hoc database storage --- especially if you have either (a) a database optimized for message queueing or (b) a schemaless database --- is going to tend to beat syslog. You're not going to realize it until you need the information you're logging, though; until then, it's going to seem like syslog is everything you could reasonably need.
Re: Another Redis case: Centralized logging
#27Earlier quoted context omitted.
Why do you want to send logs over UDP? I get that you were making a snarky allusion to syslog, but what part of the syslog UDP protocol do you feel beats Redis' TCP protocol?
I don't, really (you can use syslog pretty successfully, I guess, if you don't mind it), but UDP has certain advantages over TCP, namely that your code keeps running even if the server you're sending to goes down or is unable to respond to messages, it's faster, etc. Ideally, I'd use a function that sent things to a small server over UDP, which would then put them in Redis. This assumes you don't mind losing a few li…
If you continuously send UDP messages to a server that isn't accepting them, you will eventually get an error from the sendto() system call; the uninterested receiving host is generating ICMP messages saying "I don't want these". It's true that there is a flavor of sloppy socket coding where that error doesn't manifest itself. But if you're writing good code, TCP is no less fire-and-forget than UDP.
UDP is marginally faster than TCP, but the tradeoff for that is that under heavy load, UDP imposes more costs on the rest of your traffic. Since we're talking about logging, though: who gives a shit how fast it is? With either transport, if logs are taking more than hundreds of milliseconds to clear, you have a problem you need to fix.
Re: Another Redis case: Centralized logging
#28Holy balls, talk about going out of your way to avoid syslog.
The only time in my life I've ever been envious of syslog was when I had to build an aggregator/event correlator for a bunch of telco equipment that only talked TL1.
Pretty much any other time I've had to work with it, I've wished for something better, so I, for one, am very happy by the thought of people starting to "go out of their way to avoid syslog".
Re: Another Redis case: Centralized logging
#29Earlier quoted context omitted.
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all so…
Logs "feel" like one of those problem domains that Redis is a good fit for. When Redis is a good fit for something, it tends to really, really be a good fit. Capped list keys feel almost like they were designed to hold logs.
The thing to realize here is that Redis isn't like Riak or Mongodb or even MySQL. It is stupid simple to stand up a Redis instance. The code to push logs to it: also stupid simple. Even without clever indexing, just stuffing text crud into it, Redis is already natively a great log store.
Re: Another Redis case: Centralized logging
#30Holy balls, talk about going out of your way to avoid syslog.
Syslog is a pile of shit, Ted. It's a relic. You clearly happen to love that relic, and I think you should find a way to place it just-so in a nicely lit alcove in your apartment. The rest of us should move on from it. I don't think less of you for admiring it. I have useless old things on display in my house too. * Freeform text is a terrible way to track system events. * Periodically rotated flat files are not a gr…
Yes, syslog is a pile of shit. It's a relic.
And I'll add it tends to ship in a horrible default configuration with events scattered randomly over multiple files, no safe-guards against filling up the disk and no safeguards to ensure the stupid daemon is actually running.
However...
Freeform text is a terrible way to track system events.
Nothing stops you from logging structured text.
Periodically rotated flat files are not a great way to store log information.
Modern syslog daemons will write to pretty much anything you want.
Goofy little UDP messages are not a good way to convey system events
Modern syslog daemons offer tcp transport. Some even try to offer some delivery guarantees (disk-backed spool), although personally I wouldn't rely on that for truly critical stuff.
The syslog PRI field dates back to when we exchanged messages with UUCP.
Thanks, I always wondered where those were from...
And, well, you forgot a couple bullets:
* syslog() is available everywhere, out of the box
* It's trivial to move from file-based logging to syslog
* We have mature syslog-daemons that dispatch events pretty reliably
* Unless you're facebook you probably don't need anything more fancy.
So, I'd say syslog gets the job done quite well, as long as you don't mistake it for a message queue.