Would like to know what logging strategy people have been following in a distributed environment.
Another Redis case: Centralized logging
sunilarora.org
Another Redis case: Centralized logging
1–10 of 47 posts
Re: Another Redis case: Centralized logging
#2Depends on how much you care about latency, right?
The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on how often you rotate.
Re: Another Redis case: Centralized logging
#3Depends on how much you care about latency, right? The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on…
I prefer sending the logs to some other computer over UDP.
Re: Another Redis case: Centralized logging
#4My only question: how is this better than syslog?
Re: Another Redis case: Centralized logging
#5My only question: how is this better than syslog?
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
Re: Another Redis case: Centralized logging
#6Depends on how much you care about latency, right? The easy solution is just, y'know, write the log to a file and scp it back to some central place every so often. But then you have to either (a) keep track of how much of a file you've copied, which is a pain; or (b) only grab files that you're no longer actively writing to (as determined by naming scheme or something), but that introduces some latency, depending on…
I prefer sending the logs to some other computer over UDP.
This only works if you can tolerate loss of some log items.
Re: Another Redis case: Centralized logging
#7Re: Another Redis case: Centralized logging
#8A similar concept, though using MongoDB with a capped collection is http://graylog2.org.
Re: Another Redis case: Centralized logging
#9The article was fairly devoid of the necessary details. One of the main ones being: how many events are we talking about? 10 events/sec? 100/sec? 10_000/sec ? And what is the size of these events? How many event emitters are connecting to the Redis server?
With the details, it would be a much more interesting post.
Re: Another Redis case: Centralized logging
#10My only question: how is this better than syslog?
I agree, it is nothing different than syslog if all you want to do is just dump all logs in one file. But one advantage of this approach in our case was that we wanted to show last 1000 critical logs on a web interface and with logs stored in redis, it was pretty easy to do that. And as redis was part of our stack, it was very easy to hook it up for this specific task.
Syslog implementations like syslog-ng support both TCP and UDP relaying of all log data on a machine to a centralized Syslog server, and can even bypass storing those logs to the source machine's disk at all. Syslog-ng also supports inserting that data directly into MySQL, and there are various other backends (like Splunk, though I know it's commercial) that can accept the TCP and UDP streams and index them in all sorts of fancy ways.
I think the key point here is that all the above mentioned implementations have significant adoption and are in a sense "battle-tested". For example, what if your background worker has failed and log events are piling up in the Redis list you are using as queue? Do you have monitors in place to detect that situation, and at what value do your alarms go off? Projects like this have a way of taking a lot more time than originally thought, often at the expense of your core development time. I personally don't like spending the time writing and maintaining code for a project that isn't aligned with the problem I'm trying to solve, so I avoid it whenever possible.
On the flip side, if you are setting out to build a really robust logging system on top of Redis, and that's something of value to your organization, then more power to you!