Centralized Logging With Rsyslog
blog.urbanairship.com
Centralized Logging With Rsyslog
1–10 of 22 posts
Re: Centralized Logging With Rsyslog
#2I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server.
Has anyone else experienced this? Is it just a simple configuration tuning problem?
Re: Centralized Logging With Rsyslog
#3I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server. Has anyone else experienced this? Is it just a simple configuration tuning problem?
It definitely helped to separate the incoming queue from the actions queue on the central loghost. Unfortunately, I don't have good data comparing the two. At this time, our loghost is an EC2small, so relatively susceptible to I/O disruption, and we haven't had any problems.
If you're running a whole datacenter's worth of logs into your loghost, then you might want to consider a distributed approach. For medium sized companies, I don't see why rsyslog wouldn't do the trick.
Re: Centralized Logging With Rsyslog
#4Protip: Use syslog-ng.
Besides longer log message (arbitrarily long, with a recompile) and reliable delivery, it obviates my main use for logrotate, since it can be configured to write to a filename (including directory) based on time, date, or other variables.
Re: Centralized Logging With Rsyslog
#5Protip: Use syslog-ng. Besides longer log message (arbitrarily long, with a recompile) and reliable delivery, it obviates my main use for logrotate, since it can be configured to write to a filename (including directory) based on time, date, or other variables.
Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.
Re: Centralized Logging With Rsyslog
#6I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server. Has anyone else experienced this? Is it just a simple configuration tuning problem?
I've had to up the maximum simultaneous connections, but otherwise I've been able to log extremely large volumes of data from 1,000s of machines, with syslog-ng. I'd assume the same is true with rsyslog.
Looking at limits on open files, and considering tcp timeouts and cookies were all we needed. Total volume was in the order of high 10s of GB.
If you want to unpack offline, contact details are in my profile.
Re: Centralized Logging With Rsyslog
#7One big advantage of rsyslog over syslog-ng is that you can spool messages to disk if the remote syslog server is down (syslog-ng only offers this in their 'enterprise' paid version).
Re: Centralized Logging With Rsyslog
#8If you're interested in more volume and flexibility, check out Flume, a new open-source project from Cloudera (the Hadoop/logging experts). Solid software and community behind it. http://archive.cloudera.com/cdh/3/flume/UserGuide.html
Re: Centralized Logging With Rsyslog
#9I've had much better results from using Syslog-NG along with php-Syslog-NG.
Syslog-NG can already split your log files into subdirectories with the hostname of each server, but it also has the capability of redirecting messages to named pipes. This is great because you can pipe it into mysql and stuff all of your log messages in a database. Combine with a php front-end and now your developers and sysadmins can search logs intelligently across multiple servers, and get really fine-grained on their search strings. Want to tail the output from all Tomcat servers in your app server pool looking for a specific string? Go right ahead.
Re: Centralized Logging With Rsyslog
#10I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server. Has anyone else experienced this? Is it just a simple configuration tuning problem?
It depends - if you're logging from thousands of machines or services, you might want to use UDP logging. Then, provided you don't saturate the network connection, you should be fine. UDP is connectionless, so you definitely won't have as much overhead.
On the other hand, a lot of high security environments want to encrypt their syslog traffic using something like stunnel, which introduces OpenSSL overhead as well as TCP connection overhead. With thousands of clients and lots of encryption going on you definitely are going to hit some limits sooner rather than later. Check kernel parameter net.ipv4.ip_local_port_range (on Linux) and make sure you have a large enough range to accomodate all of the clients.