Centralized Logging With Rsyslog • Hacker News Board

Re: Centralized Logging With Rsyslog

#11

Post by moe » Tue, Oct 05, 2010, 9:08 PM UTC

psadauskas wrote: post #2

I've been pushing to implement this for our application, but I'm told that we used to, and had to turn it off because it would saturate the IO of the logging server. Has anyone else experienced this? Is it just a simple configuration tuning problem?

it would saturate the IO of the logging server.

Rather unlikely unless your deployment is very large or you're doing extraordinarily expensive filtering/binning on the sink-host.

The first bottleneck is normally diskspace, not disk I/O. Those logs pile up very quickly, depending on how long you retain them.

The raw network and disk I/O, however, are rarely of concern. Before you approach either limit you're already logging to the tune of ~300G per hour - and have probably switched to a distributed architecture of some sort long ago.

Writing broad streams of sequential text is very cheap.

Making sense of what you wrote, ideally before actually writing it, but at the very least before being forced to purge it due to storage constraints, is the difficult part. ;-)

Re: Centralized Logging With Rsyslog

#12

Post by kvs » Tue, Oct 05, 2010, 9:42 PM UTC

Has anyone tried Facebook Scribe?

Re: Centralized Logging With Rsyslog

#13

Post by mmt » Tue, Oct 05, 2010, 10:04 PM UTC

js2 wrote: post #5

mmt wrote: post #4

Protip: Use syslog-ng. Besides longer log message (arbitrarily long, with a recompile) and reliable delivery, it obviates my main use for logrotate, since it can be configured to write to a filename (including directory) based on time, date, or other variables.

Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time.

Splunk, given its cost and complexity, is almost never right for startups.

Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking?

I never quite understood the conceit that every environment is a precious-and-unique snowflake requiring careful evaluation of any given tool.

Re: Centralized Logging With Rsyslog

#14

Post by thomasknowles » Tue, Oct 05, 2010, 10:30 PM UTC

mmt wrote: post #13

js2 wrote: post #5

Earlier quoted context omitted.

Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

I'm a huge Splunk user and I agree with it's price tag it would make it very difficult to justify the costs against the gains when you're operating a small and tight ship. However, I would be very interested in the tools that they use to analyse their data. Obviously they could use a series of grep/awk scripts to pull out the data in key value pairs but what do they do with it after that?

Re: Centralized Logging With Rsyslog

#15

Post by [deleted] » Wed, Oct 06, 2010, 1:47 AM UTC

mmt wrote: post #13

js2 wrote: post #5

Earlier quoted context omitted.

Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

[deleted]

Re: Centralized Logging With Rsyslog

#16

Post by [deleted] » Wed, Oct 06, 2010, 1:53 AM UTC

thomasknowles wrote: post #14

mmt wrote: post #13

Earlier quoted context omitted.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

I'm a huge Splunk user and I agree with it's price tag it would make it very difficult to justify the costs against the gains when you're operating a small and tight ship. However, I would be very interested in the tools that they use to analyse their data. Obviously they could use a series of grep/awk scripts to pull out the data in key value pairs but what do they do with it after that?

[deleted]

Re: Centralized Logging With Rsyslog

#17

Post by mmt » Wed, Oct 06, 2010, 2:04 AM UTC

mmt wrote: post #13

js2 wrote: post #5

Earlier quoted context omitted.

Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

I have karma to burn[1], but I'm embarrassed and ashamed that pointed advice gets downvoted (with no discussion) while a platitude is upvoted.

This is an example of why Startup School speakers censor themselves: we bring it upon ourselves.

[1] Especially since it's not meaningful.

Re: Centralized Logging With Rsyslog

#18

Post by thwarted » Wed, Oct 06, 2010, 3:52 AM UTC

kvs wrote: post #12

Has anyone tried Facebook Scribe?

Yes, and it works pretty solidly. We used to use syslog-ng, writing to named pipes, but you have to be sure that there is something reading from the pipe before you start writing to it, otherwise it blocks. You don't really want to use the syslog protocol (via the syslog(3) library call) for random logging because you may end up hitting the upper bounds of the log lines. I wanted to use rsyslog, mainly for the local buffering, but it ONLY seems to support the syslog format/protocol, including prefixing all lines with a date and time and a hostname.

We use scribe and we have a stdin2scribe program (python) that can be used to hook into any log output (like apache access and error logs). We have it set up in a two tier system, all systems that we'd want to log from run a "scribe leaf" on a port on localhost, and this forwards all logs to a "scribe aggregator" (behind a load balancer), with a buffering space on the local disk when the aggregator can not be contacted. It's a pretty solid system and I recommend it.

We also have services, and command line and library interfaces to those services, that let you grab all the logs that came in on a certain time frame, or tail all the data coming into the aggregators in real time (one of them is a wrapper around a more generic tool that just tags the logs, the wrapper takes grep-style filtering arguments and the output is pipped to a pretty printer).

Re: Centralized Logging With Rsyslog

#19

Post by js2 » Wed, Oct 06, 2010, 9:19 AM UTC

mmt wrote: post #13

js2 wrote: post #5

Earlier quoted context omitted.

Protip: look at rsyslog, syslog-ng, and splunk and decide what is right for your environment.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

Turns out I am a sysadmin as well, and I'm asserting that each has various strengths. I have used syslog-ng as long ago as 2001, so I have some experience with it. Today I would recommend rsyslog. It is the default logger in Ubuntu 10.04 LTS and Fedora is also transitioning to it:

http://fedoraproject.org/wiki/Releases/FeatureRsyslog

Further, I think that RELP and on-demand disk spooling of messages are compelling features. Its performance and reliability are good enough to feed your web-server access logs through.

I wouldn't overlook rsyslog, but I'm also not saying "just use it" because syslog-ng is certainly worth evaluating as well.

Re: Centralized Logging With Rsyslog

#20

Post by mmt » Wed, Oct 06, 2010, 3:12 PM UTC

js2 wrote: post #19

mmt wrote: post #13

Earlier quoted context omitted.

The "pro" that I am is sysadmin, and I'm asserting that evaluating all three is a waste of time. Splunk, given its cost and complexity, is almost never right for startups. Non-ng syslog is, on the other hand, so simplistic that it's not worth the effort of fancy configuration. Is there some kind of compelling advantage that I've been overlooking? I never quite understood the conceit that every environment is a precio…

Turns out I am a sysadmin as well, and I'm asserting that each has various strengths. I have used syslog-ng as long ago as 2001, so I have some experience with it. Today I would recommend rsyslog. It is the default logger in Ubuntu 10.04 LTS and Fedora is also transitioning to it: http://fedoraproject.org/wiki/Releases/FeatureRsyslog Further, I think that RELP and on-demand disk spooling of messages are compelling fe…

This more in-depth discussion has more value. Thank you.

I think that RELP and on-demand disk spooling of messages are compelling features

I think we're coming at the question from different perspectives. One of my primary goals is to avoid wasting my time. Since I've already evaluated and experimentally proven syslog-ng, switching means a large time investment.

As such, features like REPL and, arguably misfeatures[1], like disk spooling, fail to compel such an investment.

Once rsyslog has matured, something that I expect will be accelerated by its inclusion in major distros, it may be a no-brainer.

For my "money," there are far more interesting and productive problems to work on than logging, which is why I do give the "just use it" advice.

Turns out I am a sysadmin as well

By choice or necessity? Just curiosity on my part.

[1] I have yet to encounter an environment of non-trivial size where the risk of losing logging outweighs the risk of disk filling up and/or performance degradation from additional contentious I/O. For me, it's a killer feature of centralized logging: elimination of a particular source of failure/degradation.