Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nginx 1.7.1 adds syslog support

nginx.org

Post reply on HN
Print view View original

31–40 of 45 posts

Re: Nginx 1.7.1 adds syslog support

#31

Post by danudey »

nrr wrote: post #25
quonn wrote: post #7

Earlier quoted context omitted.

It's useful if you want to log to a centralized logging server. This helps to have all your logs in one place and also keeps the logs safe, if someone breaks into your server.

That's a pretty weak argument considering that syslog is entirely UDP and is bound to drop log data, sometimes en masse, most likely even silently. Not a good idea. Why not use something like multilog or svlogd and wire up a tiny processor for it to kick logging data over someplace using something like rsync? To boot, syslog is annoying to tune, depending on your particular implementation. rsyslog has a default buffe…

> That's a pretty weak argument considering that syslog is entirely UDP and is bound to drop log data, sometimes en masse, most likely even silently. Not a good idea.

rsyslog and syslog-ng have support for TCP

> Why not use something like multilog or svlogd

Additional point of failure.

> and wire up a tiny processor for it to kick logging data over someplace using something like rsync?

Additional point of failure (processor); additional point of failure (rsync/ssh); non-realtime log replication (which is bad for breaking/progressive system failure/etc).

> To boot, syslog is annoying to tune, depending on your particular implementation.

All of the examples you list are easier to learn about, tweak, and monitor than the suggestions you've proposed, however.

> On an nginx server that services 2TB/mo worth of transit (which is distinctly possible since I've got infrastructure in production that does this), there's a good chance that you'll be stretching some of these limits a bit.

If you're dealing with 2 TB/mo in transit, you're probably capable enough to understand the risks with centralized log management and mitigate/monitor them ahead of time.

Re: Nginx 1.7.1 adds syslog support

#33

Post by peterwwillis »

forgottenpass wrote: post #29
akurilin wrote: post #4

Why is logging to syslog as opposed to a separate file an important use case? Genuinely curious.

The use case is people with: - A large enough deployment to want centralized logging but are: - Cheap enough not to buy nginx (for good reason or not) and - Too lazy to maintain a patcheset against distro packages and - Too bad at linux administration to use the file pipe trick to log to syslog anyway So, yeah syslog is nice but this change does have quite a narrow use case. What it did have were vocal complainers th…

This is kind of a cheap shot at people who just wanted built-in syslog support. There are plenty of reasons someone might not be able to patch or upgrade a binary, and using a fifo is a pretty bad kludge considering the whole thing can hang if they're started/stopped in the wrong order (meaning your startup scripts now have to be rewritten). Building in syslog support means just getting the damn thing working without special hacks and kludges.

But to answer OP's question, syslog is better than just appending to a file because syslog does a lot of things for you, like filtering your logs in real time and splitting them into new files, logging remotely to industry-standard aggregation devices, access control, (somewhat) standardized formatting, log rolling, etc.

Re: Nginx 1.7.1 adds syslog support

#34

Post by peterwwillis »

Thaxll wrote: post #30

Finally... it worked well with logstash but a native implementation is always better. Is that TCP only, I don't see any UDP settings?

It's UDP only. They also hard-code the port instead of looking it up with getservbyname, which is a little weird. If you want to use TCP, send syslog messages to localhost, and have a local rsyslog/RELP forwarder send messages remotely.

Re: Nginx 1.7.1 adds syslog support

#35

Post by worklogin »

forgottenpass wrote: post #29
akurilin wrote: post #4

Why is logging to syslog as opposed to a separate file an important use case? Genuinely curious.

The use case is people with: - A large enough deployment to want centralized logging but are: - Cheap enough not to buy nginx (for good reason or not) and - Too lazy to maintain a patcheset against distro packages and - Too bad at linux administration to use the file pipe trick to log to syslog anyway So, yeah syslog is nice but this change does have quite a narrow use case. What it did have were vocal complainers th…

My job as a sysadmin is to keep servers running and log their status and errors.

When a large team has many jobs:

-Yes, I want central logging

-I probably did buy nginx (you're telling me this was already available, but closed source? How rude.)

-No, managing distro patches and writing a web server are not my jobs

-No, I'd prefer avoiding hackish file redirects in lieu of real features, but I'll do what I have to.

Re: Nginx 1.7.1 adds syslog support

#36

Post by lobster_johnson »

nrr wrote: post #25
quonn wrote: post #7

Earlier quoted context omitted.

It's useful if you want to log to a centralized logging server. This helps to have all your logs in one place and also keeps the logs safe, if someone breaks into your server.

That's a pretty weak argument considering that syslog is entirely UDP and is bound to drop log data, sometimes en masse, most likely even silently. Not a good idea. Why not use something like multilog or svlogd and wire up a tiny processor for it to kick logging data over someplace using something like rsync? To boot, syslog is annoying to tune, depending on your particular implementation. rsyslog has a default buffe…

In addition to what danudey says, modern sysloggers also support Unix domain sockets, which are reliable. Typically this is /dev/log; on Linux, I believe the GNU C Library's syslog() uses this by default.

On Linux, sending UDP to localhost is very reliable and fast, essentially going through kernel buffers with very little overhead. You will only see dropped data if the system is extremely overloaded. I did some testing, a few years back, and was not able to induce packet loss on localhost.

The usual way to set up centralized logging with syslog is to have each node run a local syslog daemon (eg., RSyslog), which then buffers the data and streams it to a central syslog daemon using a more reliable protocol such as RELP [1] over TCP.

[1] http://www.rsyslog.com/doc/omrelp.html

Re: Nginx 1.7.1 adds syslog support

#37

Post by artbikes »

rahilsondhi wrote: post #20
chrisbolt wrote: post #13

Earlier quoted context omitted.

syslog -> logstash -> elasticsearch -> kibana

Can I reach you privately? I need help with my ELK set up :/

You can usually get excellent support on the Logstash IRC channel. #logstash on irc.freenode.org

Re: Nginx 1.7.1 adds syslog support

#38

Post by IgorPartola »

lobster_johnson wrote: post #36
nrr wrote: post #25

Earlier quoted context omitted.

That's a pretty weak argument considering that syslog is entirely UDP and is bound to drop log data, sometimes en masse, most likely even silently. Not a good idea. Why not use something like multilog or svlogd and wire up a tiny processor for it to kick logging data over someplace using something like rsync? To boot, syslog is annoying to tune, depending on your particular implementation. rsyslog has a default buffe…

In addition to what danudey says, modern sysloggers also support Unix domain sockets, which are reliable. Typically this is /dev/log; on Linux, I believe the GNU C Library's syslog() uses this by default. On Linux, sending UDP to localhost is very reliable and fast, essentially going through kernel buffers with very little overhead. You will only see dropped data if the system is extremely overloaded. I did some test…

While I agree with you on the whole, one minor point about UDP/datagrams: they are not reliable even on localhost under some circumstances. The point of datagrams is that they are allowed to be lost without a trace if the consumer (syslog) is not consuming fast enough. For example, if process A starts spewing 10,000 log records (UPD or datagram UNIX socket packets) a second at syslog, and syslog can only handle 5,000, then the other 5,000 records will be lost. Any other process will also get its records lost as they will not be guaranteed to be processed. The rate of loss will be controlled by how large a datagram buffer the consumer's kernel has. Moreover, the processing will not be uniform: the buffer is LIFO, so older records will be processed while newer ones will be lost.

On the other hand if you use stream sockets, the producer will either block or be told that the consumer is not ready to read any more data (beauty of TCP). In either case, TCP produces enough overhead compared to UDP to slow down the actual useful part of your application, which is often not desirable.

Neither one of these is a good solution as either your consumer or your producer needs to keep their own very large buffers to accommodate spikes in traffic. Ideally, you do this anyways to ensure that you hold onto all the packets you received.

Having said that, I don't know exactly what rsyslog does so I cannot say if this would actually be a problem for it.

Re: Nginx 1.7.1 adds syslog support

#39

Post by kolev »

nailer wrote: post #11
kolev wrote: post #9

Earlier quoted context omitted.

True, it's been available, but having it in the core means that it will get into Linux distros, i.e. people don't have to compile.

Distros have been patching things like this for a long time.

True, but I can't recall any having the syslog module out of the box.
Post reply on HN