Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

NGINX syslog-ing without breaking the bank or patching the code

syshero.org

Post reply on HN
Print view View original

11–20 of 47 posts

Re: NGINX syslog-ing without breaking the bank or patching the code

#11

Post by pilif »

staunch wrote: post #3

You can also write a pretty simple program to do essentially `tail -F /var/log/nginx/access.log | logger --server some.host`. I'm doing something like that. It works, but I still don't like it. The idea of nginx hanging while trying to write to FIFOs I like even less.

The issue of the article the OP was a response to is that the traffic they are expecting is too big for the disk nginx is running on to handle the log files.

Your solution works for centralising the log, but it doesn't solve the local i/o issue as the file is still being written (which was the problem of the original article).

Re: NGINX syslog-ing without breaking the bank or patching the code

#13

Post by staunch »

pilif wrote: post #11
staunch wrote: post #3

You can also write a pretty simple program to do essentially `tail -F /var/log/nginx/access.log | logger --server some.host`. I'm doing something like that. It works, but I still don't like it. The idea of nginx hanging while trying to write to FIFOs I like even less.

The issue of the article the OP was a response to is that the traffic they are expecting is too big for the disk nginx is running on to handle the log files. Your solution works for centralising the log, but it doesn't solve the local i/o issue as the file is still being written (which was the problem of the original article).

Easily solved by writing the logs to a filesystem in RAM (e.g. tmpfs). Have your little program truncate them as often as you need.

Re: NGINX syslog-ing without breaking the bank or patching the code

#14

Post by chc »

alrs wrote: post #2

"Devops" can be considered a philosophy, a methodology, or a movement. It doesn't mean "ops monkey smart enough to code." Referring to a human as a "devop" is equivalent to calling someone an "agile".

It seems like a reasonable shortening of "devops engineer." If we can call people "quants," I don't suppose "devop" is all that much sillier.

Re: NGINX syslog-ing without breaking the bank or patching the code

#15

Post by bashtoni »

Nice work!

We write the logs to disk and then use rsyslogs imfile feature to read from there. Your approach has the advantage of not requiring disk writes.

BTW, we're not in Toronto, but we're hiring and happy to accept remote workers from that timezone :)

http://www.bashton.com/jobs/

Re: NGINX syslog-ing without breaking the bank or patching the code

#16

Post by blibble »

bashtoni wrote: post #15

Nice work! We write the logs to disk and then use rsyslogs imfile feature to read from there. Your approach has the advantage of not requiring disk writes. BTW, we're not in Toronto, but we're hiring and happy to accept remote workers from that timezone :) http://www.bashton.com/jobs/

put your logs in /dev/shm

oh and don't remember to rotate them!

Re: NGINX syslog-ing without breaking the bank or patching the code

#17

Post by otterley »

radq wrote: post #4

Relevant comment: https://news.ycombinator.com/item?id=6799328 " CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_l…

If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service.

I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.

Re: NGINX syslog-ing without breaking the bank or patching the code

#18

Post by otterley »

rodgerd wrote: post #9
semiquaver wrote: post #5

This seems dangerous -- if rsyslog crashes then the pipe is left without a consumer and all writes to it block indefinitely, exactly the opposite of how you would expect syslog to work.

If you were using systemd, I'd imagine you'd tell it to restart your syslogger automagically, and use ExecStartPost or ExecReload to tickle nginx.

Perhaps, but there are plenty of scenarios where restarting won't help (e.g. daemon misconfiguration).

Re: NGINX syslog-ing without breaking the bank or patching the code

#19

Post by otterley »

blibble wrote: post #16
bashtoni wrote: post #15

Nice work! We write the logs to disk and then use rsyslogs imfile feature to read from there. Your approach has the advantage of not requiring disk writes. BTW, we're not in Toronto, but we're hiring and happy to accept remote workers from that timezone :) http://www.bashton.com/jobs/

put your logs in /dev/shm oh and don't remember to rotate them!

If you need guaranteed log delivery, I wouldn't do that. Unread logs won't survive a power failure or system crash.

Re: NGINX syslog-ing without breaking the bank or patching the code

#20

Post by nasalgoat »

rodgerd wrote: post #9
semiquaver wrote: post #5

This seems dangerous -- if rsyslog crashes then the pipe is left without a consumer and all writes to it block indefinitely, exactly the opposite of how you would expect syslog to work.

If you were using systemd, I'd imagine you'd tell it to restart your syslogger automagically, and use ExecStartPost or ExecReload to tickle nginx.

I'm a little concerned that these days to go-to answer for a crashing daemon is to have another daemon that watches it and restarts it.

Who watches the watchers? What happened to writing things that don't break?

Post reply on HN