NGINX syslog-ing without breaking the bank or patching the code
1–10 of 47 posts
Re: NGINX syslog-ing without breaking the bank or patching the code
#2"Devops" can be considered a philosophy, a methodology, or a movement. It doesn't mean "ops monkey smart enough to code."
Referring to a human as a "devop" is equivalent to calling someone an "agile".
Re: NGINX syslog-ing without breaking the bank or patching the code
#3You can also write a pretty simple program to do essentially `tail -F /var/log/nginx/access.log | logger --server some.host`. I'm doing something like that. It works, but I still don't like it. The idea of nginx hanging while trying to write to FIFOs I like even less.
Re: NGINX syslog-ing without breaking the bank or patching the code
#4Relevant comment: https://news.ycombinator.com/item?id=6799328
" CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx.
access_log /dev/nginx_access log_format_name buffer=64k flush=10s;
Since this is a pipe there is still some blocking IO, but no worse off then writing to local file."
Re: NGINX syslog-ing without breaking the bank or patching the code
#5This seems dangerous -- if rsyslog crashes then the pipe is left without a consumer and all writes to it block indefinitely, exactly the opposite of how you would expect syslog to work.
Re: NGINX syslog-ing without breaking the bank or patching the code
#6Working on a similar problem in our stack and didn't know about the limitations on NGINX. I used syslog-ng years ago but now we're on rsyslog. I feel like this is squarely a devops problem and one where the software producing logs is deficient.
Re: NGINX syslog-ing without breaking the bank or patching the code
#7Writing to a pipe is different from writing to a local file, only because you have to be aware of what's going on behind the scenes.
If the other side of the pipe closes (e.g. syslog-ng), your process will hang once the I/O buffer fills up, which is generally 4KB.
You especially need to keep this in mind, when syslog-ng restarts (pipe closed, pipe reopened). You'll probably glance quickly to see that syslog-ng is running, and the pipe is present, but wonder why nginx is not serving traffic.
Or worse, be fooled into thinking nginx is fine because it was serving traffic up until it served 50 requests (or however many reqs it takes to generate 4,000 bytes of access logs), then stopped.
Re: NGINX syslog-ing without breaking the bank or patching the code
#8This seems dangerous -- if rsyslog crashes then the pipe is left without a consumer and all writes to it block indefinitely, exactly the opposite of how you would expect syslog to work.
You can just send an USR1 signal to nginx when you restart rsyslog (hopefully with some kind of supervisor). With Upstart, you could use a post-start script in the rsyslog service configuration.
Re: NGINX syslog-ing without breaking the bank or patching the code
#9This seems dangerous -- if rsyslog crashes then the pipe is left without a consumer and all writes to it block indefinitely, exactly the opposite of how you would expect syslog to work.
If you were using systemd, I'd imagine you'd tell it to restart your syslogger automagically, and use ExecStartPost or ExecReload to tickle nginx.
Re: NGINX syslog-ing without breaking the bank or patching the code
#10Use Logstash it's 10 times better than 'quick' hack for those problems.