"Devops" can be considered a philosophy, a methodology, or a movement. It doesn't mean "ops monkey smart enough to code." Referring to a human as a "devop" is equivalent to calling someone an "agile".
[deleted]
NGINX syslog-ing without breaking the bank or patching the code
31–40 of 47 posts
Re: NGINX syslog-ing without breaking the bank or patching the code
#32Relevant comment: https://news.ycombinator.com/item?id=6799328 " CloudFlare generates 50gb/s of logs globally and have handled collecting this volume in two ways. Historically the logs are sent to a local syslog-ng through the use of a PIPE and the forwarded to central logger. This can be done with nginx with no patches by just treating the PIPE as file. Just make sure you do a little buffering inside nginx. access_l…
If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.
>If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service.
But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.
Re: NGINX syslog-ing without breaking the bank or patching the code
#33Earlier quoted context omitted.
I'm a little concerned that these days to go-to answer for a crashing daemon is to have another daemon that watches it and restarts it. Who watches the watchers? What happened to writing things that don't break?
Whenever I hear this complaint the first thing I think about is this Douglas Adams quote: The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair. Personally when doing operations related stuff I like to assume that everything will eventually break - though…
That quote is entirely inapplicable though. It is in fact much easier to fix the simple, do one thing and do it well unix style applications. They are not hard to get at or repair. This is one area where I think the linux philosophy winning out over the unix philosophy has really cost us a lot.
Re: NGINX syslog-ing without breaking the bank or patching the code
#34Earlier quoted context omitted.
If you were using systemd, I'd imagine you'd tell it to restart your syslogger automagically, and use ExecStartPost or ExecReload to tickle nginx.
I'm a little concerned that these days to go-to answer for a crashing daemon is to have another daemon that watches it and restarts it. Who watches the watchers? What happened to writing things that don't break?
If your PID 1 is screwed, the ordering of syslogd/nginx restarts is the least of your worries.
Re: NGINX syslog-ing without breaking the bank or patching the code
#35Earlier quoted context omitted.
Related question - how do you configure logrotate to honour size immediately? In my (limited) experience, logrotate runs once overnight. If you have a log file that expands by 1gb per hour, and you want to rotate it every 500mb, how do you instruct logrotate to handle it? Do you just crontab logrotate for short intervals, and will that trigger "daily" rotations each time for other configurations?
There's no reason why you can't run logrotate more than once a day. You can run a separate instance every 15 minutes if you like, with a separate configuration file that only rotates the short-lived logs.
Ok that's the (easy) thing I was missing. Instead of using the main logrotate.conf (that includes logrotate.d/*), use a fully separate logrotate.conf. Thanks.
Re: NGINX syslog-ing without breaking the bank or patching the code
#36Earlier quoted context omitted.
If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. I would still configure nginx to write to a file and have the consumer tail it to avoid this situation. Properly-configured log rotation can keep the file size within reasonable bounds.
>If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.
Extremely likely. I've seen the rsyslog daemon die randomly many times, for example. Whereas in 6+ years of using Nginx I've never seen it fall over a single time. Web serving is usually a primary purpose for a server. You do not want your web site to be down just because the machine's syslog daemon died.
Re: NGINX syslog-ing without breaking the bank or patching the code
#37Earlier quoted context omitted.
If you were using systemd, I'd imagine you'd tell it to restart your syslogger automagically, and use ExecStartPost or ExecReload to tickle nginx.
I'm a little concerned that these days to go-to answer for a crashing daemon is to have another daemon that watches it and restarts it. Who watches the watchers? What happened to writing things that don't break?
Oh please, spare us the fake concern. systemd is just an init daemon doing what init daemons should have been doing for a long time. The only reason sysvinit didn't do this is because it was a nasty rats nest of shell scripts with absolutely zero introspection capability.
"Who watches the watchers" is such a silly question. Simple knowledge of the halting problem (and all other problems reducible to it) should inform you of how pointless it is to ask "what if" question like this. What if cosmic rays flip a critical bit in just the right way and cause the dialysis machine to explode? Who's gonna "watch" for that?
Things DO break. Having the init daemon do whatever it can to assist is just good sense.
Re: NGINX syslog-ing without breaking the bank or patching the code
#38Earlier quoted context omitted.
Whenever I hear this complaint the first thing I think about is this Douglas Adams quote: The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at or repair. Personally when doing operations related stuff I like to assume that everything will eventually break - though…
That quote is entirely inapplicable though. It is in fact much easier to fix the simple, do one thing and do it well unix style applications. They are not hard to get at or repair. This is one area where I think the linux philosophy winning out over the unix philosophy has really cost us a lot.
I cannot abide this "X philosophy vs Y philosophy" nonsense. It's just imprecise, meaningless, appeal-to-authority drivel, the likes of which should be expected from used car salesmen, not engineers. Say what you mean, not what some other guy vaguely said 20 years ago.
Re: NGINX syslog-ing without breaking the bank or patching the code
#39Earlier quoted context omitted.
>If the pipe consumer (i.e. your syslog server) stalls, then it will be much worse than writing to a local file, as it could cause a denial of service. But how likely is that to happen in any circumstance that doesn't also bring down nginx anyways? Obviously in the case of a large setup like that they don't care, it just gets removed from the pool regardless of why it is failing.
Extremely likely. I've seen the rsyslog daemon die randomly many times, for example. Whereas in 6+ years of using Nginx I've never seen it fall over a single time. Web serving is usually a primary purpose for a server. You do not want your web site to be down just because the machine's syslog daemon died.
Isn't this a reason to replace the syslog daemon rather than the webserver?
Re: NGINX syslog-ing without breaking the bank or patching the code
#40Earlier quoted context omitted.
Extremely likely. I've seen the rsyslog daemon die randomly many times, for example. Whereas in 6+ years of using Nginx I've never seen it fall over a single time. Web serving is usually a primary purpose for a server. You do not want your web site to be down just because the machine's syslog daemon died.
Isn't this a reason to replace the syslog daemon rather than the webserver?
It's a reason to keep things as loosely coupled as possible. And when I say I've seen rsyslog crash many times, I'm talking about over the course of years and thousands of servers.