this is more confirmation of my anecdotal experience that when things fail it's 90% of the time a FS problem (run out of space, run out of inodes etc), thanks :)
The Day “/Proc” Died
61–70 of 80 posts
Re: The Day “/Proc” Died
#62Earlier quoted context omitted.
The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
That is probably being cause by your use of swap space, not an OOM issue. I've had multiple cases of the OOM killer kicking off on my system, all without it slowing way down.
Lacking swap space causes more severe symptoms in an OOM situation, not less, from my experience. I think this is because everything that can get evicted from RAM is before the OOM killer gets invoked, which means every disk access slows to a crawl.
Re: The Day “/Proc” Died
#63Earlier quoted context omitted.
Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.
> Most of the programs should by default use syslog Is that still true today? Docker wants you to log to stdout, so that's what most newer applications do. systemd also wants you to log to stdout, and will redirect stdout to journald/syslog automatically. In fact, an application that only logs to syslog can turn into a minor headache when you want to dockerize it. Which is why stuff like https://github.com/sapcc/sysl…
> Is that still true today?
I think it is.
> Docker wants you to log to stdout, so that's what most newer applications do.
According to Debian popularity contest [1], which matches my observations, Docker itself isn't a particularly common package to be found in a system.
> systemd also wants you to log to stdout
While that's an option, there is sd-journal(3), which allows proper logging with priorities and custom fields.
[1] https://popcon.debian.org/by_inst
Edit: Perhaps worth mentioning that even with systemd and logging into stdout, syslogd (and maybe journald) configuration should be sufficient to sort out the log files, as mentioned in the grandparent comment.
Re: The Day “/Proc” Died
#64Earlier quoted context omitted.
Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.
> Most of the programs should by default use syslog Is that still true today? Docker wants you to log to stdout, so that's what most newer applications do. systemd also wants you to log to stdout, and will redirect stdout to journald/syslog automatically. In fact, an application that only logs to syslog can turn into a minor headache when you want to dockerize it. Which is why stuff like https://github.com/sapcc/sysl…
If your dockerized application logs to a file you can always symlink that file to /dev/stdout as a last resort.
Re: The Day “/Proc” Died
#65Just df -h, dude.
Right, how do you know to check that? What if you're out of inodes, would you know to check for that, just randomly because a process is acting weird? Learn something everyday.
The experience came from the days when I wasn't able to debug yet, and when full disks were frequent for similar reasons.
Re: The Day “/Proc” Died
#66Earlier quoted context omitted.
The OOM killer is a terrible hack that no self respecting system should have ever employed.
The OOM killer is a “solution” to a very real, and sensible design choice: not committing physical memory and swap whenever address space is mapped - there are very good (and noticeable) reasons for not eagerly committing, but fundamentally if you have done so you have to decide what to do when you end up needing more physical space than is available. Linux went down the “if a process is trying to do this, it must be…
Problem is that something else might be actually performing a critical task.
Re: The Day “/Proc” Died
#67Earlier quoted context omitted.
The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
Pro-Tip: You can use Alt + SysRq + F to trigger the OOM-killer action immediately. It helps me avoiding pulling the plug of my desktops on multiple occasions over the years when I accidentally start RAM-eating programs. Just make sure SysRq is enabled in sysctl.
When the machine is busy -- and even when it's not, it's very uncertain thing to try sys-rq combinations.
Re: The Day “/Proc” Died
#68Earlier quoted context omitted.
Pro-Tip: You can use Alt + SysRq + F to trigger the OOM-killer action immediately. It helps me avoiding pulling the plug of my desktops on multiple occasions over the years when I accidentally start RAM-eating programs. Just make sure SysRq is enabled in sysctl.
When the machine is busy -- and even when it's not, it's very uncertain thing to try sys-rq combinations.
Yes, it's risky. But when the machine is swapping to death and pulling the plug is the only option, SysRq is a better alternative.
Re: The Day “/Proc” Died
#69Earlier quoted context omitted.
The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
That is probably being cause by your use of swap space, not an OOM issue. I've had multiple cases of the OOM killer kicking off on my system, all without it slowing way down.
No. This happens even without swap, OOM killer might as well not exist.
Re: The Day “/Proc” Died
#70Earlier quoted context omitted.
The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.
Sounds like you're starting to swap heavily. Adjusting swappiness to 0 may help there.
This also happens without swap.