Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

71–80 of 80 posts

Re: The Day “/Proc” Died

#71

Post by Avamander »

pas wrote: post #53
Avamander wrote: post #27

Earlier quoted context omitted.

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

https://lwn.net/Articles/759781/ already merged in 4.20. https://github.com/facebookincubator/oomd/blob/master/README... and here's a less complicated, but similar proactive daemon: https://github.com/rfjakob/earlyoom/blob/master/README.md

I'm on 5.0.0 and I legit haven't noticed a difference. If I run out of RAM without swap the system freezes and if I have swap then the system freezes when both are full. The only reliable solution is having a RAM+swap usage graph on my screen at all times and then closing stuff manually.

Re: The Day “/Proc” Died

#72

Post by aidenn0 »

krageon wrote: post #60
aidenn0 wrote: post #21

Earlier quoted context omitted.

I've been using illumos on and off since it was called "OpenSolaris" and illumos is rapidly becoming less relevant just due to the tiny ecosystem. dtrace and mdb are both great tools, but it's getting to the point where ZFS on Linux (which possibly can't even be shipped as a binary without violating the Linux EULA) is seeing more usage than all other downstream consumers (and ZFS/FreeBSD was probably a wider usage th…

> the Linux EULA What are you talking about? This isn't a thing that exists as far as I'm aware.

https://www.kernel.org/doc/html/v4.18/process/license-rules....

Re: The Day “/Proc” Died

#73

Post by mshook »

colechristensen wrote: post #52
Avamander wrote: post #27

Earlier quoted context omitted.

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years. (spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM…

That's pretty much why we use vm.panic_on_oom = 1 as you have no idea about what's going to be killed...

Re: The Day “/Proc” Died

#74

Post by olliej »

pjmlp wrote: post #66
olliej wrote: post #38

Earlier quoted context omitted.

The OOM killer is a “solution” to a very real, and sensible design choice: not committing physical memory and swap whenever address space is mapped - there are very good (and noticeable) reasons for not eagerly committing, but fundamentally if you have done so you have to decide what to do when you end up needing more physical space than is available. Linux went down the “if a process is trying to do this, it must be…

Problem is that something else might be actually performing a critical task.

The assumption being made is that the app going mad for memory is the critical one. Something has to die, and deciding which is hard.

Re: The Day “/Proc” Died

#75

Post by colechristensen »

mshook wrote: post #73
colechristensen wrote: post #52

Earlier quoted context omitted.

In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years. (spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM…

That's pretty much why we use vm.panic_on_oom = 1 as you have no idea about what's going to be killed...

I am in the camp of having systemd or your service manager of choice restart any killed service or service failing health check and emitting an ERROR or higher log message.

I want my systems to heal themselves. More often than not these memory problems end up being slow leaks which can be effectively permanently resolved with periodic restarts, and the engineering time to fix them is appropriately not prioritized.

I want to know that there has been a problem but I would rather not be forced to do anything about it unless absolutely necessary.

Re: The Day “/Proc” Died

#76

Post by pas »

Avamander wrote: post #71
pas wrote: post #53

Earlier quoted context omitted.

https://lwn.net/Articles/759781/ already merged in 4.20. https://github.com/facebookincubator/oomd/blob/master/README... and here's a less complicated, but similar proactive daemon: https://github.com/rfjakob/earlyoom/blob/master/README.md

I'm on 5.0.0 and I legit haven't noticed a difference. If I run out of RAM without swap the system freezes and if I have swap then the system freezes when both are full. The only reliable solution is having a RAM+swap usage graph on my screen at all times and then closing stuff manually.

You probably won't until the distro you use (or you manually) set up something other than the default OOM killer.

You probably have too much swap. More than ~10 sec * your I/O speed (so let's say 512M-1G) is probably the max for the reasons you mentioned.

Re: The Day “/Proc” Died

#77

Post by dharmab »

simonjgreen wrote: post #48
aidenn0 wrote: post #19

Since ext4 became the default, my most common cause of bizarre behaviors has been running out of inodes. df -i will show this, but a bare df command will not. Getting off topic now, but does anybody know if the ext4 utilities changed the calculation of inodes when formatting compared to ext3/ext2 utilities? Running out of inodes on those filesystems was fairly unusual, but I've seen it happen a dozen times in the pas…

Across the approximately 2000 VMs we look after as an MSP, we see this at least once a week. It's almost always badly cleaned up session files from a php app or similar.

So raise the inode allocation? It's a small amount of additional memory allocation to fend off a time consuming failure cause

Re: The Day “/Proc” Died

#78

Post by Avamander »

pas wrote: post #76
Avamander wrote: post #71

Earlier quoted context omitted.

I'm on 5.0.0 and I legit haven't noticed a difference. If I run out of RAM without swap the system freezes and if I have swap then the system freezes when both are full. The only reliable solution is having a RAM+swap usage graph on my screen at all times and then closing stuff manually.

You probably won't until the distro you use (or you manually) set up something other than the default OOM killer. You probably have too much swap. More than ~10 sec * your I/O speed (so let's say 512M-1G) is probably the max for the reasons you mentioned.

The system freezes also without any swap enabled but much more suddenly (there's no slowdown before dying). It's really just that the OOM killer triggers way way way too late.

Re: The Day “/Proc” Died

#80

Post by simonjgreen »

Symbiote wrote: post #56
simonjgreen wrote: post #48

Earlier quoted context omitted.

Across the approximately 2000 VMs we look after as an MSP, we see this at least once a week. It's almost always badly cleaned up session files from a php app or similar.

The default ratio of inodes is configured in /etc/mke2fs.conf. You should either change that, override it with the -i argument when you create the filesystem. My desktop's / filesystem is around 238,000,000,000 bytes, and with the default ratio of one inode per 16384 bytes, I have around 14,500,000 inodes. Note that with a default blocksize of 4096, you're limited to 4× as many inodes as you have at present, so if yo…

We're talking about preexisting systems. inodes are already tweaked up during build to highest, and we monitor similar to how you suggest via a checkmk script. If the apps running were ours we could make changes to make them clean up better or store through other means, alas this is just something you have to handle when doing break fix response on 100s of clients making their own decisions.
Post reply on HN