Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

51–60 of 80 posts

Re: The Day “/Proc” Died

#51

Post by Iv »

theamk wrote: post #24
Iv wrote: post #20

Slightly tangential, but I find it a bit hard to accept that systems do not have a more graceful failure mode when their disks are full. I keep a few GB free on my / but when I inadvertently fill it, it becomes almost impossible to use. Would it be so hard to keep the few last MB as reserved space for debugging purpose and refuse any space allocation that is not devoted to a 'ls' or a 'baobab' process?

They do https://odzangba.wordpress.com/2010/02/20/how-to-free-reserv...

You can reboot as root in a terminal but really, there is no good reason to fall back to this mode instead of more gracefully refuse, from userland, to use up more space.

Re: The Day “/Proc” Died

#52

Post by colechristensen »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

In ops, I've seen this happen very many times. A linux server running happy and free because the kernel OOM killer murdered the reason that server existed, leaving alone some side process with a memory leak (usually some external service agent or maintenance service run amok). I learned well how to fix that over the years.

(spoiler alert: it was much more often about becoming ornery when devs insisted that their JVM app could have a 12 GB heap on a machine with 12 GB of memory)

Re: The Day “/Proc” Died

#53

Post by pas »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

https://lwn.net/Articles/759781/ already merged in 4.20.

https://github.com/facebookincubator/oomd/blob/master/README...

and here's a less complicated, but similar proactive daemon:

https://github.com/rfjakob/earlyoom/blob/master/README.md

Re: The Day “/Proc” Died

#54

Post by jve »

bpchaps wrote: post #12

Interesting. I'm legitimately surprised the author put so much work into this research (happy they did!). By far one of the biggest culprits for "really odd behavior" is because of a full disk, or the disk is in some failing/failed state. To the point where when troubleshooting, `df -k` will be one of the first commands I'll run. Does this company not have disk monitoring?

I think it is because once you know how to look at internals, you are tempted to do it that way :) And you learn alot that way too. But yeah, probably not the most effective way.

Re: The Day “/Proc” Died

#55

Post by Symbiote »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

It triggered on one of my systems yesterday, and killed the runaway process.

When we were running tests of a new distributed system on our development (slightly underspecced) cluster, it would kill the distributed system processes when they took too much RAM.

As other write, having slow or "too much" swap can delay the OOM killer from running in reasonable time.

Re: The Day “/Proc” Died

#56

Post by Symbiote »

simonjgreen wrote: post #48
aidenn0 wrote: post #19

Since ext4 became the default, my most common cause of bizarre behaviors has been running out of inodes. df -i will show this, but a bare df command will not. Getting off topic now, but does anybody know if the ext4 utilities changed the calculation of inodes when formatting compared to ext3/ext2 utilities? Running out of inodes on those filesystems was fairly unusual, but I've seen it happen a dozen times in the pas…

Across the approximately 2000 VMs we look after as an MSP, we see this at least once a week. It's almost always badly cleaned up session files from a php app or similar.

The default ratio of inodes is configured in /etc/mke2fs.conf. You should either change that, override it with the -i argument when you create the filesystem.

My desktop's / filesystem is around 238,000,000,000 bytes, and with the default ratio of one inode per 16384 bytes, I have around 14,500,000 inodes.

Note that with a default blocksize of 4096, you're limited to 4× as many inodes as you have at present, so if you're seeing this weekly I recommend monitoring the number of remaining inodes (df -i), or changing the app to store sessions in a database.

Re: The Day “/Proc” Died

#57

Post by majewsky »

Galanwe wrote: post #39
est wrote: post #32

Earlier quoted context omitted.

I always rm the dir inside /var/log and ln them back from another disk. Because I am too lazy to change default config sparsely located on /etc of many installed programs.

Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.

> Most of the programs should by default use syslog

Is that still true today? Docker wants you to log to stdout, so that's what most newer applications do. systemd also wants you to log to stdout, and will redirect stdout to journald/syslog automatically. In fact, an application that only logs to syslog can turn into a minor headache when you want to dockerize it. Which is why stuff like https://github.com/sapcc/syslog-stdout exists.

I'm having a related problem with another project that I'm working on where I wrap an OpenLDAP server. It would be much easier to properly wrap it if OpenLDAP would just log to stdout instead of bypassing me and going for the syslog. Maybe at some point I'll set up a separate mount namespace for it to pass a /dev/syslog shim into it. But this shows that the log-to-stdout pattern is much more Unix-y because it composes better.

Re: The Day “/Proc” Died

#58

Post by majewsky »

wahern wrote: post #5

Wow, that's almost exactly the same problem I encountered in Linux last week, where simply reading /proc/$pid/cmdline would block the process attempting to read the process info. It appears to have been related to this issue: https://lkml.org/lkml/2018/2/20/576 And much like the Solaris issue, as best I could tell the original processes (this occurred multiple times on two different nodes) would seem to be blocked ei…

Please don't put things that are not code in codeblocks. In this case, it makes the link unclickable for no good reason. Working link: https://lkml.org/lkml/2018/2/20/576

Re: The Day “/Proc” Died

#59

Post by hinkley »

free652 wrote: post #28

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

I broke a machine today by deploying a docker image to it. I couldn’t even ssh in because the disk was too full.

Which probably means all of those machines have the same fucked up bad configuration. If that team had something you could mistake for humility, it wouldn’t be so bad.

Re: The Day “/Proc” Died

#60

Post by krageon »

aidenn0 wrote: post #21
roryrjb wrote: post #3

I've never heard of mdb before. I mean I've never actually used Solaris (OpenSolaris/illumos to be specific) for anything other than a few experiments here and there but nothing in production. But I've been intrigued by it mostly due to watching talks by Bryan Cantrill. It seems that at least Joyent bets heavily on it, and it seems that's mostly due to ZFS. There's also DTrace in this area, again haven't used it, but…

I've been using illumos on and off since it was called "OpenSolaris" and illumos is rapidly becoming less relevant just due to the tiny ecosystem. dtrace and mdb are both great tools, but it's getting to the point where ZFS on Linux (which possibly can't even be shipped as a binary without violating the Linux EULA) is seeing more usage than all other downstream consumers (and ZFS/FreeBSD was probably a wider usage th…

> the Linux EULA

What are you talking about? This isn't a thing that exists as far as I'm aware.

Post reply on HN