Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

11–20 of 80 posts

Re: The Day “/Proc” Died

#11

Post by souprock »

wahern wrote: post #5

Wow, that's almost exactly the same problem I encountered in Linux last week, where simply reading /proc/$pid/cmdline would block the process attempting to read the process info. It appears to have been related to this issue: https://lkml.org/lkml/2018/2/20/576 And much like the Solaris issue, as best I could tell the original processes (this occurred multiple times on two different nodes) would seem to be blocked ei…

This is part of why I resisted requests to make the Linux ps command sort processes by default. Half a process listing is a useful hint.

It also saves memory, which is great on the insane 10240-core boxes.

Re: The Day “/Proc” Died

#12

Post by bpchaps »

Interesting. I'm legitimately surprised the author put so much work into this research (happy they did!). By far one of the biggest culprits for "really odd behavior" is because of a full disk, or the disk is in some failing/failed state. To the point where when troubleshooting, `df -k` will be one of the first commands I'll run.

Does this company not have disk monitoring?

Re: The Day “/Proc” Died

#13

Post by Alupis »

icedchai wrote: post #9
tetha wrote: post #6

Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full? There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic…

Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems.

> Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems.

Yep, to the point where `df -h` has become one of the first things I run when a server starts acting funny or things stop working.

Disk being full is far too common - run away logging, weird temp files, etc, or sometimes just a box that nobody maintained for years and years.

The fun part starts after you've identified which partition or drive is full - now you have to identify the problem files!

Re: The Day “/Proc” Died

#14

Post by segmondy »

bpchaps wrote: post #12

Interesting. I'm legitimately surprised the author put so much work into this research (happy they did!). By far one of the biggest culprits for "really odd behavior" is because of a full disk, or the disk is in some failing/failed state. To the point where when troubleshooting, `df -k` will be one of the first commands I'll run. Does this company not have disk monitoring?

Did you not see that the date was 2011? I think this is lesson learned sort of article. When things go wonky, look for simple reasons. At least he didn't think it was a hardware bug.

Re: The Day “/Proc” Died

#15

Post by drudru11 »

Wow - cool example of live kernel debugging. It is very cool that the kernel is still usable while this issue occurred.

Also lucky that someone was logged into the zone without a /proc dependency. Usually people have complex shell prompts that might require /proc lookups.

It is concerning, though, that a less privileged zone could affect the entire system.

Re: The Day “/Proc” Died

#16

Post by spydum »

I've never used mdb on solaris, but its where I first learned of strace (and ptrace) - which unlocked soooo much of how unix worked for me, and lead me down the path to unix wizardy (well, that and access to open source code bases). in fact, when i started reading this, it's where I thought it was headed (in fact, I think could have come to the conclusion quicker than trying to read mdb)

Re: The Day “/Proc” Died

#17

Post by bpchaps »

segmondy wrote: post #14
bpchaps wrote: post #12

Interesting. I'm legitimately surprised the author put so much work into this research (happy they did!). By far one of the biggest culprits for "really odd behavior" is because of a full disk, or the disk is in some failing/failed state. To the point where when troubleshooting, `df -k` will be one of the first commands I'll run. Does this company not have disk monitoring?

Did you not see that the date was 2011? I think this is lesson learned sort of article. When things go wonky, look for simple reasons. At least he didn't think it was a hardware bug.

Nope, I didn't actually. Which is funny and makes a bit more sense, since I haven't seen a full disk cause bad locking like that since around the date of the article. Still, this never should have been an issue, since disk space monitoring shoulda prevented the problem from ever happening in the first place.

Re: The Day “/Proc” Died

#18

Post by mobilemidget »

Alupis wrote: post #13
icedchai wrote: post #9

Earlier quoted context omitted.

Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems.

> Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems. Yep, to the point where `df -h` has become one of the first things I run when a server starts acting funny or things stop working. Disk being full is far too common - run away logging, weird temp files, etc, or sometimes just a box that nobody maintained for years and years. The fun part starts after you've identif…

First things you run on login?

It ought to be monitored with alerting by default and inode use too.

Re: The Day “/Proc” Died

#19

Post by aidenn0 »

Since ext4 became the default, my most common cause of bizarre behaviors has been running out of inodes.

   df -i
will show this, but a bare df command will not.

Getting off topic now, but does anybody know if the ext4 utilities changed the calculation of inodes when formatting compared to ext3/ext2 utilities? Running out of inodes on those filesystems was fairly unusual, but I've seen it happen a dozen times in the past 5 years on ext4.

Re: The Day “/Proc” Died

#20

Post by Iv »

Slightly tangential, but I find it a bit hard to accept that systems do not have a more graceful failure mode when their disks are full.

I keep a few GB free on my / but when I inadvertently fill it, it becomes almost impossible to use. Would it be so hard to keep the few last MB as reserved space for debugging purpose and refuse any space allocation that is not devoted to a 'ls' or a 'baobab' process?

Post reply on HN