Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

1–10 of 80 posts

Re: The Day “/Proc” Died

#3

Post by roryrjb »

I've never heard of mdb before. I mean I've never actually used Solaris (OpenSolaris/illumos to be specific) for anything other than a few experiments here and there but nothing in production. But I've been intrigued by it mostly due to watching talks by Bryan Cantrill. It seems that at least Joyent bets heavily on it, and it seems that's mostly due to ZFS. There's also DTrace in this area, again haven't used it, but I've been itching to use bpftrace on Linux as an equivalent. Anyway, what I really came to ask is, who uses Solaris or illumos? Does it have a future? How relevant is it?

Re: The Day “/Proc” Died

#5

Post by wahern »

Wow, that's almost exactly the same problem I encountered in Linux last week, where simply reading /proc/$pid/cmdline would block the process attempting to read the process info. It appears to have been related to this issue:

  https://lkml.org/lkml/2018/2/20/576
And much like the Solaris issue, as best I could tell the original processes (this occurred multiple times on two different nodes) would seem to be blocked either in the filesystem or memory management layers flushing pages.

Re: The Day “/Proc” Died

#6

Post by tetha »

Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full?

There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic problem to monitor for.

Re: The Day “/Proc” Died

#8

Post by rkeene2 »

tetha wrote: post #6

Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full? There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic…

The great thing about Solaris is all the debugging tools, the bad thing about Solaris is you have to use all the debugging tools.

As to why you might want to do this -- it wasn't obvious that the filesystem being full was related to "echo *" in /proc hanging, until after the debugging was done.

Re: The Day “/Proc” Died

#9

Post by icedchai »

tetha wrote: post #6

Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full? There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic…

Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems.

Re: The Day “/Proc” Died

#10

Post by tetha »

rkeene2 wrote: post #8
tetha wrote: post #6

Hm. I am impressed by the casual kernel debugging. I couldn't do that. However, why would I have to do that if running check_disk on all important file systems tells me if file systems are >80% or >90% full? There'll be bugs I cannot debug, because I cannot casually debug my linux kernel. Yes. I still have to encounter them personally, but I'll get there. But this just seems weird, because this is a very, very basic…

The great thing about Solaris is all the debugging tools, the bad thing about Solaris is you have to use all the debugging tools. As to why you might want to do this -- it wasn't obvious that the filesystem being full was related to "echo *" in /proc hanging, until after the debugging was done.

Mh, I fully agree that I don't expect '(cd /proc; echo *)' to hang because of the disk being full.

However, with my current setup, the alerting checks all disks of all monitored systems every minute, and once a single of them exceeds 80%, we get alerts. That's no smart setup, that's stock nagios/iciniga/icinga2 with stock nrpe checks. This is considered a very cheap and basic setup when running servers. We've had this on our solaris systems out of the box.

And practically, this is an alert with a very high true positive rate and very few false positive alerts. Most false positives we've had to deal with had been with systems with > 3-4 Tb of storage. And systems filling up more than 5% - 10% of their storage per minute tend to trigger other stock alerts as well.

Post reply on HN