Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

21–30 of 80 posts

Re: The Day “/Proc” Died

#21

Post by aidenn0 »

roryrjb wrote: post #3

I've never heard of mdb before. I mean I've never actually used Solaris (OpenSolaris/illumos to be specific) for anything other than a few experiments here and there but nothing in production. But I've been intrigued by it mostly due to watching talks by Bryan Cantrill. It seems that at least Joyent bets heavily on it, and it seems that's mostly due to ZFS. There's also DTrace in this area, again haven't used it, but…

I've been using illumos on and off since it was called "OpenSolaris" and illumos is rapidly becoming less relevant just due to the tiny ecosystem. dtrace and mdb are both great tools, but it's getting to the point where ZFS on Linux (which possibly can't even be shipped as a binary without violating the Linux EULA) is seeing more usage than all other downstream consumers (and ZFS/FreeBSD was probably a wider usage than Illumos for a long while now).

Ultimately being a libre *nix that is better in a few ways than Linux seems to be a long-term losing proposition, as Linux will eventually check all the boxes you (even if it's not quite as nice) thus steadily shrinking your niche.

Re: The Day “/Proc” Died

#22

Post by verisimilitudes »

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening.

Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges that would protect them from this. It would be preferable for the system to die, rather than permit this debugging nonsense, considering it would perhaps actually be fixed if the machine died in this case.

This is an asinine failure case mired in 1970s malpractice. Don't you agree this is damning and unacceptable? No one has any right to be proud of this mess. There's millions of lines of code and yet basic failure cases aren't truly accounted for or are handled in the most asinine of ways, such as with the ``Out of Memory Killer''.

Re: The Day “/Proc” Died

#23

Post by theamk »

verisimilitudes wrote: post #22

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges…

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes.

This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glaring design omission.

Re: The Day “/Proc” Died

#24

Post by theamk »

Iv wrote: post #20

Slightly tangential, but I find it a bit hard to accept that systems do not have a more graceful failure mode when their disks are full. I keep a few GB free on my / but when I inadvertently fill it, it becomes almost impossible to use. Would it be so hard to keep the few last MB as reserved space for debugging purpose and refuse any space allocation that is not devoted to a 'ls' or a 'baobab' process?

They do https://odzangba.wordpress.com/2010/02/20/how-to-free-reserv...

Re: The Day “/Proc” Died

#25

Post by icedchai »

mobilemidget wrote: post #18
Alupis wrote: post #13

Earlier quoted context omitted.

> Nobody thought to run "df"? Disk being full is a common source of all sorts of weird server problems. Yep, to the point where `df -h` has become one of the first things I run when a server starts acting funny or things stop working. Disk being full is far too common - run away logging, weird temp files, etc, or sometimes just a box that nobody maintained for years and years. The fun part starts after you've identif…

First things you run on login? It ought to be monitored with alerting by default and inode use too.

True. Unfortunately good monitoring is an afterthought in most organizations.

Re: The Day “/Proc” Died

#27

Post by Avamander »

theamk wrote: post #23
verisimilitudes wrote: post #22

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges…

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

Re: The Day “/Proc” Died

#28

Post by free652 »

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

Re: The Day “/Proc” Died

#29

Post by Avamander »

Iv wrote: post #20

Slightly tangential, but I find it a bit hard to accept that systems do not have a more graceful failure mode when their disks are full. I keep a few GB free on my / but when I inadvertently fill it, it becomes almost impossible to use. Would it be so hard to keep the few last MB as reserved space for debugging purpose and refuse any space allocation that is not devoted to a 'ls' or a 'baobab' process?

Yep, things like desktop environments don't even start up when / is full even if /tmp or /home isn't. Awful.

Re: The Day “/Proc” Died

#30

Post by chmod775 »

verisimilitudes wrote: post #22

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges…

I fill my tiny 250GB SSD on my thinkpad all the time (well it's mostly logs of a certain terrible database topping it up now and then).

Everything including my usual desktop environment keeps working. I have /home on a separate partition though.

Most of the time I only notice that it filled up again when pacman -Syu fails because it can't save anything.

It probably really depends on your setup and how stuff is configured. It appears the various systemd tools keep working just fine though, otherwise my system wouldn't even boot.

If I could make a wish, I'd want my system to just buffer writes (transparently) to RAM once the SSD filled up. Then fire some events that can be intercepted by a GUI, so the user could be informed that he'll lose data if he shuts the system down now. Then as a last resort, abort any shutdown the user initiated and drop him into a shell so he can fix the mess and allow the system to save his data.

Post reply on HN