Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Day “/Proc” Died

dev.to

Post reply on HN
Print view View original

31–40 of 80 posts

Re: The Day “/Proc” Died

#31

Post by dsfyu404ed »

chmod775 wrote: post #30
verisimilitudes wrote: post #22

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges…

I fill my tiny 250GB SSD on my thinkpad all the time (well it's mostly logs of a certain terrible database topping it up now and then). Everything including my usual desktop environment keeps working. I have /home on a separate partition though. Most of the time I only notice that it filled up again when pacman -Syu fails because it can't save anything. It probably really depends on your setup and how stuff is config…

Same here only I don't have /home on a separate partition. I don't usually notice it's full until I go to download something and it fails.

Re: The Day “/Proc” Died

#32

Post by est »

free652 wrote: post #28

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

I always rm the dir inside /var/log and ln them back from another disk. Because I am too lazy to change default config sparsely located on /etc of many installed programs.

Re: The Day “/Proc” Died

#33

Post by lulouie »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

Yep, when Chromium ate up all the memory, it just hang the whole OS, waiting for OOM killer about 10+ min, then the cursor can be moved again, then freeze again...

Re: The Day “/Proc” Died

#34

Post by [deleted] »

theamk wrote: post #23
verisimilitudes wrote: post #22

Some things never change. Is there a single UNIX variant that doesn't fail in some way when it can no longer write to disk? It's unacceptable for most programs to fail if they can no longer write to disk; Firefox also fails, but of course gives no indication as to what's happening. Now, of course, no UNIX variant I'm aware of has a real notion of system programs and so there's no programs that get special privileges…

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

[deleted]

Re: The Day “/Proc” Died

#35

Post by viraptor »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

Sounds like you're starting to swap heavily. Adjusting swappiness to 0 may help there.

Re: The Day “/Proc” Died

#36

Post by AnIdiotOnTheNet »

Avamander wrote: post #27
theamk wrote: post #23

Earlier quoted context omitted.

Did you know that Linux has this special note of “root reserved space” specifically for situations like this? And the same way, there is per-process “oom_adj” which can be used to control OOM killer priority, to spare the system processes. This problem has been solved, multiple times. Of course many distributions fail to mark appropriate processes as system, so they fail anyway, but this is just a bug, not an a glari…

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

The OOM killer is a terrible hack that no self respecting system should have ever employed.

Re: The Day “/Proc” Died

#38

Post by olliej »

AnIdiotOnTheNet wrote: post #36
Avamander wrote: post #27

Earlier quoted context omitted.

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

The OOM killer is a terrible hack that no self respecting system should have ever employed.

The OOM killer is a “solution” to a very real, and sensible design choice: not committing physical memory and swap whenever address space is mapped - there are very good (and noticeable) reasons for not eagerly committing, but fundamentally if you have done so you have to decide what to do when you end up needing more physical space than is available.

Linux went down the “if a process is trying to do this, it must be important so I’ll prioritise it and kill something else”, and alternative is to kill that process when the commit fails.

Either is a valid option, the OOM killer ran against a regular desktop user’s idea of what is the correct course of action, but for a server it might not have been.

Re: The Day “/Proc” Died

#39

Post by Galanwe »

est wrote: post #32
free652 wrote: post #28

The rookie mistake a lot of admins do is never creating a partition for /var/log , I lost count how many times servers went into a weird mode when the root is getting filled to 100%

I always rm the dir inside /var/log and ln them back from another disk. Because I am too lazy to change default config sparsely located on /etc of many installed programs.

Most of the programs should by default use syslog, so you should not really have to configure much of them except enabling syslog output. Then you just have to configure your syslog implementation to write wherever you desire, rotate the files, or forward the messages to other machines for storing, etc.

Re: The Day “/Proc” Died

#40

Post by olliej »

lulouie wrote: post #33
Avamander wrote: post #27

Earlier quoted context omitted.

The OOM killer never in the twelve years I've used Linux has triggered before my system grinds to a halt and never recovers. This problem has not been solved.

Yep, when Chromium ate up all the memory, it just hang the whole OS, waiting for OOM killer about 10+ min, then the cursor can be moved again, then freeze again...

If you want hilarious fun: make a gl shader than takes ~30 seconds to run. GPUs are only very recently preemptable (if they are at all yet? I lose track of what is “planned” vs released).

Make it run that shader in a loop.

See how well your system appears to respond.

IIRC macOS has a 60s or something watchdog the hard resets the GPU, while the gpu is hung the screen is not updated. Everything is running fine, cpu isn’t pinned or anything, but the gpu is blocked so no compositing, and so no screen updating.

I’m not sure what Linux does in that case, and I think windows may be able to paint because the directx driver interfaces let it do ... something? I’ve always assume some way to dma straight to the framebuffer, but no real idea :)

Post reply on HN