Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux 7.3 improves performance when running out of vRAM

pixelcluster.dev

Post reply on HN
Print view View original

161–170 of 328 posts

Re: Linux 7.3 improves performance when running out of vRAM

#161

Post by dawnerd »

LoganDark wrote: post #146
inventor7777 wrote: post #143

I see all the comments about how Windows and Linux handle OOM, but I suppose I'll also mention macOS. On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending proce…

On my M4 Max MacBook Pro, if I run or even load an LLM more than a couple times, even a small one, general desktop performance starts to get really, really bad until a reboot and I don't know why. Genuinely, everything starts feeling super sluggish, even long after the process has terminated -- cmd+tab gets slow, apps constantly hang for a few seconds at a time, the cursor position starts lagging and videos are slow…

Same, I loaded a model that should have fit fine in my available memory and macOS progressively got more broken in the weirdest ways, like my AirPods connecting but being unresponsive and not being able to be configured. I think it started killing internal processes that failed to come back.

Re: Linux 7.3 improves performance when running out of vRAM

#162

Post by da_chicken »

mtgh2s wrote: post #92
krisknez wrote: post #34

I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.

From my experience, Windows does freeze and become completely unusable too, just not usually from RAM usage hitting 100%. I don't use Windows much, nor do I care much about that environment, but from what I've seen it seems to keep RAM usage below 100% most of the time. What I do see pretty often is the drive getting stuck at 100% usage instead, which makes the whole system ridiculously unusable anyway.

Windows does freeze and become unstable, but it's usually because of some ring 0 errors not getting trapped appropriately.

But Windows has always handled both OOM and out of disk space very well. The system will be extremely sluggish, but it typically continues operating.

The Linux design is to keep using memory, then push to swap, and then when you OOM you hard lock. The built in kernel OOM module can miss when RAM usage spikes rapidly. You can enable OOM monitors like systemd-oomd or earlyoom, but they do not run by default, and their behavior is to term the offending process.

The thing about Windows is that when you request memory, you're only granted memory if it can be guaranteed in the first place. The application will get a "not enough memory" error. On Linux, you're permitted to request more memory than the system actually has, with the idea that you won't actually use that much memory. It's optimistic that way. But if you do use it, then you're screwed.

Re: Linux 7.3 improves performance when running out of vRAM

#163

Post by rfgplk »

krisknez wrote: post #34

I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.

Configure overcommit_memory, overcommit_ratio, admin|user_reserve_kbytes, min_free_kbytes, oom_kill_allocating_task, oom_score_adj; if you're on a dev box make it such that even if you approach memory limits to just instantly start killing processes rather than swapping. It's not a solution for servers, but for dev environments it's basically perfect.

Re: Linux 7.3 improves performance when running out of vRAM

#164

Post by yencabulator »

jmbwell wrote: post #105
krisknez wrote: post #34

I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.

Solaris family kernels move this problem to run/allocation time, instead of waiting for the system to become unusable before trying to do something about it. Kinda nice

  echo 2 | sudo tee /proc/sys/vm/overcommit_memory
is there on Linux if you want it.

https://www.kernel.org/doc/html/latest/mm/overcommit-account...

Re: Linux 7.3 improves performance when running out of vRAM

#165

Post by MBCook »

LoganDark wrote: post #153
MBCook wrote: post #131

Earlier quoted context omitted.

Macs are also real bad at handling low disk space. I don’t how other OSes do but the best advice on a Mac is to always have a decent amount free. What’s a decent amount? No idea. Don’t test it :)

When my Intel Mac ran out of disk space once, it became impossible to clear the space without rebooting (due to APFS's annoying journaling crap or whatever -- it would literally say "rm: can not remove [whatever]: no space left on device" or something), which caused Google Chrome to wipe all my local storage and cookies because it loves to do that when you run out of disk space, and there's no way to get that data ba…

FWIW it didn’t do well under HPFS+ either. It’s not a new problem, though it may manifest differently.

Re: Linux 7.3 improves performance when running out of vRAM

#167

Post by inventor7777 »

LoganDark wrote: post #159
inventor7777 wrote: post #158

Earlier quoted context omitted.

Wow! That is wild. I am on 15.7.9 and I have had uptimes in the high 50s with weekly local model usage, I even tried GLM5.2 streaming from SSD and I have not noticed any slowdowns after at all. I guess you are right about 26 and 27. Dang, the more I hear about 26 I am just so glad that I did not update. What version of macOS are you on? 26 or 27? Have you considered doing a custom backup and restoring manually to 15.…

Unfortunately I don't have the spare 8 TB. Been unemployed for 6 months, have not gotten a single offer in that time, + memory/storage crisis, etc. Shit's fucked. I don't think I would downgrade though. I just hope this gets fixed someday... I can deal with the terrible memory bandwidth but I don't like to reboot. (That said, macOS is so, so much better at reboots than Windows)

I'm sorry to hear that.

But my GLM5.2 is not 8TB, it is only around 200GB and works pretty well. (0.9tok/sec from a 40Gbps NVMe)

I used https://huggingface.co/anemll/GLM-5.2-sidecar

Re: Linux 7.3 improves performance when running out of vRAM

#169

Post by cmckn »

jeffbee wrote: post #150
adrian_b wrote: post #148

Earlier quoted context omitted.

I stopped using swap almost a quarter of century ago, and since then I have not seen a PC with Linux freezing because of full RAM. Before the recent increase in RAM prices, it was not difficult to have a generous amount of RAM in a PC, that would make very unlikely the out-of-memory situations. When OOM really happened, it was much better to become aware of this immediately, and reduce the number of concurrent thread…

Unfortunately Linux makes it difficult to fully get away from swapping because any file-backed page is eligible to be evicted, and even if you have "disabled swap" the kernel will cheerfully page-out your program, even if it needs to be immediately paged in again to continue. The only way to stop it from doing so is to remap all executable mappings into anonymous memory, a thing which is possible but which the loader…

This type of thrashing was the most common OOM issue I saw on my servers, the system would effectively deadlock until I pulled the plug. In my setup this seemed to be caused by slow-ish disks, and using a small amount of zram made things more CPU-bound / less IO-bound. I haven’t seen the problem since, the OOM killer does what I expect.

Re: Linux 7.3 improves performance when running out of vRAM

#170

Post by merb »

rfgplk wrote: post #163
krisknez wrote: post #34

I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.

Configure overcommit_memory, overcommit_ratio, admin|user_reserve_kbytes, min_free_kbytes, oom_kill_allocating_task, oom_score_adj; if you're on a dev box make it such that even if you approach memory limits to just instantly start killing processes rather than swapping. It's not a solution for servers, but for dev environments it's basically perfect.

On servers with k8s the default is preempt based on prio, followed by oom kill and ‚newly’ memory qos:

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...

https://kubernetes.io/docs/concepts/configuration/manage-res...

https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#... (New feature)

So normally if you overcommit it might be undefined behavior if you not have enough memory. However if everything is fine pods can preempt based on prio to new nodes.

With the right settings only the faulty app might fail. Without it it can be a real pain to find the rouges

Post reply on HN