Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Memory Management FAQ

landley.net

Post reply on HN
Print view View original

41–50 of 75 posts

Re: Linux Memory Management FAQ

#41

Post by spijdar »

kelnos wrote: post #37
dbattaglia wrote: post #4

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

I think that's probably still true for what 32-bit systems are still out there today. And regardless, I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM. But I expect the FAQ was probably originally thinking about desktop or server systems, so, yeah, the intent there is probably out of date. Those types of systems are rarely 32-bit these days, and usually have a bit…

> I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM.

Even this is quickly becoming less and less true (for new phones). Even the Pinephone comes with 3 GB of RAM at a $200 price point, and that's inflated because of the niche, low volume nature of its production.

Samsung's "mid range" A series smartphones, for instance, start at 3GB at the absolute lowest end, with most models coming with 6 GB of memory. I expect this will be even more common in a year or two.

Re: Linux Memory Management FAQ

#42

Post by n_jd »

phtrivier wrote: post #7

The Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?

Been a while since I've read them but I recall there was info about the FSB and northbridge, which no longer exist outside the CPU. They've been replaced by internal memory controllers and PCI-E controllers.

Re: Linux Memory Management FAQ

#43

Post by thricegr8 »

barnacled wrote: post #5

For anybody who's interested I also wrote up a whole bunch of notes on this at https://github.com/lorenzo-stoakes/linux-vm-notes and superceded by far more recent https://github.com/lorenzo-stoakes/linux-mm-notes I have made a few patches into the mm subsystem some simply inspired by researching for the articles.

If someone wants to begin their journey in understanding and level of aptitude you've displayed on your sites, where should they begin?

Re: Linux Memory Management FAQ

#44

Post by dataflow »

wahern wrote: post #27
barnacled wrote: post #24

Earlier quoted context omitted.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

This depends on the OS. Solaris and Windows both do strict accounting by default, and overcommit is opt-in at a fine-grain API level. Linux is relatively extreme in its embrace of overcommit. So extreme that strict accounting isn't even possible--even if you disable overcommit in Linux, there are too many corner cases in the kernel where a process (including innocent processes) will be shot down under memory pressure…

> Solaris and Windows both do strict accounting by default, and overcommit is opt-in at a fine-grain API level.

Not sure what you mean by this - I don't think Windows has overcommit in any form, whether opt-in or opt-out. What it does have is virtual address space reservation, but that's separate from commitment; reserved virtual memory is not backed by any page, no matter how much free RAM you have, until you explicitly tell the system to commit physical memory to it.

In fact I'm not even sure 'opt-in' to overcommit is possible in principle. Because if you opt-in to overcommit, you jeopardize other applications' integrity—who likely did not opt-in.

Re: Linux Memory Management FAQ

#45

Post by wahern »

dataflow wrote: post #44
wahern wrote: post #27

Earlier quoted context omitted.

This depends on the OS. Solaris and Windows both do strict accounting by default, and overcommit is opt-in at a fine-grain API level. Linux is relatively extreme in its embrace of overcommit. So extreme that strict accounting isn't even possible--even if you disable overcommit in Linux, there are too many corner cases in the kernel where a process (including innocent processes) will be shot down under memory pressure…

> Solaris and Windows both do strict accounting by default, and overcommit is opt-in at a fine-grain API level. Not sure what you mean by this - I don't think Windows has overcommit in any form, whether opt-in or opt-out. What it does have is virtual address space reservation, but that's separate from commitment; reserved virtual memory is not backed by any page, no matter how much free RAM you have, until you explic…

I thought there was a flag or commonly used library function that would do VirtualAlloc(MEM_RESERVER) and then from an in-process page fault handler attempt VirtualAlloc(MEM_COMMIT). But I guess I was wrong? I assume it's possible, just not as common as I thought.

Re: Linux Memory Management FAQ

#46

Post by haberman »

xxpor wrote: post #23
amscanne wrote: post #17

Earlier quoted context omitted.

No, it must be sign-extended from the top bit of the valid set. Otherwise the address is non-canonical.

This is true for x86-64, not true for other architectures such as arm64. Apple uses the high bits to cryptographicly sign the pointer value.

Hmm, it appears that the top byte on arm64 is only ignored if TBI (Top Byte Ignore) is enabled.

I don't think pointer signing requires TBI though. Pointer signing uses the PAC instruction to sign a pointer, and the AUT instruction to verify and unpack the signed pointer, but in its signed/packed form it is not a usable pointer. So actual addressable pointers need not support non-canonical addresses.

Re: Linux Memory Management FAQ

#47

Post by dataflow »

wahern wrote: post #45
dataflow wrote: post #44

Earlier quoted context omitted.

> Solaris and Windows both do strict accounting by default, and overcommit is opt-in at a fine-grain API level. Not sure what you mean by this - I don't think Windows has overcommit in any form, whether opt-in or opt-out. What it does have is virtual address space reservation, but that's separate from commitment; reserved virtual memory is not backed by any page, no matter how much free RAM you have, until you explic…

I thought there was a flag or commonly used library function that would do VirtualAlloc(MEM_RESERVER) and then from an in-process page fault handler attempt VirtualAlloc(MEM_COMMIT). But I guess I was wrong? I assume it's possible, just not as common as I thought.

I don't know of a common (or uncommon) function like this, though I think you could indeed implement it if you really want to (likely via AddVectoredExceptionHandler). It still requires explicitly telling the OS to commit just-in-time, so it's not "overcommitting". The closest built-in thing to this that I know of is PAGE_GUARD, which is internally used for stack extension, but that's all I can think of. The use cases for such a thing would be incredibly niche though—like kind of high-performance sparse page management where every single memory access instruction counts. Like maybe if you're writing a VM or emulator or something. Something that's only appropriate for << 1% of programs.

Re: Linux Memory Management FAQ

#48

Post by [deleted] »

jdsully wrote: post #26
barnacled wrote: post #24

Earlier quoted context omitted.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

In the C standard malloc should return null if it can’t fulfill the request. Linux violates this but it usually works out in the end since virtual memory makes true OOM very rare.

[deleted]

Re: Linux Memory Management FAQ

#49

Post by ymbeld »

jdsully wrote: post #26
barnacled wrote: post #24

Earlier quoted context omitted.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

In the C standard malloc should return null if it can’t fulfill the request. Linux violates this but it usually works out in the end since virtual memory makes true OOM very rare.

I don’t know what true OOM means, but my desktop has crashed I think at least three times in the last four months and the console said “OOM killer”. About 15GB of usable RAM, 2GB swap drive. I just have to have the usual applications open plus another browser in addition to Firefox, namely Chrome. (But naturally I don’t try to actively reproduce the behavior since I usually have better things to do than wait 10 minutes from everything becoming unresponsive -- even switching from the graphical session to a console -- to the OOM killer finally deciding to kill Chrome.) And I don’t run any virtual machines, just a big, fat IDE and stuff like that.

Re: Linux Memory Management FAQ

#50

Post by AnimalMuppet »

dataflow wrote: post #25
barnacled wrote: post #24

Earlier quoted context omitted.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

I said "in C". You're talking "in Linux" (or glibc/whatever). Which, as I already said, plays by its own rules and defies C. It's broken by design.

So if I malloc 2 MB or 2 GB or whatever in a C program running on Linux, but I have not yet either read from or written to that memory, then what's the state? Has the C library forced Linux to actually allocate it, or has it not? Or does it depend, and if so, on what?
Post reply on HN