Live data from Hacker News

Linux Memory Management FAQ

landley.net

51–60 of 75 posts

Re: Linux Memory Management FAQ

#51

Earlier quoted context omitted.

I said "in C". You're talking "in Linux" (or glibc/whatever). Which, as I already said, plays by its own rules and defies C. It's broken by design.

So if I malloc 2 MB or 2 GB or whatever in a C program running on Linux, but I have not yet either read from or written to that memory, then what's the state? Has the C library forced Linux to actually allocate it, or has it not? Or does it depend, and if so, on what?

It depends on the overcommit setting. By default it's on, and that indicates Linux doesn't promise to back it with a physical page. Only the virtual address range is allocated (i.e. the only guarantee is that future allocations within your process won't return addresses from that range). This implies that if you try to write to it, your write might segfault due to OOM. If overcommit is turned off, then Linux promises it will be backed with a physical page if you try to it, meaning your write won't segfault due to OOM. Aside from these, I think everything else is an implementation detail, but generally OSes map unwritten pages to the same zero page as an optimization, and then when a write occurs they back it with a physical page.

Re: Linux Memory Management FAQ

#52
post #37

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

I think that's probably still true for what 32-bit systems are still out there today. And regardless, I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM. But I expect the FAQ was probably originally thinking about desktop or server systems, so, yeah, the intent there is probably out of date. Those types of systems are rarely 32-bit these days, and usually have a bit…

My OnePlus 3T is nearly 4 years old now and has 6GB (and is really showing its age...)

Re: Linux Memory Management FAQ

#53

The Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?

Highly relevant. The only part that I would discount is that he was pretty bullish on the prospects for hardware transactional memory, and his forward-looking statements about it didn't pan out. In fairness, much of the industry was bullish about HTM at that time.

In contrast, software transactional memory is still a pretty neat abstraction for some concurrency problems.

(And, of course, hardware transactional memory can be used to implement 'software' transactional memory faster than in software.)

However, STM only really works well in languages that are pure by default, like eg Haskell (or perhaps Erlang might be close enough). In a language with pervasive mutations and side effects, it's too annoying to use. Microsoft tried to make it work for .net for a while, and gave up.

Re: Linux Memory Management FAQ

#54
post #6

Earlier quoted context omitted.

I think there might be some more hardware-specific nuance here. e.g. /proc/cpuinfo says this on a couple of different x86_64 systems that I checked. address sizes : 36 bits physical, 48 bits virtual address sizes : 40 bits physical, 48 bits virtual PS: I don't understand what this means, btw.

It means that it can address 40-bits of address space worth of physical memory, but that virtual memory addresses can use 48 bits. Physical addresses are just your RAM bytes numbered 1 through whatever. Virtual address space is the address space of a process, which includes mapped physical memory, unmapped pages, guard pages, and other virtual memory tricks.

> Physical addresses are just your RAM bytes numbered 1 through whatever.

Not really. There are lots of holes in the physical address map. Look at /proc/iomem. Look at all of the gunk in there at addresses lower than the amount of RAM you have. Look at the highest “System RAM” address. It will be higher than the amount of actual physical RAM that you have.

Re: Linux Memory Management FAQ

#55

For anybody who's interested I also wrote up a whole bunch of notes on this at https://github.com/lorenzo-stoakes/linux-vm-notes and superceded by far more recent https://github.com/lorenzo-stoakes/linux-mm-notes I have made a few patches into the mm subsystem some simply inspired by researching for the articles.

If someone wants to begin their journey in understanding and level of aptitude you've displayed on your sites, where should they begin?

Thanks, I am reading the kernel source as I go and using that to answer questions as I figure things out.

Re: Linux Memory Management FAQ

#56
post #52
post #37

Earlier quoted context omitted.

I think that's probably still true for what 32-bit systems are still out there today. And regardless, I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM. But I expect the FAQ was probably originally thinking about desktop or server systems, so, yeah, the intent there is probably out of date. Those types of systems are rarely 32-bit these days, and usually have a bit…

My OnePlus 3T is nearly 4 years old now and has 6GB (and is really showing its age...)

What's the use case of having so much ram on a smartphone ? Gaming?

Re: Linux Memory Management FAQ

#57
post #56
post #52

Earlier quoted context omitted.

My OnePlus 3T is nearly 4 years old now and has 6GB (and is really showing its age...)

What's the use case of having so much ram on a smartphone ? Gaming?

App-switching (multitasking) without LRU apps getting force-closed to make room for active apps. In other words, if you like to keep apps open, more RAM will reduce the chances of an app opened a while ago having to "start fresh" when you switch back to it, losing whatever state it had when you last used it.

Re: Linux Memory Management FAQ

#58
post #40
post #18

Earlier quoted context omitted.

Oddly enough the unused bits are in the middle of the address. They're also sign-extended rather than filled with zeros, so sometimes they are ones and other times they are zeros.

In the middle of the address _space_, I should say.

If you interpret your addresses as signed values, then the entire address space can be contiguous.

I'm not suggesting that this is a good idea, but is certainly an idea.

Re: Linux Memory Management FAQ

#59

For anybody who's interested I also wrote up a whole bunch of notes on this at https://github.com/lorenzo-stoakes/linux-vm-notes and superceded by far more recent https://github.com/lorenzo-stoakes/linux-mm-notes I have made a few patches into the mm subsystem some simply inspired by researching for the articles.

Thank you, this is great!

Thank you! I wanted to reply last night but got rate limited. I've been questioning my side project recently (hence why no updates for a month) so hearing positive feedback from people does help motivate.

Re: Linux Memory Management FAQ

#60
post #39

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

>I guess this is not the most up-to-date document? it's also not correct. It doesn't have all 4GB "all to itself", because a portion of that (usually 1 or 2 GB) is mapped to the kernel.

"each process has 4 gigabytes of virtual address space all to itself"

A process does indeed have all 4GB of VIRTUAL adress space to itself. unless I'm misunderstanding you.

Post reply on HN