Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Memory Management FAQ

landley.net

Post reply on HN
Print view View original

61–70 of 75 posts

Re: Linux Memory Management FAQ

#61

Post by dezgeg »

Out_of_Characte wrote: post #60
gruez wrote: post #39

Earlier quoted context omitted.

>I guess this is not the most up-to-date document? it's also not correct. It doesn't have all 4GB "all to itself", because a portion of that (usually 1 or 2 GB) is mapped to the kernel.

"each process has 4 gigabytes of virtual address space all to itself" A process does indeed have all 4GB of VIRTUAL adress space to itself. unless I'm misunderstanding you.

Yes, VIRTUAL memory. Most operating systems (Windows, Linux) leave 2 or 3 GB for the user process and reserve the of the address space for themselves.

That way userspace-to-kernel switch does not require changing active page table (and also avoids switcharoo each time kernel needs to access userspace memory).

Re: Linux Memory Management FAQ

#62

Post by [deleted] »

dbattaglia wrote: post #4

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

[deleted]

Re: Linux Memory Management FAQ

#63

Post by dfox »

jcul wrote: post #22
sigjuice wrote: post #14

Earlier quoted context omitted.

So are the 16 leftmost bits of a virtual address always 0?

Yes generally for userspace addresses they are 0. But more importantly they can be used for other stuff, commonly referred to as pointer tagging / smuggling etc. It's a useful optimisation technique where you can add some extra metadata without having to dereference a pointer.

The reason why amd64 checks whether the addresses are “canonical” is discourage exactly this trick. On almost all platforms that simply ignored upper byte of pointer (m68k, s390, IIRC even early ARMs) this lead to significant compatibility issues.

As for storing tags in pointers on 64b platforms it is probably better to use the 3 low order bits. Another useful trick is what was used in PDP-10 MacLisp and is used by BDW GC: encode the type information in virtual memory layout itself.

Re: Linux Memory Management FAQ

#64

Post by db48x »

moonchild wrote: post #58
db48x wrote: post #40

Earlier quoted context omitted.

In the middle of the address _space_, I should say.

If you interpret your addresses as signed values, then the entire address space can be contiguous. I'm not suggesting that this is a good idea, but is certainly an idea.

Negative addresses? Anathema!

Re: Linux Memory Management FAQ

#65

Post by jcul »

dfox wrote: post #63
jcul wrote: post #22

Earlier quoted context omitted.

Yes generally for userspace addresses they are 0. But more importantly they can be used for other stuff, commonly referred to as pointer tagging / smuggling etc. It's a useful optimisation technique where you can add some extra metadata without having to dereference a pointer.

The reason why amd64 checks whether the addresses are “canonical” is discourage exactly this trick. On almost all platforms that simply ignored upper byte of pointer (m68k, s390, IIRC even early ARMs) this lead to significant compatibility issues. As for storing tags in pointers on 64b platforms it is probably better to use the 3 low order bits. Another useful trick is what was used in PDP-10 MacLisp and is used by B…

I guess it checks it when you actually try to dereference the pointer? On Intel too you still have to "repair"the pointer before you use it. It's definitely not the safest optimisation but it can be used to great effect when needed.

I think Intel is adding CPU support for pointer tagging operations in the future which should make them a lot easier / safer / more efficient to work with, though I can't find a reference now, it doesn't refer to it as pointer tagging.

Any more information on encoding the type information in virtual memory layout? Sounds cool.

I guess you have different types allocated in specific regions?

Re: Linux Memory Management FAQ

#66

Post by Agingcoder »

einpoklum wrote: post #33
Agingcoder wrote: post #19

Earlier quoted context omitted.

It's still relevant. Other than that, I also think that even when outdated, computing history is worth reading anyway, since it gives you a natural understanding of _why_ we do what we do these days. In your day job, it also gives you a different appreciation for what people did and why they did it, and why 'this horrible code' may have made sense at the time. Furthermore, performance engineering is fundamentally abo…

> computing history is worth reading anyway Sometimes, but a description of the state of the art in the past does not become a historical tract with the passage of time. The better ones do; others just become outdated.

Well, the ones which fail (and which become outdated) can also teach us valuable lessons : looking at the current state of the art doesn't necessarily tells you what happens if you do things differently.

In other words, we tend to focus on positive results, but negative ones ('don't do this or.. !') can be equally interesting and useful.

Re: Linux Memory Management FAQ

#67

Post by ben_bai »

ymbeld wrote: post #49
jdsully wrote: post #26

Earlier quoted context omitted.

In the C standard malloc should return null if it can’t fulfill the request. Linux violates this but it usually works out in the end since virtual memory makes true OOM very rare.

I don’t know what true OOM means, but my desktop has crashed I think at least three times in the last four months and the console said “OOM killer”. About 15GB of usable RAM, 2GB swap drive. I just have to have the usual applications open plus another browser in addition to Firefox, namely Chrome. (But naturally I don’t try to actively reproduce the behavior since I usually have better things to do than wait 10 minut…

Your problem is the 2GB of swap. Get rid of it and it will just crash without 10min of slowdown (while swap disk is getting written to).

Linux overcommitting memory and especially chrome/firefox beeing big-fat-memory-hogs are the problem. In fact every application which doesn't cope malloc beeing out of memory or assuming everybody has multiple gigs of memory to spare should "reevaluate".

Re: Linux Memory Management FAQ

#68

Post by rrss »

dataflow wrote: post #25
barnacled wrote: post #24

Earlier quoted context omitted.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

I said "in C". You're talking "in Linux" (or glibc/whatever). Which, as I already said, plays by its own rules and defies C. It's broken by design.

I don't think the c standard specifies this behavior. malloc must return either a pointer where you can store an object, or null. I think platform details about when accesses to that pointer might fail are outside the scope of the language / stdlib standard.

Are failures when accessing the allocated pointer due to overcommit substantially different than failures due to ECC errors or other hardware failure, with regard to what is specified in the c standard?

(FWIW I don't particularly like overcommit-by-default either)

Re: Linux Memory Management FAQ

#69

Post by ta988 »

forty wrote: post #56
mhh__ wrote: post #52

Earlier quoted context omitted.

My OnePlus 3T is nearly 4 years old now and has 6GB (and is really showing its age...)

What's the use case of having so much ram on a smartphone ? Gaming?

Allowing app developers to not worry about optimization, put more trackers and more annoying ads...

Re: Linux Memory Management FAQ

#70

Post by dfox »

jcul wrote: post #65
dfox wrote: post #63

Earlier quoted context omitted.

The reason why amd64 checks whether the addresses are “canonical” is discourage exactly this trick. On almost all platforms that simply ignored upper byte of pointer (m68k, s390, IIRC even early ARMs) this lead to significant compatibility issues. As for storing tags in pointers on 64b platforms it is probably better to use the 3 low order bits. Another useful trick is what was used in PDP-10 MacLisp and is used by B…

I guess it checks it when you actually try to dereference the pointer? On Intel too you still have to "repair"the pointer before you use it. It's definitely not the safest optimisation but it can be used to great effect when needed. I think Intel is adding CPU support for pointer tagging operations in the future which should make them a lot easier / safer / more efficient to work with, though I can't find a reference…

Most general purpose ISAs (eg. SPARC and IIRC RiscV has something similar) with some kind of intrinsic support for tagged pointers also prefer the tags in low order bits.

And you are right that the tag inside address trick involves allocating objects of same type in different continuous regions. Usually such that whole page contains object of same type (as far as the tagging scheme is concerned) and by either masking off lower ten-ish bits of pointer you get to type header or you have some global out-of-line map of page frame->type.

Post reply on HN