Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linux Memory Management FAQ

landley.net

Post reply on HN
Print view View original

31–40 of 75 posts

Re: Linux Memory Management FAQ

#31

Post by wahern »

barnacled wrote: post #24
dataflow wrote: post #21

Earlier quoted context omitted.

Well, it does mean that in C. But some folks prefer to play by their own rules.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

> Also differs if you turn overcommit off but since so much in linux assumes it your system will soon break if you try it.

I agree, reliance on overcommit has resulted in stability problems in Linux. But IME stability problems aren't induced by disabling overcommit, they're induced by disabling swap. The stability problems occur precisely because by relying on magical heuristics to save the day, we end up with an overall MM architecture that reacts extremely poorly under memory pressure. Whether or not overcommit is enabled, physical memory is a limited resource, and when Linux can't relieve physical memory pressure by dumping pages to disk, bad things happen, especially when under heavy I/O load (e.g. the buffer cache can grab pages faster than the OOM killer can free them).

Re: Linux Memory Management FAQ

#32

Post by quotemstr »

barnacled wrote: post #24
dataflow wrote: post #21

Earlier quoted context omitted.

Well, it does mean that in C. But some folks prefer to play by their own rules.

Actually no, the malloc doesn't allocate any memory it just updates the process's VMA to say that the allocated virtual range is valid. The pages are then faulted in on write. This is where things like OOM killer become very confusing for people. In linux (in sane configurations) allocations are just preorders. EDIT: I can't reply below due to rate limiting: I'd argue that overcommit just makes the difference between…

And that's why strict allocation tracking (no overcommit) should be the default. But those of us in favor of guaranteed forward progress and sensible resource accounting lost this fight a long time ago.

Re: Linux Memory Management FAQ

#33

Post by einpoklum »

Agingcoder wrote: post #19
phtrivier wrote: post #7

The Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?

It's still relevant. Other than that, I also think that even when outdated, computing history is worth reading anyway, since it gives you a natural understanding of _why_ we do what we do these days. In your day job, it also gives you a different appreciation for what people did and why they did it, and why 'this horrible code' may have made sense at the time. Furthermore, performance engineering is fundamentally abo…

> computing history is worth reading anyway

Sometimes, but a description of the state of the art in the past does not become a historical tract with the passage of time. The better ones do; others just become outdated.

Re: Linux Memory Management FAQ

#34

Post by brandmeyer »

phtrivier wrote: post #7

The Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?

Highly relevant. The only part that I would discount is that he was pretty bullish on the prospects for hardware transactional memory, and his forward-looking statements about it didn't pan out. In fairness, much of the industry was bullish about HTM at that time.

Re: Linux Memory Management FAQ

#35

Post by danielheath »

everybodyknows wrote: post #28
xxpor wrote: post #23

Earlier quoted context omitted.

This is true for x86-64, not true for other architectures such as arm64. Apple uses the high bits to cryptographicly sign the pointer value.

Fascinating. Does this confer some of the benefits of ECC RAM, for pointer data only — without the hardware cost?

Some (but I believe the advantage is that it’s much harder to inject valid code from a buffer overflow).

Re: Linux Memory Management FAQ

#36

Post by my123 »

everybodyknows wrote: post #28
xxpor wrote: post #23

Earlier quoted context omitted.

This is true for x86-64, not true for other architectures such as arm64. Apple uses the high bits to cryptographicly sign the pointer value.

Fascinating. Does this confer some of the benefits of ECC RAM, for pointer data only — without the hardware cost?

It's for a different purpose. (as in mitigate to some extent security bugs) And isn't an Apple feature only but an Arm one. (that is only rolling out on Cortex with Cortex-A78C and A78AE)

A paper on it from Qualcomm: https://www.qualcomm.com/media/documents/files/whitepaper-po...

And there's also MTE which is upcoming.

Re: Linux Memory Management FAQ

#37

Post by kelnos »

dbattaglia wrote: post #4

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

I think that's probably still true for what 32-bit systems are still out there today.

And regardless, I think the majority of systems running Linux today are phones, which usually have 4GB or less of RAM.

But I expect the FAQ was probably originally thinking about desktop or server systems, so, yeah, the intent there is probably out of date. Those types of systems are rarely 32-bit these days, and usually have a bit more than 4GB of RAM.

Re: Linux Memory Management FAQ

#38

Post by mhh__ »

phtrivier wrote: post #7

The Drepper series of article dates from 2007. Is it still relevant or has anything fundamental changed in memory handling in the last 13 years ?

The prefetching aspects are basically bunk as hardware prefetching is much better now. The rest is still gold.

Re: Linux Memory Management FAQ

#39

Post by gruez »

dbattaglia wrote: post #4

"Virtual addresses are the size of a CPU register. On 32 bit systems each process has 4 gigabytes of virtual address space all to itself, which is often more memory than the system actually has." I guess this is not the most up-to-date document?

>I guess this is not the most up-to-date document?

it's also not correct. It doesn't have all 4GB "all to itself", because a portion of that (usually 1 or 2 GB) is mapped to the kernel.

Re: Linux Memory Management FAQ

#40

Post by db48x »

db48x wrote: post #18
sigjuice wrote: post #14

Earlier quoted context omitted.

So are the 16 leftmost bits of a virtual address always 0?

Oddly enough the unused bits are in the middle of the address. They're also sign-extended rather than filled with zeros, so sometimes they are ones and other times they are zeros.

In the middle of the address _space_, I should say.
Post reply on HN