Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malloc Never Fails (2012)

scvalex.net

Post reply on HN
Print view View original

131–140 of 165 posts

Re: Malloc Never Fails (2012)

#131

Post by mark-r »

Crinus wrote: post #93
TheCondor wrote: post #75

Earlier quoted context omitted.

What is your definition of graceful in this case? Presumably there was a “business need” for the allocation and the software isn’t just allocating for fun. An allocation failure is a hard fault, it’s not possible to honor the business need so something expected to happen cannot happen. The program could segfault when the null pointer returned is written to, that is kind of crappy. It could report the error “hey we ar…

> It could report the error “hey we are out of memory and couldn’t do xyz” or better “hey we are out of memory and couldn’t do xyz, this is probably because of abc, maybe you can adjust the tuning.” What if you cannot do that report because the reporting itself needs memory that can fail? > but yes you should put a check on every toolbar allocation for two reason: one you can gracefully report why your app isn’t doin…

> What if you cannot do that report because the reporting itself needs memory that can fail?

Allocate a big block of memory at startup. Free it just before doing the reporting.

Re: Malloc Never Fails (2012)

#132

Post by systemBuilder »

Seems like a needless feature, overcommit, "oh let's make our app faster by pre allocating 2GB of RAM!" says the naive programmers, "Oh let's make our operating system more powerful by allowing overcommit.". Overall, you moved the allocation delay from the app to the OS, which has much less end to end knowledge of how to do it effectively, overcomplicated both layers, achieved practically nothing overall, like a dog chasing it's tail, except now both layers have more garbage code, so it's like an obese dog chasing it's tail!

Re: Malloc Never Fails (2012)

#133

Post by bscphil »

dooglius wrote: post #94
burntsushi wrote: post #90

Earlier quoted context omitted.

There is some support for this actually. Namely, when overcommit_memory is set to its default value (0), then it actually implements a heuristic for overcommit. (Where as the value `1` corresponds to "always overcommit.") Namely, only when overcommit_memory is set to 0, if you allocate memory with mmap and pass the MAP_NORESERVE option, then allocation behaves as if overcommit is always enabled and there is no check.…

The OOM killer can strike at any time, regardless of the mapping options you've set, even if you aren't explicitly or implicitly allocating new memory. What I'd like, and what I think twic is asking for, is a way to "opt-out" of the overcommit paradigm at a process level, so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it.

>so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it

The issue with this is that there are other processes on the system. If I start a program that used your idea for allocation, and it used 90% of my memory, a small growth in my memory usage might mean I run out. But because it's guaranteed not to OOM, another program gets killed rather than the one using 90% of the memory. If you want no OOM kills, you've got to disable overcommit globally.

Re: Malloc Never Fails (2012)

#134

Post by JasuM »

gmueckl wrote: post #96
JasuM wrote: post #76

Earlier quoted context omitted.

Oh yeah, I didn't think of that. I wonder if you could write a signal handler carefully to not allocate any memory, stack or otherwise, or is some return address or an internal structure being allocated transparently...

Just trying to allocate stack space for the signal handler may cause the stack to spill into a new page. That is absolutely out of anybody's control. And if that new page cannot be provided, it's game over.

Makes sense. I was thinking that maybe signal handlers could use the regular stack of the thread, but that would of course make everything fall down if the "real" code would write to the stack before updating the stack pointer.

Re: Malloc Never Fails (2012)

#135

Post by monocasa »

antirez wrote: post #123

Interesting topic with many things to say about, but wrong content. The point is that in most C programs, it is not worth to handle OOM errors, because what you can do during OOM is of very little value, on the other hand handling OOM correctly is very hard . However you can't do this in libraries, because you don't know how the library is going to be used. So for instance in order to make my Radix tree library resis…

When writing C libraries, it's good form to be able to init your instance with allocation and logging function pointers. That lets you play nice with most any env you're being pulled into, and gives your consumers an obvious place for nice hooks for debugging.

Re: Malloc Never Fails (2012)

#136

Post by lalaithion »

amelius wrote: post #124
Paul-ish wrote: post #108

When I use Python sometimes I run into a MemoryError when working with large datasets. How does the Python runtime know I am out of memory if the kernel won't tell it. Does it try a write and catch the signal?

Perhaps any write can fail, and if one does then it triggers some code which frees some pre-reserved space which allows the interpreter to continue, and properly unwind the stack.

I believe that the python interpreter creates a MemoryError at startup, and throws that when you run out of memory.

Re: Malloc Never Fails (2012)

#137

Post by monocasa »

lifthrasiir wrote: post #23
metrxqin wrote: post #20

Earlier quoted context omitted.

That's incorrect, 2^48 bits = 262144GB actually.

But x86-64 divides the 48-bit address space into two halves, only one of which is available in the user mode.

To be fair, there's nothing that says the higher half _has_ to be entirely kernel mode. Only that bits 63 through 48 have to be the same value.

And Intel has a spec out for a PML5 page table, giving you 57 total virtual address bits.

Re: Malloc Never Fails (2012)

#138

Post by antirez »

monocasa wrote: post #135
antirez wrote: post #123

Interesting topic with many things to say about, but wrong content. The point is that in most C programs, it is not worth to handle OOM errors, because what you can do during OOM is of very little value, on the other hand handling OOM correctly is very hard . However you can't do this in libraries, because you don't know how the library is going to be used. So for instance in order to make my Radix tree library resis…

When writing C libraries, it's good form to be able to init your instance with allocation and logging function pointers. That lets you play nice with most any env you're being pulled into, and gives your consumers an obvious place for nice hooks for debugging.

Yep, that's what I do in order to be able to fuzz test with an OOM returning malloc() in my lib. Agreed on the fact it's good form.

Re: Malloc Never Fails (2012)

#139

Post by dragontamer »

shereadsthenews wrote: post #125
dragontamer wrote: post #83

Earlier quoted context omitted.

What exactly is "wrong" about the article? Linux's Overcommit behavior is non-obvious to many programmers. Its one of those issues that very few programmers I've come across in the workplace understand properly. This blogpost properly understands the issues associated with Overcommit, and have done some preliminary investigations that describe the behavior. Its a really good blogpost. The general point of the blogpos…

What's wrong with the article is that malloc is not even a feature of Linux. There's mmap and brk, both of which have documented failure modes.

Most typical programmers will be using malloc, or some mechanism built on top of malloc (C++ new). MMap and brk are useful for certain situations (explicitly getting Huge Pages), but I don't think that the typical programmer necessarily needs to know about those.

Since glibc malloc is built on top of mmap and brk, I think your distinction is mostly academic. For any programmer using Linux and glibc... malloc's failure mode IS mmap and brk failure modes.

Re: Malloc Never Fails (2012)

#140

Post by zonidjan »

shereadsthenews wrote: post #125
dragontamer wrote: post #83

Earlier quoted context omitted.

What exactly is "wrong" about the article? Linux's Overcommit behavior is non-obvious to many programmers. Its one of those issues that very few programmers I've come across in the workplace understand properly. This blogpost properly understands the issues associated with Overcommit, and have done some preliminary investigations that describe the behavior. Its a really good blogpost. The general point of the blogpos…

What's wrong with the article is that malloc is not even a feature of Linux. There's mmap and brk, both of which have documented failure modes.

It is a feature of your chosen libc, which has to then interact with those system calls, in a way that gives malloc a peculiar behavior on Linux specifically as opposed to other OS.
Post reply on HN