Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malloc Never Fails (2012)

scvalex.net

Post reply on HN
Print view View original

71–80 of 165 posts

Re: Malloc Never Fails (2012)

#71

Post by mehrdadn »

simias wrote: post #70
mehrdadn wrote: post #63

Earlier quoted context omitted.

> If you have some specific part of the C standard in mind please do tell See here: https://news.ycombinator.com/item?id=20145604 Also note the POSIX standard: Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null…

Thank you for your reply but I still don't buy it. As far as the program is concerned it is returned a memory block, what this "memory" is effectively behind the scenes is none of the standard's business. As long as the implementation manages to maintain the illusion it's perfectly fine AFAIK. The problem is when this breaks down and the kernel realizes that it can no longer maintain the masquerade. If at this point…

> If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business.

We're not, and we never were, debating the situation where dereferencing the non-null pointer returned by malloc succeeds but takes long time due to your Amazon order. We've been talking about the situation where it fails. malloc is not allowed to return a non-null pointer to a memory block that cannot be written to. Linux does it anyway, and in doing so blatantly violates the standard.

Re: Malloc Never Fails (2012)

#72

Post by saagarjha »

gmueckl wrote: post #53
saagarjha wrote: post #12

Earlier quoted context omitted.

I had always thought they'd be running a 64-bit OS, given that the new hardware is 64-bit. TIL that Raspbian is still 32-bit!

Unless you need the address space, there is little gain with a 64 bit OS. The wider pointers use more RAM and eat up more of the available memory bandwidth and caches. So not swotching to 64 bits is likely the best use if the hardware.

It’s a toss up, I think: actual results depend on the task at hand. 64-bit has the benefit of more, wider registers, which can speed up certain tasks.

Re: Malloc Never Fails (2012)

#73

Post by mmillin »

pornel wrote: post #68

This assumption lead to Rust's standard library not having a way to catch allocation failures (which is only now being rectified, and only partially). It's very Linux-centric and presumes a certain config+usage pattern. Not true on Windows. Not quite true on macOS. Not true in WASM. Definitely not true on embedded platforms.

Is there somewhere to read about how rust is tackling this problem?

Re: Malloc Never Fails (2012)

#74

Post by mannykannot »

twic wrote: post #57
dooglius wrote: post #7

The title here is just blatantly false; there are certainly some scenarios in which it won't fail, but many in which it will. The author is aware of this since he fixes the claim toward the end: > To clarify, the surprising behaviour malloc has does not mean we should ignore its return value. We just need to be careful because malloc returning successfully does not always mean that we can use the requested memory. It…

It's a shame that turning off overcommit is done at the system level, rather than the malloc call level. Some applications, and perhaps some allocations within otherwise naive applications, might prefer to have an allocation fail early, where failure could be sensibly handled.

I think that would ultimately have undesirable consequences.

If this were an option on a malloc call, any such call would have to reserve the allocated memory at that point, reducing the usefulness of overcommit for other processes. This would set up a 'tragedy of the commons' scenario, where every application developer defensively uses this feature because other applications are using it.

Re: Malloc Never Fails (2012)

#75

Post by TheCondor »

Crinus wrote: post #36
chrisseaton wrote: post #15

Earlier quoted context omitted.

> Not that you can do much when malloc fails... Tell the user they can’t do that operation? Use on-disk storage instead? Re-use memory you already have (such as evicting a cache and taking the memory it already had allocated)? Abandon what you were doing if it was only an optimisation and wasn’t essential (such as allocating memory as part of a speculative execution)? Run a garbage collector and try again? Lots of op…

In practice i haven't seen any application to gracefully handle out of memory situations. Even back in Windows 3.x days when running out of memory was common, most applications simply hanged or crashed (which also took the entire GUI with them) and those that didn't often had a "no memory, kthxbye" dialog that shut down the application. A very few rare cases would try to display some "sorry, no memory, close some pro…

What is your definition of graceful in this case? Presumably there was a “business need” for the allocation and the software isn’t just allocating for fun. An allocation failure is a hard fault, it’s not possible to honor the business need so something expected to happen cannot happen. The program could segfault when the null pointer returned is written to, that is kind of crappy. It could report the error “hey we are out of memory and couldn’t do xyz” or better “hey we are out of memory and couldn’t do xyz, this is probably because of abc, maybe you can adjust the tuning.” Or it could be a completely temporal thing, maybe you flush some caches or something to free up some memory and reschedule the work.

I don’t know, maybe this is old school, but yes you should put a check on every toolbar allocation for two reason: one you can gracefully report why your app isn’t doing it’s job and two it is a very very slippery slope when you start ignoring those errors as a matter of practice

Re: Malloc Never Fails (2012)

#76

Post by JasuM »

gmueckl wrote: post #43
JasuM wrote: post #41

What happens if you really run out of physical memory (including swap) after overcommitting? Does your process get a signal, or will OOM killer just run without notifying the process that triggered the condition?

Ig you try to allocate more than the system is willing to overcommit (e.g. a huge block all at once), malloc will fail. But if phyaical memory gets exhausted by accessing previously allocated pages, the OOM killer will evebtuslly come around and kill processes without signalling. Signal handlers could still make thenprocess (unknowingly) request more memory, so there is 0 guarantee that a handler could even run succe…

Oh yeah, I didn't think of that. I wonder if you could write a signal handler carefully to not allocate any memory, stack or otherwise, or is some return address or an internal structure being allocated transparently...

Re: Malloc Never Fails (2012)

#77

Post by dooglius »

mannykannot wrote: post #74
twic wrote: post #57

Earlier quoted context omitted.

It's a shame that turning off overcommit is done at the system level, rather than the malloc call level. Some applications, and perhaps some allocations within otherwise naive applications, might prefer to have an allocation fail early, where failure could be sensibly handled.

I think that would ultimately have undesirable consequences. If this were an option on a malloc call, any such call would have to reserve the allocated memory at that point, reducing the usefulness of overcommit for other processes. This would set up a 'tragedy of the commons' scenario, where every application developer defensively uses this feature because other applications are using it.

It's already quite easy to implement manually though, just do a write after allocation.

Re: Malloc Never Fails (2012)

#78

Post by brynet »

The malloc(3) family will absolutely fail on OpenBSD, for many reasons: login.conf/ulimits, calloc(3) for when integer overflow is detected (nmemb * size), same for OpenBSD's reallocarray extension. And yes, because the system was unable to satisfy the requested allocation.

Linux overcommit, and developer mindset is a detriment to software quality and portability.

Re: Malloc Never Fails (2012)

#79

Post by klodolph »

mehrdadn wrote: post #63
simias wrote: post #50

Earlier quoted context omitted.

Linux is a kernel, not a C runtime so that's not really relevant. You say you realize that but then what are you arguing for exactly? Arguably you could complain that the glibc (or whatever libc you're using) is not working around that by, for instance, scrubbing the pages in malloc() to force the kernel to allocate memory. But even then I'm not convinced that anything here is non-standard, at worse maybe we're in a…

> If you have some specific part of the C standard in mind please do tell See here: https://news.ycombinator.com/item?id=20145604 Also note the POSIX standard: Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null…

Allocated "space", i.e., virtual memory space. Malloc allocates virtual memory space, not physical RAM.

Re: Malloc Never Fails (2012)

#80

Post by jvanderbot »

What bothers me is that I read this train wreck without any red flags until I saw his correction at the end. Even the headline was wrong given the article, which itself was wrong.

I really should have coffee before HN

Post reply on HN