Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malloc Never Fails (2012)

scvalex.net

Post reply on HN
Print view View original

151–160 of 165 posts

Re: Malloc Never Fails (2012)

#151

Post by kj4ips »

joosters wrote: post #61
dooglius wrote: post #7

The title here is just blatantly false; there are certainly some scenarios in which it won't fail, but many in which it will. The author is aware of this since he fixes the claim toward the end: > To clarify, the surprising behaviour malloc has does not mean we should ignore its return value. We just need to be careful because malloc returning successfully does not always mean that we can use the requested memory. It…

n.b. turning over-commit off comes with its own set of problems, such as causing programs to fail long before all memory has really been exhausted. For example, fork() will have to ensure that there is enough memory for a complete copy of the running process. If you have a large process, e.g. using 4GB of a 8GB machine, then fork() won't be able to run, even if you just want to fork and run a tiny program. With over-…

Redis is an example of this. When redis saves in the background, it will for and use COW to shapshot a point in time, while the main redis thread keeps going with the normal redis. If you disable overcommit, redis can fail to fork, when there is actually sufficient memory to save the db.

Re: Malloc Never Fails (2012)

#152

Post by mehrdadn »

simias wrote: post #97
mehrdadn wrote: post #71

Earlier quoted context omitted.

> If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business. We're not, and we never were, debating the situation where dereferencing the non-null pointer returned by malloc succeeds but takes long time due to your Amazon order. We've been talking about the situation where it fails . malloc is not allowed…

That is my point, it doesn't fail. Either the kernel finds out a way to map the memory and it succeeds, or it kill the program and the instruction never runs. Code that doesn't run can't violate the standard. When the code is allowed to run all the invariants are guaranteed to be respected. If I write this code: char *b = malloc(2); if (b == NULL) { return 0; } b[0] = 'A'; b[1] = '\0'; printf("%s\n", b); free(b); The…

Okay this is a far more reasonable argument but I'm not convinced it's right. The standard does define normal and abnormal program termination. It also defines . SIGINT addresses the keyboard case (or the implementation can provide another signal). SIGABRT, SIGTERM, etc. are raised upon termination. For the keyboard case, then the program would be made aware, and it can indeed ignore the Ctrl+C request if it desires. That's perfectly normal and nothing absurd. For other types of termination, the other signals are raised. But in this case the program is terminated before any signal is raised at all. Now, as a practical matter I would argue the hosted environment should still be able to kill the program in the face of user request simply because nobody wants a host that's the slave of the program even if it's against the standard, but this is going far, far, far beyond that. It's happening at the request of neither the user nor the program; it's just happening because the host feels like it. Now that's both a violation of the standard and an uncool one at that!

P.S. I don't think indefinite hold is "functionally the same thing" as termination. A caller system(), for one, would need to return in one case, but not the other.

Re: Malloc Never Fails (2012)

#153

Post by [deleted] »

klodolph wrote: post #81
mehrdadn wrote: post #65

Earlier quoted context omitted.

The standard literally could not care less if there's a kernel underneath. It doesn't care how malloc is implemented or what the kernel, if any, looks like "from its perspective". The perspective that has relevance is that of the caller of malloc. From your own quotes: > The pointer returned if the allocation succeeds is suitably aligned so that [...] > If the space cannot be allocated, a null pointer is returned. If…

The space IS allocated, it's just that this space (virtual memory) is not backed by physical RAM.

[deleted]

Re: Malloc Never Fails (2012)

#154

Post by cesarb »

cheerlessbog wrote: post #98
dooglius wrote: post #94

Earlier quoted context omitted.

The OOM killer can strike at any time, regardless of the mapping options you've set, even if you aren't explicitly or implicitly allocating new memory. What I'd like, and what I think twic is asking for, is a way to "opt-out" of the overcommit paradigm at a process level, so that allocations by a given process reserve memory and may fail, but in return the kernel promises not to OOM kill it.

Most of my programming experience is in Windows. Can someone briefly explain why it doesn't seem to need an OOM killer? Is it happier to page? Or it always commits on allocation? Or something else.

AFAIK, Windows always commits on allocation, growing the swap file if necessary.

Re: Malloc Never Fails (2012)

#155

Post by cesarb »

Paul-ish wrote: post #108

When I use Python sometimes I run into a MemoryError when working with large datasets. How does the Python runtime know I am out of memory if the kernel won't tell it. Does it try a write and catch the signal?

If your large dataset leads to a single huge memory allocation, the kernel will fail the allocation even though overcommit is enabled. Early versions of libvpx had a similar issue, where it allocated a huge amount of memory at once (actually, told the dynamic linker to do so) which failed on low-memory machines (https://bugzilla.redhat.com/show_bug.cgi?id=600663 "libvpx.so.0 fails to load on low-memory machines due to ridiculously oversized .bss").

Re: Malloc Never Fails (2012)

#156

Post by cesarb »

mark-r wrote: post #129
Crinus wrote: post #114

Earlier quoted context omitted.

No, i did not implied that, please do not put words in my mouth. My implication was software that if it fails it will kill people. Otherwise if a program crashes because of a situation happens once every 100000000 runs, not only is worthless to worry about it, it actually is preferable to not do that as to keep the codebase clean and hence easier to maintain for bugs that actually do affect people.

Keeping the codebase clean is why I prefer that allocation errors throw an exception rather than returning NULL.

But then you have to worry about exception safety on every code which might call an allocating function, making the code more complex. This is worse on C++ where even innocent-looking code could be allocating memory in the middle of an operation (for instance, by calling a copy constructor which allocates).

Re: Malloc Never Fails (2012)

#157

Post by cesarb »

mehrdadn wrote: post #62
potyl wrote: post #39

Earlier quoted context omitted.

> [...] Linux blatantly violates the language specification [...] We can't expect a kernel to be aware of all language specifications. Yes, I know that in this case C is both the program's and the Kernel's language in this example but even then. Languages go through iterations (versions) and a Kernel can't be required to obey all languages which might run under it.

If you're suggesting Linux did not intend its malloc implement the corresponding function in the C standard then you're just lying to yourself. That's exactly why it's there and that's exactly what it's used for.

> If you're suggesting Linux did not intend its malloc implement

Linux (the kernel) doesn't have malloc. It's part of the C library, which is a completely separate project. What Linux implements is brk and mmap.

Re: Malloc Never Fails (2012)

#158

Post by mehrdadn »

cesarb wrote: post #157
mehrdadn wrote: post #62

Earlier quoted context omitted.

If you're suggesting Linux did not intend its malloc implement the corresponding function in the C standard then you're just lying to yourself. That's exactly why it's there and that's exactly what it's used for.

> If you're suggesting Linux did not intend its malloc implement Linux (the kernel) doesn't have malloc. It's part of the C library, which is a completely separate project. What Linux implements is brk and mmap.

Playing loose with the word Linux here. Play along a bit maybe?

Re: Malloc Never Fails (2012)

#159

Post by mehrdadn »

kazinator wrote: post #150
mehrdadn wrote: post #27

Translation: This is a euphemism for saying Linux blatantly violates the language specification with no remorse. (Yes, I realize Linux is the kernel, etc.)

> Linux blatantly violates the language specification with no remorse. We may be able to argue that it doesn't. ISO C says in 1. Scope , this: This International Standard does not specify [...] — the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor. It seems that these weasel words have an interpretation that ca…

No it wouldn't. They clearly included the ability to return null to allow for when the data exceeds the capacity of the system. There is no provision for returning a pointer to memory that cannot be used.

Re: Malloc Never Fails (2012)

#160

Post by kazinator »

mehrdadn wrote: post #159
kazinator wrote: post #150

Earlier quoted context omitted.

> Linux blatantly violates the language specification with no remorse. We may be able to argue that it doesn't. ISO C says in 1. Scope , this: This International Standard does not specify [...] — the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor. It seems that these weasel words have an interpretation that ca…

No it wouldn't. They clearly included the ability to return null to allow for when the data exceeds the capacity of the system. There is no provision for returning a pointer to memory that cannot be used.

You might think, but that's not how how conformance (of an ISO C implementation to the standard) works.

For an implementation to be deemed conforming, it just has to be demonstrated to successfully translate and execute one program that tests each of the minimum implementation limits.

See 5.2.4.1 in http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Only if no such program can be found can we then conclude that the implementation is nonconforming (and if the reason for not finding such a program is this overcommit issue, then we can blame that issue).

Your Linux system is indeed nonconforming if it is so low on memory that, for instance, no C program can allocate a 65535 byte object (that can be actually initialized, and used: a real object). Basically the memory situation has to be so severe that it takes the implementation below the minimum limits, whereby we can clearly demonstrate nonconformance.

Post reply on HN