Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Story Of realloc (And Laziness)

blog.httrack.com

Post reply on HN
Print view View original

51–60 of 65 posts

Re: A Story Of realloc (And Laziness)

#51

Post by asveikau »

fprawn wrote: post #50
asveikau wrote: post #42

Earlier quoted context omitted.

> In the general case, however, if allocating 100 bytes fails, reporting that error is also likely to fail. An actual memory allocation failure on a modern computer running a modern OS is a very rare and very bad situation. It's rarely recoverable. I call BS on this. First of all, it's not the 100 byte allocation that is likely to fail; chances are it's going to be bigger than 100 bytes and the 100 byte allocation wi…

If you have time, can you expound on this? Is there, perhaps, an open source project that handles NULL returns from malloc in this way you could point me to?

My first instinct is to say look at something kernel-related. If an allocation fails, taking down the entire system is usually not an option (or not a good one anyway). Searching http://lxr.linux.no/ for "kmalloc" you see a lot of callers handling failure.

Re: A Story Of realloc (And Laziness)

#52

Post by nknighthb »

fprawn wrote: post #32
fprawn wrote: post #10

Earlier quoted context omitted.

Memory allocation failures are virtually non-existent in modern desktop computers. Good practice is to not test return values from malloc, new, etc. Memory can be allocated beyond RAM size, so by the time a failure occurs your program really should crash and return its resources. Embedded systems have fewer resources and some will not have virtual memory and so the situation will be different. But unless you know bet…

I'll clarify this. I'm not saying you shouldn't ever check return values, that's obviously not the right thing to do. And of course there are exceptions to the general rule. If you're allocating a large chunk of memory and there's a reasonable expectation that it could fail, that should be reported, of course. In the general case, however, if allocating 100 bytes fails, reporting that error is also likely to fail. An…

I'm not sure what you imagine when you say "modern computer running a modern OS". Does this not include anything but desktop PCs and laptops? Because phones and tablets have some rather nasty memory limits for applications to deal with, which developers run into frequently.

The space I work in deals with phones and tablets, as well as other embedded systems (TVs, set-top boxes, etc.) that tend to run things people think of as "modern" (recentish Linux kernels, userlands based on Android or centered around WebKit), while having serious limits on memory and storage. My desktop calendar application uses more memory than we have available on some of these systems.

In these environments, it is essential to either avoid any possibility of memory exhaustion, or have ways to gracefully deal with the inevitable. This is often quite easy in theory -- several megabytes of memory might be used by a cached data structure that can easily be re-loaded or re-downloaded at the cost of a short wait when the user backs out of whatever screen they're in.

But one of the consequences of this cavalier attitude to memory allocation is that even in these constrained systems, platform owners have mandated apps sit atop an ever-growing stack of shit that makes it all but impossible for developers to effectively understand and manage memory usage.

Re: A Story Of realloc (And Laziness)

#54

Post by yaur »

htns wrote: post #33
yaur wrote: post #27

Earlier quoted context omitted.

Ugh. I would much rather know that the process died because of allocation failure than try to figure out why some code is trying to write to a random null pointer as these are two very different types of bugs.

I'm having a hard time picturing a situation where it would be tough to figure out. Typically you allocate memory to use it right after. errno will be set too. Of course, there is no reason to not do all your allocation through a wrapper function which does check and abort on failure. I think the point was that surviving malloc failures is a dubious approach - instead go all in, or if it's a long-running service, pro…

In the case of my day job (CCTV application 80% C#, 20% C and C++) writing to a bad pointer will get reported as an AccessViolationException with no hope of getting a dump or a stack trace of the native code. An allocation failure will get translated into an OutOfMemoryException and typically includes stats of what is consuming RAM.

Re: A Story Of realloc (And Laziness)

#55

Post by mgraczyk »

optimiz3 wrote: post #12

Code in the article for realloc is dangerous and wrong: void *realloc(void *ptr, size_t size) { void *nptr = malloc(size); if (nptr == NULL) { free(ptr); return NULL; } memcpy(nptr, ptr, size); // KABOOM free(ptr); return nptr; } Line marked KABOOM copies $DEST_BYTE_COUNT, rather than $SOURCE_BYTE_COUNT. Say you want to realloc a 1 byte buffer to a 4 byte buffer - you just copied 4 bytes from a 1 byte buffer which me…

Not to mention that this realloc incorrectly frees the input ptr on failure. The standard says

    "If the space cannot be allocated, the object [*ptr] shall remain unchanged."

Re: A Story Of realloc (And Laziness)

#56

Post by __david__ »

crackerz wrote: post #21

And this is why OpenSource is awesome.

Agreed. Not sure why people are downvoting you. It blows my mind every time I think, "I wonder how works?" and I'm able to just "apt-get source some_program" and check it out.

Working with Linux and having the source (and the ability to change it) for entire stack all the way down to and including the kernel is liberating. As a programmer it feels like the entire world is open to me.

I guess that's GNU's dream brought to life, really.

Re: A Story Of realloc (And Laziness)

#57

Post by ams6110 »

twic wrote: post #14
rfrey wrote: post #6

Earlier quoted context omitted.

Serious question from a guy made soft by garbage collection: how frequent is memory allocation failure nowadays, with large memories and virtual memory? Were I to guess from my state of ignorance I'd think that if allocs began to fail, there was no recovery anyhow... so leaking in this case would be one leak right before a forced quit. Wrong? Are there lots of ways allocation can fail besides low memory conditions?

On Linux, somewhat infamously, malloc never fails. It will always return a pointer to some fresh part of the address space. It is able to do this because, in turn, sbrk/anonymous mmap never fails - it always allocates some fresh address space. It is able to do this because Linux does not allocate physical memory (or swap) when it assigns address space, but when that address space is actually used. It will happily all…

oom killer always seems to target sshd in my encounters. Handy when you're at home.

Re: A Story Of realloc (And Laziness)

#58

Post by binarycrusader »

twic wrote: post #14
rfrey wrote: post #6

Earlier quoted context omitted.

Serious question from a guy made soft by garbage collection: how frequent is memory allocation failure nowadays, with large memories and virtual memory? Were I to guess from my state of ignorance I'd think that if allocs began to fail, there was no recovery anyhow... so leaking in this case would be one leak right before a forced quit. Wrong? Are there lots of ways allocation can fail besides low memory conditions?

On Linux, somewhat infamously, malloc never fails. It will always return a pointer to some fresh part of the address space. It is able to do this because, in turn, sbrk/anonymous mmap never fails - it always allocates some fresh address space. It is able to do this because Linux does not allocate physical memory (or swap) when it assigns address space, but when that address space is actually used. It will happily all…

  I don't know about what other operating systems do.
  Apparently all modern unixes overcommit address space in
  much the same way as Linux. However, i can't believe that
  FreeBSD handles this as crassly as Linux does.
Solaris does not (generally, unless you use MAP_NORESERVE w/ mmap).

In general, the Linux kernel's default OOM behaviour is undesirable for the vast majority of enterprise use cases. That's why RedHat and many other vendors used to disable it by default (unknown who still does).

Why is it bad? Simple; imagine your giant database is running with a large address space mapped. Another program decides to allocate a large amount of memory. The Linux kernel sees the tasty database target, kills it, and gives the smaller program its memory. Congratulations, your database just went poof.

There's an article that discusses the advantages/disadvantages with respect to Solaris here:

http://www.oracle.com/technetwork/server-storage/solaris10/s...

The article (while old) still applies today.

Re: A Story Of realloc (And Laziness)

#59

Post by KayEss »

BugBrother wrote: post #22
plorkyeran wrote: post #13

Earlier quoted context omitted.

It always calls (the equivalent of) malloc + memcpy + free for each grow, so it can be anywhere from the exact same speed (when realloc does the same thing internally) to absurdly slower (in the given case of a large array that has to be paged in). The first case is by far the most common case, but it is something that sometimes matters.

Huh, is it really implemented like that?! Why use realloc when growing an array you have an interface to? Just add another allocated buffer to the previous allocated areas. When there are too many small areas, consolidate with realloc/free. Much faster (yes yes, almost always). (Disclaimer: Last time I used C++ I had hair. :-) ) Edit: OK, thanks plorkyeran.

What you're talking about is some sort of rope data structure which is pretty simple to implement on top of std::vector and std::list. The std::vector guarantees contiguous memory locations so you can't do it for that particular container.

Re: A Story Of realloc (And Laziness)

#60

Post by aktau »

fprawn wrote: post #10
rfrey wrote: post #6

Earlier quoted context omitted.

Serious question from a guy made soft by garbage collection: how frequent is memory allocation failure nowadays, with large memories and virtual memory? Were I to guess from my state of ignorance I'd think that if allocs began to fail, there was no recovery anyhow... so leaking in this case would be one leak right before a forced quit. Wrong? Are there lots of ways allocation can fail besides low memory conditions?

Memory allocation failures are virtually non-existent in modern desktop computers. Good practice is to not test return values from malloc, new, etc. Memory can be allocated beyond RAM size, so by the time a failure occurs your program really should crash and return its resources. Embedded systems have fewer resources and some will not have virtual memory and so the situation will be different. But unless you know bet…

See xmalloc [1] and friends, courtesy of the Git project.

[1]: https://github.com/git/git/blob/master/wrapper.c

Post reply on HN