Live data from Hacker News

Why does calloc exist?

vorpus.org

61–70 of 141 posts

Re: Why does calloc exist?

#61
post #59

Earlier quoted context omitted.

>Anything else is amateur hour. Such as writing the optimizing compilers that make it feasible for you to use C at all?

I don't understand your reply. How is this not a non sequitur?

The output of a modern C compiler is unpredictable in terms of performance. Yet mannykannot suspects you still use such tools. Please explain your inconsistency.

Re: Why does calloc exist?

#62
post #58

Earlier quoted context omitted.

Predictability sometimes trumps optimizations. For a striking illustration of this, see timing attacks.

Exactly. I would avoid using calloc simply because I don't know what it actually does .

The implementation details of malloc aren't specified as part of its interface either...

Re: Why does calloc exist?

#63
post #36
post #30

Sorry, but this is just goofy and bad. If you depend on copy-on-write functionality, then you need to use an API that is specced to guarantee copy-on-write functionality. If that means you use an #ifdef per platform and do OS-specific stuff, then that is what you do. Anything else is amateur hour. If copy-on-write is a desirable feature, then as the API creator, your job is to expose this functionality in the cleares…

Neither copy on write nor size checking are specced as part of the calloc() definition. Here's the specification of calloc from the ISO standard: 7.22.3.2 The calloc function Synopsis #include void *calloc(size_t nmemb, size_t size); Description The calloc function allocates space for an array of nmemb objects, each of whose size is size. The space is initialized to all bits zero. Returns The calloc function returns…

An implementation that lets the size overflow and returns a pointer to a block that isn't large enough for "an array of nmemb objects, each of whose size is size" is not conforming with that specification.

That specification gives the implementation exactly two options: return NULL, or return a pointer to a block of sufficient size.

Re: Why does calloc exist?

#64

I've always been surprised that memset is usually just a nonmagical for loop. I used to expect that the OS does things to magically make it faster (running lazily, etc).

CPUs are pretty great at running nonmagical for loops, so you'd have be zeroing a pretty giant block of memory before it made sense to get the OS involved at all.

Re: Why does calloc exist?

#65
Let’s see what happens after the allocation.

With malloc + memset, the OS will likely allocate that memory in huge pages, on PC that would be 2-4MB / page depending on the architecture, https://en.wikipedia.org/wiki/Page_(computer_memory)#Huge_pa...

If I calloc then write, the OS can’t give me huge pages because of that copy on write thing. Instead, the OS will gradually give me the memory in tiny 4kb pages. For large buffers you should expect TLB cache misses, therefore slowing down all operations with that memory.

Re: Why does calloc exist?

#66

I you calloc some memory and then the first thing you do is write to it, can the compiler optimize away the initial write of zeros since they will just be overwritten?

Possibly, for small allocations: a possible way this can happen is a small-value `calloc()` inlined as a `malloc()` / store zeroes pair, and then the "store zeroes" part of that discarded by a later optimisation pass as a dead store.

On modern server/desktop/mobile CPUs, this won't make much difference anyway because the second write to the same location is essentially 'free' due to the store buffer.

And of course if you're calling calloc() in a tight loop, then the zeroing is the least of your performance concerns!

Re: Why does calloc exist?

#67

Let’s see what happens after the allocation. With malloc + memset, the OS will likely allocate that memory in huge pages, on PC that would be 2-4MB / page depending on the architecture, https://en.wikipedia.org/wiki/Page_(computer_memory)#Huge_pa... If I calloc then write, the OS can’t give me huge pages because of that copy on write thing. Instead, the OS will gradually give me the memory in tiny 4kb pages. For larg…

I don't think your average kerbel can tell the difference between malloc+memset and calloc.

Re: Why does calloc exist?

#68
post #33
post #3

That's a nice alternative history fiction. Here's an early implementation: https://github.com/dspinellis/unix-history-repo/blob/Researc...

You haven't proven it wrong. Here's the earliest implementation in that repo (in Research UNIX V6; your link in V7): https://github.com/dspinellis/unix-history-repo/blob/Researc... calloc(n, s) { return(alloc(n*s)); } There are several interesting things we learn from poking around V6 though: - `calloc` originated not on UNIX, but as part of Mike Lesk's "iolib", which was written to make it easier to write C programs…

It appears that only calloc was in Lesk's Portable C Library [0] while malloc was the name Thompson gave the kernel's memory allocator in V6 [1]. When Ritchie rewrote Lesk's library for V7, he may have simply retained calloc for backward compatibility with existing user space code. [0] http://roguelife.org/~fujita/COOKIES/HISTORY/V6/iolib.html [1] https://github.com/hephaex/unix-v6/blob/master/ken/malloc.c

Re: Why does calloc exist?

#69
post #33
post #3

That's a nice alternative history fiction. Here's an early implementation: https://github.com/dspinellis/unix-history-repo/blob/Researc...

You haven't proven it wrong. Here's the earliest implementation in that repo (in Research UNIX V6; your link in V7): https://github.com/dspinellis/unix-history-repo/blob/Researc... calloc(n, s) { return(alloc(n*s)); } There are several interesting things we learn from poking around V6 though: - `calloc` originated not on UNIX, but as part of Mike Lesk's "iolib", which was written to make it easier to write C programs…

OpenBSD added calloc overflow checking on July 29th, 2002. glibc added calloc overflow checking on August 1, 2002. Probably not a coincidence. I'm going to say nobody checked for overflow prior to the August 2002 security advisory.

https://github.com/openbsd/src/commit/c7b2af4b3f7e78424f8943...

https://github.com/bminor/glibc/commit/0950889b810736fe7ad34...

http://cert.uni-stuttgart.de/ticker/advisories/calloc.html

Re: Why does calloc exist?

#70
I don't get it. The two behaviors are completely orthogonal. Why can't I have a malloc() that does lazy copy-on-write for large arrays and why can't I have an error checking malloc() and why can't I have a calloc() that allocates the memory up front and doesn't zero it out? I get the "it's historic" argument, but this seems like a silly distinction. Sounds like what you want to do practically is basically just make your malloc() wrap a calloc() with size 1, and stop explicitly memset()ing. Or just introduce your own functions:

    moarmem(n) // malloc(n)
    moarmemslower(n) // p = malloc(n); memset(p, 0);
    moarmemfaster(n) // calloc(n, 1)
    evenmoarmem(p, n); // realloc()
    fuggetaboutit(p) // free()
Post reply on HN