Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Memory Allocators 101 – Write a simple memory allocator (2016)

arjunsreedharan.org

Post reply on HN
Print view View original

41–50 of 56 posts

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#41

Post by flurrything »

wongarsu wrote: post #33
badminton1 wrote: post #4

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

In the past I've sometimes achieved 20% or better speedups by writing custom allocators (speedup on overall performance, not just malloc performance). tcmalloc or jemalloc are great for the general case, but sometimes you know invariants about object sizes, alloc patterns and free patterns that allow much more performant allocation. The simplest case is if you know that you will free everything at once, or nothing at…

> but sometimes you know invariants about object sizes, alloc patterns and free patterns that allow much more performant allocation.

jemalloc allows you to query these invariants if you don't know them, and to use the information to re-configure the allocator to match them :/

I'm pretty sure most modern allocators allow you to do this as well.

> The simplest case is if you know that you will free everything at once, or nothing at all.

That's pretty much a one liner with jemalloc.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#42

Post by q3k »

Please don't write custom memory allocators for production code without seriously considering the security implications of doing so. The glibc allocator has benefited from years of security hardening against very creative attack vectors.

You don't want a simple double-free to lead to an RCE bug, do you...

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#43

Post by pjc50 »

K&R includes an example memory allocator as well: https://stackoverflow.com/questions/13159564/explain-this-im...

To me, the K&R one seems much less readable and also doesn't include the global malloc lock, since even the updated edition of K&R predates standardised threading.

I note it also does the "bp = (Header *)ap - 1;" trick, so if that's undefined behaviour then it's a good example of how hard it is to write C without relying on UB.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#44

Post by [deleted] »

q3k wrote: post #42

Please don't write custom memory allocators for production code without seriously considering the security implications of doing so. The glibc allocator has benefited from years of security hardening against very creative attack vectors. You don't want a simple double-free to lead to an RCE bug, do you...

[deleted]

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#46

Post by Thaxll »

lasagnaphil wrote: post #13
badminton1 wrote: post #4

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

I’ve heard many gamedevs make specialized allocators for some of their code (the most being arena allocators)...

All major game engines have custom memory allocator.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#47

Post by slashvar2701 »

While I strongly support the idea that no one should write a generic allocator in production, writing it as an exercise is a very good idea.

The article looks at lot like the tutorial I wrote a long time ago ... (Every now and then, I see my old PDF in post about implementing allocators, which is disturbing since I wrote it in a hurry as a quick support for a lecture and I found it very poorly written ... )

I think it's interesting to note that using sbrk(2) is a bit deprecated, it's way easier to implement malloc(3) using mmap(2) ...

There's also better ways to handle pointers arithmetic and list management. Someday I'll put only cleaner version only ... Someday ...

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#48

Post by simcop2387 »

paavoova wrote: post #22
simcop2387 wrote: post #10

Earlier quoted context omitted.

You can't use memcpy here, you have to modify the original memory to mark it as freed. You also can't allocate a new buffer to do the copy into because you are the allocator. The C standard specifically states that a cast (T ) to and from (void ) is validly defined behavior and that you must get the original pointer back. It's also valid to go from (T ) to (U ) and back again if and only if T and U have the same alig…

> The caller might turn the element after the (struct header_t ) into something else Isn't this exactly the issue with this malloc implementation? The pointer returned that points past the header by sizeof(header) may not be aligned for subsequent types.

This is a problem inherit to all malloc() implementations, since it never gets any type information it can't ever make any adjustments about alignment of the final type that's involved. I'm not actually sure what the fully correct way to ensure that would be.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#49

Post by simcop2387 »

burfog wrote: post #20
simcop2387 wrote: post #10

Earlier quoted context omitted.

You can't use memcpy here, you have to modify the original memory to mark it as freed. You also can't allocate a new buffer to do the copy into because you are the allocator. The C standard specifically states that a cast (T ) to and from (void ) is validly defined behavior and that you must get the original pointer back. It's also valid to go from (T ) to (U ) and back again if and only if T and U have the same alig…

You too are losing your asterisks. Maybe a bit of Unicode can help. The only thing not stripped out by Hacker News seems to be this: ⁎ e2 81 8e

Damn markdown :). Looks like it's far too late for me to fix it now, but I'll have to keep that in mind in the future.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#50

Post by typon »

apankrat wrote: post #32
badminton1 wrote: post #4

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

Not sure why you are downvoted, but this is a toy allocator.

It's downvoted because it's missing the entire point of this article?
Post reply on HN