Live data from Hacker News

Memory Allocators 101 – Write a simple memory allocator (2016)

arjunsreedharan.org

11–20 of 56 posts

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#11
post #3

header = (struct header_t*)block - 1; Isn't this UB?

I think it would be better to go with char* instead of void*. A much better solution is to use a hashtable for your blocks, but I understand this is just a toy example.

I think he also forgot to implement memalign(3) and posix_memalign(3).

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#12
post #9

Earlier quoted context omitted.

So UB, then? All complaint solutions I could find to this use memcpy() instead. Some use __attribute__ ((__packed__)) for structs, but that seems to have its own gotchas ( https://stackoverflow.com/a/7956942 ), but not relevant here.

You can't use memcpy here, you have to modify the original memory to mark it as freed. You also can't allocate a new buffer to do the copy into because you are the allocator. The C standard specifically states that a cast (T ) to and from (void ) is validly defined behavior and that you must get the original pointer back. It's also valid to go from (T ) to (U ) and back again if and only if T and U have the same alig…

"You also can't allocate a new buffer to do the copy into because you are the allocator."

You can use the stack in this case.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#13

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

I’ve heard many gamedevs make specialized allocators for some of their code (the most being arena allocators)...

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#14

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

Or even the standard memory allocator provided by your system. I'm pretty sure this article was meant as a way to understand how malloc works and not as a high-performance replacement for the one you're currently using.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#15
Good timing. I literally had to do this on a whiteboard in an interview about 11 hours ago.

Interesting things to consider: Fragmentation prevention, real-time performance, minimizing locking(lock-free techniques, or per-thread free lists), and reusing the freed memory to contain the free list structure. I basically started out whiteboarding what the article lays out and by the end of the interview realized everything wrong with it. It's a good starting point though.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#16

Good for didactic purposes but in the real world you may want to try an allocator like tcmalloc or jemalloc.

I’ve heard many gamedevs make specialized allocators for some of their code (the most being arena allocators)...

It's fairly common to avoid naked malloc()/free() in systems with real-time requirements. Memory pools are a great way to go if you want deterministic behavior and better reliability.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#17
post #3

header = (struct header_t*)block - 1; Isn't this UB?

The cast is defined by ISO/IEC 9899:1990 §6.3.2.3.1, since block is a pointer to void, and struct header_t is an object type:

"A pointer to void may be converted to or from a pointer to any incomplete or object type."

The subtraction is defined by §6.5.6.8, provided that block points to an element of a large enough array object:

"When an expression that has integer type is added to or subtracted from a pointer, the result has the type of the pointer operand. If the pointer operand points to an element of an array object, and the array is large enough, the result points to an element offset from the original element such that the difference of the subscripts of the resulting and original array elements equals the integer expression."

(There's similar text in other versions of the C standard.)

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#18
post #3

header = (struct header_t*)block - 1; Isn't this UB?

Not quite, (void *) is special in this regard IIRC, specifically for cases like this. It's a valid pointer but you're required to ensure that any platform requirements like alignment and size will match the requirements of what you're doing.

No, (void⁎) is not special in this way. It just makes things look nicer.

The code also isn't undefined behavior... but you are really asking to hit compiler bugs! This is an easy way to confuse gcc into wrongly determining that the code has undefined behavior, and if gcc gets confused then it may determine that a code path can't be taken. Code paths that can't be taken may be deleted.

The main rule here is that memory has a type which is determined by what was last written into it, and you may only read or examine the memory using that type. (for the type, we ignore attributes like the distinction between signed and unsigned) There is a minor exception that is just enough to implement something like memcpy by using a (char⁎) to read and then write as a char. You still aren't supposed to look at that char. These rules apply to memory accessed via pointers, no matter how you cast them, and to memory accessed via union members.

Real compilers differ from that:

Every compiler I'm aware of will not enforce the rules for unions. The gcc compiler promises not to enforce the rules in this case.

Every compiler I'm aware of will let you look at any data that has been read as a char, so the memcpy trick works and you can do things like determine endianness at runtime.

It is legit to initialize a type X variable, take the address of it, cast it from (X⁎) to (Y⁎), pass it through arbitrary data structures and functions to hide the origin from the compiler, cast the (Y⁎) back to (X⁎), and then access the type X variable. If you do this, gcc may generate bad code.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#19
post #3

header = (struct header_t*)block - 1; Isn't this UB?

The cast is defined by ISO/IEC 9899:1990 §6.3.2.3.1, since block is a pointer to void, and struct header_t is an object type: "A pointer to void may be converted to or from a pointer to any incomplete or object type." The subtraction is defined by §6.5.6.8, provided that block points to an element of a large enough array object: "When an expression that has integer type is added to or subtracted from a pointer, the r…

That part of the standard only covers the cast. It means that you won't mangle a pointer if you cast it to a void pointer and then back to the original pointer type.

Accessing the data that is being pointed at is another matter entirely. You must satisfy alignment constraints. You also must not read any memory as a type other than what it was written as, aside from a very limited exception for type char.

Re: Memory Allocators 101 – Write a simple memory allocator (2016)

#20
post #9

Earlier quoted context omitted.

So UB, then? All complaint solutions I could find to this use memcpy() instead. Some use __attribute__ ((__packed__)) for structs, but that seems to have its own gotchas ( https://stackoverflow.com/a/7956942 ), but not relevant here.

You can't use memcpy here, you have to modify the original memory to mark it as freed. You also can't allocate a new buffer to do the copy into because you are the allocator. The C standard specifically states that a cast (T ) to and from (void ) is validly defined behavior and that you must get the original pointer back. It's also valid to go from (T ) to (U ) and back again if and only if T and U have the same alig…

You too are losing your asterisks. Maybe a bit of Unicode can help. The only thing not stripped out by Hacker News seems to be this:

⁎ e2 81 8e

Post reply on HN