Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Malloc Challenge

vicsydev.blogspot.com

Post reply on HN
Print view View original

51–60 of 109 posts

Re: Malloc Challenge

#51

Post by codr4life »

MichaelMoser123 wrote: post #49
codr4life wrote: post #15

Earlier quoted context omitted.

Not if they're writing C/C++; it doesn't really matter how hard you try to hide memory management when the whole language is designed around memory and pointers. What kind of programmer is that, anyway? Who can only program specific api's, as long as they don't touch the wrong parts of the language.

There are different strategies for handling perf problems of memory allocation - you can cache blocks of the same size in some linked list within the application (free list) or you can reconsider the allocator/allocation strategy. Both get the job done. Preferance of the first solution does not make you a worse programmer.

What you're describing is essentially a pool, which could be thought of as a kind of allocator since it provides an allocation strategy. There's even one in the challenge that does just that. Nothing makes you a worse programmer except refusing to learn.

Re: Malloc Challenge

#52

Post by fulafel »

EdSharkey wrote: post #41

I recall reading in the last year or two a recount of how a game developer got their PS3 engine to run at 30 or 60Hz framerate by aggressive triple-buffering of their scenes. One of the interesting bits about the article was their memory allocation scheme. Each game frame they'd allocate a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool. I think this is what you descri…

You are describing an arena allocator. Slab is more like a set of free lists per object size.

Re: Malloc Challenge

#53

Post by Jweb_Guru »

codr4life wrote: post #47
EdSharkey wrote: post #41

I recall reading in the last year or two a recount of how a game developer got their PS3 engine to run at 30 or 60Hz framerate by aggressive triple-buffering of their scenes. One of the interesting bits about the article was their memory allocation scheme. Each game frame they'd allocate a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool. I think this is what you descri…

The slab reference allocator from the challenge does that, but adds a segmented approach where the memory is allocated in N sized blocks. Go has decent support for doing these kinds of tricks; but nothing really runs faster since there's still too much magic behind the scenes. I suspect the story is the same in Rust land but I honestly never got passed the lifetime hump, too much ceremony for me.

There's not really much runtime magic in Rust (pretty comparable to C++), and arenas are definitely a performance benefit in many cases.

Re: Malloc Challenge

#54

Post by flamedoge »

greenshackle2 wrote: post #34
_RPM wrote: post #31

Earlier quoted context omitted.

Python does not look like C. The first thing I can think of is using traditional loop counters instead of the for..in construct.

That's a minor detail. I'm talking about writing in a procedural style. Rewriting a for..in as a for with counters is relatively trivial, compared to re-implementing a complex class hierarchy in C. You can write procedural Python, and it will look kinda like C if you squint. Cython even lets you use C types directly, with a simple syntax. As I said, maybe not the sanest thing to do, everything is an object in Python…

Nobody stops one from writing Python like C. CLikeMemoryModel = []

Re: Malloc Challenge

#55

Post by wahern »

fulafel wrote: post #52
EdSharkey wrote: post #41

I recall reading in the last year or two a recount of how a game developer got their PS3 engine to run at 30 or 60Hz framerate by aggressive triple-buffering of their scenes. One of the interesting bits about the article was their memory allocation scheme. Each game frame they'd allocate a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool. I think this is what you descri…

You are describing an arena allocator. Slab is more like a set of free lists per object size.

IME arena doesn't really have a fixed definition. The term can and has been used to refer to something like an object stack or bump allocator, but also to something that supports deallocation and reallocation. The term goes back over 30 years and nothing specific has really stuck.

I think the most you can say of an arena is that it's usually a contiguous region of memory from which smaller allocations are made, and which can be efficiently freed as a whole. An arena may only support fixed-size allocations, or a range of sizes; it may or may not support deallocation. However, in many cases it's natural to require multiple regions to satisfy all allocation requests for a particular context (task, generation, etc), so don't be surprised if an implementation labels a collection of contiguous regions an "arena".

The term pool is similarly ambiguous, but usually implies support for deallocation and recycling of memory. It does not necessarily imply a contiguous region, but that's a natural optimization in a language like C.

Slab is less ambiguous because it has a very specific origin in SunOS--allocation and deallocation of fixed-size, often typed objects (to optimize initialization).

Re: Malloc Challenge

#56

Post by rurban »

jwatte wrote: post #9

How about: mmap() a few terabytes of virtual space let malloc() be pointer addition and let free() be a no-op?

Or add a simple GC to that, if you need to recycle memory during longer runs, and your stack is too small to hold most allocs.

Re: Malloc Challenge

#57

Post by nullc »

EdSharkey wrote: post #41

I recall reading in the last year or two a recount of how a game developer got their PS3 engine to run at 30 or 60Hz framerate by aggressive triple-buffering of their scenes. One of the interesting bits about the article was their memory allocation scheme. Each game frame they'd allocate a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool. I think this is what you descri…

> a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool

Maybe a register could be used to keep track of the current pointer into this pool, and every time you called into a new function the call could increment the pointer enough for all the usage in a function, and when the function returns it could set it back, automatically deallocating the usage with almost no cost at all?

Re: Malloc Challenge

#58

Post by toolslive »

fulafel wrote: post #52
EdSharkey wrote: post #41

I recall reading in the last year or two a recount of how a game developer got their PS3 engine to run at 30 or 60Hz framerate by aggressive triple-buffering of their scenes. One of the interesting bits about the article was their memory allocation scheme. Each game frame they'd allocate a single huge memory pool and then allocate from it by simply incrementing a pointer into the pool. I think this is what you descri…

You are describing an arena allocator. Slab is more like a set of free lists per object size.

names vary. It's also called a scratchpad or region.

Re: Malloc Challenge

#59

Post by codr4life »

wahern wrote: post #55
fulafel wrote: post #52

Earlier quoted context omitted.

You are describing an arena allocator. Slab is more like a set of free lists per object size.

IME arena doesn't really have a fixed definition. The term can and has been used to refer to something like an object stack or bump allocator, but also to something that supports deallocation and reallocation. The term goes back over 30 years and nothing specific has really stuck. I think the most you can say of an arena is that it's usually a contiguous region of memory from which smaller allocations are made, and w…

They're all out there floating around :) What I refer to as a pool allocator is a set of separate allocations, could be same size. While a slab is a single block of memory that's dished out as separate pointers, that could likewise be same size.

Re: Malloc Challenge

#60

Post by nothrabannosir »

codr4life wrote: post #25
xenadu02 wrote: post #23

Earlier quoted context omitted.

I'm sure it inflates one's ego but the chances that they've avoided writing any exploitable security vulnerabilities is almost zero.

Show me a software stack without those kinds of bugs; they're all written in C somewhere down the line; enormous amounts of complicated C, written by coders of varying competence. At least in my stack I can keep it simple and fix anything that needs fixing.

He/she meant: two programmers of equal skill and security conscientiousness will create less exploitable security vulnerabilities per "unit of functionality", using C and respectively, ceteris paribus.

"Easier languages attract less skilled devs", "higher level languages allow you to create more complex programs", yada yada. Could all be true, but it's besides the point.

Post reply on HN