Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Arena allocator tips and tricks

nullprogram.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Arena allocator tips and tricks

#91

Post by gnubison »

astrange wrote: post #85
squeek502 wrote: post #52

Earlier quoted context omitted.

There's support for exactly this type of OOM testing in Zig via std.testing.checkAllAllocationFailures: - https://github.com/ziglang/zig/blob/1606717b5fed83ee64ba1a91... - https://www.ryanliptak.com/blog/zig-intro-to-check-all-alloc...

This is a clear sign of a badly designed language. You should never see a fixed-size (less than page size) allocation fail, simply because there's nothing you can reasonably do if it does fail. Either you should crash or it should block until it is possible again. (Where crash means a worker process or something limited to something less than the entire system. See Erlang for the logical extension of this.) I realize…

Is Linux designed in such a way that less-than-page size allocations never fail? How would that work?

Re: Arena allocator tips and tricks

#92

Post by astrange »

matheusmoreira wrote: post #89
astrange wrote: post #85

Earlier quoted context omitted.

This is a clear sign of a badly designed language. You should never see a fixed-size (less than page size) allocation fail, simply because there's nothing you can reasonably do if it does fail. Either you should crash or it should block until it is possible again. (Where crash means a worker process or something limited to something less than the entire system. See Erlang for the logical extension of this.) I realize…

So if my browser tries to allocate memory for a tab and the allocation fails it should just crash or block instead of handling the failure gracefully by not creating a new tab and telling me the system is running low on memory?

Tabs are separate processes in every modern browser.

Actually since you skipped all the less than a page stuff I'm not sure you actually understand what my post is about…?

Re: Arena allocator tips and tricks

#93

Post by astrange »

gnubison wrote: post #91
astrange wrote: post #85

Earlier quoted context omitted.

This is a clear sign of a badly designed language. You should never see a fixed-size (less than page size) allocation fail, simply because there's nothing you can reasonably do if it does fail. Either you should crash or it should block until it is possible again. (Where crash means a worker process or something limited to something less than the entire system. See Erlang for the logical extension of this.) I realize…

Is Linux designed in such a way that less-than-page size allocations never fail? How would that work?

As long as the syscalls don't themselves fail, it's no problem until you run out of address space. At which point you should crash because you probably need to allocate to handle any reasonable errors.

Re: Arena allocator tips and tricks

#94

Post by variadix »

matheusmoreira wrote: post #31
gpderetta wrote: post #20

Earlier quoted context omitted.

char can alias everything, i.e. you can deference a char pointer with impunity, even if the actual dynamic type[1] of an object is a different type. The reverse is not true: if the dynamic type of an object is char, just by using the alias rules, you can't deference it as an object of a different type. In C++ you can just use placement new to change the dynamic type of (part of ) a char array (but beware of pointer p…

> The reverse is not true: if the dynamic type of an object is char, just by using the alias rules, you can't deference it as an object of a different type. A limitation like that simply makes no sense to me. Everything is a valid char array but I can't place structs on top of one? Oh well, nothing I can do about it. I'll just keep strict aliasing disabled. If we're writing C, it's because we want to do stuff like th…

I feel like there should be a builtin that takes a void* and returns a void* which basically marks the input as aliased by the returned pointer regardless of the TBAA rules.

Re: Arena allocator tips and tricks

#95

Post by variadix »

variadix wrote: post #94
matheusmoreira wrote: post #31

Earlier quoted context omitted.

> The reverse is not true: if the dynamic type of an object is char, just by using the alias rules, you can't deference it as an object of a different type. A limitation like that simply makes no sense to me. Everything is a valid char array but I can't place structs on top of one? Oh well, nothing I can do about it. I'll just keep strict aliasing disabled. If we're writing C, it's because we want to do stuff like th…

I feel like there should be a builtin that takes a void* and returns a void* which basically marks the input as aliased by the returned pointer regardless of the TBAA rules.

Turns out there is! __builtin_launder seems mostly equivalent to what the author proposes but probably more reliable. Facebook’s folly library appears to use both, preferring the builtin.

Re: Arena allocator tips and tricks

#96

Post by teo_zero »

If alloc() changes .beg, how do you free (release to the OS, not mark as unused) the whole arena when it's not needed anymore? Wouldn't changing .end be better, so that you can implement a destroyarena(a) which calls free(a.beg)?

Typical use: set up a specific arena just to read in the config file. Once it's done, release the whole arena.

Re: Arena allocator tips and tricks

#97

Post by cassepipe »

astrange wrote: post #93
gnubison wrote: post #91

Earlier quoted context omitted.

Is Linux designed in such a way that less-than-page size allocations never fail? How would that work?

As long as the syscalls don't themselves fail, it's no problem until you run out of address space. At which point you should crash because you probably need to allocate to handle any reasonable errors.

So is there no point to checking malloc return ?

Re: Arena allocator tips and tricks

#99

Post by SoftTalker »

varispeed wrote: post #23
corysama wrote: post #5

A very old trick for running Lua in your PlayStation 2 game (where the PS2 is a machine with 32MB of RAM and no memory paging) is to hook Lua’s realloc function into the venerable Doug Lea’s Malloc ( https://gee.cs.oswego.edu/dl/html/malloc.html ) set up to run in arena mode (ONLY_MSPACES? It’s been a decade or two…). That way Lua can fragment the arena all it wants without making a mess of the rest of the tiny addre…

I recently needed to write a memory allocator and being lazy asked ChatGPT for help. Interestingly it came up with implementation eerily similar to what is described in that document. Nonetheless everything worked like a charm from the start.

Writing a memory allocator is a standard homework assignment in any CS program. The solutions are mostly similar, and ChatGPT has probably learned hundreds of examples.
Post reply on HN