51–60 of 70 posts
I never like names that require a “pronounced like” note, but cool project regardless.
Earlier quoted context omitted.
If you're single threaded then you'll never have mutex contention so they'll always be fast-path. I'd suggest you actually prove that the memory barriers are actually a problem for you via profiling, since it's somewhat unlikely it is.
> If you're single threaded then you'll never have mutex contention so they'll always be fast-path. Concurrent algorithms (to which multi-thread capable allocators belong) typically necessitate design and performance compromises which aren't nullified by creating only a single thread. > I'd suggest you actually prove that the memory barriers are actually a problem for you via profiling, since it's somewhat unlikely i…
Not as rude as I wanted to be. I expect answers like that from Stack Overflow or Reddit - just another condescending ego play without actually addressing the question.
Regardless, thank you for your reply.
I don't suppose you know a good single-threaded (or share-nothing across threads) allocator worth looking at?
Is anyone aware of a good/fast single threaded allocator for cases where you don't need/want to pay for thread safety?
Earlier quoted context omitted.
If you're single threaded then you'll never have mutex contention so they'll always be fast-path. I'd suggest you actually prove that the memory barriers are actually a problem for you via profiling, since it's somewhat unlikely it is.
> If you're single threaded then you'll never have mutex contention so they'll always be fast-path. Concurrent algorithms (to which multi-thread capable allocators belong) typically necessitate design and performance compromises which aren't nullified by creating only a single thread. > I'd suggest you actually prove that the memory barriers are actually a problem for you via profiling, since it's somewhat unlikely i…
> It's rather arrogant to assume that the poster doesn't know what they're doing
Given the question they asked I don't believe it was arrogant at all to assume they don't really know what they were doing. Their response seems to justify the push to start from the basics as well.
Are there functions available with which I can at run-time query how much OS memory is used, how much handed out in allocations, how many mmap()ed pools are used, and so on? I find that one of the most important features of a malloc library to debug memory usage. glibc has these functions (like malloc_info()) -- they are very bugged in that they return wrong results, but after patching them to be correct, they are su…
Any more info/link on how they are incorrect and the patches you need to fix them?
https://sourceware.org/bugzilla/show_bug.cgi?id=24026 - "malloc_info() returns wrong numbers"
https://sourceware.org/bugzilla/show_bug.cgi?id=21556 - "malloc_stats printing size_t fields as unsigned int"
If you're interested in this topic:
After finding one bug after the other due to 32-bit integer overflow in malloc.c, I just searched for "unsigned int" in that file for fun, and 30 seconds later found what I consider a security vulnerability in realloc():
https://sourceware.org/bugzilla/show_bug.cgi?id=24027
In certain situations, if you realloc() e.g. 32G + 5 bytes (reallocs this large can happen in large programs e.g. for data analysis), it'll copy only 5 bytes, and leave the rest as memory garbage.
That experience taught me that open source code being old doesn't mean anybody ever read it.
Are there functions available with which I can at run-time query how much OS memory is used, how much handed out in allocations, how many mmap()ed pools are used, and so on? I find that one of the most important features of a malloc library to debug memory usage. glibc has these functions (like malloc_info()) -- they are very bugged in that they return wrong results, but after patching them to be correct, they are su…
On Linux, there's a bit of the info you're looking for under /proc/self/*. (Not a great answer; the info under /proc/ isn't exactly well structured, but doesn't change super often).
It describes only the kernel-userspace parts of memory usage. For example, malloc fragmentation is not inspectable in it.
(If you enable the mallopt() setting that turns each malloc() into a separate mmap(), then it tells you a lot, but that is not a default setting.)
Earlier quoted context omitted.
> If you're single threaded then you'll never have mutex contention so they'll always be fast-path. Concurrent algorithms (to which multi-thread capable allocators belong) typically necessitate design and performance compromises which aren't nullified by creating only a single thread. > I'd suggest you actually prove that the memory barriers are actually a problem for you via profiling, since it's somewhat unlikely i…
You can get large gains tuning an allocator for a specific use case, yes, but that's very different from tuning it for thread safety. Particularly since most major allocators use a first-level allocator that's thread-local and is therefore not paying any thread-safety tax in the first place. > It's rather arrogant to assume that the poster doesn't know what they're doing Given the question they asked I don't believe…
Earlier quoted context omitted.
Any more info/link on how they are incorrect and the patches you need to fix them?
Sure, my patches are linked in the bugs I filed: https://sourceware.org/bugzilla/show_bug.cgi?id=24026 - "malloc_info() returns wrong numbers" https://sourceware.org/bugzilla/show_bug.cgi?id=21556 - "malloc_stats printing size_t fields as unsigned int" If you're interested in this topic: After finding one bug after the other due to 32-bit integer overflow in malloc.c, I just searched for "unsigned int" in that file f…
The tricky part with allocators is always the multi-threaded setups. Even something as simple as a bunch of threads doing malloc-free in a loop will drop performance of a lot of allocators to the floor, due to some sort of central locking or excessive cache thrashing. This is typically solved by adding per-thread block pools, free lists or some such. If you go further down the rabbit hole, there's a case when blocks…
What if I'm writing code that's strictly single-threaded? Presumably malloc could be simpler and/or faster in this special case? Is there a production-ready allocator that's optimized for single-threaded use?
[1] https://twitter.com/samsaffron/status/1143048590555697152