Is this applicable for Go as well ?
Rust: Dropping heavy things in another thread can make your code 10000x faster
161–170 of 285 posts
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#162Speedup numbers should be given when optimizing constant factors -- e.g. "I made this operation 5X faster using SIMD" or "By employing readahead, I sped up this file copy by 10X".
The points raised in this article are really different:
* don't do slow stuff in your latency-critical path
* threads are a nice way to unload slow stuff that you don't need done right away (especially if you have spare cores)
* dropping can be slow
The first and second points are good, but not really related to rust, deallocations, or the number 10000.
The last point is worth discussing, but still not really related to the number 10000 and barely related to rust. Rust encourages an eager deallocation strategy (kind of like C), whereas many other languages would use a more deferred strategy (like many GCs).
It seems like deferred (e.g. GC) would be better here, because after the main object is dropped, the GC doesn't bother to traverse all of the tiny allocations because they are all dead (unreachable by the root), and it just discards them. But that's not the full story either.
It's not terribly common to build up zillions of allocations and then immediately free them. What's more common is to keep the structure (and its zillions of allocations) around for a while, perhaps making small random modifications, and then eventually freeing them all at once. If using a GC, while the large structure is alive, the GC needs to scan all of those objects, causing a pause each time, which is not great. The eager strategy is also not great: it only needs to traverse the structure once (at deallocation time), but it needs to individually deallocate.
The answer here is to recognize that all of the objects in the structure will be deallocated together. Use a separate region/arena/heap for the entire structure, and wipe out that region/arena/heap when the structure gets dropped. You don't need to traverse anything while the structure is alive, or when it gets dropped.
In rust, probably the most common way to approximate this is by using slices into a larger buffer rather than separate allocations. I wish there was a little better way of doing this, though. It would be awesome if you could make new heaps specific to an object (like a hash table), then allocate the keys/values on that heap. When you drop the structure, the memory disappears without traversal.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#163I've seen variations on this trick multiple times. Using threads, using a message sent to self, using a list and a timer to do the work "later", using a list and waiting for idle time...
They all have one thing in common: pampering over a bad design.
In the particular example given, the sub-vector probably come from a common source. One could keep a big buffer (a single allocation) and an array of internal pointers. For example of such a design to hold a large array of text strings, see for example this blog entry and its associated github repo:
https://www.spiria.com/en/blog/desktop-software/optimizing-shared-data/
https://github.com/pierrebai/FastTextContainer
struct TextHolder
{
const char* common_buffer;
std::vector internal_pointers;
};
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#164This is the standard problem with tracing data structures to free them. You frequently run into it with systems based on malloc/free or reference counting. The underlying problem is that freeing the structure takes time proportional to the number of pointers in the structure it has to chase. Generational/compacting GC has the opposite problem. Garbage collection takes time proportional to the live set, and the amount…
"Generational/compacting GC has the opposite problem. Garbage collection takes time proportional to the live set, and the amount of memory collected is unimportant."
Takes time proportional the live set times the number of GC runs that happen while the objects are alive. In other words, the longer the objects live, the more GC runs have to scan that object (assuming there is enough activity to trigger the GC), and the worse GC looks.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#165The obvious solution would be to borrow the HeavyThing instead of having it dropped inside the function.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#166Earlier quoted context omitted.
>As an aside, compilers have used the trick of not free-ing data structures before, because it provides a significant performance boost. Instead of calling free on all those billions of tiny data structures a compiler would generate during its lifetime, they just let them leak. Since a compiler is short lived its not a problem, they get a free lunch (pun unintended), and the OS takes care of cleaning up after all is…
Well, then that’s not the original use case anymore, and it’ll have to be re-engineered. In the meantime it may have been used for years and the perf difference may have saved many developer-years collectively across its user base. Surely you’re not suggesting that the compiler developers should be prematurely optimizing for future use cases that they may not even have envisioned.
I am suggesting they apply good practices. I'd never imagine that compilers were actually doing what was stated -- sounds awful.
I understand it's tradeoffs and we all have real-world limitations to contend with -- but again, of all the corners that could be cut that's exactly the one I didn't imagine they would.
Nasty.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#167Some important things I think people should note before blindly commenting: * The example code is obviously contrived. The real gist is that massive deallocations in the UI thread cause lag, which the example code proves. That very thing can easily happen in the real world. * I didn't see any difference on my machine between a debug build and a release build. * The example is preforming 1 _million_ deallocations. Tha…
This deallocation trick is neat but in C and C++ you could use a memory pool to do this. In theory, you could also use a memory pool in Rust but I think the standard library uses malloc without some way of overriding this behaviour.
In a toy raytracer I once wrote in C++, switching from malloc to custom memory pools for small fixed-size objects was a big performance boost. Making free() a noop was another big performance boost, both for deallocation and allocation. Turns out sequentially handing out memory from a big chunk of memory is much easier than keeping track of and reusing empty slots, and it keeps sequentially allocated objects in sequential memory locations.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#168Some important things I think people should note before blindly commenting: * The example code is obviously contrived. The real gist is that massive deallocations in the UI thread cause lag, which the example code proves. That very thing can easily happen in the real world. * I didn't see any difference on my machine between a debug build and a release build. * The example is preforming 1 _million_ deallocations. Tha…
One of my “favorite” snags in perf analysis is that periodicity in allocations can misattribute the cost of allocations to the wrong function. If I allocate just enough memory, but not too much, then pauses for defragmentation of free space may be costed to the code that calls me. A solution to this that I’ve seen in soft real time systems is to amortize cleanups across all allocations. Every allocation performs n st…
Doing unrelated cleanup sounds like flushing CPU cache per every allocation.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#169Earlier quoted context omitted.
Well, then that’s not the original use case anymore, and it’ll have to be re-engineered. In the meantime it may have been used for years and the perf difference may have saved many developer-years collectively across its user base. Surely you’re not suggesting that the compiler developers should be prematurely optimizing for future use cases that they may not even have envisioned.
Avoiding leaks is not optimisation, it's a matter of correctness - not freeing memory is an optimisation based on a very shortsighted assumption that is not practical for any new language (modern languages are expected to come with language server support)
You have not provided any refutation to the OPs argument.
Re: Rust: Dropping heavy things in another thread can make your code 10000x faster
#170Earlier quoted context omitted.
Well, then that’s not the original use case anymore, and it’ll have to be re-engineered. In the meantime it may have been used for years and the perf difference may have saved many developer-years collectively across its user base. Surely you’re not suggesting that the compiler developers should be prematurely optimizing for future use cases that they may not even have envisioned.
I am suggesting they apply good practices. I'd never imagine that compilers were actually doing what was stated -- sounds awful. I understand it's tradeoffs and we all have real-world limitations to contend with -- but again, of all the corners that could be cut that's exactly the one I didn't imagine they would. Nasty.
Deallocation at the end of a program's execution can substantially add to its runtime, and it's entirely waste. It's a much more common strategy than you might think.