Live data from Hacker News

Rust: Dropping heavy things in another thread can make your code 10000x faster

abramov.io

231–240 of 285 posts

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#231

Some important things I think people should note before blindly commenting: * The example code is obviously contrived. The real gist is that massive deallocations in the UI thread cause lag, which the example code proves. That very thing can easily happen in the real world. * I didn't see any difference on my machine between a debug build and a release build. * The example is preforming 1 _million_ deallocations. Tha…

> That very thing can easily happen in the real world.

Only if badly designed. That is why it is contrived!

> While that may seem contrived, consider a vector of 1 million strings, something that's not too uncommon

A program dealing with a million elements of any kind should not be performing naive allocations to begin with.

> we do deallocation trickery in the real world

Skipping deallocations is an optimization, not a design pattern.

In other words, the code needs to keep the ability to perform the deallocation for debugging, testing, usage as a library, etc.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#232

It seems that this would be a great reason to not pass the entire heavy object through your function, and to instead pass it as a reference. When passing an object (rather than a reference to an object) there's a lot more work going on both in function setup, and in object dropping. I'm not a rust guru, so I don't know the precise wording, but it's simple enough to realize that if this function, as claimed, must drop…

> but it's simple enough to realize that if this function, as claimed, must drop all the sub-objects within the `HeavyObject` type, then those objects must have been copied from the original object.

Untrue. Rust uses move semantics (or shallow copys for types that implement the Copy trait via e.g. memcpy - no you can't customize this!) Deep copies require explicitly calling methods like ".clone()". So HashMap's pointers and sizes do get memcpyed... 56 bytes on the 64-bit playground currently.

This is similar to how std::move(...)ing a std::unordered_map in C++ nulls out the old object and just copies the pointers of the container - not a deep copy of the subobjects - which in similar C++ code would turn the main thread's destructor into a noop.

The main difference from C++ is: Rust handles this at the language level instead, and doesn't call the dtor at all on the main thread at all if the value was moved. No need for manual movement logic - it is the default, for everything. Also unlike C++, it also prohibits you from using the old moved-from object at compile time, preventing bugs.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#233

I've seen variations on this trick multiple times. Using threads, using a message sent to self, using a list and a timer to do the work "later", using a list and waiting for idle time... They all have one thing in common: pampering over a bad design. In the particular example given, the sub-vector probably come from a common source. One could keep a big buffer (a single allocation) and an array of internal pointers.…

Yes. There's also a number of pool/arena allocators in rust which could be used here instead to drop All entries at once.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#234
post #8
post #6

Earlier quoted context omitted.

In C, if you forget to clean up, you have a memory leak which is hard to track down. In Rust, if you don't do this, you're not sacrificing memory leaks, only performance. A profiler can tell you when you should drop asynchronously.

>A profiler can tell you when you should drop asynchronously Is there any profiler that does this today? What are the drawbacks with asynchronous drops?

> Is there any profiler that does this today?

It requires interpretation, but yes.

> What are the drawbacks with asynchronous drops?

You pay some overhead for enqueuing work for later. Taken too far, lock contention / false sharing can make performance way worse.

The allocators and destructors must be thread safe if offloading to a worker thread.

The core running your thread is likely to have some of this data in L1/L2/L3 cache, which might not be true for whatever core would deque the work for asyncronously dropping.

It can be harder to attribute the costs of dropping to the right code if it all gets mixed up into a single work queue cleaned up by opaque worker threads when profiling.

If you don't use some kind of backpressure mechanism, allocations can potentially outpace deallocations and run you out of memory.

----

So, concrete example: Using Telemetry - a flamegraph style realtime profiler requiring invasive annotations - I was able to track down the cause of a framerate hitch in a game I was working on, to the sudden release of several graphics resources in a game. During events which would significantly restyle the look of some of the terrain, we'd eat several 10s/100s of milliseconds of overhead freeing things - more than enough to cause us to miss vsync. Would've stuck out like a sore thumb in any profiler capable of giving you a rough idea of the stack(s) involved in a 100ms timeframe that you can correlate to a vsync miss / missed frames.

D3D9 isn't thread safe (although freeing resources might've been?), but I didn't need to offload the work onto another thread just to amortize the cost over a few frames. Instead, a simple work queue did the trick. Problem solved! New problem: level transitions took significantly longer when doing mass frees of the same resources - more than doubling the cost of deallocation IIRC, for reasons I never did fully understand. Cache thrashing of some sort? We were still maxing out the core running the main thread with mostly cleanup logic...

Final code we shipped with used a hybrid solution that would choose between syncronous (high-throughput) and asyncronous (non-stalling) cleanup logic depending on what was happening in-game. Worked like a charm. Of course, this logic was hideously project specific and unable to be automatically chosen correctly for you by the programming language...

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#235
post #190

Earlier quoted context omitted.

Correctness means adherence to the spec, not some contrived absolute truth.

But the spec usually has some implicit assumptions. Usually it's "app doesn't leak memory" in the same way nobody explicitly specifies "result of an addition of natural numbers should match ...". We don't go around saying "oh, you didn't want modulo 5 arithmetic? You should've put that in the spec, not rely on some contrived absolute truth".

But the memory doesn't leak, it gets freed when the program exits.

The user doesn't care that each allocation is paired with another call to free it up. They just care if the program runs quickly and doesn't use too much memory overall.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#236

Earlier quoted context omitted.

But the spec usually has some implicit assumptions. Usually it's "app doesn't leak memory" in the same way nobody explicitly specifies "result of an addition of natural numbers should match ...". We don't go around saying "oh, you didn't want modulo 5 arithmetic? You should've put that in the spec, not rely on some contrived absolute truth".

But the memory doesn't leak, it gets freed when the program exits. The user doesn't care that each allocation is paired with another call to free it up. They just care if the program runs quickly and doesn't use too much memory overall.

Unless the system doesn't have enough memory to allocate for large compilation units and starts swapping allocated/dead memory and slowing down massively. Unless the user runs a CI where as many things should be compiling at the same time as possible and slowly freeing memory in 3 concurrent processes is better than running only 1 at a time. Unless they have spare cores, but no spare memory. Unless ...

> They just care ...

It all depends on who "they" are and what they want to achieve.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#237
post #195

Earlier quoted context omitted.

But the spec usually has some implicit assumptions. Usually it's "app doesn't leak memory" in the same way nobody explicitly specifies "result of an addition of natural numbers should match ...". We don't go around saying "oh, you didn't want modulo 5 arithmetic? You should've put that in the spec, not rely on some contrived absolute truth".

Okay, but we're talking about an application, a piece of software who's primary intention is to run for a short period of time, parse a text file and transform it. It will be ran many many many times a day by developers whose time is expensive. Language servers are a very new concept in terms of being part of the day-to-day tool chains for most developers. Trading garbage collection for compile time was absolutely a…

"We made dodgy choices to be competitive" is a valid explanation. They don't start being dodgy when new use cases come around and technical debt bites you in the butt.

I see not freeing as a clever bug/workaround with nice positive side effects. Not as a clever solution.

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#239

Earlier quoted context omitted.

But the memory doesn't leak, it gets freed when the program exits. The user doesn't care that each allocation is paired with another call to free it up. They just care if the program runs quickly and doesn't use too much memory overall.

Unless the system doesn't have enough memory to allocate for large compilation units and starts swapping allocated/dead memory and slowing down massively. Unless the user runs a CI where as many things should be compiling at the same time as possible and slowly freeing memory in 3 concurrent processes is better than running only 1 at a time. Unless they have spare cores, but no spare memory. Unless ... > They just ca…

[deleted]

Re: Rust: Dropping heavy things in another thread can make your code 10000x faster

#240
post #66

Just be careful, because moving heavy things to be dropped to another thread can change the semantics of the program. For instance, consider what happens if within that heavy thing you had a BufWriter: unless its buffer is empty, dropping it writes the buffer, so now your file is being written and closed in a random moment in the future, instead of being guaranteed to have been sent to the kernel and closed when the…

A while ago I stumbled over a proposal to move a shared pointer (this was C++ code) to a thread in order to trigger the freeing of a legacy data structure there (the multi-thousand delete calls caused the watchdog of the main thread to fail). However, keeping the shared pointer reference in the main thread for too long resulted in the possibility that the "clean-up" thread ran while the main thread still had a hold on the shared pointer... Resulting in a low chance of the "clean-up" thread doing nothing and the main thread still locking up. People here got taught to use shared pointers to prevent memory management errors, but it can really cause a lot of unexpected non-determinism when used blindly.
Post reply on HN