It would be nice if the author also compared different runtimes (e.g. NodeJS vs Deno, or cpython vs pypy) and core language engines (e.g. v8 vs spider monkey vs JavaScript core)
How much memory do you need in 2024 to run 1M concurrent tasks?
31–40 of 205 posts
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#32depends on the tasks.
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#33I feel this benchmark compares apples to oranges in some cases. For example, for node, the author puts a million promises into the runtime event loop and uses `Promise.all` to wait for them all. This is very different from, say, the Go version where the author creates a million goroutines and puts `waitgroup.Done` as a defer call. While this might be the idiomatic way of concurrency in the respective languages, it do…
The requirement is to run 1 million concurrent tasks.
Of course each language will have a different way of achieving this task each of which will have their unique pros/cons. That's why we have these different languages to begin with.
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#34No baseline against UNIX processes?
A million processes?
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#35The C# version will copy the list into an array during Task.WhenAll, it may save some memory to use an array directly. Souce: https://github.com/microsoft/referencesource/blob/master/msc...
It doesn't take that much space, and not all languages have option to easily map an initial range onto an iterator that produces tasks. Most are dominated by the size of state machines/virtual threads.
Please note that the link above leads to old code from .NET Framework.
The up-to-date .NET implementation lives here: https://github.com/dotnet/runtime/blob/main/src/libraries/Sy...
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#36This benchmark is nonsense. Apart from the fact that Go has an average Goroutine overhead of a 4kB stack (meaning an average usage of 3.9GB for 1M tasks), the code written is also in a closure, and scheduling a 2nd Goroutine in the wg.Done(), so unlike some of the others it had at least 2M function calls on the event loop stack in addition to at least 1M closure references. So yeah, it’s a great example of bad code in any language.
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#37The C# version will copy the list into an array during Task.WhenAll, it may save some memory to use an array directly. Souce: https://github.com/microsoft/referencesource/blob/master/msc...
The referencesource repository is only relevant if you are using the legacy .NET Framework. Modern .NET has a special case for passing a List and avoids the allocation:
https://github.com/dotnet/runtime/blob/1f01bee2a41e0df97089f...
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#38This depends a lot on how you define "concurrent tasks", but the article provides a definition: Let's launch N concurrent tasks, where each task waits for 10 seconds and then the program exists after all tasks finish. The number of tasks is controlled by the command line argument. Leaving aside semantics like "since the tasks aren't specified as doing anything with side effects, the compiler can remove them as dead c…
> I have no idea what Go is doing to get 2500 bytes per task.
TFA creates a goroutine (green thread) for each task (using a waitgroup to synchronise them). IIRC goroutines default to 2k stacks, so that’s about right.
One could argue it’s not fair and it should be timers which would be much lighter. There’s no “efficient wait” for them but that’s essentially the same as the appendix rust program.
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#39The C# version will copy the list into an array during Task.WhenAll, it may save some memory to use an array directly. Souce: https://github.com/microsoft/referencesource/blob/master/msc...
[deleted]
Re: How much memory do you need in 2024 to run 1M concurrent tasks?
#40This benchmark is nonsense. Apart from the fact that Go has an average Goroutine overhead of a 4kB stack (meaning an average usage of 3.9GB for 1M tasks), the code written is also in a closure, and scheduling a 2nd Goroutine in the wg.Done(), so unlike some of the others it had at least 2M function calls on the event loop stack in addition to at least 1M closure references. So yeah, it’s a great example of bad code i…
Here's an implementation in C# that more faithfully matches what you have to do in Go:
var cnt = int.Parse(args[0]);
var evt = new CountdownEvent(cnt);
for (var i = 0; i