Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much memory do you need in 2024 to run 1M concurrent tasks?

hez2010.github.io

Post reply on HN
Print view View original

21–30 of 205 posts

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#22

Post by SPascareli13 »

AkshitGarg wrote: post #9

I feel this benchmark compares apples to oranges in some cases. For example, for node, the author puts a million promises into the runtime event loop and uses `Promise.all` to wait for them all. This is very different from, say, the Go version where the author creates a million goroutines and puts `waitgroup.Done` as a defer call. While this might be the idiomatic way of concurrency in the respective languages, it do…

As far as I know there is no way to do Promise like async in go, you HAVE to create a goroutine for each concurrent async task. If this is really the case then I believe the submition is valid.

But I do think that spawning a goroutine just to do a non-blocking task and get its return is kinda wasteful.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#23

Post by dboreham »

erik_seaberg wrote: post #12
ankit70 wrote: post #8

NodeJS is better at memory than go?

I'd expect that because Promises are small Javascript objects while goroutines each get a stack that grows from at least 2 KB.

Otoh Go actually supports concurrency.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#24

Post by jpgvm »

dboreham wrote: post #23
erik_seaberg wrote: post #12

Earlier quoted context omitted.

I'd expect that because Promises are small Javascript objects while goroutines each get a stack that grows from at least 2 KB.

Otoh Go actually supports concurrency.

Well they are all concurrent. I think what you mean is Go is also parallel. As is C#, Rust and Java in this bench.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#26

Post by jpgvm »

fuzzybear3965 wrote: post #16
amazingamazing wrote: post #2

Yet again nodejs surpasses my pre-read expectations 3rd best (generalized) for a million? Wow. I must be missing something - isn’t Go supposed to be memory efficient? Perhaps promises and goroutines aren’t comparable?

I'm not sure what "memory efficient" means. But, Go sprung as a competitor to Java (portability, language stability, corporate language support/development) and C++ (faster compile times). Can't beat C++ in terms of memory management (performance, guys, not safety) much. But, you can fare well against the JVM, I'm guessing.

In this benchmark actually no, Go doesn't fare well. There is actually higher static overhead per goroutine than JVM VirtualThread. I presume this is because of a larger initial stack size though/

This probably doesn't matter in the real world as you will actually use the tasks to do some real work which should really dwarf the static overhead is almost all cases.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#28

Post by neonsunset »

To add a data point for Elixir: https://gist.github.com/neon-sunset/8fcc31d6853ebcde3b45dc7a...

Note 1: The gist is in Ukrainian, and the blog post by Steve does a much better job, but hopefully you will find this useful. Feel free to replicate the results and post them.

Note 2: The absolute numbers do not necessarily imply good/bad. Both Go and BEAM focus on userspace scheduling and its fairness. Stackful coroutines have their own advantages. I think where the blog post's data is most relevant is understanding the advantages of stackless coroutines when it comes to "highly granular" concurrency - dispatching concurrent requests, fanning out to process many small items at once, etc. In any case, I did not expect sibling comments to go onto praising Node.js, is it really that surprising for event loop based concurrency? :)

Also, if you are an Elixir aficionado and were impressed by C#'s numbers - know that they translate ~1:1 to F# now that it has task CE, just sayin'.

Here's how the program looks in F#:

    open System
    open System.Threading.Tasks

    let argv = Environment.GetCommandLineArgs()

    [1..int argv[1]]
    |> Seq.map (fun _ -> Task.Delay(TimeSpan.FromSeconds 10.0))
    |> Task.WaitAll
Post reply on HN