Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How much memory do you need in 2024 to run 1M concurrent tasks?

hez2010.github.io

Post reply on HN
Print view View original

101–110 of 205 posts

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#101

Post by flockonus »

jakewins wrote: post #66
masklinn wrote: post #59

Earlier quoted context omitted.

Go is absolutely using that ram, it’s not available to other services on the system.

You know what I mean. If this was a real world program where those million tasks actually performed work, then this stack space is available for the application to do that work. It’s not memory that’s consumed by the runtime, it’s memory the runtime expects the program to use - it’s just that this program does no useful work.

I'll yield it would be interesting to have a similar benchmark but instead of sleeping - which indeed by itself is nonsense, to instead each task compute a small fib sequence, or write a small file; something like that.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#102

Post by pbhjpbhj »

Quothling wrote: post #73
Aeolun wrote: post #69

Earlier quoted context omitted.

Is that the ideomatic way to do it, or the best way you can imagine?

> Is that the ideomatic way to do it Well... I'm actually not sure what ideomatic means (English isn't my first language), but it's the standard way of doing it. You'll even find it as step 2 and 3 here: https://go.dev/tour/concurrency/1 > or the best way you can imagine I would do a lot much more to tune it if you were in a position where you'd know it would run that many "tasks". I think what many non-Go programmer…

Idiomatic is the word the parent was looking for. The base word is idiom.

It was probably the intent of the parent to mean 'making use of the particular features of the language that are not necessarily common to other languages'.

I'm not a programmer, but you appear to give good examples.

I hope I'm not teaching you to suck eggs... {That's an idiom, meaning teaching someone something they're already expert in. Like teaching your Grandma to suck eggs - which weirdly means blowing out the insides of a raw egg. That's done when using the egg to paint; which is a traditional Easter craft.}

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#105

Post by pjmlp »

mrweasel wrote: post #98
pjmlp wrote: post #97

While the idea in general is interesting, the whole site is unreadable in Firefox, text gets all over the place the graphics.

Seems fine to me.

Strange, killing Firefox and going again did it for me.

I guess some FF bug then.

Thanks.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#106

Post by rtpg »

hamandcheese wrote: post #95
rtpg wrote: post #90

Earlier quoted context omitted.

But you wouldn't call a million tasks with `Promise.all` in Node, right? That's just not a thing that one does. Instead, there's usually going to be some queue outside the VM that will leave you with _some_ sort of chunking and otherwise working in smaller, more manageable bits (that might, incidentally, be shaped in ways that the VM can handle in interesting ways). It's definitely true to say that the "idioamatic" w…

> But you wouldn't call a million tasks with `Promise.all` in Node, right? That's just not a thing that one does. But neither would you wait on a waitgroup of size 1 million in Go... right?

yeah, right? I mean I don't have a dog in this race, just wished we could get into "normal" repros without having to wonder if some magic is kicking in

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#107

Post by Quothling »

guitarbill wrote: post #83
Quothling wrote: post #73

Earlier quoted context omitted.

> Is that the ideomatic way to do it Well... I'm actually not sure what ideomatic means (English isn't my first language), but it's the standard way of doing it. You'll even find it as step 2 and 3 here: https://go.dev/tour/concurrency/1 > or the best way you can imagine I would do a lot much more to tune it if you were in a position where you'd know it would run that many "tasks". I think what many non-Go programmer…

The post was edited, previously it just said roughly this part: "step 2 and 3 here: https://go.dev/tour/concurrency/1 ". Which - as far as I can tell - does not mention worker pools...

You're right. It is using channels and buffers, but you're right.

It's not part of the actual documentation either, at least not exactly: https://go.dev/doc/effective_go#concurrency You will achieve much the same if you follow it, but my answer should have been yes and no as far as being the "standard" Go way.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#108

Post by bekantan »

aba_cz wrote: post #70

Regarding Java I'm pretty sure that benchmark is broken at least a little bit and testing something else as not specifying initial size for ArrayList means list of size 10 which gets resized all the time when `add()` is called, leading to big amount of unused objects needing garbage collection.

It would indeed be better to create appropriately sized storage.

However, I don't think that underlying array is resized every time `add` is called. I'd expect that resize will happen less than 30 times for 1M adds (capacity grows geometrically with a=10 and r=1.5)

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#109

Post by sfn42 »

brabel wrote: post #89
aba_cz wrote: post #70

Regarding Java I'm pretty sure that benchmark is broken at least a little bit and testing something else as not specifying initial size for ArrayList means list of size 10 which gets resized all the time when `add()` is called, leading to big amount of unused objects needing garbage collection.

Yeah that is a junior mistake... They should've pre-sized the ArrayList, or better, used an array because that's more memory efficient (and I would say would be what any decent dev would do when the size of tasks is known beforehand). > Some folks pointed out that in Rust (tokio) it can use a loop iterating over the Vec instead of join_all to avoid the resize to the list Right, but some folks also pointed out you sho…

The difference between an arraylist with correct initial size and an array is almost nothing. Arraylist itself is just a wrapper around an array.

Re: How much memory do you need in 2024 to run 1M concurrent tasks?

#110

Post by Izkata »

liontwist wrote: post #42
masklinn wrote: post #34

Earlier quoted context omitted.

A million processes?

I've seen 100k. What happens at a million? How many is unworkable?

  $ cat /proc/sys/kernel/pid_max 
  4194304
My computer can handle that many processes, after that no new processes can be spawned (see: forkbomb)
Post reply on HN