Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread pools: How do I use them?

jvns.ca

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Thread pools: How do I use them?

#21

Post by emmelaich »

Great writing.

I wonder if a ForkJoinPool [1] would be helpful here as it does work-stealing giving better utilisation.

Although I've read admonitions that they should not be used when doing I/O. [2] That article seems a little ranty and overblown though, so I'd like opinions on that.

1. https://docs.oracle.com/javase/8/docs/api/java/util/concurre... 2. http://coopsoft.com/ar/CalamityArticle.html

Re: Thread pools: How do I use them?

#23

Post by ambicapter »

Pirate-of-SV wrote: post #2

> Or maybe there is a totally simple way and this could take me 5 minutes! At that paragraph I immediately thought of http://aadrake.com/command-line-tools-can-be-235x-faster-tha...

Holy shit.

> I have been in technology roles for over 17 years

I guess this is what experience gets you, huh.

Re: Thread pools: How do I use them?

#24

Post by chongli »

Smircio- wrote: post #5
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

This is how everybody feels at one point or another. It's called imposter syndrome[0] and it's extremely common!

[0] https://en.wikipedia.org/wiki/Impostor_syndrome

Re: Thread pools: How do I use them?

#25

Post by aftbit »

delroth wrote: post #13
bpicolo wrote: post #9

> Here's what that looks like in Python. Although python multithreading outside of IO tasks or dropping into C code that drops the GIL isn't going to parallelize, the latter of which is the only way you're going to burn up cores. The API does make it seem simple though.

The example is using multiprocessing.pool.ThreadPool which does not have the GIL problem you mention (at the cost of making it harder to share data).

You're thinking multiprocessing.Pool. I think ThreadPool actually uses threads and does hit the GIL.

Re: Thread pools: How do I use them?

#26

Post by cmsd2 »

emmelaich wrote: post #21

Great writing. I wonder if a ForkJoinPool [1] would be helpful here as it does work-stealing giving better utilisation. Although I've read admonitions that they should not be used when doing I/O. [2] That article seems a little ranty and overblown though, so I'd like opinions on that. 1. https://docs.oracle.com/javase/8/docs/api/java/util/concurre... 2. http://coopsoft.com/ar/CalamityArticle.html

A coworker showed me Akka Streams recently. It allows you to overcome the impedence mismatch between threadpools processing jobs at different speeds where one task feeds into another. Could be a neat way to join up the i/o bound tasks to the cpu bound tasks.

https://opencredo.com/introduction-to-akka-streams-getting-s...

Re: Thread pools: How do I use them?

#27

Post by teh_klev »

Smircio- wrote: post #5
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

Don't be intimidated. I've got 30 years of being in the business of software development, telecoms, networking etc under my belt and every day there's still something new to learn, sometimes too many new things :) Also don't be frightened to ask about things you don't understand, I find the folks around here are generally quite an affable bunch.

Re: Thread pools: How do I use them?

#28

Post by megamouse »

metachris wrote: post #15

I recently ran into the issue that the multiprocessing.pool.ThreadPool class is not cancellable in Python 2. It seems that only in Python 3 can you exit a ThreadPool with eg. a KeyboardInterrupt before all tasks are finished. Since then I've been using this code, which works great across all platforms and Python versions: https://github.com/metachris/pdfx/blob/master/pdfx/threadpoo...

I've experienced this bug in Python 2 using "pool.map" and "pool.apply", but KeyboardInterrupt does work if you're waiting for a finite amount of time. So as a workaround you can use "result = pool.map_async(...); result.get(sys.maxint)". A little hacky but functional.

Re: Thread pools: How do I use them?

#29

Post by bronxbomber92 »

ambicapter wrote: post #23
Pirate-of-SV wrote: post #2

> Or maybe there is a totally simple way and this could take me 5 minutes! At that paragraph I immediately thought of http://aadrake.com/command-line-tools-can-be-235x-faster-tha...

Holy shit. > I have been in technology roles for over 17 years I guess this is what experience gets you, huh.

Or if you know some basic performance characteristics of your machine (i.e. memory bandwidth + processor speed + number of cores) and roughly how much work you need to do (i.e. number cycles to do X), some back of the envelope calculations can often give you the same insight (i.e. "is X feasible with compute resource Y?").

Re: Thread pools: How do I use them?

#30

Post by lambda »

Smircio- wrote: post #5
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

Heh. Keep in mind that people are going to be more likely to comment on something which they have knowledge of or an opinion on, so you will see a certain amount of selection bias in that most of the people who comment on something know something about it, even if there are large numbers of people reading and interested who are just keeping quiet (though of course there are people who also make comments on things they know little about).

Also, I know that for some things that I write about, I'm not actually an expert on it beforehand. Instead, I see an interesting question, say to myself "hmm, that's interesting, and I don't actually know the answer", go out to Google, books I own, or just try some experimentation out myself, and then comment on what I found. I usually do try to provide sources and some details about what I did when doing this, but I realize that it may sometimes make me look like more of an expert on topics than I actually am. For example, I have a silver badge in JavaScript on StackOverflow, even though I've only ever written a few dozen lines of JavaSript in my life (in fact, I've probably written more lines of it on StackOverflow than off), since I tend to be good at quickly researching and experimenting with a question and providing a good explanation, not because I'm an expert on JavaScript.

So yeah, it's great the Julia writes down her thought and search process while doing this, as it helps people realize that they too can go through the same process and learn new things.

Post reply on HN