Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread pools: How do I use them?

jvns.ca

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Thread pools: How do I use them?

#11

Post by barrkel »

throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

If you, like, don't mind the, like, valley/bro/speak lingo. AWESOME, right?

I like structured arguments against a status quo - that's what's really required here. This article is one notch earlier in the mental digestion chain; a kind of experiential report, an extended anecdote. It's harder to tell if the problem is the library or the user. It asks the reader to work more, because they need to go on a journey, rather than just parse an argument.

I enjoy reported experience too, but I have less time for them unless they're specific to something I have a strong interest in. Otherwise, give me the executive summary.

Yes, I read less of the internet than I used to.

Re: Thread pools: How do I use them?

#12

Post by abalashov »

throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Yes, the narrative form gives the impression that the author is a very! excited! person! full! of! energy!

But that's kind of refreshing sometimes. It's easy to get cynical about software engineering like I am. And I followed her stream of consciousness a lot better than I follow a lot of recondite technical prose, particularly when it concerns things mired in proprietary abstractions (Java).

Re: Thread pools: How do I use them?

#13

Post by delroth »

bpicolo wrote: post #9

> Here's what that looks like in Python. Although python multithreading outside of IO tasks or dropping into C code that drops the GIL isn't going to parallelize, the latter of which is the only way you're going to burn up cores. The API does make it seem simple though.

The example is using multiprocessing.pool.ThreadPool which does not have the GIL problem you mention (at the cost of making it harder to share data).

Re: Thread pools: How do I use them?

#14

Post by tatterdemalion »

Smircio- wrote: post #5
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

A lot of people have discovered that speaking confidently can be a solid substitute for actual expertise.

Re: Thread pools: How do I use them?

#15

Post by metachris »

I recently ran into the issue that the multiprocessing.pool.ThreadPool class is not cancellable in Python 2. It seems that only in Python 3 can you exit a ThreadPool with eg. a KeyboardInterrupt before all tasks are finished.

Since then I've been using this code, which works great across all platforms and Python versions: https://github.com/metachris/pdfx/blob/master/pdfx/threadpoo...

Re: Thread pools: How do I use them?

#16

Post by someguy1234 »

delroth wrote: post #13
bpicolo wrote: post #9

> Here's what that looks like in Python. Although python multithreading outside of IO tasks or dropping into C code that drops the GIL isn't going to parallelize, the latter of which is the only way you're going to burn up cores. The API does make it seem simple though.

The example is using multiprocessing.pool.ThreadPool which does not have the GIL problem you mention (at the cost of making it harder to share data).

ThreadPool would lock the GIL, wouldn't it? I think multiprocessing.Pool uses processes and that's where you can avoid the GIL (each process gets its own) at the cost of making data sharing harder.

Even with the GIL, ThreadPools are still fine performance-wise for tasks that release the GIL such as I/O or a number of functions in numpy and scipy.

Re: Thread pools: How do I use them?

#17

Post by allcentury »

throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

According to her bio, she works in Stripe's machine learning department so clearly a smart gal and appears to have a great amount of humility - enjoyed her post, was easy to digest and a light read.

Re: Thread pools: How do I use them?

#18

Post by charlesism »

radicalbyte wrote: post #10
Smircio- wrote: post #5

Earlier quoted context omitted.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

It's not that, it's just that experts tend to contribute to threads where they can share their expertise. So while it appears that everyone here is amazing; that isn't the case: we just tend to have something that we're very good at and comment on it. The good thing is that you can learn a lot here as long as you don't get sidelined by some of the inane arguments and fighting :)

Also, an awful lot of people who sound like infallible "experts" in tech are just people who are dealing poorly with imposter syndrome.

Re: Thread pools: How do I use them?

#19

Post by charlesism »

tatterdemalion wrote: post #14
Smircio- wrote: post #5

Earlier quoted context omitted.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

A lot of people have discovered that speaking confidently can be a solid substitute for actual expertise.

They're called "consultants"

Re: Thread pools: How do I use them?

#20

Post by charlesism »

barrkel wrote: post #11
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

If you, like, don't mind the, like, valley/bro/speak lingo. AWESOME, right? I like structured arguments against a status quo - that's what's really required here. This article is one notch earlier in the mental digestion chain; a kind of experiential report, an extended anecdote. It's harder to tell if the problem is the library or the user. It asks the reader to work more, because they need to go on a journey, rathe…

I've read a lot worse on that score. At least the author didn't parrot a bunch of memes, or throw animated gifs everywhere.
Post reply on HN