Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thread pools: How do I use them?

jvns.ca

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Thread pools: How do I use them?

#31

Post by DigitalJack »

teh_klev wrote: post #27
Smircio- wrote: post #5

Earlier quoted context omitted.

Heck yea. Im kind of new here and it can be so intimidating. I feel like everyone is an expert at everything.

Don't be intimidated. I've got 30 years of being in the business of software development, telecoms, networking etc under my belt and every day there's still something new to learn, sometimes too many new things :) Also don't be frightened to ask about things you don't understand, I find the folks around here are generally quite an affable bunch.

The trick is figuring out what you don't know to ask the question in the first place. Often not trivial.

It helps to try to tackle a project that you don't already know how to accomplish. Then the Q&A can work like a reverse dictionary. You know the answer, and work backwards to find the question. Every stumbling block is a question to ask.

Re: Thread pools: How do I use them?

#32

Post by maherbeg »

emmelaich wrote: post #21

Great writing. I wonder if a ForkJoinPool [1] would be helpful here as it does work-stealing giving better utilisation. Although I've read admonitions that they should not be used when doing I/O. [2] That article seems a little ranty and overblown though, so I'd like opinions on that. 1. https://docs.oracle.com/javase/8/docs/api/java/util/concurre... 2. http://coopsoft.com/ar/CalamityArticle.html

The article is definitely super ranty and a bit overblown. The Fork/Join framework is actually pretty solid and works for a wide variety of use cases and has a relatively simple API. It's an excellent article though and I highly recommend anyone implementing the use of Fork/Join to read it and get more in depth knowledge.

Java 8 made some improvements in code quality and safety and the framework underlies all of the parallel streaming abstractions used going forward.

It's definitely worth learning all of the concurrency abstractions available in the JDK too though!

Re: Thread pools: How do I use them?

#33

Post by hibikir »

abalashov wrote: post #12
throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

Yes, the narrative form gives the impression that the author is a very! excited! person! full! of! energy! But that's kind of refreshing sometimes. It's easy to get cynical about software engineering like I am. And I followed her stream of consciousness a lot better than I follow a lot of recondite technical prose, particularly when it concerns things mired in proprietary abstractions (Java).

And if you had actually met Julia (I am lucky enough to work with her) you would realize that this impression is 100% correct. It is not just an impression, but her whole approach to technology.

While it is good to have more cynical people too (especially for things like software stability), a company that forgets enthusiasm ends up losing its soul.

Re: Thread pools: How do I use them?

#34

Post by kevinavery »

cmsd2 wrote: post #26
emmelaich wrote: post #21

Great writing. I wonder if a ForkJoinPool [1] would be helpful here as it does work-stealing giving better utilisation. Although I've read admonitions that they should not be used when doing I/O. [2] That article seems a little ranty and overblown though, so I'd like opinions on that. 1. https://docs.oracle.com/javase/8/docs/api/java/util/concurre... 2. http://coopsoft.com/ar/CalamityArticle.html

A coworker showed me Akka Streams recently. It allows you to overcome the impedence mismatch between threadpools processing jobs at different speeds where one task feeds into another. Could be a neat way to join up the i/o bound tasks to the cpu bound tasks. https://opencredo.com/introduction-to-akka-streams-getting-s...

Yep, it's a really cool library. I'm pretty new to it myself, but I made a little gist of how one might approach this problem: https://gist.github.com/kevinavery/941e7d67c4f8b104f610

I was kinda surprised by how tricky it was to make a Flow that turned ByteStrings into separate String lines, but it seems like the custom GraphStage approach generalizes really well to more complicated stages.

Re: Thread pools: How do I use them?

#35

Post by whalesalad »

My favorite blogger of 2016 thus far. Every post is great.

Also the bit that rings home the strongest is just how difficult it is to do trivial things. Even a senior level programmer with ten years of experience has to sit and say, why is this such a pain in the ass?

I guess that's one reason I really like clojure. Core.async has a really hideous API at first glance but using channels and buffers makes doing this sort of thing a lot more enjoyable. Anyway the intent of the article was to dive into abstractions not to suggest more of them so I'll shush.

Re: Thread pools: How do I use them?

#36

Post by savanaly »

hibikir wrote: post #33
abalashov wrote: post #12

Earlier quoted context omitted.

Yes, the narrative form gives the impression that the author is a very! excited! person! full! of! energy! But that's kind of refreshing sometimes. It's easy to get cynical about software engineering like I am. And I followed her stream of consciousness a lot better than I follow a lot of recondite technical prose, particularly when it concerns things mired in proprietary abstractions (Java).

And if you had actually met Julia (I am lucky enough to work with her) you would realize that this impression is 100% correct. It is not just an impression, but her whole approach to technology. While it is good to have more cynical people too (especially for things like software stability), a company that forgets enthusiasm ends up losing its soul.

I think a perfect team would be one Julia, three or four cynical types, and maybe one fresh faced newbie to ask the dumb questions that nevertheless sometimes lead to something innovative.

Re: Thread pools: How do I use them?

#38

Post by wsargent »

cmsd2 wrote: post #26
emmelaich wrote: post #21

Great writing. I wonder if a ForkJoinPool [1] would be helpful here as it does work-stealing giving better utilisation. Although I've read admonitions that they should not be used when doing I/O. [2] That article seems a little ranty and overblown though, so I'd like opinions on that. 1. https://docs.oracle.com/javase/8/docs/api/java/util/concurre... 2. http://coopsoft.com/ar/CalamityArticle.html

A coworker showed me Akka Streams recently. It allows you to overcome the impedence mismatch between threadpools processing jobs at different speeds where one task feeds into another. Could be a neat way to join up the i/o bound tasks to the cpu bound tasks. https://opencredo.com/introduction-to-akka-streams-getting-s...

https://rnduja.github.io/2016/03/25/a_first_look_to_akka_str... is also pretty good.

Re: Thread pools: How do I use them?

#39

Post by wsargent »

The easiest way to use thread pools in Scala is to use an ExecutionContext. Akka will let you define an ExecutionContext from a configuration file using a wrapper called a Dispatcher -- you can ramp up the parallelism-max setting for more cores. Then you do actorSystem.dispatchers.lookup("my-dispatcher") and pass it to all the actor based code that you are using.

You can use a work pulling set of actors to handle computation after that, or use Akka Streams to provide backpressure, to avoid the OOM problem.

Re: Thread pools: How do I use them?

#40

Post by xerophyte12932 »

throweway wrote: post #3

I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.

I have been following her for well over an year now for exactly this reason. She never preaches. It's more like taking her readers along on this journey of discovery. Very very humble and yet insightful
Post reply on HN