Good write up.
Thread pools: How do I use them?
41–48 of 48 posts
Re: Thread pools: How do I use them?
#42As is sometimes the case, people start with the wrong abstraction (thread pool) and then optimize it.
Java/Scala streams look like they would solve most of the problems in the post on their own.
e.g. Java streams
http://radar.oreilly.com/2015/02/java-8-streams-api-and-para...
private void runParallel() {
trades
.stream()
.parallel().forEach(t->doSomething(t) );
}
Streams, Parallel collections, Futures, Actors, Fork/Join, ExecutionContext/ExecutorService, Threadpools, Threads, synchronized, Barriers, Atomic values and many more.
Re: Thread pools: How do I use them?
#43As is sometimes the case, people start with the wrong abstraction (thread pool) and then optimize it. Java/Scala streams look like they would solve most of the problems in the post on their own. e.g. Java streams http://radar.oreilly.com/2015/02/java-8-streams-api-and-para... private void runParallel() { trades .stream() .parallel().forEach(t->doSomething(t) ); } Java/Scala has a hierarchy of concurrency abstractions…
Something not so obvious for beginners: Java/Scala does not use all the available memory. If you get out of memory problems with the 10gb data set and you have much more RAM available, you need to tell Java/Scala to increase the Heap.
Re: Thread pools: How do I use them?
#44Earlier quoted context omitted.
It's not that, it's just that experts tend to contribute to threads where they can share their expertise. So while it appears that everyone here is amazing; that isn't the case: we just tend to have something that we're very good at and comment on it. The good thing is that you can learn a lot here as long as you don't get sidelined by some of the inane arguments and fighting :)
Also, an awful lot of people who sound like infallible "experts" in tech are just people who are dealing poorly with imposter syndrome.
Hang on; Imposter Syndrome would mean that you would come across as a fallible, non-expert, since you believe yourself to really be an imposter. Do you really mean something like the Dunning-Kruger effect, which is sort of the opposite of Imposter Syndrome, people with little knowledge believing themselves to be infallible experts?
Re: Thread pools: How do I use them?
#45Earlier quoted context omitted.
Also, an awful lot of people who sound like infallible "experts" in tech are just people who are dealing poorly with imposter syndrome.
Hang on; Imposter Syndrome would mean that you would come across as a fallible, non-expert, since you believe yourself to really be an imposter. Do you really mean something like the Dunning-Kruger effect, which is sort of the opposite of Imposter Syndrome, people with little knowledge believing themselves to be infallible experts?
Impostor syndrome means that the person thinks of themselves as a fraud, not that other people do. So someone dealing poorly with impostor syndrome might well only chime in when they are exceedingly confident, or only when they do research and can be certain that the information/position they are providing is correct. If you keep your mouth shut except when you're 100% certain, you'll likely look like an "infallible expert" because you're always correct from an external perspective, even if you are quietly filled with self-doubt.
Re: Thread pools: How do I use them?
#46Earlier quoted context omitted.
Hang on; Imposter Syndrome would mean that you would come across as a fallible, non-expert, since you believe yourself to really be an imposter. Do you really mean something like the Dunning-Kruger effect, which is sort of the opposite of Imposter Syndrome, people with little knowledge believing themselves to be infallible experts?
Impostor syndrome means that the person thinks of themselves as a fraud, not that other people do. So someone dealing poorly with impostor syndrome might well only chime in when they are exceedingly confident, or only when they do research and can be certain that the information/position they are providing is correct. If you keep your mouth shut except when you're 100% certain, you'll likely look like an "infallible…
Pretty much. Some people who lack self-confidence seem afraid to ever admit "I don't know" lest it somehow reveal their "fraud" to the world.
Re: Thread pools: How do I use them?
#47I like the style of writing in this post. Its fresh in the programming world where everyone seems to know it all for someone to admit something is hard or confusing.
While i admit it was a thrilling take on a concept as abstract as thread-pools, i think the issue lies in the nature in which we programmers talk about computing theory.90% of the time we are either telling (stack-overflow help, or receiving (asking for help on stack overflow) which requires us either to be the teacher or the learner, never in between like this author is.
Re: Thread pools: How do I use them?
#48> In my case, I was reading a bunch of data off disk. maybe 10GB of data. And I was submitting all of that data into the ExecutorService work queue. Unsurprisingly, the queue exploded and crashed my program.
In the spirit of brainstorming:
1) Abstract out the I/O step so that you can pass around a bunch of lightweight and lazy "I'll get the lines for you when you ask" objects. Then you can safely queue up a large number of them.
2) Choose to break the "job" apart into two parts, and submit the "get lines from filename" jobs into a different, smaller pool, with the results flowing into the larger (CPU-core-limited) pool.