Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Go package: fanout – make writing parallel code even easier

github.com

Post reply on HN
Print view View original

11–20 of 26 posts

Re: Go package: fanout – make writing parallel code even easier

#11

Post by drtse4 »

sunfmin wrote: post #10
drtse4 wrote: post #8

>How does it make the program run faster? Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with 20 cores (ignoring the limited overhead), regardless of the number of workers your queue of jobs will be consumed in number_of_jobs*single_job_duration/GOMAXPROCS.

Thanks, but there will be I/O waiting stuff right? For that whois command example, I find the CPU usage is zero if I am not running it parallel.

Yup, if your jobs are i/o heavy the cpu time is wasted waiting if you execute them in a strictly sequential way. With your worker pool the cpu intensive part of the jobs is instead being executed concurrently GOMAXPROCS jobs at a time.

Re: Go package: fanout – make writing parallel code even easier

#12

Post by goykasi »

I wrote a similar Go package for running work loads in parallel, but I used beanstalkd for job/result transport. This allows me a bit more freedom to spread the workers/requesters across my network. It's a bit rough around the edges and could use some refactoring, but it works well for my uses.

https://github.com/eramus/worker

Re: Go package: fanout – make writing parallel code even easier

#14

Post by sunfmin »

goykasi wrote: post #12

I wrote a similar Go package for running work loads in parallel, but I used beanstalkd for job/result transport. This allows me a bit more freedom to spread the workers/requesters across my network. It's a bit rough around the edges and could use some refactoring, but it works well for my uses. https://github.com/eramus/worker

Bookmarked and Stared!

Re: Go package: fanout – make writing parallel code even easier

#15

Post by aikah »

latch wrote: post #4
StavrosK wrote: post #2

I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?

it's a small abstraction, but I could see it being useful. My gripe, and I don't know how to say this without people saying I'm whining about generics, is the interface{} return. Such a pain.

Go has generics,you just can't define new ones,which is the heart of the problem.

It's not about whinning.There is a need for that yet no real will to implement user defined one.

Re: Go package: fanout – make writing parallel code even easier

#16

Post by coldtea »

TheDong wrote: post #7

Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.

>Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code.

Or actually, it's not a generic (as in generics) package, which is why you lose type safety.

Re: Go package: fanout – make writing parallel code even easier

#19

Post by chimeracoder »

sunfmin wrote: post #9
TheDong wrote: post #7

Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.

Yes, I often find myself lost in this trivial channels + goroutines + WaitGroup combination. For my self I'd prefer cast interface{}, :-)

interface{} is different from []interface{}.

You can do a type assertion (not cast - these are different concepts) from interface{} to another type. You cannot do a type assertion on []interface{}; you have to iterate over each element and assert individually.

I agree with GP - the built-in primitives for synchronization, along with sync from the stdlib, are much cleaner once you know how to use them, and they're more idiomatic.

Re: Go package: fanout – make writing parallel code even easier

#20

Post by chimeracoder »

coldtea wrote: post #16
TheDong wrote: post #7

Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.

> Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Or actually, it's not a generic (as in generics) package, which is why you lose type safety.

> Or actually, it's not a generic (as in generics) package, which is why you lose type safety.

Can we please skip just one opportunity to begin this endless flamewar on a HN post about Go? Just one? Please?

This topic has been beaten to death and beyond, and is not relevant to TFA at all, as the word "generic" can be used in many contexts, and it's more than clear which context OP meant.

Post reply on HN