hide goroutines and channels from developer.
Go package: fanout – make writing parallel code even easier
github.com
Go package: fanout – make writing parallel code even easier
1–10 of 26 posts
Re: Go package: fanout – make writing parallel code even easier
#2I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?
Re: Go package: fanout – make writing parallel code even easier
#3I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?
I guess it is a parallel queue, It saves the trouble to worry about goroutines and channels and synchronizations, encapsulated the logic that talked in http://blog.golang.org/pipelines last part. To make use of that pattern easier.
I have quite a few use cases myself:
1. Say there is a article on the web, that contains dozens of images embed in the html. Which I want to crawl the article, and Also the images, and put the image into S3 for faster serve. I can use the fanout concurrency pattern to put upload all the images simultaneously to S3 to make it faster.
2. Say I have 20 mysql node sharded data, I want to fetch top 50 latest created data from all of them, and sort it to and show it to user after.
Re: Go package: fanout – make writing parallel code even easier
#4I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?
it's a small abstraction, but I could see it being useful. My gripe, and I don't know how to say this without people saying I'm whining about generics, is the interface{} return. Such a pain.
Re: Go package: fanout – make writing parallel code even easier
#5Why does this need to be a dependency that you bring on? This is a trivial pattern.
Re: Go package: fanout – make writing parallel code even easier
#6I'm not sure what the usefulness of this is (apart from saving you from writing one or two functions yourself), isn't it just a parallel queue?
I guess it is a parallel queue, It saves the trouble to worry about goroutines and channels and synchronizations, encapsulated the logic that talked in http://blog.golang.org/pipelines last part. To make use of that pattern easier. I have quite a few use cases myself: 1. Say there is a article on the web, that contains dozens of images embed in the html. Which I want to crawl the article, and Also the images, and put…
Oh, yes, there are lots of use cases for parallel queues, for sure.
Re: Go package: fanout – make writing parallel code even easier
#7Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code.
Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.
Re: Go package: fanout – make writing parallel code even easier
#8>How does it make the program run faster?
Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with 20 cores (ignoring the limited overhead), regardless of the number of workers your queue of jobs will be consumed in number_of_jobs*single_job_duration/GOMAXPROCS.
Re: Go package: fanout – make writing parallel code even easier
#9Oh, hey, it's a generic package that means you lose type safety and saves you from writing roughly 10 lines of code. Buffered channels + goroutines + WaitGroup already allow you to implement this trivially, and because channels are builtin generic, you can do it without the nasty type casts. Really, []interface{} is a terrible type to work with.
Yes, I often find myself lost in this trivial channels + goroutines + WaitGroup combination. For my self I'd prefer cast interface{}, :-)
Re: Go package: fanout – make writing parallel code even easier
#10>How does it make the program run faster? Sunfmin, wrong answer, you'd have a 20x improvement with 20 workers only with 20 cores (ignoring the limited overhead), regardless of the number of workers your queue of jobs will be consumed in number_of_jobs*single_job_duration/GOMAXPROCS.
Thanks, but there will be I/O waiting stuff right? For that whois command example, I find the CPU usage is zero if I am not running it parallel.