Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Playing with Go: Embarrassingly Parallel Scripts

collectiveidea.com

Post reply on HN
Print view View original

21–30 of 53 posts

Re: Playing with Go: Embarrassingly Parallel Scripts

#21

Post by tptacek »

jnazario wrote: post #19

in the DNS case asynchronous event handling would be super easy to do. in python asyncore with something like dpkt to construct and read DNS lookups works like a champ, as does twisted. i did a simple async DNS resolver in pure python (asyncore, dpkt) and can sustain thousands of lookups a second. GNU adns also has bindings in various languages. you can get Go's parallelisms via CSP (e.g. python-csp, ruby-csp) and re…

You wouldn't do DNS lookups asynchronously in Go to begin with. Modeling concurrency of any sort in Go the way you would with an event loop is usually a code smell.

Re: Playing with Go: Embarrassingly Parallel Scripts

#22

Post by jnazario »

tptacek wrote: post #21
jnazario wrote: post #19

in the DNS case asynchronous event handling would be super easy to do. in python asyncore with something like dpkt to construct and read DNS lookups works like a champ, as does twisted. i did a simple async DNS resolver in pure python (asyncore, dpkt) and can sustain thousands of lookups a second. GNU adns also has bindings in various languages. you can get Go's parallelisms via CSP (e.g. python-csp, ruby-csp) and re…

You wouldn't do DNS lookups asynchronously in Go to begin with. Modeling concurrency of any sort in Go the way you would with an event loop is usually a code smell.

agreed, and i should have been more clear. the author's blog post states that one of the reasons he explored Go was that his initial sketch of a solution in his language of choice (ruby) was sequential. my point was that you can get performance in ruby with asynchronous operations, and that you don't need to go parallel for something like this.

then again if it was a matter of "well, i had a problem to solve and i had a desire to explore another language, so solving it in that new language was a way to explore" then the point is moot.

however agreed 100% or more on the "code smell" of doing an event loop in Go.

Re: Playing with Go: Embarrassingly Parallel Scripts

#23

Post by danneu »

My attempt with Ruby and Celluloid.

https://gist.github.com/a803d86234e8d1fc5496

I also include a list of 100 domains in a domains.txt if anyone wants to try for themselves.

    require "socket"
    require "celluloid"

    class IPGetter
      include Celluloid

      def get(url)
        Socket.getaddrinfo(url, "http")[0][2]
      end
    end

    pool = IPGetter.pool(size: 100)
    ips = {}

    File.open("domains.txt").each_line do |line|
      line.chomp!
      ips[line] = pool.future.get(line)
    end

    ips.each do |url, ip_future|
      puts "#{url} => #{ip_future.value}"
    end

Re: Playing with Go: Embarrassingly Parallel Scripts

#24

Post by darkhelmetlive »

jameskilton wrote: post #5
dadkins wrote: post #3

Earlier quoted context omitted.

There's a library for that pattern: http://golang.org/pkg/sync/#WaitGroup Instead of counting and using a done channel, import "sync" ... var wg sync.WaitGroup for ... { wg.Add(1) go dowork() } wg.Wait()

Nice. Yeah I'm very new to Go so I know almost nothing about the stdlib libraries. Thanks for the pointer.

Tooting my own horn here, but I'm working on a book covering the Go Standard Library. Still in progress, and not at the sync package yet, but it's coming along. Check it out if you feel inclined.

http://thestandardlibrary.com/go.html

Re: Playing with Go: Embarrassingly Parallel Scripts

#26

Post by scurry »

skeltoac wrote: post #14
hannibalhorn wrote: post #4

Go is great for stuff like this, especially when part of an actual system. That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host

Until last week I didn't know that xargs could invoke commands in parallel. xargs -n1 -P8 dig A hosts-matched.txt

So what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?

Re: Playing with Go: Embarrassingly Parallel Scripts

#27

Post by lsb »

darkhelmetlive wrote: post #24
jameskilton wrote: post #5

Earlier quoted context omitted.

Nice. Yeah I'm very new to Go so I know almost nothing about the stdlib libraries. Thanks for the pointer.

Tooting my own horn here, but I'm working on a book covering the Go Standard Library. Still in progress, and not at the sync package yet, but it's coming along. Check it out if you feel inclined. http://thestandardlibrary.com/go.html

You know what'd be cool? To take arbitrary code in a language, pattern match on the implementation in that language of each std lib function, and actively recommend substitutes for duplicated code.

Re: Playing with Go: Embarrassingly Parallel Scripts

#28

Post by stock_toaster »

darkhelmetlive wrote: post #24
jameskilton wrote: post #5

Earlier quoted context omitted.

Nice. Yeah I'm very new to Go so I know almost nothing about the stdlib libraries. Thanks for the pointer.

Tooting my own horn here, but I'm working on a book covering the Go Standard Library. Still in progress, and not at the sync package yet, but it's coming along. Check it out if you feel inclined. http://thestandardlibrary.com/go.html

Looks interesting (thanks). Will certainly check it out.

Re: Playing with Go: Embarrassingly Parallel Scripts

#29

Post by prodigal_erik »

scurry wrote: post #26
skeltoac wrote: post #14

Earlier quoted context omitted.

Until last week I didn't know that xargs could invoke commands in parallel. xargs -n1 -P8 dig A hosts-matched.txt

So what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?

For what it's worth, GNU Parallel seems to be fairly new; at work I have an ubuntu distro from this year and it's not in the package repo yet. xargs is POSIX so you can expect it everywhere, though no parallel option is specified (merely encouraged).

Re: Playing with Go: Embarrassingly Parallel Scripts

#30

Post by eggoa »

scurry wrote: post #26
skeltoac wrote: post #14

Earlier quoted context omitted.

Until last week I didn't know that xargs could invoke commands in parallel. xargs -n1 -P8 dig A hosts-matched.txt

So what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?

It's mostly about how they handle special characters.

http://www.gnu.org/software/parallel/man.html#differences_be...

Post reply on HN