Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Playing with Go: Embarrassingly Parallel Scripts

collectiveidea.com

Post reply on HN
Print view View original

11–20 of 53 posts

Re: Playing with Go: Embarrassingly Parallel Scripts

#11

Post by etrain »

y0ghur7_xxx wrote: post #6
jgrahamc wrote: post #2

Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…

I know it's about Go, but bash is awesome for stuff like this: while read line; do host $line | head -n1 | awk '{print $1 " -> " $4}' & done

I realize it's a one-off, but you could potentially mix lines of output with a script like that.

If you gnu parallel (xargs++) installed - turn that second line into a script (and replace $line with $1).

  cat domains.txt | parallel -n 1 -P 50 script.sh

Re: Playing with Go: Embarrassingly Parallel Scripts

#12

Post by lazyjones »

To be fair to languages without such great parallelism support: you can do this using asynchronous/event-loop-based code because the parallelism will be limited by the nameserver anyway (the calling code does almost nothing, it mostly waits for the net / the nameserver).

Re: Playing with Go: Embarrassingly Parallel Scripts

#14

Post by skeltoac »

hannibalhorn wrote: post #4

Go is great for stuff like this, especially when part of an actual system. That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host

Until last week I didn't know that xargs could invoke commands in parallel.

  xargs -n1 -P8 dig A hosts-matched.txt

Re: Playing with Go: Embarrassingly Parallel Scripts

#15

Post by stusmall »

drivebyacct2 wrote: post #9
jgrahamc wrote: post #2

Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…

So many ignored errors. :(

This is one of the things I love about go. If he wants the return value but doesn't want to address the errors, he has to actively do it. It makes you think twice when you are putting a _ for an error return. And now to another programmer coming in to maintain it, they stick out like a sore thumb.

Re: Playing with Go: Embarrassingly Parallel Scripts

#16

Post by icebraining »

etrain wrote: post #11
y0ghur7_xxx wrote: post #6

Earlier quoted context omitted.

I know it's about Go, but bash is awesome for stuff like this: while read line; do host $line | head -n1 | awk '{print $1 " -> " $4}' & done

I realize it's a one-off, but you could potentially mix lines of output with a script like that. If you gnu parallel (xargs++) installed - turn that second line into a script (and replace $line with $1). cat domains.txt | parallel -n 1 -P 50 script.sh

Notice that with parallel, you don't need the -n 1, since that's the default.

Re: Playing with Go: Embarrassingly Parallel Scripts

#17

Post by crusso »

etrain wrote: post #11
y0ghur7_xxx wrote: post #6

Earlier quoted context omitted.

I know it's about Go, but bash is awesome for stuff like this: while read line; do host $line | head -n1 | awk '{print $1 " -> " $4}' & done

I realize it's a one-off, but you could potentially mix lines of output with a script like that. If you gnu parallel (xargs++) installed - turn that second line into a script (and replace $line with $1). cat domains.txt | parallel -n 1 -P 50 script.sh

Thanks for the note on parallel, it looks to be an excellently useful tool. I can find a lot of ways to save time with it.

FYI: On OSX, there is a Homebrew formula (brew install parallel).

Re: Playing with Go: Embarrassingly Parallel Scripts

#18

Post by etrain »

icebraining wrote: post #16
etrain wrote: post #11

Earlier quoted context omitted.

I realize it's a one-off, but you could potentially mix lines of output with a script like that. If you gnu parallel (xargs++) installed - turn that second line into a script (and replace $line with $1). cat domains.txt | parallel -n 1 -P 50 script.sh

Notice that with parallel, you don't need the -n 1, since that's the default.

Ah, didn't realize that - throwback to my xargs days! Thanks for the tip.

Re: Playing with Go: Embarrassingly Parallel Scripts

#19

Post by jnazario »

in the DNS case asynchronous event handling would be super easy to do. in python asyncore with something like dpkt to construct and read DNS lookups works like a champ, as does twisted. i did a simple async DNS resolver in pure python (asyncore, dpkt) and can sustain thousands of lookups a second. GNU adns also has bindings in various languages.

you can get Go's parallelisms via CSP (e.g. python-csp, ruby-csp) and replace a lot of fragile threading/parallel code with it. i've been doing that in lieu of learning Go (i know i know .. i'm lazy) and been very pleased.

anyhow, many ways to skin cats. those are just two or three.

Re: Playing with Go: Embarrassingly Parallel Scripts

#20

Post by dkhenry »

Scala is really great for this too. The downside is the spin up time for the JVM, but the upside is that if you use SBT's script launcher it will compile and cache the script transparently for you , you can still pull in _any_ JVM dependency and you can run it just like a shell script. I needed to test for a port being open in parallel and it was a cake walk to use NIO's SocketSelector to do it reactivly.
Post reply on HN