Playing with Go: Embarrassingly Parallel Scripts
collectiveidea.com
Playing with Go: Embarrassingly Parallel Scripts
1–10 of 53 posts
Re: Playing with Go: Embarrassingly Parallel Scripts
#2Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing:
package main
import (
"fmt"
"net"
"io/ioutil"
"strings"
)
func main() {
file_in, _ := ioutil.ReadFile("domains.txt")
domain_list := string(file_in)
done := make(chan bool)
count := 0
for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") {
go func(d string) {
ipAddresses, _ := net.LookupIP(d)
ip := ""
if len(ipAddresses) > 0 {
ip = ipAddresses[0].String()
}
fmt.Println("Mapping: ", d, "->", ip)
done Re: Playing with Go: Embarrassingly Parallel Scripts
#3Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
There's a library for that pattern:
http://golang.org/pkg/sync/#WaitGroup
Instead of counting and using a done channel,
import "sync"
...
var wg sync.WaitGroup
for ... {
wg.Add(1)
go dowork()
}
wg.Wait()Re: Playing with Go: Embarrassingly Parallel Scripts
#4Go is great for stuff like this, especially when part of an actual system.
That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host
Re: Playing with Go: Embarrassingly Parallel Scripts
#5Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
There's a library for that pattern: http://golang.org/pkg/sync/#WaitGroup Instead of counting and using a done channel, import "sync" ... var wg sync.WaitGroup for ... { wg.Add(1) go dowork() } wg.Wait()
Nice. Yeah I'm very new to Go so I know almost nothing about the stdlib libraries. Thanks for the pointer.
Re: Playing with Go: Embarrassingly Parallel Scripts
#6Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
I know it's about Go, but bash is awesome for stuff like this:
while read line; do
host $line | head -n1 | awk '{print $1 " -> " $4}' &
done Re: Playing with Go: Embarrassingly Parallel Scripts
#7Go is great for stuff like this, especially when part of an actual system. That said, if this was just an adhoc job (to figure out which domains point to a specific IP address) you can just use "xargs -P" or GNU parallel and it becomes a pretty basic shell script, along the lines of: cat domains.txt | xargs -P 1000 -n 1 host
xargs -P is a godsend for one-off jobs needing parallelism
Re: Playing with Go: Embarrassingly Parallel Scripts
#8Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
There's a library for that pattern: http://golang.org/pkg/sync/#WaitGroup Instead of counting and using a done channel, import "sync" ... var wg sync.WaitGroup for ... { wg.Add(1) go dowork() } wg.Wait()
I'm embarrassed how many times I've implemented the above without thinking to look for a standard lib solution. Thanks!
Re: Playing with Go: Embarrassingly Parallel Scripts
#9Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
So many ignored errors. :(
Re: Playing with Go: Embarrassingly Parallel Scripts
#10Yes, Go does make writing stuff like that nice, but I think the actual code given is pretty heavyweight. This does the same thing: package main import ( "fmt" "net" "io/ioutil" "strings" ) func main() { file_in, _ := ioutil.ReadFile("domains.txt") domain_list := string(file_in) done := make(chan bool) count := 0 for _, domain := range strings.Split(strings.TrimSpace(domain_list), "\n") { go func(d string) { ipAddress…
So many ignored errors. :(
It's a one-off script, I didn't really care about errors. If this was something run regularly inside of a bigger application yes I'd have full error handling.