Playing with Go: Embarrassingly Parallel Scripts • Hacker News Board

Re: Playing with Go: Embarrassingly Parallel Scripts

#31

Post by hannibalhorn » Mon, Dec 03, 2012, 8:42 PM UTC

scurry wrote: post #26

skeltoac wrote: post #14

Earlier quoted context omitted.

Until last week I didn't know that xargs could invoke commands in parallel. xargs -n1 -P8 dig A hosts-matched.txt

So what's the difference between xargs and parallel? I thought the point of parallel was that it was xargs with the addition of running things in parallel. But if xargs can do that already, is there any reason to use one over the other?

In addition to the above, it's worth nothing that parallel also supports running the jobs on multiple remote systems via ssh, giving you an easy way to take advantage of a whole cluster.

Re: Playing with Go: Embarrassingly Parallel Scripts

#32

Post by jlgreco » Mon, Dec 03, 2012, 8:46 PM UTC

lsb wrote: post #27

darkhelmetlive wrote: post #24

Earlier quoted context omitted.

Tooting my own horn here, but I'm working on a book covering the Go Standard Library. Still in progress, and not at the sync package yet, but it's coming along. Check it out if you feel inclined. http://thestandardlibrary.com/go.html

You know what'd be cool? To take arbitrary code in a language, pattern match on the implementation in that language of each std lib function, and actively recommend substitutes for duplicated code.

That would be very cool. I wonder how hard it would be though. At least in Go I know that the standard lib contains lots of duplicate code (primarily so that things that should be small don't require larger things as dependencies. I think the time package's String() functions use reimplemented fmt package functionality for example, since fmt is a much larger dependency than time should have.)

Re: Playing with Go: Embarrassingly Parallel Scripts

#33

Post by skybrian » Mon, Dec 03, 2012, 9:02 PM UTC

jlgreco wrote: post #32

lsb wrote: post #27

Earlier quoted context omitted.

You know what'd be cool? To take arbitrary code in a language, pattern match on the implementation in that language of each std lib function, and actively recommend substitutes for duplicated code.

That would be very cool. I wonder how hard it would be though. At least in Go I know that the standard lib contains lots of duplicate code (primarily so that things that should be small don't require larger things as dependencies. I think the time package's String() functions use reimplemented fmt package functionality for example, since fmt is a much larger dependency than time should have.)

Yes, but outside the Go standard libraries, adding a dependency on a standard library isn't a big deal and won't add a cycle.

Re: Playing with Go: Embarrassingly Parallel Scripts

#34

Post by jlgreco » Mon, Dec 03, 2012, 9:06 PM UTC

skybrian wrote: post #33

jlgreco wrote: post #32

Earlier quoted context omitted.

That would be very cool. I wonder how hard it would be though. At least in Go I know that the standard lib contains lots of duplicate code (primarily so that things that should be small don't require larger things as dependencies. I think the time package's String() functions use reimplemented fmt package functionality for example, since fmt is a much larger dependency than time should have.)

Yes, but outside the Go standard libraries, adding a dependency on a standard library isn't a big deal and won't add a cycle.

Yes. What I mean though is that if you are reimplementing, say, fmt.Printf, such a suggestion system might correctly suggest you use fmt.Printf instead, but also suggest you can use func (m Month) String() string from time, or something equally silly.

Since the standard libs in Go duplicate code, you would have to be careful that your suggestion system isn't picking up false positives. I think the idea has a lot of promise though.

Re: Playing with Go: Embarrassingly Parallel Scripts

#35

Post by ImJasonH » Mon, Dec 03, 2012, 9:23 PM UTC

jameskilton wrote: post #10

drivebyacct2 wrote: post #9

Earlier quoted context omitted.

So many ignored errors. :(

It's a one-off script, I didn't really care about errors. If this was something run regularly inside of a bigger application yes I'd have full error handling.

But tiny throwaway scripts get built up into huge applications all the time, sometimes by other people who don't have the mental TODO to go back and handle errors.

Re: Playing with Go: Embarrassingly Parallel Scripts

#36

Post by chimeracoder » Mon, Dec 03, 2012, 9:36 PM UTC

lazyjones wrote: post #12

To be fair to languages without such great parallelism support: you can do this using asynchronous/event-loop-based code because the parallelism will be limited by the nameserver anyway (the calling code does almost nothing, it mostly waits for the net / the nameserver).

> To be fair to languages without such great parallelism support: you can do this using asynchronous/event-loop-based code

Well you can do with callbacks anything that you can do with channels and goroutines. Go's primary appeal is that it makes concurrent[1] code easy to reason about, not that it enables you to do anything that you "couldn't do" otherwise.

Continuations are just GOTOs, and just like GOTOs, some people love them and some people hate them, but even people who like them can find them difficult in large doses. Goroutines and channels are nice, because they fit the structure of imperative code, whereas callbacks sort of resemble imperative code but "inside out".

[1] Note that I didn't say parallel!

Re: Playing with Go: Embarrassingly Parallel Scripts

#37

Post by ericmoritz » Mon, Dec 03, 2012, 9:45 PM UTC

danneu wrote: post #23

My attempt with Ruby and Celluloid. https://gist.github.com/a803d86234e8d1fc5496 I also include a list of 100 domains in a domains.txt if anyone wants to try for themselves. require "socket" require "celluloid" class IPGetter include Celluloid def get(url) Socket.getaddrinfo(url, "http")[0][2] end end pool = IPGetter.pool(size: 100) ips = {} File.open("domains.txt").each_line do |line| line.chomp! ips[line] = pool.fu…

Or Erlang

https://gist.github.com/bb01a85404e6b445dcb3#file_resolve_do...

    % -*- erlang -*-                                                                                                                              
    %%! -smp enable                                                                                                                               
                                                                                                                                                  
    worker(Hostname) ->                                                                                                                           
        {ok, IP} = inet:getaddr(Hostname, inet),                                                                                                  
        io:format(                                                                                                                                
          "~s => ~s~n",                                                                                                                                 
          [ip_to_string(IP)]                                                                                                                      
         ).                                                                                                                                       
                                                                                                                                                  
    ip_to_string({N1,N2,N3,N4}) ->                                                                                                                
        io_lib:format(                                                                                                                            
          "~w.~w.~w.~w",                                                                                                                          
          [N1,N2,N3,N4]                                                                                                                           
         ).                                                                                                                                       
                                                                                                                                                  
    main([DomainFile]) ->                                                                                                                         
        {ok, Bin} = file:read_file(DomainFile),                                                                                                   
        String = binary_to_list(Bin),                                                                                                             
        Domains = string:tokens(String, "\n"),                                                                                                    
        plists:foreach(                                                                                                                           
          fun(Domain) -> worker(Domain) end,                                                                                                      
          Domains                                                                                                                                 
        ).

This uses https://github.com/eveel/plists/

Re: Playing with Go: Embarrassingly Parallel Scripts

#38

Post by blablabla123 » Mon, Dec 03, 2012, 10:11 PM UTC

tptacek wrote: post #21

jnazario wrote: post #19

in the DNS case asynchronous event handling would be super easy to do. in python asyncore with something like dpkt to construct and read DNS lookups works like a champ, as does twisted. i did a simple async DNS resolver in pure python (asyncore, dpkt) and can sustain thousands of lookups a second. GNU adns also has bindings in various languages. you can get Go's parallelisms via CSP (e.g. python-csp, ruby-csp) and re…

You wouldn't do DNS lookups asynchronously in Go to begin with. Modeling concurrency of any sort in Go the way you would with an event loop is usually a code smell.

Not sure if I understand your point, can you explain a bit more please? Are you saying event loops with select {} are considered code smell?

Re: Playing with Go: Embarrassingly Parallel Scripts

#39

Post by d0mine » Mon, Dec 03, 2012, 10:23 PM UTC

j2labs wrote: post #13

This kind of a thing is also very easy in Python if you use Gevent. I have used Gevent a lot over the last couple years and see a lot of similarities in Go's concurrency, which I think is great. Concurrency doesn't have to be about insane looking code!

With concurrent.futures http://docs.python.org/3/library/concurrent.futures.html#con...

it could look like:

  from concurrent.futures import ThreadPoolExecutor as Pool
  from socket import getaddrinfo

  def lookup(domain):
        try:
            result = getaddrinfo(domain, 80)
        exception Exception as e:
            print("error %s -> %s" % (domain, e))
        else:
            print("done  %s -> %s" % (domain, result))

  nconcurrent = 20
  with open('domains.txt') as file, Pool(nconcurrent) as pool:
        for domain in (line.strip() for line in file):
            pool.submit(lookup, domain)

To run multiple processes instead of threads, change the import to ProcessPoolExecutor.

To support multiprocessing.Pool (for Python 2 where concurrent.futures is not in stdlib), replace pool.submit() with pool.apply_async() and use contextlib.closing() around the Pool().

Re: Playing with Go: Embarrassingly Parallel Scripts

#40

Post by tptacek » Mon, Dec 03, 2012, 11:19 PM UTC

blablabla123 wrote: post #38

tptacek wrote: post #21

Earlier quoted context omitted.

You wouldn't do DNS lookups asynchronously in Go to begin with. Modeling concurrency of any sort in Go the way you would with an event loop is usually a code smell.

Not sure if I understand your point, can you explain a bit more please? Are you saying event loops with select {} are considered code smell?

If I had to simultaneously generalize the idea and make it specific enough to explain it further, I'd say fiddly callback state machines are a code smell in Go.