Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Command Line Tools in D

dlang.org

Post reply on HN
Print view View original

51–60 of 100 posts

Re: Faster Command Line Tools in D

#51

Post by gshulegaard »

throwaway7645 wrote: post #50
gshulegaard wrote: post #17

Earlier quoted context omitted.

Is it though? Un-optimized Python vs. a D script iterated on 5 times? Certainly eye-catching but I wouldn't call it conclusive.

No, it isn't debatable which is generally faster (D), but I was impressed that you could just prototype in Python (probably significantly faster than D) and run it through PyPy when you're done and get 90% of the performance of D. Of course Python has the bloat of the interpreter and Jitter, while D is just a binary. Point is I was expecting more speed from D. I'm curious if this is just luck or the two would be neck…

Optimizing/squeezing performance out of Python is a rabbit hole:

https://www.ibm.com/developerworks/community/blogs/jfp/entry...

I would speculate using numba or Cython would yield further performance gains over PyPy...but that's mostly just based on anecdotal comparisons:

https://cardinalpeak.com/blog/faster-python-with-cython-and-...

I just think it is a bit dishonest to try and make a claim as pointed as this article's in 2017 by stopping at simply running an un-optimized CPython script with PyPy.

Re: Faster Command Line Tools in D

#52

Post by voltagex_ »

biomcgary wrote: post #6

Just FYI, I tried using a couple of the pre-compiled binaries in bash on Ubuntu on Windows and got a segmentation fault. Same binaries worked fine in real Linux.

I wonder if that's a WSL bug you should report.

Edit: do you mean the tsv utilities? Because they're working fine here on the Creator's Update.

Re: Faster Command Line Tools in D

#53

Post by bachmeier »

throwaway7645 wrote: post #45
ramses0 wrote: post #8

This article is trash. They start with "obvious" python at 12s, run it with pypy instead for 3s, and then rewrite and optimize a D version from 3s to 1s w/o attempting any further optimization of the python version(!?!). In my opinion omit all of the discussion on python and just talk about "how to optimize a D program" b/c that's what this article is.

I was excited with D's performance before I realized it is barely faster than PyPy. Almost not much of a point unless it saves in other ways like concurrency and parallelism?

This problem is not designed to compare language speed. You can get C speeds (and sometimes better) using D.

Re: Faster Command Line Tools in D

#54

Post by e12e »

systems wrote: post #3

well, i dont think that there were much doubt that d was faster than python i am thought impressed with how fast pypy did

I tried the python programs under pypy and python3 (after running through 2to3) -- and got similar speeds as the author. I was a little surprised that my little awk script was slower than than pypy (completing in 6 to 7 seconds):

  $ cat sum.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      if (a[i] > max) {
        maxk = i
        max=a[i]
      }
    }
    print "max_key:", maxk, "sum:", max
  }

  $ time gawk -f sum.awk -O
(This under Linux subsystem for windows)

According to the gawk profiler, and strace -c (count) -- the awk program mainly spends its time reading the file (without the loop at the end, looking for the max value, the runtime is essentially the same).

In fact, on the surface, pypy and python3 are quite similar on the syscall/strace front - with roughly 23k "read" calls -- awk did 375k. And adding cat in front sped it up by about two seconds:

  $ time (cat ngrams.tsv |awk -f sum.awk )
  max_key: 2006 sum: 22569013

  real    0m3.969s
  user    0m3.719s
  sys     0m0.516s

  $ time awk -f sum.awk ngrams.tsv
  max_key: 2006 sum: 22569013

  real    0m6.465s
  user    0m3.609s
  sys     0m2.859s

Re: Faster Command Line Tools in D

#55

Post by e12e »

e12e wrote: post #54
systems wrote: post #3

well, i dont think that there were much doubt that d was faster than python i am thought impressed with how fast pypy did

I tried the python programs under pypy and python3 (after running through 2to3) -- and got similar speeds as the author. I was a little surprised that my little awk script was slower than than pypy (completing in 6 to 7 seconds): $ cat sum.awk { a[$2] += $3 } END { for (i in a) { if (a[i] > max) { maxk = i max=a[i] } } print "max_key:", maxk, "sum:", max } $ time gawk -f sum.awk -O (This under Linux subsystem for win…

Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before:

  (cat ngrams.tsv \
    |parallel --pipe awk -f map.awk \
    |awk -f reduce.awk )
  max_key: 2006 sum: 22569013
This now runs in 17 to 18 seconds... :-/

  $ cat map.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      print "ignore", i, a[i]
    }
  }

  $ cat reduce.awk
  { a[$2] += $3 }
  END {
    for (i in a) {
      if (a[i] > max) {
        maxk = i
        max=a[i]
      }
    }
    print "max_key:", maxk, "sum:", max
  }
[ed: However, there are faster awks than gawk:

  $ time mawk -f sum.awk ngrams.tsv
  max_key: 2006 sum: 22569013

  real    0m2.826s
  user    0m2.391s
  sys     0m0.422s
mawk is (a little) faster than pypy on my machine.

]

Re: Faster Command Line Tools in D

#57

Post by nerdponx »

ramses0 wrote: post #8

This article is trash. They start with "obvious" python at 12s, run it with pypy instead for 3s, and then rewrite and optimize a D version from 3s to 1s w/o attempting any further optimization of the python version(!?!). In my opinion omit all of the discussion on python and just talk about "how to optimize a D program" b/c that's what this article is.

Not to mention using .split(delim) instead of the proper CSV parsing library that ships with Python.

Re: Faster Command Line Tools in D

#58

Post by smitherfield »

e12e wrote: post #55
e12e wrote: post #54

Earlier quoted context omitted.

I tried the python programs under pypy and python3 (after running through 2to3) -- and got similar speeds as the author. I was a little surprised that my little awk script was slower than than pypy (completing in 6 to 7 seconds): $ cat sum.awk { a[$2] += $3 } END { for (i in a) { if (a[i] > max) { maxk = i max=a[i] } } print "max_key:", maxk, "sum:", max } $ time gawk -f sum.awk -O (This under Linux subsystem for win…

Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before: (cat ngrams.tsv \ |parallel --pipe awk -f map.awk \ |awk -f reduce.awk ) max_key: 2006 sum: 22569013 This now runs in 17 to 18 seconds... :-/ $ cat map.awk { a[$2] += $3 } END { for (i in a) { pri…

> GNU parallel

Which is a 15000-line Perl script, so to be expected.

Re: Faster Command Line Tools in D

#59

Post by giancarlostoro »

steveklabnik wrote: post #33
maxpert wrote: post #30

Makes me wonder, how many people actually use D-lang in production. Specifically with HTTP stack what kind of numbers/performance benchmarks we are looking at? Other than toy projects is there a company out there running D on massive scale (millions per day)?

They have a page here: http://dlang.org/orgs-using-d.html

Based on that I would look at Funatics since they seem to have the "bigger" traffic, or even AdRoll...

http://tech.adroll.com/blog/data/2014/11/17/d-is-for-data-sc...

Edit:

To be fair a lot of the companies there look like they handle heavier loads. Also Garbage Collection is optional, and there are alternatives to the "standard library" for D that others have made that are probably usable without GC. Some people have done successful embedded systems programming without the GC, I remember one guy talking about it on the D irc channel.

Re: Faster Command Line Tools in D

#60

Post by gwu78 »

"The task is to sum the values for each key and print the key with the largest sum."

What is the smart way to do this in kdb+?

This is my naive, sloppy 15min approach.

Warning: Noob. May offend experienced k programmers.

   k)`t insert+:`k`v!("CI";"\t")0:`:tsvfile
   k)f:{select (*:k),(sum v) from t where k=x}
   k)a:f["A"]
   k)b:f["B"]
   k)c:f["C"]
   k)select k from a,b,c where v=(max v)
Post reply on HN