Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Command Line Tools in D

dlang.org

Post reply on HN
Print view View original

81–90 of 100 posts

Re: Faster Command Line Tools in D

#81

Post by qesa »

gwu78 wrote: post #60

"The task is to sum the values for each key and print the key with the largest sum." What is the smart way to do this in kdb+? This is my naive, sloppy 15min approach. Warning: Noob. May offend experienced k programmers. k)`t insert+:`k`v!("CI";"\t")0:`:tsvfile k)f:{select (*:k),(sum v) from t where k=x} k)a:f["A"] k)b:f["B"] k)c:f["C"] k)select k from a,b,c where v=(max v)

Using the file from the original,

    1#desc sum each group (!/) (" II";"\t") 0: `:tsvfile
Took about 3 seconds, 2.5 of which was reading the file

EDIT:

    q)\ts d: (!/) (" II";"\t") 0: `:tsvfile
    2489 134218576
    q)\ts 1#desc sum each group d
    486 253055104

Re: Faster Command Line Tools in D

#82

Post by nenreme »

destructionator wrote: post #63
milcron wrote: post #43

Earlier quoted context omitted.

D's standard library uses garbage collection.

So do Java, C#, Python, Ruby, PHP, Javascript, and virtually everything else and they are very heavily used. Garbage collection is a smashing success in the real world and D made the right decision to follow that success. Of course, it is also true that much of the standard library doesn't actually use it... but these objections are never actually about facts.

> So do Java, C#, Python, Ruby, PHP, Javascript, and virtually everything else and they are very heavily used.

That's the point. There are already many popular languages with GC. Why would people switch from C++ (this was the original question) to D instead of one of those much more popular languages that you mentioned?

Re: Faster Command Line Tools in D

#83

Post by agumonkey »

AceJohnny2 wrote: post #42
agumonkey wrote: post #20

Earlier quoted context omitted.

Yeah code only dealing with elegant cases is elegant. News at 20. Sometimes I wonder if software should error first, then when you bounded the failure space, you iterate on the success space as you see fit.

You've just defined Test-Driven-Development.

TDD is informal, I was thinking about something a bit more mathematical. It's hard to reason about spaces with TDD.

Re: Faster Command Line Tools in D

#84

Post by ole_tange »

e12e wrote: post #55
e12e wrote: post #54

Earlier quoted context omitted.

I tried the python programs under pypy and python3 (after running through 2to3) -- and got similar speeds as the author. I was a little surprised that my little awk script was slower than than pypy (completing in 6 to 7 seconds): $ cat sum.awk { a[$2] += $3 } END { for (i in a) { if (a[i] > max) { maxk = i max=a[i] } } print "max_key:", maxk, "sum:", max } $ time gawk -f sum.awk -O (This under Linux subsystem for win…

Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before: (cat ngrams.tsv \ |parallel --pipe awk -f map.awk \ |awk -f reduce.awk ) max_key: 2006 sum: 22569013 This now runs in 17 to 18 seconds... :-/ $ cat map.awk { a[$2] += $3 } END { for (i in a) { pri…

--pipe is well know for being slow.

Try --pipe-part instead:

    parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk |
      awk -f reduce.awk

Re: Faster Command Line Tools in D

#85

Post by mhh__ »

ajbonkoski wrote: post #62

I don't get articles like this.. they seem to miss the bigger point. Typically there are two modes in my computing: (1) Scripting / Command-line get stuff done and throw-away, and (2) Serious applications that are heavily used, need to process lots of data and be as fast as possible (e.g. processing millions of files like this one where the algorithm constant factor really matters). In case 1: Hack something together…

The point here is that you can write both the python and the C version in the same language. Your C version, excluding the preprocessor, might already compile as a D program anyway.

Re: Faster Command Line Tools in D

#86

Post by d33 »

paulddraper wrote: post #69
Twirrim wrote: post #66

Earlier quoted context omitted.

Out of curiosity, I gave it a shot. I came out roughly 20% faster using python's inbuilt csv library. When I switched to pypy the csv library actually made it nearly 2x slower than pypy using .split(delim)

And you were using the pure Python CSV library, not the C one?

How can you tell?

Re: Faster Command Line Tools in D

#87

Post by nerdponx »

Twirrim wrote: post #66
nerdponx wrote: post #57

Earlier quoted context omitted.

Not to mention using .split(delim) instead of the proper CSV parsing library that ships with Python.

Out of curiosity, I gave it a shot. I came out roughly 20% faster using python's inbuilt csv library. When I switched to pypy the csv library actually made it nearly 2x slower than pypy using .split(delim)

Of course it's faster. Sorry, I wasn't clear. But good luck handling a tab in a quoted field, then.

Re: Faster Command Line Tools in D

#88

Post by gshulegaard »

throwaway7645 wrote: post #75
gshulegaard wrote: post #51

Earlier quoted context omitted.

Optimizing/squeezing performance out of Python is a rabbit hole: https://www.ibm.com/developerworks/community/blogs/jfp/entry... I would speculate using numba or Cython would yield further performance gains over PyPy...but that's mostly just based on anecdotal comparisons: https://cardinalpeak.com/blog/faster-python-with-cython-and-... I just think it is a bit dishonest to try and make a claim as pointed as this arti…

I think they're just giving you bounds on what to expect and not selling anything. The D optimizations looked a lot easier (write it slightly different) than mucking with Cython or Numba. Simply running through PyPy is another thing all together.

I don't know, a lot of benefit can be had from Cython by just declaring types and flagging for compilation:

http://cython.readthedocs.io/en/latest/src/tutorial/cython_t...

http://cython.readthedocs.io/en/latest/src/quickstart/cython...

But that is just my opinion.

Re: Faster Command Line Tools in D

#89

Post by EvenThisAcronym »

nemo1618 wrote: post #61
penpapersw wrote: post #12

Earlier quoted context omitted.

Honest question: what strengths does Go have over D? I became very proficient at Go several years ago, and was reading about D for hours and hours last night, and it looks like D is overall a much better language.

Simplicity and consistency is the biggest strength of Go. D is a complicated language with a ton of features. That makes it harder to learn, and it makes it more likely that you'll encounter "cleverness" in other people's code. As a tiny nit, it's interesting to compare D's `out` keyword to Go's multiple return values. An `out` keyword seems like a prime example of "thinking in blub."

I find that kind of ironic. I know there's a lot of people that love Go and it's a solid language, but I personally see it as the ultimate blub language.

Re: Faster Command Line Tools in D

#90

Post by e12e »

ole_tange wrote: post #84
e12e wrote: post #55

Earlier quoted context omitted.

Out of curiosity (and with the danger of hijacking this thread as a stackoverflow discussion), I changed the awk code so that it could be used with GNU parallel - the "reduce"-step is essentially the same program as before: (cat ngrams.tsv \ |parallel --pipe awk -f map.awk \ |awk -f reduce.awk ) max_key: 2006 sum: 22569013 This now runs in 17 to 18 seconds... :-/ $ cat map.awk { a[$2] += $3 } END { for (i in a) { pri…

--pipe is well know for being slow. Try --pipe-part instead: parallel -a ngrams.tsv --pipe-part --block -1 awk -f map.awk | awk -f reduce.awk

Thanks for the tip, always nice to see the author of tools commenting on hn :-)

The (old) version of parallel packaged with Ubuntu 16.04 (linux subsystem for windows) - doesn't have --pipe-part -- but running from upstream, the speed is more reasonable:

  $ time (./parallel-20170522/src/parallel -a ngrams.tsv \
    --pipe-part --block -1 -j4 mawk -f map.awk \
    | mawk -f reduce.awk )
  max_key: 2006 sum: 22569013

  real    0m2.265s
  user    0m4.672s
  sys     0m1.672s
(Tried a few variants with/without -jN -- and this seems typical for the fast end of the spectrum).

  $ time (cat ngrams.tsv \
     | mawk -f map.awk \
     | mawk -f reduce.awk )
  max_key: 2006 sum: 22569013

  real    0m3.472s
  user    0m2.891s
  sys     0m2.406s
[ed: btw, did a double-take when I saw your Gnu Privacy Guard id: 0x88888888 :-) ]
Post reply on HN