Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

benhoyt.com

Post reply on HN
Print view View original

111–120 of 234 posts

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#111

Post by anonu »

q solution in kdb, runs in about 2.5s on my machine vs simple.py which takes 4.7s. Rough interpolation with the results table puts this on par with Rust/Go/OCaml or other compiled languages.

  \t desc count each group `$lower " " vs raze read0 `:kjvbible_x10.txt
It's also 1 line!

I am sure q pros or k purists can optimize this even more...

EDIT: Moving the lower earlier brings this down to 1.9s

  desc count each group `$ " " vs lower raze read0 `:kjvbible_x10.txt

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#112

Post by ijidak »

> We usually think of I/O as expensive, but I/O isn’t the bottleneck here...The tokenization and hash table operations are the bottleneck by a long shot

Interesting. This is something I'll have to keep in mind.

Best practice is always been to consider I/O the slowest.

But, it's true... times have changed.

Maybe we shouldn't make this assumption anymore.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#113

Post by gibsonf1 »

Jtsummers wrote: post #106
gibsonf1 wrote: post #93

What code was used for Common Lisp? I'm guessing a well written version would be in the top contenders for speed.

https://github.com/benhoyt/countwords/blob/master/simple.lis... From Hoyt's repository itself.

Right, this came up a year ago with a version that ranks among the top here: https://news.ycombinator.com/item?id=26465857

  (defmethod performance-count ((path-file string))
    (let ((map (make-hash-table :test 'equal)))
      (with-open-file (stream path-file :direction :input :if-does-not-exist nil)
        (when stream
          (loop for line = (read-line stream nil 'end)
         until (eq line 'end)
         do
         (let ((split (split-string #\space (string-downcase line))))
           (dolist (word split)
      (let ((index (gethash word map)))
        (if index
            (setf (gethash word map) (incf index))
          (setf (gethash word map) 1))))))
          (let ((keys (sort (alexandria:hash-table-keys map) 
  (lambda(x y)(> (gethash x map)(gethash y map))))))
     (dolist (key keys)
       (format t "~A ~A~%" key (gethash key map))))))))

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#114

Post by Jtsummers »

gibsonf1 wrote: post #109
Jtsummers wrote: post #106

Earlier quoted context omitted.

https://github.com/benhoyt/countwords/blob/master/simple.lis... From Hoyt's repository itself.

Which lisp compiler did you use?

I haven't run it, just tracked down the source because I was similarly curious about the slow runtime. If I do run it it will be with SBCL. Looking at the repo, Hoyt did as well.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#116

Post by Jtsummers »

gibsonf1 wrote: post #113
Jtsummers wrote: post #106

Earlier quoted context omitted.

https://github.com/benhoyt/countwords/blob/master/simple.lis... From Hoyt's repository itself.

Right, this came up a year ago with a version that ranks among the top here: https://news.ycombinator.com/item?id=26465857 (defmethod performance-count ((path-file string)) (let ((map (make-hash-table :test 'equal))) (with-open-file (stream path-file :direction :input :if-does-not-exist nil) (when stream (loop for line = (read-line stream nil 'end) until (eq line 'end) do (let ((split (split-string #\space (string-do…

You don't need those extra newlines if you'd prefix every code line with two spaces, by the way. It makes it much more readable if you do that.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#117

Post by grumpyprole »

mustache_kimono wrote: post #20

Re: the Rust optimized implementation, I was able to get ~20-25% better performance by rewriting the for loops as iterators, using Rust's byte range pattern matching, and a buffered writer, which seems crazy, but it's true. I chalked it up to some crazy ILP/SIMD tricks the compiler is doing. I even submitted a PR[0], but Ben decided he was tired of maintaining and decided to archive the project (which fair enough!).…

Iterators are potentially a much better design too as they allow for more modular code.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#118

Post by gibsonf1 »

Jtsummers wrote: post #114
gibsonf1 wrote: post #109

Earlier quoted context omitted.

Which lisp compiler did you use?

I haven't run it, just tracked down the source because I was similarly curious about the slow runtime. If I do run it it will be with SBCL. Looking at the repo, Hoyt did as well.

That code is slow because it's working with a really long list instead of a very fast hash table

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#119

Post by gibsonf1 »

Jtsummers wrote: post #116
gibsonf1 wrote: post #113

Earlier quoted context omitted.

Right, this came up a year ago with a version that ranks among the top here: https://news.ycombinator.com/item?id=26465857 (defmethod performance-count ((path-file string)) (let ((map (make-hash-table :test 'equal))) (with-open-file (stream path-file :direction :input :if-does-not-exist nil) (when stream (loop for line = (read-line stream nil 'end) until (eq line 'end) do (let ((split (split-string #\space (string-do…

You don't need those extra newlines if you'd prefix every code line with two spaces, by the way. It makes it much more readable if you do that.

Nice! I didn't know that trick.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#120

Post by bestinterest »

compumike wrote: post #69

How about extending this to comparing programming language efficiency from a developer's standpoint, looking at bytes of code, versus the execution time? I took the source code file size from the repository, and the runtime from the blog post. Then I made an arbitrary overall "PAIN SCORE" (lower is better) by multiplying code size * runtime. I suggest this is a worthwhile metric simply because lower is better on both…

Also a Crystal fan but from afar. The compile time speeds are a real killer of productivity.
Post reply on HN