Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

benhoyt.com

Post reply on HN
Print view View original

11–20 of 234 posts

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#11

Post by mistrial9 »

chrismorgan wrote: post #7

> Incidentally, this problem set the scene for a wizard duel between two computer scientists several decades ago. In 1986, Jon Bentley asked Donald Knuth to show off “literate programming” with a solution to this problem, and he came up with an exquisite, ten-page Knuthian masterpiece. Then Doug McIlroy (the inventor of Unix pipelines) replied with a one-liner Unix shell version using tr, sort, and uniq. Since this w…

1986 was very early days for modern operating systems. I suspect that the hardware environment(low resource) would be surprising to many phone owners here. So the efficacy had to do with the tools low-level bindings, too

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#12

Post by FpUser »

I would not call it performance comparison at all. When Python call functions written in C it is not Python's performance. Write those functions using plain Python and then see the results. Sure for this basic example in the article it does not matter from a practical standpoint. But when you need to step away from canned cases suddenly Python's performance sucks big time.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#13

Post by Twirrim »

As they call out in the article, accurately counting words is a bit more complicated that splitting by space. You need to account for all sorts of punctuation etc.

When I've been asked this in programming interviews, I've almost always been expected to produce something like the code in the article (and do), but usually I'd point to something like the NLTK library as a better approach. It's polished and highly capable, and handles probably just about every edge case there is.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#14

Post by mayoff »

gary17the wrote: post #5

Wow. Swift, touted as "safe by design and (...) runs lightning-fast"[1] is more of a screw-up than I thought. Almost twice as slow as Lua and behind even Pascal and Forth. [1] https://developer.apple.com/swift/

Note that only an unoptimized Swift solution was provided, and Swift String operations (like `lowercased`) are fully Unicode-compliant.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#15

Post by [deleted] »

chrismorgan wrote: post #7

> Incidentally, this problem set the scene for a wizard duel between two computer scientists several decades ago. In 1986, Jon Bentley asked Donald Knuth to show off “literate programming” with a solution to this problem, and he came up with an exquisite, ten-page Knuthian masterpiece. Then Doug McIlroy (the inventor of Unix pipelines) replied with a one-liner Unix shell version using tr, sort, and uniq. Since this w…

[deleted]

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#16

Post by machinekob »

gary17the wrote: post #5

Wow. Swift, touted as "safe by design and (...) runs lightning-fast"[1] is more of a screw-up than I thought. Almost twice as slow as Lua and behind even Pascal and Forth. [1] https://developer.apple.com/swift/

Swift is in the same or better performance bracket then C#/Java and unsafe usage can be close to performance of C (maybe 1.3-2x slower). I'm not sure about ranting for poor single test results :P

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#17

Post by bhauer »

gary17the wrote: post #5

Wow. Swift, touted as "safe by design and (...) runs lightning-fast"[1] is more of a screw-up than I thought. Almost twice as slow as Lua and behind even Pascal and Forth. [1] https://developer.apple.com/swift/

I only skimmed the article, but I suspect they are including the time necessary to spin up the executable and any runtime environment prior to executing the relevant code.

"Lightning fast" can mean a lot of things, and it might not mean "executables start quickly."

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#18

Post by djhworld »

EDIT: I think the use of the term "allocation" might be overloaded here from the original article, after thinking about this a bit more I think the author is referring to a different way of accessing the map. If you look in godbolt https://godbolt.org/z/Yf64rodW6 you can see that the pointer version uses

  CALL    runtime.mapaccess2_fast64(SB)
whereas the 'raw' version uses

  CALL    runtime.mapassign_fast64(SB)
When reading the Go solution this bit stood out to me

> To reduce the allocations, we’ll use a map[string]*int instead of map[string]int so we only have to allocate once per unique word, instead of for every increment

I just tried benchmarking this with a simple setup and I get zero allocations for both approaches, although the "pointer" approach is slightly faster

  package main
  
  import (
   "testing"
  )
  
  func BenchmarkIncrementMapRawInt(b *testing.B) {
   var data = make(map[int]int)
  
   b.ResetTimer()
   for i := 0; i

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#19

Post by dan-robertson »

djhworld wrote: post #18

EDIT: I think the use of the term "allocation" might be overloaded here from the original article, after thinking about this a bit more I think the author is referring to a different way of accessing the map. If you look in godbolt https://godbolt.org/z/Yf64rodW6 you can see that the pointer version uses CALL runtime.mapaccess2_fast64(SB) whereas the 'raw' version uses CALL runtime.mapassign_fast64(SB) When reading t…

Presumably this would allocate if bits weren’t “primitive”, eg if you had 128 bit ints implemented as a tuple of two ints.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#20

Post by mustache_kimono »

Re: the Rust optimized implementation, I was able to get ~20-25% better performance by rewriting the for loops as iterators, using Rust's byte range pattern matching, and a buffered writer, which seems crazy, but it's true. I chalked it up to some crazy ILP/SIMD tricks the compiler is doing.

I even submitted a PR[0], but Ben decided he was tired of maintaining and decided to archive the project (which fair enough!).

[0]: https://github.com/benhoyt/countwords/pull/115

Post reply on HN