Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

benhoyt.com

Post reply on HN
Print view View original

141–150 of 234 posts

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#141

Post by sk0g »

fractalb wrote: post #50
sk0g wrote: post #9

Earlier quoted context omitted.

It's a single implementation, I wouldn't put too much on it. I can definitely write a C++ implementation that's slower and less optimised than most of those results, particularly if I was using Boost. iPhone and Mac apps run pretty well in my experience, and Swift is definitely faster (in general) than Python at least. There were some serious considerations to port over ML libraries to Swift due to its ease of use, s…

> There were some serious considerations to port over ML libraries to Swift due to its ease of use, similar to Python, while providing much better execution speed. Google abandoned that Swift work (if that's what you're referring to)

Sure, but the fact that it was a project at all does imply Swift had a not-insignificant speedup vs Python, meanwhile in these benchmarks Python is about2x the speed of Swift.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#142

Post by Jtsummers »

mustache_kimono wrote: post #139
Jtsummers wrote: post #102

Earlier quoted context omitted.

Is what a compiled language? For Knuth's program in the Programming Pearls article? He wrote it in Pascal, so not a scripting language.

I thought he wrote it in his own language, WEB.

WEB is not a totally different programming language, it is a way to write programs. It is a hybrid (in the form presented in that article, though later versions were centered on other languages or language agnostic) of TeX and Pascal. The code that was written was proper Pascal, the documentation was proper TeX, and there was extra syntax for WEB use in naming and referencing code definitions. The WEB system had two additional components (and these terms are still used in many WEB-derived systems): WEAVE, TANGLE.

WEAVE takes a WEB source file and generates proper TeX out of it. TANGLE takes a WEB source file and generates proper Pascal.

The code written in WEB (and variants) is not necessarily in proper order for compilation, and can contain references to other blogs which are included as text-inclusions. I've never used WEB proper, but in org-mode there is org-babel. Its syntax is something like this (from memory, I use shortcuts so I don't have to memorize all the details and type them out):

  To handle user input, the program will read from a file.

  #+NAME: open-file (a better name would be used in a real program)
  #+BEGIN_SRC lisp :noweb yes
    (with-open-file (f filepath)
      >)
  #+END_SRC
elsewhere

  The actual parsing will look like:

  #+NAME: parsing
  #+BEGIN_SRC lisp :noweb yes
    ...
  #+END_SRC
And the order of these can be reversed, the correct output will be produced with a few other settings and options. With org-babel, when you tangle the org file it will generate one or more source files based on the options and settings you've used throughout the org file itself. Instead of WEAVE, you would just use the standard org export settings.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#143

Post by kovac »

HarHarVeryFunny wrote: post #36

This is a rather meaningless comparison, since the differences are going to be dominated by: 1) The choice of libraries/datatypes used for strings and word->count map 2) How the source file is split into words - probably library function again, although in C/C++ one could choose to implement a super-optimized low level version that would blow the others away IMO a performance comparison between languages is only mean…

With C I rarely use libs outside of stdlib and when I do I pay attention to performance more than convenience.

I think even if it's dominated by libs and data types I think it has value as it might reflect what actually happens out there in the wild rather than a purely academic exercise.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#144

Post by zX41ZdbW »

I've checked with ClickHouse and the result is better than I expect... it runs in 0.043 sec. on my machine, which is faster than any other result.

The code:

SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, count() AS c FROM file('kjvbible.txt', LineAsString) WHERE notEmpty(word) GROUP BY word ORDER BY c DESC FORMAT Null

or:

clickhouse-local --query "SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, count() AS c FROM file('kjvbible.txt', LineAsString) WHERE notEmpty(word) GROUP BY word ORDER BY c DESC" > /dev/null

It is using only a single thread.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#145

Post by zX41ZdbW »

zX41ZdbW wrote: post #144

I've checked with ClickHouse and the result is better than I expect... it runs in 0.043 sec. on my machine, which is faster than any other result. The code: SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, count() AS c FROM file('kjvbible.txt', LineAsString) WHERE notEmpty(word) GROUP BY word ORDER BY c DESC FORMAT Null or: clickhouse-local --query "SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, cou…

I forgot to multiply the file 10 times. When I do, the result is 0.209 sec. which is still better than every other result.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#146

Post by zX41ZdbW »

zX41ZdbW wrote: post #144

I've checked with ClickHouse and the result is better than I expect... it runs in 0.043 sec. on my machine, which is faster than any other result. The code: SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, count() AS c FROM file('kjvbible.txt', LineAsString) WHERE notEmpty(word) GROUP BY word ORDER BY c DESC FORMAT Null or: clickhouse-local --query "SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, cou…

https://github.com/ClickHouse/countwords/pull/1/files

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#148

Post by [deleted] »

mustache_kimono wrote: post #20

Re: the Rust optimized implementation, I was able to get ~20-25% better performance by rewriting the for loops as iterators, using Rust's byte range pattern matching, and a buffered writer, which seems crazy, but it's true. I chalked it up to some crazy ILP/SIMD tricks the compiler is doing. I even submitted a PR[0], but Ben decided he was tired of maintaining and decided to archive the project (which fair enough!).…

[deleted]

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#149

Post by centilliard »

Areading314 wrote: post #132
yb303 wrote: post #131

I had a 1 minute look at the "optimized" C code - it's calling malloc for every word.

Doesn't malloc return a pointer? Sounds unsafe

Ooo... spooooky pointers... ancient prophecies speak of the Earth splitting in half and swallowing everyone who's ever thought about allocating memory.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#150

Post by thayne »

I'm curious why go is faster than rust in the benchmarks. My best theory is that the program doesn't run long enough to trigger a GC, so rust is paying the cost of freeing memory, but go isn't. (Using an arena allocator in rust would probably produce similar behavior).

Or maybe the difference is just noise? It's hard to tell without more details on how the benchmarks were run.

Post reply on HN