Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

benhoyt.com

Post reply on HN
Print view View original

191–200 of 234 posts

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#191

Post by gcanyon »

HarHarVeryFunny wrote: post #36

This is a rather meaningless comparison, since the differences are going to be dominated by: 1) The choice of libraries/datatypes used for strings and word->count map 2) How the source file is split into words - probably library function again, although in C/C++ one could choose to implement a super-optimized low level version that would blow the others away IMO a performance comparison between languages is only mean…

My language of choice, LiveCode, is a superset of HyperTalk. The concept of "words" has been built-in (not a library) since the '80s.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#192

Post by gcanyon »

My language of choice, LiveCode, is a superset of HyperTalk. The concept of "words" has been built-in (not a library) since the '80s. Hash tables since the '90s.

A solution to every aspect of this problem but the line-by-line bit took just over a minute to write. Adding that took another minute.

The trade-off is that LC is slower than almost any of the languages listed, and optimization is almost impossible -- the idiomatic way is the fastest (almost). And if you don't like the way LC parses words, there is almost no way to change it without going into the source code (a big pain).

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#193

Post by 1980phipsi »

compumike wrote: post #69

How about extending this to comparing programming language efficiency from a developer's standpoint, looking at bytes of code, versus the execution time? I took the source code file size from the repository, and the runtime from the blog post. Then I made an arbitrary overall "PAIN SCORE" (lower is better) by multiplying code size * runtime. I suggest this is a worthwhile metric simply because lower is better on both…

I think another interesting "pain score" is whether the code is written with built-ins from the language or the standard library (i.e. not new data structures or functions).

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#194

Post by HarHarVeryFunny »

kitd wrote: post #88
HarHarVeryFunny wrote: post #36

This is a rather meaningless comparison, since the differences are going to be dominated by: 1) The choice of libraries/datatypes used for strings and word->count map 2) How the source file is split into words - probably library function again, although in C/C++ one could choose to implement a super-optimized low level version that would blow the others away IMO a performance comparison between languages is only mean…

I understand what you're saying. However if you look at it not from the point of the performance figures per se , but rather which language, when written idiomatically, can solve a realistic problem performantly, then I think it makes more sense.

True - there's still value in these types of comparison, although the click-baity "C++ vs Python" title rather detracts from that.

I'm not sure how much value there is in this specific benchmark though since in the real world you'd be using a utility to do this (e.g. linux wc - word count) and it's not obvious what's dominating the runtime (I'd guess reading file and/or splitting into words), so what you're takeaway should be (idiomatic X is good for what, exactly?) if wanting to extrapolate this to some other task.

For that matter, in the real world people don't really choose language on a task specific basis... It's either use language X because the project demands it, or write some limited use utility in a scripting language if performance isn't a major concern.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#195

Post by oconnor663 »

camel-cdr wrote: post #171
oconnor663 wrote: post #90

Earlier quoted context omitted.

Do you have a particular example problem in mind that might demonstrate language performance better? One of the Benchmark Game programs maybe?

What is there to benchmark between C++, C, Rust, or any other low level compiled language though? With any one of them you can essentially achieve the same assembly code generation, if you try hard enough. In the end it either boils down to compiler optimisations or library implementations.

I think it's more interesting to try to measure the languages that are designed to be "fast but also garbage collected", like Java and Go.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#196

Post by zX41ZdbW »

fauigerzigerk wrote: post #167
zX41ZdbW wrote: post #144

I've checked with ClickHouse and the result is better than I expect... it runs in 0.043 sec. on my machine, which is faster than any other result. The code: SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, count() AS c FROM file('kjvbible.txt', LineAsString) WHERE notEmpty(word) GROUP BY word ORDER BY c DESC FORMAT Null or: clickhouse-local --query "SELECT arrayJoin(splitByChar(' ', lower(line))) AS word, cou…

>it runs in 0.043 sec. on my machine, which is faster than any other result. Did you run the other benchmarks on your machine as well?

Yes (but only scripted, without compilation):

`grep` | 0.03 | 0.03 | `grep` baseline; optimized sets `LC_ALL=C`

`wc -w` | 0.18 | 0.25 | `wc` baseline; optimized sets `LC_ALL=C`

SQL | 0.26 | | by Alexey Milovidov

Perl | 1.22 | | by Charles Randall

Python | 1.42 | 0.86 |

Tcl | 5.30 | | by William Ross

Shell | 9.66 | 1.79 | optimized does `LC_ALL=C sort -S 2G`

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#198

Post by igouy »

svat wrote: post #6

See also a different comparison (on ~the same problem) at StackExchange: https://codegolf.stackexchange.com/questions/188133/bentleys... (Copying from my comment the last time this was posted: https://news.ycombinator.com/item?id=26467684 ) There's also a nice book "Exercises in Programming Style" about just this problem.

See also Doug Bagley's "The Great Computer Language Shootout"

https://web.archive.org/web/20010616231931/http://www.bagley...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#200

Post by kortex »

Bogdanp wrote: post #190
kortex wrote: post #67

Earlier quoted context omitted.

But you are still wrong. As mentioned, Dicts are incredibly efficient data structures in Python (because they underpin everything ) and the Counter class is pure python. That's 100% pure python. Saying dicts "don't count" because they are implemented in C would disqualify the entire language of CPython, as virtually everything under the hood is a PyObject struct pointer. It just so happens that "counting abstract obj…

I have no dog in this fight, but I do want to point out that’s it’s not exactly true that Counter is implemented in pure Python: * it’s a subclass of dict * its update method (used by the code in the post) dispatches to a C implementation on its fast path

Yes, that's exactly what I said. dict.update is in C, because it's a core feature of the python vm. It's pure CPython. What do you think "pure python" is? There's no python hardware ISA (afaik). All cpython is manipulating data structures in C via Python VM opcodes. It just so happens that whatever opcodes that are dispatched in the course of solving this problem are quite efficient.

If you say "it does not count as Real Python if you dispatch to C", then you literally cannot execute any CPython vm opcodes, because it's all dispatching to C under the hood.

Post reply on HN