Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

benhoyt.com

Post reply on HN
Print view View original

181–190 of 234 posts

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#181

Post by mustache_kimono »

arijun wrote: post #174
mustache_kimono wrote: post #160

Earlier quoted context omitted.

In case anyone is interested, I did an optimized, but much more simple, Rust implementation just today[0], which is faster than the optimized implementation on my machine. No indexing into arrays of bytes, no unsafe, etc., no "code golf" measures. Of course, credit where it's due -- these are simply tweaks to Andrew Gallant's code. Looks like idiomatic Rust, which I think is interesting. Shows there is more than one…

Out of curiosity, what's the difference in runtime when compared to Andrew's optimized version on your machine? For your other solution, if the time save is consistent from your machine to OP's (a big if), the Rust solution bumps up to 4th place.

> Out of curiosity, what's the difference in runtime when compared to Andrew's optimized version on your machine?

Results on an M1: my "idiomatic" version is 1.32 times faster than Andrew's original optimized version, whereas the optimized C version is 1.13 times faster than my "idiomatic" version. So, all things being equal, that'd make Rust 3rd, just ahead of C++, and behind Zig and C, if I'm reading the results correctly.

More important, to me, would be the other thing -- it's readable, idiomatic Rust. That is, if it had been 5% slower, I think we'd probably all prefer to maintain this code.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#182

Post by nerdponx »

anonymoushn wrote: post #66
codeflo wrote: post #44

Earlier quoted context omitted.

>> ASCII: it’s okay to only support ASCII for the whitespace handling and lowercase operation That's a somewhat specific list -- at least I didn't read that as a general "the program can assume that the input is only ASCII". But then, the author seems to have accepted solutions that crash on non-UTF8 sequences and ones that byte-compare them, so probably either behavior was meant to be fine. I just don't get that fro…

> That's a somewhat specific list -- at least I didn't read that as a general "the program can assume that the input is only ASCII". I don't think it assumes the input is only ASCII. If the problem is "given UTF-8 text, split on ASCII whitespace and convert ASCII uppercase letters to lowercase," you can do that correctly (and produce correct UTF-8 output) without really being UTF-8 aware. For why, see here: https://e…

The issue is that this is a weird requirement. I have seen real world data sets that had all manner of exotic whitespace like nonbreaking spaces and vertical tabs peppered throughout, so I am sympathetic. But this situation isn't common.

That said, for a CLI program like this, usually approximate results are good enough anyway. And realistically, most text should use ASCII whitespace for pretty much all text.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#183

Post by gary17the »

benibela wrote: post #180
gary17the wrote: post #5

Wow. Swift, touted as "safe by design and (...) runs lightning-fast"[1] is more of a screw-up than I thought. Almost twice as slow as Lua and behind even Pascal and Forth. [1] https://developer.apple.com/swift/

>even Pascal and Forth. Even? Pascal is touted as being as fast as C

Perhaps bad phrasing on my part. I meant that in the are-they-still-even-maintaining-modern-optimizing-compilers-for-Pascal-and-Forth sense.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#184

Post by samuell »

Fun stuff! Did run a similar thing with a simple bioinformatics problem before (calculating the ratio of G and Cs against A+G+C+T), also with a whole bunch of contributors:

https://github.com/samuell/gccontent-benchmark#readme

Really hard - or impossible - to arrive at a definitive single number for one language, but the whole exercise is a lot of fun and quite informative IMO :)

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#185

Post by Izkata »

raffraffraff wrote: post #91

I love using the UNIX shell (pipes and programs that do "one thing well") to the point where it hurts my ability to improve at other other languages. But sometimes all you need is a throwaway command to do something once, not a beautifully optimized function that will be written, improved, compiled, profiled, reoptimized and then executed a billion times. The utter tersity of a bash one-liner just tickles me.

> The utter tersity of a bash one-liner just tickles me.

To anyone else confused, the repo has both "sh" and "bash" versions, and the bash one isn't a one liner - it's the sh one.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#186

Post by anonymoushn »

nerdponx wrote: post #182
anonymoushn wrote: post #66

Earlier quoted context omitted.

> That's a somewhat specific list -- at least I didn't read that as a general "the program can assume that the input is only ASCII". I don't think it assumes the input is only ASCII. If the problem is "given UTF-8 text, split on ASCII whitespace and convert ASCII uppercase letters to lowercase," you can do that correctly (and produce correct UTF-8 output) without really being UTF-8 aware. For why, see here: https://e…

The issue is that this is a weird requirement. I have seen real world data sets that had all manner of exotic whitespace like nonbreaking spaces and vertical tabs peppered throughout, so I am sympathetic. But this situation isn't common. That said, for a CLI program like this, usually approximate results are good enough anyway. And realistically, most text should use ASCII whitespace for pretty much all text.

It is a context-specific requirement, but "fully general lowercasing" would be an impossible requirement.

For e.g. Japanese text, I think you'd only have to add 1 or 2 characters to the set of whitespace characters. You also have to solve Japanese text segmentation, which is hard-to-impossible. If you want to canonicalize the words by transforming half-width katakana to full-width, transforming full-width romaji to ascii, etc., that's a lot of work, and which of those transformations are desired will be specific to the actual use of the program. If you want to canonicalize the text such that the same word written using kanji or using only hiragana end up in the same bucket, or that words that are written the same way in hiragana but written differently when using kanji end up in different buckets, or that names that are written the same way in kanji but written differently in hiragana end up in different buckets, or that loanwords incorrectly written using hiragana are bucketed with the katakana loanword, or that words written using katakana for emphasis are bucketed with the hiragana word (but katakana loanwords are not converted to hiragana and bucketed with the non-loanword that is made up of the same moras), well, that all sounds even more challenging than the hard-to-impossible problem you already had to solve to decide where words begin and end :)

Edit: One of the first concerns I mentioned, about full width romaji and half width katakana, and additionally concerns about diacritics, can be addressed using unicode normalization, so these things are pretty easy[0]. An issue you may still face after normalizing is that you may receive inputs that have incorrectly substituted tsu ツ for sokuon ッ (these are pronounced differently), because for example Japanese banking software commonly transmits people's names using a set of characters that does not include sokuon.

My point is that this is not just a hard problem but many different, incompatible problems, many of which are hard, and because of the incompatibilities you have to pick one and give up on the others. An English-speaking end user may not want their wordcount to perform full width romaji -> ascii conversion.

[0]: https://towardsdatascience.com/difference-between-nfd-nfc-nf...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#187

Post by philosopher1234 »

kramerger wrote: post #165

> When optimizing this, the first thing to do is compile with optimizations enabled (g++ -O2). I kind of like the fact that with Go you don’t have to worry about this – optimizations are always on. I think it is more fair to say that in Go optimizations are always OFF.

The results indicate otherwise

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#188

Post by anonymoushn »

tialaramex wrote: post #125
Kiwikwi wrote: post #92

Earlier quoted context omitted.

Such "overlong" encodings are not valid UTF-8.

This was down-voted, which isn't really the best way to handle things that are wrong Your parent was wondering about a hypothetical UTF-8 sequence [all sequences in this post are hexadecimal bytes] XX 20 XX in which the ASCII space character encoded 20 is actually somehow part of a UTF-8 character. That's not a thing. UTF-8 is deliberately designed so that nothing like this can happen, along with several other clever…

I would expect a wordcount to pass these through as-is, along with any other invalid sequences of bytes with the high bits set. Performing unexpected conversions just seems like a way to make my future self have a bad time debugging.

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#190

Post by Bogdanp »

kortex wrote: post #67
FpUser wrote: post #59

Earlier quoted context omitted.

I am not criticizing Python. It does well enough what it was made to do. It just make no sense to call that particular example a "language performance comparison". It is anything but.

But you are still wrong. As mentioned, Dicts are incredibly efficient data structures in Python (because they underpin everything ) and the Counter class is pure python. That's 100% pure python. Saying dicts "don't count" because they are implemented in C would disqualify the entire language of CPython, as virtually everything under the hood is a PyObject struct pointer. It just so happens that "counting abstract obj…

I have no dog in this fight, but I do want to point out that’s it’s not exactly true that Counter is implemented in pure Python:

* it’s a subclass of dict

* its update method (used by the code in the post) dispatches to a C implementation on its fast path

Post reply on HN