These days I tend to use R for these type problems. It’s slow, but vectors as first-class data types simplify things a lot
readLines(file) |> strsplit(“ +”) |> unlist() |> table() |> sort(desc = TRUE)
Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
101–110 of 234 posts
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#102Earlier quoted context omitted.
Indeed. In 1986, Knuth was exclusively using the SAIL DEC10, running the WAITS OS. This environment limited him to approximately half a megabyte of data space (heap plus stack plus globals), and half a megabyte of code space. All statically linked, so all library and runtime stuff had to fit. Also of note is that most CPUs of the day didn't execute instructions orders of magnitude faster than bandwidth to main memory…
and, is it a compiled language at all? versus a script language of some kind.
Is what a compiled language? For Knuth's program in the Programming Pearls article? He wrote it in Pascal, so not a scripting language.
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#103Your Perl implementation is almost as fast as c++? Something weird is going on here. Maybe you are bottlenecked on disk read speeds or something.
Not really surprising. Perl approaches a domain specific language for text processing with a heavily-optimized 34 year old implementation.
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#104Curious if anyone knows why the swift version is slow? Is it the casting from subsequence to a string?
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#105Re: the Rust optimized implementation, I was able to get ~20-25% better performance by rewriting the for loops as iterators, using Rust's byte range pattern matching, and a buffered writer, which seems crazy, but it's true. I chalked it up to some crazy ILP/SIMD tricks the compiler is doing. I even submitted a PR[0], but Ben decided he was tired of maintaining and decided to archive the project (which fair enough!).…
Very nice. That’s in fact really clean, and (at least to me) indeed surprising. As rejection reasons go, “I wanted to name-drop person X” is interesting. But as you said, that’s the maintainer’s decision to make.
[deleted]
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#106What code was used for Common Lisp? I'm guessing a well written version would be in the top contenders for speed.
https://github.com/benhoyt/countwords/blob/master/simple.lis...
From Hoyt's repository itself.
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#107Earlier quoted context omitted.
It's a single implementation, I wouldn't put too much on it. I can definitely write a C++ implementation that's slower and less optimised than most of those results, particularly if I was using Boost. iPhone and Mac apps run pretty well in my experience, and Swift is definitely faster (in general) than Python at least. There were some serious considerations to port over ML libraries to Swift due to its ease of use, s…
> There were some serious considerations to port over ML libraries to Swift due to its ease of use, similar to Python, while providing much better execution speed. Google abandoned that Swift work (if that's what you're referring to)
Google and abandoning things, name a more iconic duo
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#108Doesn't the Unix Shell implementation break the "threading" constraint, in that each program in the pipe will have a main thread each, and all the programs will run concurrently? Or should the "threading" constraint really have been stated as "you can use multiple threads, but only if they all have their own address space"? Alternatively, given the multi-core capabilities of even budget/mobile/small systems these day…
If I were doing the benchmark, I would just run it on a VM with a single physical core. There are good reasons for comparing single threaded performance since it may not be necessary or desirable to scale that way. For example, suppose it's a wc API or something; limiting each request to a single thread can be better since you get fairer scheduling and less data sharing which is almost always better for multiprocessi…
> If I were doing the benchmark, I would just run it on a VM with a single physical core.
I think that's definitely an environment worth benchmarking on - but I don't think that it should be the only environment to benchmark on.
Also, I don't think it's a good reason to limit implementations to a single thread, even if that is your benchmark environment. It can be worth seeing how well an implementation that's capable of taking advantage of multiple cores/CPUs, does when it's only given one core to work with.
It's probably worth optimising for and benchmarking different threading strategies too - does your implementation create one thread per "work unit" and let the scheduler work them all out, or does it create one a one thread per core (or maybe, one thread per core, plus one), thread pool, and assign work units to each each thread until they're done? And how do those strategies work if they're only given a single core?
The single core case is definitely worth testing, but it seems odd to limit implementations to a single thread because of it. If you think you can go faster with a threaded implementation, you should be able to try that out.
Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#109Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more
#110C# really needs a GetOrAdd method on the regular Dictionary class (as exists on ConcurrentDictionary). This common pattern (from the optimized version) requires the hash function on "word" to be evaluated twice (once for TryGetValue and once for Add). Then, if there is a collision on the 32 bit hash then the equality check needs to be run twice as well. Not good as all of these are O(N) operations (where N is the size of "word"). I suspect the optimized version would be materially faster if they implemented their own dictionary that includes this.
if (!counts.TryGetValue(word, out var wordCountRef))
{
counts.Add(word, new Ref(1));
}