Performance comparison: counting words in Python, C/C++, Awk, Rust, and more • Hacker News Board

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#201

Post by igouy » Mon, Jul 25, 2022, 5:23 PM UTC

jamincan wrote: post #179

camel-cdr wrote: post #171

Earlier quoted context omitted.

What is there to benchmark between C++, C, Rust, or any other low level compiled language though? With any one of them you can essentially achieve the same assembly code generation, if you try hard enough. In the end it either boils down to compiler optimisations or library implementations.

I think the interesting question there (and really any language comparison) is to compare the result of idiomatic implementations. Unfortunately, there is no clear answer on what is idiomatic, even with something like Python that is more opinionated than most on the question.

When there is "no clear answer on what is idiomatic", we are left with uninteresting "yes it is" / "no it isn't" Dead Parrot assertions.

http://montypython.50webs.com/scripts/Series_1/53.htm

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#202

Post by igouy » Mon, Jul 25, 2022, 5:41 PM UTC

crummy wrote: post #172

bhauer wrote: post #17

Earlier quoted context omitted.

I only skimmed the article, but I suspect they are including the time necessary to spin up the executable and any runtime environment prior to executing the relevant code. "Lightning fast" can mean a lot of things, and it might not mean "executables start quickly."

If that were the case, that makes Java's results fairly impressive, given the JVM's slow start time.

Maybe for tiny tiny programs "the JVM's slow start time" is only a few tenths of a second?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#203

Post by igouy » Mon, Jul 25, 2022, 5:52 PM UTC

kmonsen wrote: post #126

bhauer wrote: post #124

Earlier quoted context omitted.

As I wrote elsewhere in this thread (though I was downvoted), I suspect the reason is that this benchmark is measuring the total execution time of a single run of an executable. This would include any time spent by the executable to bootstrap its runtime environment, initiate a virtual machine, and whatever else it needs to do in addition to the relevant code at hand to process the string. Such a test will favor impl…

I was going to make a general comment about this too. It’s a huge penalty for Java too and has no correlation for how well it performs outside of toy benchmarks

> a huge penalty

Is that just your assumption or have you measured that penalty for this tiny tiny program?

Might "the JVM's slow start time" in this case be insignificant?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#204

Post by igouy » Mon, Jul 25, 2022, 5:59 PM UTC

samuell wrote: post #184

Fun stuff! Did run a similar thing with a simple bioinformatics problem before (calculating the ratio of G and Cs against A+G+C+T), also with a whole bunch of contributors: https://github.com/samuell/gccontent-benchmark#readme Really hard - or impossible - to arrive at a definitive single number for one language, but the whole exercise is a lot of fun and quite informative IMO :)

That seems familiar —

k-nucleotide

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#205

Post by jcelerier » Mon, Jul 25, 2022, 6:01 PM UTC

Got a bit better C++ version here which uses a couple libraries instead of std:: stuff - https://gist.github.com/jcelerier/74dfd473bccec8f1bd5d78be5a... ; boost, fmt and https://github.com/martinus/robin-hood-hashing

    $ g++ -I robin-hood-hashing/src/include -O2 -flto -std=c++20 -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -lfmt

    $ time ./a.out  /dev/null
    0,19s user 0,01s system 99% cpu 0,197 total

with the same build flags, optimize.cpp gives me

    0,22s user 0,01s system 99% cpu 0,233 total

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#206

Post by igouy » Mon, Jul 25, 2022, 6:13 PM UTC

Karellen wrote: post #108

slaymaker1907 wrote: post #95

Earlier quoted context omitted.

If I were doing the benchmark, I would just run it on a VM with a single physical core. There are good reasons for comparing single threaded performance since it may not be necessary or desirable to scale that way. For example, suppose it's a wc API or something; limiting each request to a single thread can be better since you get fairer scheduling and less data sharing which is almost always better for multiprocessi…

> If I were doing the benchmark, I would just run it on a VM with a single physical core. I think that's definitely an environment worth benchmarking on - but I don't think that it should be the only environment to benchmark on. Also, I don't think it's a good reason to limit implementations to a single thread, even if that is your benchmark environment. It can be worth seeing how well an implementation that's capabl…

> … worth seeing how well an implementation that's capable of taking advantage of multiple cores/CPUs, does when it's only given one core to work with.

Did something like that — programs written for multi-core forced onto one core, alongside programs not written for multi-core.

iirc That difference wasn't something anyone ever expressed interest in.

https://web.archive.org/web/20121231010227/http://benchmarks...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#207

Post by igouy » Mon, Jul 25, 2022, 6:36 PM UTC

zmmmmm wrote: post #173

It seems like this is just measuring a single pass at reading a file with about 100k words in it. I ran the benchmark and it finished almost instantly. So languages with startup overhead and warmup time (eg: JIT etc) will totally underperform here. Makes sense then why Java is almost identical between optimised and non-optimised versions.

otoh the similar k-nucleotide program in the benchmarks game processes bigger files to amortize any startup overhead —

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

otoh instead of asserting "will totally underperform" please measure and share your measurements!

Sometimes "startup overhead" turns-out to be insignificant.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#208

Post by igouy » Mon, Jul 25, 2022, 6:43 PM UTC

bjourne wrote: post #137

ijidak wrote: post #112

> We usually think of I/O as expensive, but I/O isn’t the bottleneck here...The tokenization and hash table operations are the bottleneck by a long shot Interesting. This is something I'll have to keep in mind. Best practice is always been to consider I/O the slowest. But, it's true... times have changed. Maybe we shouldn't make this assumption anymore.

He didn't clear the disk cache when profiling so of course the IO overhead was negligible. Had he flushed the cache between runs the numbers would have been very different.

Measurements of the similar k-nucleotide programs show that effect — N=250,000 compared to N=2,500,000

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#209

Post by tannhaeuser » Mon, Jul 25, 2022, 7:58 PM UTC

benhoyt wrote: post #100

tannhaeuser wrote: post #73

Earlier quoted context omitted.

TFA only talks about gawk, though, making this comparison meaningless ie. you can't benchmark languages but only language implementations . Same goes for other programming languages ofc unless those have only a single implementation. mawk is an order of magnitude faster than gawk, and gawk isn't even the default on many Linuxen.

> TFA only talks about gawk, though, making this comparison meaningless Not exactly! To quote from TFA (I'm the author): > Another “optimization” is to run it using mawk, a faster AWK interpreter than gawk. In this case it’s about 1.7 times as fast as gawk -b. I’m using mawk in the benchmarks for the optimized version.

d'oh sorry didn't read that far

Re: Performance comparison: counting words in Python, C/C++, Awk, Rust, and more

#210

Post by mustache_kimono » Mon, Jul 25, 2022, 8:34 PM UTC

thayne wrote: post #150

I'm curious why go is faster than rust in the benchmarks. My best theory is that the program doesn't run long enough to trigger a GC, so rust is paying the cost of freeing memory, but go isn't. (Using an arena allocator in rust would probably produce similar behavior). Or maybe the difference is just noise? It's hard to tell without more details on how the benchmarks were run.

The naive implementation is slower mostly because of the DOS resistant hasher in the Rust stdlib, and the smallish buffer a line represents.