ARMv7 vs. x86-64: Pathfinding benchmark of C++, D, Go, Nim, Ocaml, and more

I think the Nim code is also pretty readable: https://github.com/def-/LPATHBench/blob/master/nim.nim

And Nim doesn't look bad in the other statistics (on an x86_64 Intel Core2Quad Q9300):

    Lang    Time [ms]  Memory [KB]  Compile Time [ms]  Compressed Code [B]
    Nim          1400         1460                893                  486
    C++          1478         2717                774                  728
    D            1518         2388               1614                  669
    Rust         1623         2632               6735                  934
    Java         1874        24428                812                  778
    OCaml        2384         4496                125                  782
    Go           3116         1664                596                  618
    Haskell      3329         5268               3002                 1091
    LuaJit       3857         2368                  -                  519
    Lisp         8219        15876               1043                 1007
    Racket       8503       130284              24793                  741

Code size with gzip -9 < nim.nim | wc -c. Removed unused functions in Haskell.

I have no particular love for Mozilla (no more than any other corporation), but your points all seem facile.

1) Regardless of whether anyone thinks Firefox's UI changes have been for the better, it's not possible to alienate Firefox users via UI changes because Firefox still offers the most customizable UI of any browser you've heard of. What are users going to say? "Damn you Mozilla, you made your browser look just like Chrome! I hate that so much, I'm switching to Chrome!"

2) Many of the remaining Firefox users point out that they have switched back to Firefox because Chrome has become bloated and slow. (Personally I think that everyone making this argument, on all sides, merely fails to appreciate what sort of benefit it brings to a browser to have a totally fresh user profile.) In overall benchmarks of memory usage and browser engine/Javascript engine speed, neither Chrome nor Firefox is significantly better by any significant margin.

3) Firefox for Android has between 50 and 100 million downloads on Google Play, and has a higher user rating than Chrome for Android (4.4 to 4.2).

4) Mozilla has announced earlier this month that they'll be shipping a Firefox for iOS, but given the crippling of third-party browsers on iOS I doubt Firefox will be any less hobbled than Chrome for iOS, and will certainly be worse than Safari.

5) I have no sales stats on Firefox OS, but given that they're still persisting in setting up new carrier partnerships I'd say they're better off than at least the Ubuntu phone. I'll probably never need a Firefox OS phone, but honestly if it weren't for the audacity of Mozilla trying to penetrate the OS market (and hence trying to end their reliance on the willing participation of third-party platforms to host their browser (which isn't so "willing" these days with the advent of locked-down platforms like iOS and WinRT (is that still even a thing?))), then I'd have already written off Firefox as dead in the water.

6) Thunderbird was never a moneymaker nor key to Mozilla's strategy, especially after the meteoric rise of web-based email clients.

7) Rust is the most interesting systems language to emerge in years, and its influence will be felt on every future systems programming language to come (though I am certainly biased here).

8) Servo is Mozilla's other project, aside from Firefox OS, that is so unbelievably audacious that I can't help but cheer them on. I have spoken with its developers and they're all astounded with the performance they're seeing, though they're holding off on releasing concrete numbers until the feature set is comparable with more complete browsers. Having seen the Servo devs in action, I can assure you they are on top of their game.

All this said, your original point was that you don't think that Mozilla has a top-notch team of C++ developers. And here's the thing: all of this is irrelevant to whether or not Mozilla's C++ developers are top-notch. I bet John Carmack's team at Id software was as top-notch a team of C++ developers as will ever be assembled, and yet Rage was still a commercial failure. To know whether or not the team is good, you have to look at their code and you have to look at their process.

Not that the article does this, but since my current bandwagon is that assessing performance by comparing the source of two programs without consideration of the compiler and target processor is silly, I decided to try out the C++ version with several compilers and options. Renaming the file to 'lpath.cpp' and compiling with 'cc lpath.cpp -std=gnu++11 -Wall -Oxxx -march=native -o lpath-cc-Oxxx' here's what I found an an i7 Haswell running at 3.4 GHz:

  lpath-clang3.5-O2 8981 LANGUAGE C++ 763
  lpath-gcc4.7-O2 8981 LANGUAGE C++ 769
  lpath-gcc4.8-O2 8981 LANGUAGE C++ 746
  lpath-icpc14-O2 8981 LANGUAGE C++ 750
  lpath-icpc15-O2 8981 LANGUAGE C++ 735
  lpath-clang3.5-O3 8981 LANGUAGE C++ 734
  lpath-gcc4.7-O3 8981 LANGUAGE C++ 943
  lpath-gcc4.8-O3 8981 LANGUAGE C++ 946
  lpath-icpc14-O3 8981 LANGUAGE C++ 664
  lpath-icpc15-O3 8981 LANGUAGE C++ 655

The last column is the time reported by the program in milliseconds. What this shows is that the same source compiled with Intel's icpc 14 or 15 -O3 is about 50% faster than the same source compiled with g++ 4.7 or 4.8 and -O3, and about 20% faster against g++ -O2. The point isn't that Intel's compiler is so much better, but that this degree of variation is normal for a benchmark like this. There are times when Clang or GCC will come out ahead by the same margin. The lesson is that you aren't benchmarking source code, you are benchmarking a particular compiler with particular options running on a particular processor.

The article is wonderfully specific about what was used, but one should be very careful extrapolating to different combinations. In addition to the compiler differences, note for example that although I did my tests on a processor running less than 1.5x faster, I got runtimes that were almost 2-3x faster. Most likely, this is because the Haswell processor I tested on is more efficient than the several generation old Westmere that the author used. There's nothing right or wrong about either choice, but the degree of difference is why it's always important to specify.

I glanced briefly at the code with 'perf record -F10000', and my quick conclusion was the the Intel version was running faster because it was making better use of the branchless cmov's than the other compilers, and thus has 10,000,000 fewer branch prediction errors. At 20 cycles per miss, this accounts for over half the difference between icpc and the others. The use of the new fast variable shift instructions (shlx) and the once-again fast bit test (bt) instruction is probably the rest of the difference.

The difference between g++ -O2 and -O3 seems to be that the -O3 version is doing almost everything off the stack rather than in registers. It's bad enough that this is probably a performance bug rather than intended behavior.