Live data from Hacker News

Programming language ray tracing benchmarks project

github.com

71–80 of 90 posts

Re: Programming language ray tracing benchmarks project

#71
post #23

Earlier quoted context omitted.

I wouldn’t give much weight to those benchmark numbers at this point. Some of those language versions are quite out of date...

The numbers are indeed out of date. I re-ran the tests for some of the C and Nim programs using Nim 0.19.4 and gcc 7.3.0 on Windows 10. Here are the results: crb-omp 0m22.949s crb 0m23.240s crb_opt 0m31.404s nimrb_pmap 0m8.828s nimrb_fn 0m22.988s nimrb 0m26.556s The base C code is faster than base Nim code. The optimised C code is significantly slower than everything else(!?). The Nim program that uses a threadpool i…

It's surprising that the optimized C is slower; it's much faster on my machine. Like 20s (optimized C) vs 1m20s (unoptimized C).

Re: Programming language ray tracing benchmarks project

#72
post #33

Ordered by realtime, fastest to slowest for those like me who got annoyed by the scrolling up and down trying to compare: Rust (1.13.0-nightly) 1m32.392s Nim (0.14.2) 1m53.320s C 1m59.116s Julia (0.4.6) 2m01.166s Crystal (0.18.7) 2m01.735s C Double Precision 2m26.546s Java (1.7.0_111) 2m36.949s Nim Double Precision (0.14.2) 3m19.547s OCaml 3m59.597s Go 1.6 6m44.151s node.js (6.2.1) 7m59.041s node.js (5.7.1) 8m49.170s…

Julia was astonishing. It's a high level language that's performing almost like C. Last time I checked, many years back, the spec was changing and the run time did crash. Guess it has gone a long way since. The other one is Lua. My assumption was that it's one of the lightest and fastest language around. Looks like "fastest" isn't true in some cases.

That's using version 0.4 of Julia too (current is 1.1). Current version has a a lot of improved optimization passes that would likely benefit this benchmark.

Re: Programming language ray tracing benchmarks project

#73

Ordered by realtime, fastest to slowest for those like me who got annoyed by the scrolling up and down trying to compare: Rust (1.13.0-nightly) 1m32.392s Nim (0.14.2) 1m53.320s C 1m59.116s Julia (0.4.6) 2m01.166s Crystal (0.18.7) 2m01.735s C Double Precision 2m26.546s Java (1.7.0_111) 2m36.949s Nim Double Precision (0.14.2) 3m19.547s OCaml 3m59.597s Go 1.6 6m44.151s node.js (6.2.1) 7m59.041s node.js (5.7.1) 8m49.170s…

I rewrote the Go benchmark to be a mechanical translation of C and it performs much better.

    C        (gcc -O3):       23.8s
    Julia    (julia 1.1.0):   32.8s
    Go (alt) (go 1.12):       39.3s
    Java     (java 1.8.0_60): 44.2s
    Go (org) (go 1.12):       64.8s
    OCaml    (ocaml 4.07.1):  79.1s
    JS       (node 11.14.0):  137.0s
    Pypy     (pypy 6.0.0):    139.4s
    C#       (mono 4.2.1):    187.3s

    Rust: DOES NOT COMPILE
So Go is only twice as slow as C, not thrice as slow. This puts it just ahead of Java and just behind Julia.

Re: Programming language ray tracing benchmarks project

#74
post #68
post #24

Earlier quoted context omitted.

That's a big jump between OCaml and Go. I'm not familiar with ray tracing, but skimming the source code it mostly looks like it's doing floating point math; it doesn't look like it's using the runtime (no allocations, no virtual function calls, no scheduling, etc), so I'm surprised that Go is performing relatively poorly. I wonder if the performance gap is attributable to some overhead in Go's function calls? I know…

Profiling results: - initial time: 49s - replacing default thread-safe RNG with rand.New sliced 6 seconds off. (the default RNG uses mutexes), = 43s - use float64 instead of float32 and remove many type conversions. Another two seconds off. = 41s. As others suggested, go still lacks many compile time optimizations and the implementation could be improved.

I did some profiling too--I shaved off 10-15s by using the C xorshift implementation instead of rand.Float32() (which spends a lot of time locking a mutex).

Re: Programming language ray tracing benchmarks project

#75
post #46

Earlier quoted context omitted.

Different languages' benchmarks might not be equally well-written / optimized. In particular, I'd expect C and Rust to be very close to each other, and a 20% gap between them is a red flag. Rules like "code should be simple, as in, easy to read and understand" are also hard to judge, especially near the top of the list where there's a lot of pressure to optimize. Is SIMD easy to understand? What if it's in a library?…

Not necessarily. Because C allows the pointer manipulation, the compiler can in general not make assumptions about pointer aliasing. This prevents some optimizations. In Rust, the compiler has more information/control over memory layout/lifetime and can therefore make stronger optimizations. Automatic vectorization is an area where this helps a lot, and raytracing can benefit a lot here. 20% sounds reasonable to me.

Well, generally, perhaps. But any performance oriented C programmer worth his or her salt would be aware of aliasing issues and write code in such a way that it doesn't cause problems for the compiler. Plus, this is a toy benchmark of a few hundred lines so the compiler can do full-program analysis. So the 20% difference is indeed a smell.

Looking at the crb*.c files, structs are passed as pointers and not by value. This makes it harder for the compiler to analyze the data flow which I would bet is part of the reason Rust is faster here.

Re: Programming language ray tracing benchmarks project

#76
post #53

Earlier quoted context omitted.

shouldn't rust being faster than C be something of a red flag that they aren't quite the same algorithm? Or that the algorithm is sub-optimal?

It got me thinking as well, so I ventured and did some experiments on this, and found that the main difference is the algorithm used for the RNG; C's std lib uses a slower one (which also is thread safe, and butchered OpenMP performance). You can take a look at a more apples to apples comparison in the latest update for crb.c which uses a xor128 rng; rust is still a little faster (especially when going multithreaded)…

Have you tried passing everything by value? That is, instead of:

    bool hit_sphere(const struct sphere* sp, const struct ray* ray, struct hit* hit)
you write:

    static bool
    hit_sphere(struct sphere sp, struct ray ray, struct hit hit)
IME, clang is insanely good at optimizing pass by value calls.

Re: Programming language ray tracing benchmarks project

#77
I looked over the Common Lisp version at https://github.com/niofis/raybench/blob/master/lisprb.lisp and it's… really bad, in a lot of ways.

    (declaim (optimize (speed 3) (safety 0) (space 0) (debug 0) (compilation-speed 0)))
Never use `(optimize (safety 0))` in SBCL — it throws safety completely out the window. We're talking C-levels of safety at that point. Buffer overruns, the works. It might buy you 10-20% speed, but it's not worth it. Lisp responsibly, use `(safety 1)`.

    (defconstant WIDTH 1280)
People generally name constants in CL with +plus-muffs+. Naming them as uppercase doesn't help because the reader uppercases symbol names by default when it reads. So `(defconstant WIDTH ...)` means you can no longer have a variable named `width` (in the same package).

    (defstruct (vec
                 (:conc-name v-)
                 (:constructor v-new (x y z))
                 (:type (vector float)))
      x y z)
Using `:type (vector float)` here is trying to make things faster, but failing. The type designator `float` covers all kinds of floats, e.g. both `single-float`s and `double-float`s in SBCL. So all SBCL knows is that the struct contains some kind of float, and it can't really do much with that information. This means all the vector math functions below have to fall back to generic arithmetic, which is extremely slow. SBCL even warns you about this when it's compiling, thanks to the `(optimize (speed 3))` declaration, but I guess they ignored or didn't understand those warnings.

    (defconstant ZERO (v-new 0.0 0.0 0.0))
This will cause problems because if it's ever evaluated more than once it'll try to redefine the constant to a new `vec` instance, which will not be `eql` to the old one. Use `alexandria:define-constant` or just make it a global variable.

All the vector math functions are slow because they have no useful type information to work with:

    (disassemble 'v-add)
    ; disassembly for V-ADD
    ; Size: 160 bytes. Origin: #x52D799AF
    ; 9AF:       488B45F8         MOV RAX, [RBP-8]                ; no-arg-parsing entry point
    ; 9B3:       488B5001         MOV RDX, [RAX+1]
    ; 9B7:       488B45F0         MOV RAX, [RBP-16]
    ; 9BB:       488B7801         MOV RDI, [RAX+1]
    ; 9BF:       FF1425A8001052   CALL QWORD PTR [#x521000A8]     ; GENERIC-+
    ; 9C6:       488955E8         MOV [RBP-24], RDX
    ; 9CA:       488B45F8         MOV RAX, [RBP-8]
    ; 9CE:       488B5009         MOV RDX, [RAX+9]
    ; 9D2:       488B45F0         MOV RAX, [RBP-16]
    ; 9D6:       488B7809         MOV RDI, [RAX+9]
    ; 9DA:       FF1425A8001052   CALL QWORD PTR [#x521000A8]     ; GENERIC-+
    ; 9E1:       488BDA           MOV RBX, RDX
    ; 9E4:       488B45F8         MOV RAX, [RBP-8]
    ; 9E8:       488B5011         MOV RDX, [RAX+17]
    ; 9EC:       488B45F0         MOV RAX, [RBP-16]
    ; 9F0:       488B7811         MOV RDI, [RAX+17]
    ; 9F4:       48895DE0         MOV [RBP-32], RBX
    ; 9F8:       FF1425A8001052   CALL QWORD PTR [#x521000A8]     ; GENERIC-+
    ; 9FF:       488B5DE0         MOV RBX, [RBP-32]
    ; A03:       49896D40         MOV [R13+64], RBP               ; thread.pseudo-atomic-bits
    ; A07:       498B4520         MOV RAX, [R13+32]               ; thread.alloc-region
    ; A0B:       4C8D5830         LEA R11, [RAX+48]
    ; A0F:       4D3B5D28         CMP R11, [R13+40]
    ; A13:       772E             JNBE L2
    ; A15:       4D895D20         MOV [R13+32], R11               ; thread.alloc-region
    ; A19: L0:   C600D9           MOV BYTE PTR [RAX], -39
    ; A1C:       C6400806         MOV BYTE PTR [RAX+8], 6
    ; A20:       0C0F             OR AL, 15
    ; A22:       49316D40         XOR [R13+64], RBP               ; thread.pseudo-atomic-bits
    ; A26:       7402             JEQ L1
    ; A28:       CC09             BREAK 9                         ; pending interrupt trap
    ; A2A: L1:   488B4DE8         MOV RCX, [RBP-24]
    ; A2E:       48894801         MOV [RAX+1], RCX
    ; A32:       48895809         MOV [RAX+9], RBX
    ; A36:       48895011         MOV [RAX+17], RDX
    ; A3A:       488BD0           MOV RDX, RAX
    ; A3D:       488BE5           MOV RSP, RBP
    ; A40:       F8               CLC
    ; A41:       5D               POP RBP
    ; A42:       C3               RET
    ; A43: L2:   6A30             PUSH 48
    ; A45:       FF142520001052   CALL QWORD PTR [#x52100020]     ; ALLOC-TRAMP
    ; A4C:       58               POP RAX
    ; A4D:       EBCA             JMP L0
If they had done the type declarations correctly, it would look more like this:

    ; disassembly for V-ADD
    ; Size: 122 bytes. Origin: #x52C33A78
    ; 78:       F30F104A05       MOVSS XMM1, [RDX+5]              ; no-arg-parsing entry point
    ; 7D:       F30F105F05       MOVSS XMM3, [RDI+5]
    ; 82:       F30F58D9         ADDSS XMM3, XMM1
    ; 86:       F30F104A0D       MOVSS XMM1, [RDX+13]
    ; 8B:       F30F10670D       MOVSS XMM4, [RDI+13]
    ; 90:       F30F58E1         ADDSS XMM4, XMM1
    ; 94:       F30F104A15       MOVSS XMM1, [RDX+21]
    ; 99:       F30F105715       MOVSS XMM2, [RDI+21]
    ; 9E:       F30F58D1         ADDSS XMM2, XMM1
    ; A2:       49896D40         MOV [R13+64], RBP                ; thread.pseudo-atomic-bits
    ; A6:       498B4520         MOV RAX, [R13+32]                ; thread.alloc-region
    ; AA:       4C8D5820         LEA R11, [RAX+32]
    ; AE:       4D3B5D28         CMP R11, [R13+40]
    ; B2:       7734             JNBE L2
    ; B4:       4D895D20         MOV [R13+32], R11                ; thread.alloc-region
    ; B8: L0:   66C7005903       MOV WORD PTR [RAX], 857
    ; BD:       0C03             OR AL, 3
    ; BF:       49316D40         XOR [R13+64], RBP                ; thread.pseudo-atomic-bits
    ; C3:       7402             JEQ L1
    ; C5:       CC09             BREAK 9                          ; pending interrupt trap
    ; C7: L1:   C7400103024F50   MOV DWORD PTR [RAX+1], #x504F0203  ; #
    ; CE:       F30F115805       MOVSS [RAX+5], XMM3
    ; D3:       F30F11600D       MOVSS [RAX+13], XMM4
    ; D8:       F30F115015       MOVSS [RAX+21], XMM2
    ; DD:       488BD0           MOV RDX, RAX
    ; E0:       488BE5           MOV RSP, RBP
    ; E3:       F8               CLC
    ; E4:       5D               POP RBP
    ; E5:       C3               RET
    ; E6:       CC0F             BREAK 15                         ; Invalid argument count trap
    ; E8: L2:   6A20             PUSH 32
    ; EA:       E8F1C64CFF       CALL #x521001E0                  ; ALLOC-TRAMP
    ; EF:       58               POP RAX
    ; F0:       EBC6             JMP L0
The weirdness continues:

    (defstruct (ray
                 (:conc-name ray-)
                 (:constructor ray-new (origin direction))
                 (:type vector))
      origin direction)
The `:conc-name ray-` is useless, that's the default conc-name. And again with the `:type vector`… just make it a normal struct. I was going to guess that they were doing it so they could use vector literals to specify the objects, but then why are they bothering to define a BOA constructor here? And the slots are untyped, which, if you're looking for speed, is not doing you any favors.

I took a few minutes over lunch to add some type declarations to the slots and important functions, inlined the math, cleaned up the broken indentation and naming issues:

https://gist.github.com/sjl/005f27274adacd12ea2fc7f0b7200b80...

The old version runs in 5m12s on my laptop, the new version runs in 58s. So if we unscientifically extrapolate that to their 24m time, it puts it somewhere around 5m in their list. This matches what I usually see from SBCL: for numeric-heavy code generic arithmetic is very slow, and some judicious use of type declarations can get you to within ~5-10x of C. Getting more improvements beyond that can require really bonkers stuff that often isn't worth it.

Re: Programming language ray tracing benchmarks project

#78

I looked over the Common Lisp version at https://github.com/niofis/raybench/blob/master/lisprb.lisp and it's… really bad, in a lot of ways. (declaim (optimize (speed 3) (safety 0) (space 0) (debug 0) (compilation-speed 0))) Never use `(optimize (safety 0))` in SBCL — it throws safety completely out the window. We're talking C-levels of safety at that point. Buffer overruns, the works. It might buy you 10-20% speed, b…

I did some quick changes to your code (inlining, stack allocating) and got a further ~2x speedup which makes SBCL performance equivalent to Julia.

Re: Programming language ray tracing benchmarks project

#79

The results are more or less in line with what I would have expected, except for SBCL and Luajit, which I would have expected to be much faster.

The Lisp code is terrible performance-wise and written by someone who obviously doesn't know Common Lisp very well.

Look at Steve Losh's comment here for something a lot better. My own (further) improvements put SBCL performance in the same order as Julia.

Re: Programming language ray tracing benchmarks project

#80

I looked over the Common Lisp version at https://github.com/niofis/raybench/blob/master/lisprb.lisp and it's… really bad, in a lot of ways. (declaim (optimize (speed 3) (safety 0) (space 0) (debug 0) (compilation-speed 0))) Never use `(optimize (safety 0))` in SBCL — it throws safety completely out the window. We're talking C-levels of safety at that point. Buffer overruns, the works. It might buy you 10-20% speed, b…

I did some quick changes to your code (inlining, stack allocating) and got a further ~2x speedup which makes SBCL performance equivalent to Julia.

Yeah I considered trying some dynamic-extent declarations but just didn't care all that much. Can you post your version? I'm curious how far into the declaration weeds you need to go to get that extra 2x.

EDIT: I'm also curious how much using an optimized vector math library (e.g. sb-cga) would buy you instead of hand-rolling your own vector math. It would certainly be easier.

Post reply on HN