Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking 20 programming languages on N-queens and matrix multiplication

github.com

Post reply on HN
Print view View original

181–190 of 194 posts

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#181

Post by bjourne »

mratsim wrote: post #179
bjourne wrote: post #176

Earlier quoted context omitted.

Can you explain how to build your project and how to run the benchmarks? Cause I just spent a few hours disproving another poster's claim of getting OpenBLAS-like performance and I won't want to waste more time ( https://news.ycombinator.com/item?id=38867009 ). While I don't know Nim very well, I dare claim that you don't get anywhere near OpenBLAS performance.

First we can use Laser, which was my initial BLAS experiment in 2019. At the time in particular, OpenBLAS didn't properly use the AVX512 VPUs. (See thread in BLIS https://github.com/flame/blis/issues/352 ), It has made progress since then, still, on my current laptop perf is in the same range Reproduction: - Assuming x86 and preferably Linux. - Install Nim - Install a C compiler with OpenMP support (not the default M…

The compilation command errors out for me:

/home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(77, 8) Warning: use `std/os` instead; ospaths is deprecated [Deprecated] /home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(101, 8) template/generic instantiation of `bench` from here /home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(106, 21) template/generic instantiation of `gemm_nn_fallback` from here /home/bjourne/p/laser/benchmarks/gemm/arraymancer/blas_l3_gemm.nim(85, 34) template/generic instantiation of `newBlasBuffer` from here /home/bjourne/p/laser/benchmarks/gemm/arraymancer/blas_l3_gemm_data_structure.nim(30, 6) Error: signature for '=destroy' must be proc[T: object](x: var T) or proc[T: object](x: T)

Anyway the reason for your competitive performance is likely that you are benchmarking with very small matrices. OpenBLAS spends some time preprocessing the tiles which doesn't really pay off until they become really huge.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#182

Post by vsskanth »

jorgelopes wrote: post #171
vsskanth wrote: post #120

Earlier quoted context omitted.

its 2x slower than C in matmul which is something I wouldn't expect mojo to be slow at

2x slower than C while not optimal at all is not something I would call 'Slow' in this context. Still pretty strong considering how much more friendly it is to write code on for the average user of the languages in the context discussed here

IMO it looks way too verbose compared to the (faster) Julia implementation in this repo.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#183

Post by dsharlet »

bjourne wrote: post #180
dsharlet wrote: post #177

Earlier quoted context omitted.

The makefile asks for -O2 with clang. I find that -O3 almost never helps in clang. (In gcc it does.) Here's what I see: $ clang++ --version clang version 18.0.0 $ time make bin/matrix mkdir -p bin clang++ -I../../include -I../ -o bin/matrix matrix.cpp -O2 -march=native -ffast-math -fstrict-aliasing -fno-exceptions -DNDEBUG -DBLAS -std=c++14 -Wall -lstdc++ -lm -lblas 1.25user 0.29system 0:02.74elapsed 56%CPU (0avgtext…

-O2 did improve performance significantly, but it's still 0.7 s for NumPy and 5.1 seconds for your code on 4096x4096 matrices. Either you're using a slow version of BLAS or you are benchmarking with matrices that are comparatively tiny (384x1536 is nothing).

BLAS is getting almost exactly 100% of the theoretical peak performance of my machine (CPU frequncy * 2 fmadd/cycle * 8 lanes * 2 ops/lane), it's not slow. I mean, just look at the profiler output...

You're probably now comparing parallel code to single threaded code.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#184

Post by mratsim »

bjourne wrote: post #181
mratsim wrote: post #179

Earlier quoted context omitted.

First we can use Laser, which was my initial BLAS experiment in 2019. At the time in particular, OpenBLAS didn't properly use the AVX512 VPUs. (See thread in BLIS https://github.com/flame/blis/issues/352 ), It has made progress since then, still, on my current laptop perf is in the same range Reproduction: - Assuming x86 and preferably Linux. - Install Nim - Install a C compiler with OpenMP support (not the default M…

The compilation command errors out for me: /home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(77, 8) Warning: use `std/os` instead; ospaths is deprecated [Deprecated] /home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(101, 8) template/generic instantiation of `bench` from here /home/bjourne/p/laser/benchmarks/gemm/gemm_bench_float32.nim(106, 21) template/generic instantiation of `gemm_nn_fallback`…

Ah,

It was from an older implementation that wasn't compatible with Nim v2. I've commented it out.

If you pull again it should work.

> Anyway the reason for your competitive performance is likely that you are benchmarking with very small matrices. OpenBLAS spends some time preprocessing the tiles which doesn't really pay off until they become really huge.

I don't get why you think it's impossible to reach BLAS speed. The matrix sizes are configured here: https://github.com/mratsim/laser/blob/master/benchmarks/gemm...

It defaults to 1920x1920 * 1920x1920. Note, if you activate the benchmarks versus PyTorch Glow, in the past it didn't support non-multiple of 16 or something, not sure today.

Packing is done here: https://github.com/mratsim/laser/blob/master/laser/primitive...

And it also support pre-packing which is useful to reimplement batch_matmul like what CuBLAS provides and is quite useful for convolution via matmul.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#185

Post by igouy »

airstrike wrote: post #136
igouy wrote: post #134

Earlier quoted context omitted.

> many charts doing one thing each Would not be a summary.

It would if you have some summary charts at the top that don't include all of the data, but just the key takeaways, with other charts providing more detail for discussion elsewhere

My apologies. One chart for nqueens and one chart for matmul would be fine.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#186

Post by igouy »

nerdponx wrote: post #154
bjourne wrote: post #153

Having checked the C implementations of the algorithms, I'm a little skeptical because the implementations aren't optimized. Tiling matrix multiplication exploiting SIMD could easily improve performance 100-fold or more. At those speeds the cost of memory transfer usually dominate so the languages that give you the most fine-grained control over how data is laid out in memory tend to win. And it may not be the same l…

I think a benchmark of "naive" implementations is interesting too, because it shows you how fast your code is usually going to run, not how fast it theoretically could run at its best.

> shows you how fast your code is usually going to run

Why would you think that?

Maybe the "naive" implementation just shows how fast the easily removed hotspot in your code is going to run.

"Swap the order of two statements and see the Java code slow down … Swap globals for local variables in a function and see the Python code speed up. Swap language implementations and see the C code speed up."

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#187

Post by igouy »

Jaxan wrote: post #173

One thing I always wonder with these benchmarks is: do we want to know the best possible runtime for a language or the runtime of idiomatic/average implementations? Even with C there are loads of compiler flags (and compilers) to choose from.

The data has nothing definite to say about language implementations that were not measured. Let's focus on what we might say about those that we're measured.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#188

Post by bjourne »

dsharlet wrote: post #183
bjourne wrote: post #180

Earlier quoted context omitted.

-O2 did improve performance significantly, but it's still 0.7 s for NumPy and 5.1 seconds for your code on 4096x4096 matrices. Either you're using a slow version of BLAS or you are benchmarking with matrices that are comparatively tiny (384x1536 is nothing).

BLAS is getting almost exactly 100% of the theoretical peak performance of my machine (CPU frequncy * 2 fmadd/cycle * 8 lanes * 2 ops/lane), it's not slow. I mean, just look at the profiler output... You're probably now comparing parallel code to single threaded code.

No, multi-threaded OpenBLAS improves performance to 0.15s.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#189

Post by dsharlet »

bjourne wrote: post #188
dsharlet wrote: post #183

Earlier quoted context omitted.

BLAS is getting almost exactly 100% of the theoretical peak performance of my machine (CPU frequncy * 2 fmadd/cycle * 8 lanes * 2 ops/lane), it's not slow. I mean, just look at the profiler output... You're probably now comparing parallel code to single threaded code.

No, multi-threaded OpenBLAS improves performance to 0.15s.

I dunno man. My claim was that for specific cases with unique properties, it's not hard to beat BLAS, without getting too exotic with your code. BLAS doesn't have routines for multiplies with non-contiguous data, various patterns of sparsity, mixed precision inputs/outputs, etc. The example I gave is for a specific case close-ish to the case I cared about.

You're changing it to a very different case, presumably one that you cared about, although 4096x4096 is oddly square and a very clean power of 2... I said right at the beginning of this long digression that what is hard about reproducing BLAS is its generality.

Re: Benchmarking 20 programming languages on N-queens and matrix multiplication

#190

Post by igouy »

xnorswap wrote: post #88
1letterunixname wrote: post #57

For the love of god, log graph tiny values with large values please. :)

Another alternative here is to plot the reciprocal (Op / Sec) which has the benefit of both the more natural "Bigger is Better" and easier comparison of the fastest runtimes.

"Bigger is Better" may be "more natural" for Op / Sec but secs is "more natural" for "comparison of the fastest runtimes".
Post reply on HN