Using D and std.ndslice as a Numpy Replacement

Post by stevieboy » Sun, Jan 03, 2016, 7:23 AM UTC

jboy wrote: post #30

stevieboy wrote: post #29

Earlier quoted context omitted.

jboy, Can nim-pymod be used as VLA's for nim? I'm not too fond of the nim seq'type (bit slow for my usage) and prefer arrays, but need their length allocated at runtime. Can this be done via (albeit a clunky route) through nim-pymod? i.e arrays created and accessed all in nim (no python)?

Hi, the short answer is "Yes, but ...". Yes, Pymod's PyArrayObject can be created & accessed entirely in Nim; yes, its length (actually, shape) is specified at runtime; and yes, it can be resized after creation. However, Pymod's PyArrayObject is designed for maximum binary compatibility with Numpy. As such: 1. It uses Numpy's array creation functions (such as `createSimpleNew`), which in turn uses the Python runtime…

Post by srean » Sun, Jan 03, 2016, 8:41 AM UTC

tadlan wrote: post #9

bionsuba wrote: post #7

Earlier quoted context omitted.

The benchmarked code is ALREADY a comiled C function called from Python and it still lost.

Numba would still be faster. It would fuse away any intermediates in the code and remove any Overhead to the compiled code. also have the option of devecting to loops. both of which are generally faster than vectorized jumpy code.

Post by skariel1 » Sun, Jan 03, 2016, 9:37 AM UTC

9il wrote: post #24

tadlan wrote: post #21

Earlier quoted context omitted.

What would one need to use c or c++ to write a julia package? It's as fast as native code so no need to use multiple languages.

Julia is really fast in 95% cases, but 5% still "make the weather". The pairwise summation is an example. I will post benchmarks D vs Julia next week ;)

Post by tadlan » Sun, Jan 03, 2016, 10:05 AM UTC

p4wnc6 wrote: post #28

tadlan wrote: post #9

Earlier quoted context omitted.

Numba would still be faster. It would fuse away any intermediates in the code and remove any Overhead to the compiled code. also have the option of devecting to loops. both of which are generally faster than vectorized jumpy code.

While I agree broadly that the benchmark example in the post is not representative or useful as a comparison between D and NumPy, I disagree with your strong insistence on Numba in this case. There is still a lot of Python code that the Numba-to-LLVM compiler cannot handle. Yes, it is true that Numba can do a decent job of removing CPython virtual machine overhead, even for functions in which you statically type the…

Post by semi-extrinsic » Sun, Jan 03, 2016, 10:41 AM UTC

bionsuba wrote: post #22

zardeh wrote: post #20

Earlier quoted context omitted.

That still doesn't matter. So much of the work is being done in python that you're benchmarking the overhead of python, not the numpy speed. I mean this is a terrible benchmark. Comparing to pure python, the numpy code is about 4-5x faster on my machine, whereas in reasonable real world benchmarks, numpy is hundreds of times faster than pure python. For reference my benchmark was %timeit [sum(row)/len(row) for row in…

you're benchmarking is python's speed of array instantiation vs. Ds mathematical speed. Its obvious that D will win. Not true, the D code also has the overhead of array initialization. std.array.array is called which allocates the results of the range into an array on the GC, which everyone bemoans as being slow as a dog. Plus, I don't see why this is an invalid benchmark when this is perfectly normal Numpy code, the…

  program p
  real, dimension(100,1000) :: data
  real, dimension(1000) :: means
  
  n=1
  forall(i=1:100,j=1:1000)
    data(i,j)=n
    n=n+1
  end forall

  forall(i=1:1000000,j=1:1000)
    means(j) = sum(data(:,j))/size(data(:,j))
  end forall