Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using D and std.ndslice as a Numpy Replacement

jackstouffer.com

Post reply on HN
Print view View original

51–59 of 59 posts

Re: Using D and std.ndslice as a Numpy Replacement

#51

Post by Gtifn »

p4wnc6 wrote: post #48
srean wrote: post #43

Earlier quoted context omitted.

I see your excitement about Numba, I guess you follow it closely, or perhaps have other valid reasons. So all the best. However, the last time I tried to use it, about 4 months ago, installation was a bitch and it would crap out compiling some functions. It holds promise of course, so does alternatives. What I think the current post is about is that similar nice and performant abstractions for D. I love Python a lot…

Numba certainly does not obviate all of these issues. I think user `tadlan` is referring to some of the compiler optimizations, like loop unrolling or fusion (e.g. noticing that two subsequent loops can be 'fused' into the subordinate execution block of one single loop). These things can offer speed-ups even beyond NumPy, and they can work even when the Python code you start out with already uses NumPy. The thing is,…

As a former member of the Numba team, what is your outlook on the technology and the broader continuum ecosystem?

I'm looking at Numba and friends (Dynd, blaze) for a new stack, but I'm not sure where the development arc will end up vs say Julia. I'm also curious about the sustainability of Continuum's business model and practices in the medium and long term.

Any thoughts on this? I understand if you are limited in what you can say, but I'm open to any nuggets.

Re: Using D and std.ndslice as a Numpy Replacement

#52

Post by p4wnc6 »

Gtifn wrote: post #51
p4wnc6 wrote: post #48

Earlier quoted context omitted.

Numba certainly does not obviate all of these issues. I think user `tadlan` is referring to some of the compiler optimizations, like loop unrolling or fusion (e.g. noticing that two subsequent loops can be 'fused' into the subordinate execution block of one single loop). These things can offer speed-ups even beyond NumPy, and they can work even when the Python code you start out with already uses NumPy. The thing is,…

As a former member of the Numba team, what is your outlook on the technology and the broader continuum ecosystem? I'm looking at Numba and friends (Dynd, blaze) for a new stack, but I'm not sure where the development arc will end up vs say Julia. I'm also curious about the sustainability of Continuum's business model and practices in the medium and long term. Any thoughts on this? I understand if you are limited in w…

This is one of those questions that is extremely hard to predict. Even though I was part of that team, it doesn't mean I have any special insight into what will happen.

A lot will depend on sources of funding. Will folks like Nvidia start sponsoring Numba, and if so what will it mean for support of Nvidia alternatives like OpenCL?

It seems like NumbaPro as a stand-alone for-pay product is not viable on its own, but that claim could be wrong based on more recent data that I don't have access to. So external sponsorship may be necessary.

One form of this could be through Continuum's already established business model of consulting and support services. But then the question is whether the nature of those consulting and support projects will allow for developers to actually further the cause of Numba, or just merely hack in poorly conceived features that are demanded by the consulting and support customers? Since Numba is open source, it should be easy enough for anyone to follow along with commits and discussions on GitHub and make their own opinion about what direction that is going.

The other question that is always hard is staffing. Far and away the colleagues I had the chance to work with on the Numba team were amazingly good. But it's not clear if working solely on Numba can justify the sort of salary that would be required to attract very top engineers and grow the team. You might start to see more interns and/or post-doc type labor feeding into Numba, and again I don't know what that will mean for the project ... could be good or bad.

At the same time, you've also got a lot of active development for Julia, PyPy, and a lot of people still prefer to use Cython rather than jitting functions. Some people even call into question the entire goal of making something that is "easy" but also a "black box" -- like the way just dropping in `jit` works for people who merely use, but don't understand the inner workings of, CPython.

It's an exciting area, and the Numba team has as much talent and ability to claim a significant piece of the tool space surrounding high performance computing as anyone else. Whether that will pan out for them is still really hard to predict.

Re: Using D and std.ndslice as a Numpy Replacement

#53

Post by Gtifn »

p4wnc6 wrote: post #52
Gtifn wrote: post #51

Earlier quoted context omitted.

As a former member of the Numba team, what is your outlook on the technology and the broader continuum ecosystem? I'm looking at Numba and friends (Dynd, blaze) for a new stack, but I'm not sure where the development arc will end up vs say Julia. I'm also curious about the sustainability of Continuum's business model and practices in the medium and long term. Any thoughts on this? I understand if you are limited in w…

This is one of those questions that is extremely hard to predict. Even though I was part of that team, it doesn't mean I have any special insight into what will happen. A lot will depend on sources of funding. Will folks like Nvidia start sponsoring Numba, and if so what will it mean for support of Nvidia alternatives like OpenCL? It seems like NumbaPro as a stand-alone for-pay product is not viable on its own, but t…

Thanks very much for your thoughts.

What do you think about the foundational tech of numba itself?

Do you think it is any more of a black box than say Julia? Is there anything about it that would impede extension into a more stable, predictable and feature rich product?

BTW looks like Intel is doing some stuff with Julia: https://github.com/IntelLabs/ParallelAccelerator.jl

There has also been alot of recent funding to Continuum and dev of numba seems to be going strong. Also some recent work with AMD.

Re: Using D and std.ndslice as a Numpy Replacement

#54

Post by semi-extrinsic »

bionsuba wrote: post #50
semi-extrinsic wrote: post #49

Earlier quoted context omitted.

Ok. So three points remain: * how do I know that the calculations aren't actually optimized away in all that code I can't understand? E.g. what happens if you make f0() return the means array instead of being a void function? * Given the same number of lines (or characters), are you sure you can't write equally fast Python code? * I tested the Fortran version on a slightly slower computer (got 14.7 msec on the Python…

"how do I know that the calculations aren't actually optimized away in all that code I can't understand? E.g. what happens if you make f0() return the means array instead of being a void function?" If you can't understand the code, how did you know it was a void function. Please stop with the hyperbole, it's not adding anything to the discussion. Updated code: import std.range : iota; import std.array : array; import…

> If you can't understand the code, how did you know it was a void function. Please stop with the hyperbole, it's not adding anything to the discussion.

I understand "void" perfectly fine, but understanding how D optimizes your code is a completely different matter. Playing devil's advocate further, you

> IMO program size is an almost meaningless statistic outside of code golf challenges. LOC is not an indicative measure of code readability, usefulness, or organization.

No, but the numpy code is undoubtedly much simpler, and very general. How does the D example look for a 3D array where you want to average over the second dimension?

> For example, your Fortran code was 11 lines while the D function (with the return) is eight lines.

You're neglecting the library imports.

>So when your program needs to do anything other than number crunching, it's normally done in a separate language. Using D you can have everything in one code base.

I do agree other languages are much better for e.g. string processing. But for the applications where you can afford to trade simpler code for 50x slower performance, I'd say you're not really caring about performance at all, so why not just use Python? If performance is mission critical, a two-language code base (Python + C/C++/Fortran/CUDA) is not that hard to do, fairly common, and will give you the required performance.

> Bringing up Fortran's speed here is like commenting on how much faster C++ is in a thread about Ruby.

No; once you start saying "I want more performance than Python", it's obviously interesting to see what level of performance a "low-level" language gets, to put the result in perspective.

Now, I'm not trying to beat down on D, so please try to interpret this as constructive criticism. Finding "where does it fit in the scientific toolbox" is what makes or breaks a language's adoption in the scientific community. Just look at R, it's at the same level of performance/abstraction but in a separate niche from Python, and doing very well. The same goes for Matlab (which has e.g. Simulink).

Re: Using D and std.ndslice as a Numpy Replacement

#55

Post by carljv »

jboy wrote: post #45
cannam wrote: post #44

I occasionally rewrite Python+NumPy signal processing code in C++ for purposes of packaging and integration with native apps, so I read these examples with an eye to how they compare with typical C++, rather than with NumPy. They compare very well, and it would never have occurred to me to look into D as a possibility for this sort of code. I'm guessing the GC might rule it out for many cases where you do signal proc…

You might also be interested to check out Nim. It transpiles to C before invoking the C compiler, so it runs as fast as C++ and has excellent C-compatibility (and by extension, excellent C++-compatibility). Compiling a shared library is as easy as passing the "--app:lib" option to the Nim compiler: http://nim-lang.org/docs/nimc.html#compiler-usage-command-li... The GC is optional; you can manage your memory manually…

Please stop this. It's approaching spam. Someone put a lot of work into building a new capability into a language and writing up a lengthy blog post about it. If you want to do the same for what you think is good about Nim and submit it to HN, please do. Repeatedly intercepting people's question about D with "you should look at Nim" is obnoxious. If D users did this in a thread about Nim, I imagine you would find it frustrating.

Re: Using D and std.ndslice as a Numpy Replacement

#56

Post by zardeh »

srean wrote: post #33
zardeh wrote: post #20

Earlier quoted context omitted.

That still doesn't matter. So much of the work is being done in python that you're benchmarking the overhead of python, not the numpy speed. I mean this is a terrible benchmark. Comparing to pure python, the numpy code is about 4-5x faster on my machine, whereas in reasonable real world benchmarks, numpy is hundreds of times faster than pure python. For reference my benchmark was %timeit [sum(row)/len(row) for row in…

You may have a point that the benchmarks do not rule out the possibility that D code might be asymptotically slower. But benchmarks that are designed to hide the fact that CPython function call overheads and object creation overheads are a godawful abomination are far from a fair benchmark. Those overheads are real and have real effects.

Sure, but the claim he's making is that D is faster than the numpy numerical algorithms. This is patently false. The C or fortan code that is doing the numerical calculation is as fast or faster than the D code he's running. Al of the difference comes from the initialization of O(n) pyobjects to represent the returned values.

If he claimed that handwritten D code was faster than equivalent python, I'd say "duh", but he claimed that the numerical libraries are faster. I take issue to the statement that D will be faster than numpy for real world calculations, because the numpy code will run the actual math faster, and the python overhead will still be on the order of nano or milliseconds.

Re: Using D and std.ndslice as a Numpy Replacement

#57

Post by zardeh »

bionsuba wrote: post #47
semi-extrinsic wrote: post #40

Earlier quoted context omitted.

He's absolutely right, this is a crappy benchmark. Increase the array sizes by at least a factor of 100 to get anything meaningful. The fact that someone who is "the review manager for std.ndslice's inclusion into the standard library" does not understand how to profile numerical algorithms makes me very skeptical of using D for any numerical project. Plus, the syntax looks god-awfully unintuitive. A main advantage o…

"Increase the array sizes by at least a factor of 100 to get anything meaningful." Ok, let's do that and see what happens: python -m timeit -s 'import numpy; data = numpy.arange(10000000).reshape((1000, 10000))' 'means = numpy.mean(data, axis=0)' D code import std.range : iota; import std.array : array; import std.algorithm; import std.experimental.ndslice; import std.datetime; import std.conv : to; import std.stdio;…

Is D or the compiler doing something sneaky and inlining something, because I'd expect doing 100x more things to take 100x more time for a linear algorithm,but it takes you on the order of 10x more time. Does the original function run in Edit:

Because as I showed above, Pypy managed to JIT inline the entire benchmark, so I wouldn't be surprised at all if a clever compiler managed to do something sneaky.

Re: Using D and std.ndslice as a Numpy Replacement

#58

Post by zardeh »

bionsuba wrote: post #22
zardeh wrote: post #20

Earlier quoted context omitted.

That still doesn't matter. So much of the work is being done in python that you're benchmarking the overhead of python, not the numpy speed. I mean this is a terrible benchmark. Comparing to pure python, the numpy code is about 4-5x faster on my machine, whereas in reasonable real world benchmarks, numpy is hundreds of times faster than pure python. For reference my benchmark was %timeit [sum(row)/len(row) for row in…

you're benchmarking is python's speed of array instantiation vs. Ds mathematical speed. Its obvious that D will win. Not true, the D code also has the overhead of array initialization. std.array.array is called which allocates the results of the range into an array on the GC, which everyone bemoans as being slow as a dog. Plus, I don't see why this is an invalid benchmark when this is perfectly normal Numpy code, the…

Its that microbenchmarks are useless. If it was worth my time, I could come back and use any of the available python tools (cython, numba, etc.) to compile that function to native code from python syntax and achieve native speed. Or I could write a D implementation of {{some relatively complex matrix algorithm}} and compare it to numpy, showing it was slower.

You could then respond by saying that I wasn't using D optimally, and I'd agree.

Re: Using D and std.ndslice as a Numpy Replacement

#59

Post by dallbee »

dsharlet wrote: post #42
dallbee wrote: post #14

Earlier quoted context omitted.

A lot of that is up and coming. https://github.com/DlangScience [x] Plotting: http://code.dlang.org/packages/plot2kill [+] Optimization: Nothing directly for the purpose, but the intermediate steps are done. And a little bit of work from http://code.dlang.org/packages/atmosphere [x] Probability Distributions: http://dlangscience.github.io/dstats/api/dstats/random.html and http://dlang.org/phobos/std_mathspecial.html…

> [?] QR: Afraid I'm not sure what you're referring to. > [+] Cholesky Decomposition: Trivial to implement How is it that you can judge Cholesky decomposition to be trivial to implement, while being unfamiliar with QR decomposition? Cholesky decomposition is not trivial to implement well . > Your post comes across to me as being rather cynical - but why not be supportive of the good work that's being done? I had a si…

I should have provided a key. Where I listed a +, I basically meant that there was either work underway or it was something that could be done yourself without hindering a project too much. I did not intend to imply that symbolic expression was a solved problem, or anything of that sort.

As for QR decomposition, I am actually rather familiar with it, my brain simply decided to fart on the symbols QR without being followed by decomposition.

For Cholesky Decomposition, I'm sticking with "trivial to implement", and I'll even throw in "trivial to implement well": https://github.com/numpy/numpy/blob/v1.10.1/numpy/linalg/lin...

You asked if D had code for a list of math capabilities that you use on a regular basis. My answer says "In many cases, no, but there's work being done toward that goal." No one is saying "we have a feature-complete math library!". There's much more needed than what you listed, and even the things you listed have many items with zero work done toward them. I am not attempting to say "D can do all these things!", and I don't think the author of the article was either.

Post reply on HN