Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Making Cython as easy as Python

github.com

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Making Cython as easy as Python

#11

Post by pathsjs »

osullivj wrote: post #9
pathsjs wrote: post #8

Earlier quoted context omitted.

The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...). Some of them try to reintroduce types in order to compile to native code, thereby losing the advantages of dynamic typing. In short, it is a mess, and everyone is trying to add on this because the existing ecosystem is big and the cost of…

So is this official Unicredit open source? I'm impressed if it is...

Yes, there are a few open-source projects that you can check on our github :-)

Re: Making Cython as easy as Python

#12

Post by dagw »

python_que wrote: post #7

I am using Python to solve combinatorial problems, and so my code relies heavily on the itertools library. My question is if I would still get a considerable speed-up if I rewrite some of my code in Cython, given my reliance on itertools?

The calls to itertools won't be sped up significantly. However if then you loop over the results of a call to itertools and do something to each element, then that loop might end up being much faster.

Re: Making Cython as easy as Python

#13

Post by dagw »

zzleeper wrote: post #4
vegabook wrote: post #3

Cython, Numpy, Numba (and others) are what make me skeptical that any of the numerical computing competitors to Python (Julia, or to a lesser extent Lua/JS/Clojure even compiled Scala etc) can displace it at. Why would you abandon this wonderfully friendly, malleable language with easily the largest set of options for anything you might like to do, for something nominally faster on an artificial benchmark (which inev…

I've seen some very impressive Julia libraries that perform almost at the level of C code, so I'm not sure if the gap that Julia needs to fill is that large. Also, a quick question: which one is preferred nowadays, Cython or Numba?

Cython is much more mature, works on basically all python code and basically never leads to slowdowns compared to pure python. Unfortunately non of that can really be said about Numba.

That being said, when Numba works it's great and is much easier to work with than cython.

Re: Making Cython as easy as Python

#14

Post by shoyer »

pathsjs wrote: post #8
vegabook wrote: post #3

Cython, Numpy, Numba (and others) are what make me skeptical that any of the numerical computing competitors to Python (Julia, or to a lesser extent Lua/JS/Clojure even compiled Scala etc) can displace it at. Why would you abandon this wonderfully friendly, malleable language with easily the largest set of options for anything you might like to do, for something nominally faster on an artificial benchmark (which inev…

The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...). Some of them try to reintroduce types in order to compile to native code, thereby losing the advantages of dynamic typing. In short, it is a mess, and everyone is trying to add on this because the existing ecosystem is big and the cost of…

> The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...).

I actually disagree completely with your specific examples. These projects have mostly orthogonal goals and are actually quite compatible with each other -- they all speak NumPy arrays. In fact, three of them (Blaze, Numba and Dask) are sponsored by the same company (Continuum Analytics).

Yes, the SciPy ecosystem is a complex beast. There are lots of complementary projects and its not always clear what the best tool for the job is. There are certainly improvements we could make for easier cross-compatibility between libraries (and there are likely projects that could be consolidated), but the number of options you have for scientific computing in Python is an indication of a very robust ecosystem.

Re: Making Cython as easy as Python

#15

Post by shoyer »

zzleeper wrote: post #4
vegabook wrote: post #3

Cython, Numpy, Numba (and others) are what make me skeptical that any of the numerical computing competitors to Python (Julia, or to a lesser extent Lua/JS/Clojure even compiled Scala etc) can displace it at. Why would you abandon this wonderfully friendly, malleable language with easily the largest set of options for anything you might like to do, for something nominally faster on an artificial benchmark (which inev…

I've seen some very impressive Julia libraries that perform almost at the level of C code, so I'm not sure if the gap that Julia needs to fill is that large. Also, a quick question: which one is preferred nowadays, Cython or Numba?

It really depends on your use case. Cython is more flexible and suitable for libraries, but Numba is easier to write. I elaborated on this in a recent blog post: http://eng.climate.com/2015/04/09/numba-vs-cython-how-to-cho...

Re: Making Cython as easy as Python

#17

Post by versteegen »

python_que wrote: post #7

I am using Python to solve combinatorial problems, and so my code relies heavily on the itertools library. My question is if I would still get a considerable speed-up if I rewrite some of my code in Cython, given my reliance on itertools?

As dagw said, if your inner loops aren't inside a .pyx file, then Cython can do little to help. Even if there are, if it involves lots of fiddling with data structures like dicts and lists, or making use of dynamic or high level features of Python like generators or classes or list comprehensions, then Cython would provide only a small speed increase (say, 20%), because almost all the running time will be spent inside the Python runtime. To get the biggest speed ups out of Cython you need to write C-like code with type annotations. E.g. "for x in range(...):" loops.

Re: Making Cython as easy as Python

#18

Post by chuckbot »

cbsmith wrote: post #16

"Runcython aims to simplify the process of using Cython without sacrificing scalability." I know this is being a semantic weenie, but I hate when people use scalability when they mean efficiency.

And I hate when people use efficiency when they mean speed.

Python -> Cython speeds up you program but does not change the complexity of your algorithm.

Re: Making Cython as easy as Python

#19

Post by oliwarner »

chuckbot wrote: post #18
cbsmith wrote: post #16

"Runcython aims to simplify the process of using Cython without sacrificing scalability." I know this is being a semantic weenie, but I hate when people use scalability when they mean efficiency.

And I hate when people use efficiency when they mean speed. Python -> Cython speeds up you program but does not change the complexity of your algorithm.

How do you suppose it speeds it up then?

I'll give you "the complexity of your algorithm" likely doesn't change, but how that algorithm is processed is usually more efficient when compiled through C, rather than running through a Python interpreter, if only because of how it's running.

Re: Making Cython as easy as Python

#20

Post by pathsjs »

shoyer wrote: post #14
pathsjs wrote: post #8

Earlier quoted context omitted.

The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...). Some of them try to reintroduce types in order to compile to native code, thereby losing the advantages of dynamic typing. In short, it is a mess, and everyone is trying to add on this because the existing ecosystem is big and the cost of…

> The problem with the Python ecosystem is that there exist a lot of libraries for scientific computation mostly incompatible with each other (blaze, numpy, numba, numexpr, dask...). I actually disagree completely with your specific examples. These projects have mostly orthogonal goals and are actually quite compatible with each other -- they all speak NumPy arrays. In fact, three of them (Blaze, Numba and Dask) are…

I am not quite sure. Let me be more specific.

Dask requires you to encode computations as a graph explicitly, essentially forcing you to write an AST manually. This means that you are sidestepping the normal language mechanisms to encode program flow, and doing so is essentially incompatible with anything else. Moreover, it does not use numpy arrays - it uses its own internal format that you can convert to a numpy array when needed as explained the overview: http://dask.pydata.org/en/latest/array-overview.html

Numba is another cool project, but - again - it deviates from the mainstream Python. Not every Python function is compilable by Numba, and arithmetic is fixed-size. This means that existing Python code may or may not be compilable by Numba, and even if it works one has to carefully check that arbitrary precision arithmetic is not used. So, it is nice when it works, but is little more than a way to write C-like code with a Pythonish syntax (in which case I greatly prefer Nim, that has actual dispatching on types, generics and so on).

Blaze is in a strange position which I do not fully understand. Apparently it uses Numpy arrays, but at the same time the Numpy author states it should be a Numpy replacement: http://technicaldiscovery.blogspot.it/2012/12/passing-torch-...

Numexpr requires you to write your code as strings, so it is essentially a separate interpreter. It is nice that it is fast, but it does not play with normal Python code.

Not a single project of these works on PyPy, which is the only fast interpreter for non-numeric Python code, nor on Jython, which is the only multithreaded Python interpreter.

Do not misunderstand me: I enjoy Python for many things, and internally we use it a lot. But I am frustrated that a lot of effort seems to be dedicated to fix things at the language level starting from a powerful library ecosystem, where I would like to see something solid at the language level that evolves libraries instead

Post reply on HN