Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Compiler for Big Data

continuum.io

Post reply on HN
Print view View original

21–30 of 36 posts

Re: A Python Compiler for Big Data

#21

Post by kkuduk »

travisoliphant wrote: post #13
rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

Cython is helpful, but you have to spell out a lot of type-information that is not specifically necessary. You might also try Numba --- easiest way to get it is via Anaconda CE or Wakari. Both at http://continuum.io

Here is a pretty nice example how to use Numba and some expectations about the speedup you may get: https://www.wakari.io/nb/travis/numba

Re: A Python Compiler for Big Data

#22

Post by draegtun »

freyrs3 wrote: post #10
chadillac83 wrote: post #9

Earlier quoted context omitted.

LUA might be good for this too, it's pretty fast on it's own but from what I've read (not tested mind you) their C API is supposed to be pretty great. http://benchmarksgame.alioth.debian.org/u32/which-programs-a... http://en.wikipedia.org/wiki/Lua_(programming_language)#C_AP...

Lua has some areas where it excels in performance, but using Python you can leverage 10 years of work on numeric libraries that unrivaled in any other general purpose language. NumPy and SciPy are extremely powerful.

One rival would be PDL which has been around for over 16 years on Perl.

Some refs:

- http://pdl.perl.org/

- http://blogs.perl.org/users/david_mertens/2012/06/adapting-p...

- http://blogs.perl.org/users/david_mertens/2011/06/cuda-and-t...

Re: A Python Compiler for Big Data

#23

Post by law »

cmccabe wrote: post #19
law wrote: post #16

Earlier quoted context omitted.

I use Python for nearly all of my ETL processes that involve text processing. Even in production systems, I'd be hard-pressed to admit any significant performance issues. Python facilitates implementing algorithms in a functional style, which I tend to prefer over the imperative style (i.e., Java). With C++11 and boost, I'm able to translate my Python code to C++ while preserving the functional style, which has immen…

Have you tried Scala? It might let you write in a functional style and then not have to translate it to something else. Please don't interpret this as a troll; I'm genuinely curious what the pros/cons of these approaches are.

I've never tried Scala, but I suppose I should give it a chance. I'm a fan of Lisp, and the two languages seem to have a lot in common. Scala's expressive type system seems like it has the potential to be both a blessing and a curse, but admittedly, I know next to nothing about the language.

Re: A Python Compiler for Big Data

#24

Post by law »

greenonion wrote: post #17
law wrote: post #16

Earlier quoted context omitted.

I use Python for nearly all of my ETL processes that involve text processing. Even in production systems, I'd be hard-pressed to admit any significant performance issues. Python facilitates implementing algorithms in a functional style, which I tend to prefer over the imperative style (i.e., Java). With C++11 and boost, I'm able to translate my Python code to C++ while preserving the functional style, which has immen…

You got me a bit confused here. If I understand correctly what you 're saying, you 're still using Python for prototyping the core algorithms and C++ in actual production systems. I'm not saying Python is not good for production systems in general, I'm wondering whether it is good enough for real-world implementations of machine learning algorithms. Also, I believe most people would consider Java as an alternative to…

I use Python in production for text pre-processing and other ETL-related processes, which is part of a larger reinforcement learning approach. Additionally, I use Python to prototype the core ML algorithms, which I sometimes re-implement in C++. However, for many of those algorithms, numpy actually performs identically to BLAS in C++.

Re: A Python Compiler for Big Data

#25

Post by greenonion »

law wrote: post #24
greenonion wrote: post #17

Earlier quoted context omitted.

You got me a bit confused here. If I understand correctly what you 're saying, you 're still using Python for prototyping the core algorithms and C++ in actual production systems. I'm not saying Python is not good for production systems in general, I'm wondering whether it is good enough for real-world implementations of machine learning algorithms. Also, I believe most people would consider Java as an alternative to…

I use Python in production for text pre-processing and other ETL-related processes, which is part of a larger reinforcement learning approach. Additionally, I use Python to prototype the core ML algorithms, which I sometimes re-implement in C++. However, for many of those algorithms, numpy actually performs identically to BLAS in C++.

I get it now, thanks. It's very interesting, maybe I will give Python for ML a chance!

Re: A Python Compiler for Big Data

#26

Post by andrewcooke »

how does this compare to theano? it seems like some of the ideas are similar?

http://deeplearning.net/software/theano/

in general, i like (ie i don't see a better solution than) the idea of having an AST constructed via an embedded language that is implemented by a library. but it does have downsides - integration with other python features is going to be much more limited (it gives the illusion of a python solution, but in practice you're off in some other world that only looks like python).

are there more details? i guess the AST is fed to something that does the work. and that something will have an API and be replaceable. but is that something also composable? does it have, say, a part related to moving data and another to evaluating data? so that you can combine "distributed across local machines" with "evaluate on GPU"?

Re: A Python Compiler for Big Data

#27

Post by freyrs3 »

andrewcooke wrote: post #26

how does this compare to theano? it seems like some of the ideas are similar? http://deeplearning.net/software/theano/ in general, i like (ie i don't see a better solution than) the idea of having an AST constructed via an embedded language that is implemented by a library. but it does have downsides - integration with other python features is going to be much more limited (it gives the illusion of a python solution,…

> how does this compare to theano? it seems like some of the ideas are similar?

It's quite similar, we just take some of the ideas farther and try to generalize the data storage to include storage backends that data scientists use more frequently ( i.e. SQL, CSV, S3, etc ). We're very friendly with the Theano developers and hope to bridge the projects with a compatibility layer at some point.

> (it gives the illusion of a python solution, but in practice you're off in some other world that only looks like python).

I would argue that's what make Python a great numeric language, and NumPy so succesfull. You get this high level language where you can express domain knowledge but also this 1:1 mapping between fast code execution at the C level. Blaze is the continuation of that vision

> i guess the AST is fed to something that does the work. and that something will have an API and be replaceable.

Precisely, we build up a intermediate form called ATerm out of the construction expression objects, do type inference, graph rewriting, and then pattern match our layout, metadata, and type information against a number of backends to find the most optimal one to perform execution. Or if needed we build a custom kernel with Numba informed by all this type and data layout information we've inferred from the graph.

We don't aim to solve all the subproblems in this area ( expression optimization passes, distributed scheduling ) but I think we have a robust enough system that others can build extensions to Blaze to do expression evaluation in whatever fashion they like.

> are there more details?

Yes! See: http://blaze.pydata.org/

Re: A Python Compiler for Big Data

#28

Post by dwiel »

greenonion wrote: post #12

So is there anyone using Python for machine learning in production systems (i.e. not just for prototyping). I would love to do it but seems Java/Mahout is a safer choice, performance-wise. I wonder whether Blaze is a step towards that direction.

We also use python in production at plotwatt for machine learning. We started by prototyping in matlab and then porting to c++, but have since found it much much easier to just do everything in python and numpy. When speed was an issue, we slightly changed the way we implemented the algorithm rather than implement the same algorithm in a faster language. Admittedly this isn't always possible.
Post reply on HN