Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Compiler for Big Data

continuum.io

Post reply on HN
Print view View original

11–20 of 36 posts

Re: A Python Compiler for Big Data

#11

Post by frozenport »

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

I would go with C/C++ as the ways to address performance are well studied. There are many tools out there like callgrind or nvvp that will make it pain-free.

I can narrow down performance in C/C++ quite quickly, but neither I nor anybody I know has done much of this for Python. Many people who I work with consider a Python implementation a prototype, while Fortran/C/C++ is mature real code worthy of attention.

The only real downside is that C/C++ requires a little knowledge of the POSIX/LINUX or Windows. This represents a learning curve, but when you are over it, it represents quite durable long lasting skills.

Re: A Python Compiler for Big Data

#12

Post by greenonion »

So is there anyone using Python for machine learning in production systems (i.e. not just for prototyping). I would love to do it but seems Java/Mahout is a safer choice, performance-wise.

I wonder whether Blaze is a step towards that direction.

Re: A Python Compiler for Big Data

#13

Post by travisoliphant »

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

Cython is helpful, but you have to spell out a lot of type-information that is not specifically necessary. You might also try Numba --- easiest way to get it is via Anaconda CE or Wakari. Both at http://continuum.io

Re: A Python Compiler for Big Data

#14

Post by hyperbovine »

omni wrote: post #5
Caligula wrote: post #4

I read about continuum after the fellow who developed numpy left a few days ago to work on continuum. I am curious to see actual projects using continuum. So some sort of writeups.

You're being downvoted because Travis Oliphant, the original author of Numpy, is also a co-founder of Continuum Analytics.

I figure he's talking about http://news.ycombinator.com/item?id=4931027. Not sure where the downvoting comes in...

Re: A Python Compiler for Big Data

#15

Post by rerere »

I'm going to get downvoted for saying this, but you guy are really wasting your time. Python was never intended to be fast; even Guido considers it "fast enough," which is a polite way of saying it's going to stay slow. Yeah, you could use weird hacks or write half your program in C, but why?

There are a ton of good languages out there that can give good performance on these kind of problems AND don't have the problems of C++. I mean you've got Golang, Scala, or Erlang for starters. Just sayin'.

Re: A Python Compiler for Big Data

#16

Post by law »

greenonion wrote: post #12

So is there anyone using Python for machine learning in production systems (i.e. not just for prototyping). I would love to do it but seems Java/Mahout is a safer choice, performance-wise. I wonder whether Blaze is a step towards that direction.

I use Python for nearly all of my ETL processes that involve text processing. Even in production systems, I'd be hard-pressed to admit any significant performance issues. Python facilitates implementing algorithms in a functional style, which I tend to prefer over the imperative style (i.e., Java). With C++11 and boost, I'm able to translate my Python code to C++ while preserving the functional style, which has immensely simplified prototyping/deploying NLP/ML algorithms while simultaneously begetting enormous performance gains. I see Python as an extremely viable alternative to Java.

Re: A Python Compiler for Big Data

#17

Post by greenonion »

law wrote: post #16
greenonion wrote: post #12

So is there anyone using Python for machine learning in production systems (i.e. not just for prototyping). I would love to do it but seems Java/Mahout is a safer choice, performance-wise. I wonder whether Blaze is a step towards that direction.

I use Python for nearly all of my ETL processes that involve text processing. Even in production systems, I'd be hard-pressed to admit any significant performance issues. Python facilitates implementing algorithms in a functional style, which I tend to prefer over the imperative style (i.e., Java). With C++11 and boost, I'm able to translate my Python code to C++ while preserving the functional style, which has immen…

You got me a bit confused here. If I understand correctly what you 're saying, you 're still using Python for prototyping the core algorithms and C++ in actual production systems. I'm not saying Python is not good for production systems in general, I'm wondering whether it is good enough for real-world implementations of machine learning algorithms.

Also, I believe most people would consider Java as an alternative to C++, hence all the Java-based Apache projects, such as Mahout, Solr etc.

Re: A Python Compiler for Big Data

#18

Post by skriticos2 »

frozenport wrote: post #11
rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

I would go with C/C++ as the ways to address performance are well studied. There are many tools out there like callgrind or nvvp that will make it pain-free. I can narrow down performance in C/C++ quite quickly, but neither I nor anybody I know has done much of this for Python. Many people who I work with consider a Python implementation a prototype, while Fortran/C/C++ is mature real code worthy of attention. The on…

I think from the development effort a more sensible approach is to build the whole thing in Python, then profile your application and find the performance bottlenecks. Then get your hands a little dirty with the Python C API. This way you can gain good performance without wasting too much time.

http://docs.python.org/3/extending/extending.html

http://docs.python.org/3/c-api/index.html

Re: A Python Compiler for Big Data

#19

Post by cmccabe »

law wrote: post #16
greenonion wrote: post #12

So is there anyone using Python for machine learning in production systems (i.e. not just for prototyping). I would love to do it but seems Java/Mahout is a safer choice, performance-wise. I wonder whether Blaze is a step towards that direction.

I use Python for nearly all of my ETL processes that involve text processing. Even in production systems, I'd be hard-pressed to admit any significant performance issues. Python facilitates implementing algorithms in a functional style, which I tend to prefer over the imperative style (i.e., Java). With C++11 and boost, I'm able to translate my Python code to C++ while preserving the functional style, which has immen…

Have you tried Scala? It might let you write in a functional style and then not have to translate it to something else. Please don't interpret this as a troll; I'm genuinely curious what the pros/cons of these approaches are.

Re: A Python Compiler for Big Data

#20

Post by winter_blue »

frozenport wrote: post #11
rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

I would go with C/C++ as the ways to address performance are well studied. There are many tools out there like callgrind or nvvp that will make it pain-free. I can narrow down performance in C/C++ quite quickly, but neither I nor anybody I know has done much of this for Python. Many people who I work with consider a Python implementation a prototype, while Fortran/C/C++ is mature real code worthy of attention. The on…

> Fortran/C/C++ is mature real code worthy of attention

Just be prepared for Drew Houston, Paul Graham et al. to come after you whipping their lashes.. (tongue in cheek)

Post reply on HN