Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Python Compiler for Big Data

continuum.io

Post reply on HN
Print view View original

31–36 of 36 posts

Re: A Python Compiler for Big Data

#31

Post by IanOzsvald »

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

I taught High Performance Python covering the tools you mention at PyCon 2012 (and EuroPython last year), maybe my videos+write-up will be helpful. I also cover profiling, shedskin, pyCUDA etc:

http://ianozsvald.com/2012/03/18/high-performance-python-1-f...

Re: A Python Compiler for Big Data

#32

Post by lrem »

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

Cython is good, but sometimes it's a bit tricky to bend it to do exactly what you want[1]. You'll probably still want to write that hot piece in C... But gluing it with Cython is IMHO much nicer than using the plain Python API.

[1] - on the other hand, it comes with a tool explaining exactly how each of your lines of Cython looks in resulting C, with color-coding for high level overview of which pieces translated smoothly

Re: A Python Compiler for Big Data

#33

Post by Erwin »

rpm4321 wrote: post #8

Bit of a tangent, but I'm wondering if anyone here has had any luck with Cython? I'm starting to run into some performance bottlenecks with Python, and so I'm just now looking at Cython, PyPy, Psyco, and... gasp... C. From what little I've read, Cython is supposed to be as easy as adding some typing and modifying a few loops here and there, and you are in business.

Depends on your application. Ideally you want to change your code so as much computation as possible can happen in pure-C code and pure-C data types (using Cython). If you have a big class tree with many callbacks and work spread over hundreds of method, that can be difficult.

Before you go that far, I'd recommend making sure you know all the Python gotchas (for example, maybe you have some inner loop that does for x in range(100000) all the time), that you algorithms are in order. Sometimes even silly microoptimization can make a difference if a small function is a significant amount of your runtime. Using multiple processes with e.g. the multiprocessing module can be an option too.

Depending on what data types you operate on, numpy (and now this new thing) can do some amazing things.

PS: check things like http://packages.python.org/line_profiler/ beyond the ordinary profiling.

Re: A Python Compiler for Big Data

#34

Post by frozenport »

winter_blue wrote: post #20
frozenport wrote: post #11

Earlier quoted context omitted.

I would go with C/C++ as the ways to address performance are well studied. There are many tools out there like callgrind or nvvp that will make it pain-free. I can narrow down performance in C/C++ quite quickly, but neither I nor anybody I know has done much of this for Python. Many people who I work with consider a Python implementation a prototype, while Fortran/C/C++ is mature real code worthy of attention. The on…

> Fortran/C/C++ is mature real code worthy of attention Just be prepared for Drew Houston, Paul Graham et al. to come after you whipping their lashes.. (tongue in cheek)

I'm not scared of a man who speak with a Lisp!

Re: A Python Compiler for Big Data

#35

Post by peatmoss »

law wrote: post #23
cmccabe wrote: post #19

Earlier quoted context omitted.

Have you tried Scala? It might let you write in a functional style and then not have to translate it to something else. Please don't interpret this as a troll; I'm genuinely curious what the pros/cons of these approaches are.

I've never tried Scala, but I suppose I should give it a chance. I'm a fan of Lisp, and the two languages seem to have a lot in common. Scala's expressive type system seems like it has the potential to be both a blessing and a curse, but admittedly, I know next to nothing about the language.

I may be missing something here, but if you're a fan of lisp and want easy interaction with libraries on the JVM, please tell me you've heard of Clojure. It's a modern lisp that strongly favors functional programming, and that has great concurrency support. Plus, there is already a data analysis / statistical platform built on top of it called Incanter.

Re: A Python Compiler for Big Data

#36

Post by lucian1900 »

piqufoh wrote: post #30
lucian1900 wrote: post #29

Interesting approach to modelling data that lives elsewhere, in fact quite similar to SQLAlchemy's.

... but you can't use numpy operations efficiently on SQLAlchemy data

That's not what I meant. Both this and SA turn python expressions into expressions to be run elsewhere, on data that isn't necessarily in the process' memory.
Post reply on HN