Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How fast can we make interpreted Python?

phi-node.com

Post reply on HN
Print view View original

61–70 of 83 posts

Re: How fast can we make interpreted Python?

#61

Post by pekk »

bsimpson wrote: post #4

As a web developer, I've often heard neckbeards bickering about Python's performance, but haven't had a real point-of-reference to understand how bad it can be until recently. I've started working on a side project that processes geo data in AppEngine. My dataset includes many long lists of numbers (lats, longs, altitudes, timestamps, etc.). A 700 route dataset is about 25MB in a sqlite database, but trying to access…

On the contrary, App Engine bears responsibility for the design of db.Model while Python does not.

Re: How fast can we make interpreted Python?

#62

Post by pekk »

lmm wrote: post #54
Fomite wrote: post #49

Earlier quoted context omitted.

I'm sure this will come as a surprise to everyone using Python's rather well developed scientific computing stack.

You mean the one that's implemented mostly in FORTRAN with python as a mere coordinating layer on top? (And don't get me wrong, it's an effective approach that plays to the strengths of both languages. But it's not doing "heavy lifting" in python)

Why should language choices be guided by the bizarre artificial scenario where other languages do not exist?

Re: How fast can we make interpreted Python?

#63

Post by pekk »

mateo411 wrote: post #38
dexterous wrote: post #36

Earlier quoted context omitted.

I'm not denying that IO is important to a certain class of Python programs too; but I still can't see what leads to the conclusion that IO bound programs are the primary use case for Python. There are enough evented and async io libraries being built for just about every platform right now, that doesn't make IO the center of any of them.

And then there is the GIL. So if you want to squeeze more performance out of your CPU bound task. You need to use the multiprocess module, because Python threading does not work well for CPU bound tasks.

Since you are talking about CPU-bound tasks, it is relevant that numpy array operations release the GIL.

Re: How fast can we make interpreted Python?

#64

Post by pekk »

sherjilozair wrote: post #28
dschiptsov wrote: post #23

The answer is very simple: 1) Know what ought to be done - do it and send the patches. 2) Need "speed" - write that part in C.)

2) Need "speed" - write that part in C.) That's not so easy. Interfacing Python and C code is also incredibly hard, and no one true way exists.

It might be incredibly hard if you do not know C.

Re: How fast can we make interpreted Python?

#65

Post by tehwalrus »

bsimpson wrote: post #4

As a web developer, I've often heard neckbeards bickering about Python's performance, but haven't had a real point-of-reference to understand how bad it can be until recently. I've started working on a side project that processes geo data in AppEngine. My dataset includes many long lists of numbers (lats, longs, altitudes, timestamps, etc.). A 700 route dataset is about 25MB in a sqlite database, but trying to access…

I often find this problem with python, although usually it is the parsing code; the code that loads all the data up, that actually uses the high-water-mark of RAM.

For example, I had a 100MB JSON file that I tried to use the stdlib json library to load. It quickly used >8GB (my machine's RAM) and started paging, dragging everything to a halt. This is partly because the stdlib JSON parser is written in python.

Now, if you switch to a small, clever implementation called cjson[1], it can load the whole thing without bumping 3-400MB in RAM, and the high watermark is the data at the end. Much better!

So, in summary, be careful that the important part of your code is the one that uses all the RAM - and that it's not some "hello world" quality stdlib code that's killing you. If it is, and there isn't a cjson for the job, I've found wrapping C/C++ libraries with Cython[2] a simple way to solve the problem without too much hassle (generally only a couple of days work at a time if you're tight, and only wrap the functions you actually need to use yourself.)

[1] https://pypi.python.org/pypi/python-cjson - although there's a 1.5.1 out there somewhere with a fix for a bug that loses precision on floats...which is the only one I use personally. It's so hard to find that I keep a copy of the source in my Dropbox for when I need it!

[2] http://cython.org/ - although of course actually using cython means you can't take advantage of pypy, IronPython, and other "faster" implementations because you're tied to the cpython C interface forever.

Re: How fast can we make interpreted Python?

#66

Post by MostAwesomeDude »

laurencerowe wrote: post #43
cwzwarich wrote: post #32

Earlier quoted context omitted.

Thanks for the correction, I wasn't aware that Python uses floats. I would guess that most new languages starting today would use doubles instead of floats.

'''...almost all platforms map Python floats to IEEE-754 “double precision”.''' http://docs.python.org/2/tutorial/floatingpoint.html

Python as a language requires the float type to have at least double precision. This is sadly not documented explicitly, but the big four Python implementations all respect this.

Re: How fast can we make interpreted Python?

#67

Post by MostAwesomeDude »

iskander wrote: post #19
jerf wrote: post #15

It seems to me over the past ten years I've heard this story so many times: "Python sucks. Let's do the obvious thing that makes it faster." Then, a month or two later, "I did the obvious thing and it's sometimes faster but often slower, net no gain or possible loss." to which the response is obviously "No sale." I say this merely as an interesting observation. I've come to consider this a de facto counterargument to…

>It may be theoretically true, but in practice, as nice as Python may be to use, it has proved a very difficult language to speed up. I disagree with you. Python isn't much harder to speed up than Lua and in some ways it's better behaved than JavaScript. Still, both of those languages enjoy implementations significantly faster than CPython. Really, it's not the semantics of the language which hold back Python's perfo…

You are wrong.

Python's object model is incredibly rich in ways that JavaScript and Lua don't even come close to touching. Let me list some things that you'll see in Python code that you're not gonna see in JS or Lua:

* Objects that don't extend the object hierarchy (you don't have to extend from `object`)

* Types that don't extend the type hierarchy (Python has full metaclassing)

* Any object can elect to become callable; calls are almost message passes

* Two different levels of message-passing method/attribute rewriting (__getattr__ and __getattribute__)

* Descriptors, such as properties (no, real properties) are baked into the object model

* The table of globals can be altered at any time, frustrating static analysis

* The table of locals can be altered too!

* The table of builtins can be altered!! (Is nothing sacred?)

In addition, PyPy did not start out as a partial evaluator and meta-tracing JIT generator. What you're seeing is the result of about a decade of work and a half-dozen iterations. They started out with something much like the thing that you would expect to see, but just like every other Python JIT project, they learned that Python is complex and difficult to optimize.

So, uh, you're wrong. Sorry.

Re: How fast can we make interpreted Python?

#68

Post by JoshTriplett »

cwzwarich wrote: post #32
JoshTriplett wrote: post #24

Earlier quoted context omitted.

> 3) Pick your value encoding carefully. You almost always want fast immediate integers. On 64-bit platforms it is quite common these days to repurpose some of the NaN range in IEEE doubles for type tags to enable storing doubles in immediate values. That technique applies more to JavaScript, which uses doubles as the standard number type, than in Python, which has both integers and floats. Still a good idea to make…

Thanks for the correction, I wasn't aware that Python uses floats. I would guess that most new languages starting today would use doubles instead of floats.

I didn't mean that Python uses "float" instead of "double"; I meant that Python has both integer and floating-point types, not just floating-point types as in JavaScript.

Re: How fast can we make interpreted Python?

#69

Post by silverlake »

MostAwesomeDude wrote: post #67
iskander wrote: post #19

Earlier quoted context omitted.

>It may be theoretically true, but in practice, as nice as Python may be to use, it has proved a very difficult language to speed up. I disagree with you. Python isn't much harder to speed up than Lua and in some ways it's better behaved than JavaScript. Still, both of those languages enjoy implementations significantly faster than CPython. Really, it's not the semantics of the language which hold back Python's perfo…

You are wrong. Python's object model is incredibly rich in ways that JavaScript and Lua don't even come close to touching. Let me list some things that you'll see in Python code that you're not gonna see in JS or Lua: * Objects that don't extend the object hierarchy (you don't have to extend from `object`) * Types that don't extend the type hierarchy (Python has full metaclassing) * Any object can elect to become cal…

is Python more complicated than Lisp or Scheme or Smalltalk or Self? All are dynamically typed and infinitely flexible.

Re: How fast can we make interpreted Python?

#70

Post by toolslive »

xenonflash wrote: post #51

I don't think its correct to say the CPython is slow. What you can more accurately say about CPython is that the performance is highly variable. Some things are very fast, while others are comparatively slow. The slow things tend to be the sort of numerical loops that you see in micro-benchmarks. It's no coincidence that the version of Python in the linked article saw its greatest speed up in a numerical loop, but on…

it really is a slow interpreter. Python comes with a pystone benchmark, and CPython is invariably the slowest of all interpreters. This being said, if you take a look at it's implementation, then it's immediately obvious why. The interpreter is a basically a simple C switch, with no optimization whatsoever. Simply threading the interpreter would make it about a factor 2 faster (at least that's what the experts claim you gain by threading)
Post reply on HN