11–20 of 83 posts
Python is typically used to build applications that are IO bound. Squeezing more performance out of the interpreter is not going to translate to any real gains for most Python users these days.
As a web developer, I've often heard neckbeards bickering about Python's performance, but haven't had a real point-of-reference to understand how bad it can be until recently. I've started working on a side project that processes geo data in AppEngine. My dataset includes many long lists of numbers (lats, longs, altitudes, timestamps, etc.). A 700 route dataset is about 25MB in a sqlite database, but trying to access…
Earlier quoted context omitted.
A dataset with "many long lists of numbers" sounds like an ideal use case for NumPy, have you tried using that?
GAE allows only pure Python. No binary modules like NumPy.
Python is typically used to build applications that are IO bound. Squeezing more performance out of the interpreter is not going to translate to any real gains for most Python users these days.
I'm curious, what leads you to the conclusion that 'Python is typically used to build applications that are IO bound'? Also, what exactly are you referring to when you say 'IO bound'?
I say this merely as an interesting observation. I've come to consider this a de facto counterargument to the claim that languages aren't slow, only implementations are. It may be theoretically true, but in practice, as nice as Python may be to use, it has proved a very difficult language to speed up. (PyPy has taken a very good run at it, but it sure wasn't a case of "I'll just do this easy, obvious thing." PyPy seems to have hit Python's performance with multiple PhD-thesis level attacks, and it's still certainly not C in the general case.)
Earlier quoted context omitted.
I'm curious, what leads you to the conclusion that 'Python is typically used to build applications that are IO bound'? Also, what exactly are you referring to when you say 'IO bound'?
Python is commonly used in web services, where retrieving information from a database and transmitting the results via slow network connections are what takes up the most time, not processing the data in between with Python.
Earlier quoted context omitted.
Python is commonly used in web services, where retrieving information from a database and transmitting the results via slow network connections are what takes up the most time, not processing the data in between with Python.
I don't buy that. I've had enough people relate Python's roots in the sysad tool belt and it's application in processing large data sets to believe that it's intended use case could be that limited.
Python is used just about everywhere for just about everything (sometimes properly, sometimes poorly). While there are a lot of web sites that run Python, there are also countless other applications that use it that are unrelated to the web. See Scipy as an example.
That said, the earlier poster mentioning that many people are using Python for IO bound processes is not too far a stretch. Why else would Twisted Python exist, and why would the new Tulip async IO stuff be developed?
Python is typically used to build applications that are IO bound. Squeezing more performance out of the interpreter is not going to translate to any real gains for most Python users these days.
Because Python is slow, Python is not used in scenarios where speed is crucial. That much is true.
However, if Python was faster, it would be used in those scenarios, so more people would be using it for speed critical code so it would provide real gains for a great many Python programmers.
This is exactly what happened with JavaScript: before V8 JavaScript was in exactly the same position as Python. Not many people were writing large programs in JavaScript because JavaScript was too slow. V8 sped up JavaScript 10x+ and people started writing much larger apps that do require that speed. If JavaScript speed suddenly dropped to pre-V8 speeds, we would all find the most popular web apps unusably slow.
It seems to me over the past ten years I've heard this story so many times: "Python sucks. Let's do the obvious thing that makes it faster." Then, a month or two later, "I did the obvious thing and it's sometimes faster but often slower, net no gain or possible loss." to which the response is obviously "No sale." I say this merely as an interesting observation. I've come to consider this a de facto counterargument to…
I disagree with you. Python isn't much harder to speed up than Lua and in some ways it's better behaved than JavaScript. Still, both of those languages enjoy implementations significantly faster than CPython. Really, it's not the semantics of the language which hold back Python's performance but rather the fact that extension modules are extremely tightly coupled with a particular interpreter implementation. Lua has a clean interface with C, JavaScript implementations generally force the outside world to use doubly indirect handles on objects. CPython, on the other hand, is shameless in flaunting its internals for the whole world to see.
The only reason that PyPy has taken "multiple PhD-thesis level attacks" to near completion is because their approach is insanely ambitious. They didn't write a JIT. Instead they wrote a toolkit for partially evaluating interpreters on source files and generate native code by tracing an interpreter while it itself runs a program. It's nuts! It's amazing that PyPy works and the amount of effort is totally unsurprising.
Had they gone a more traditional route, the whole thing could have been done in a year or two. They would, however, still face resistance from a Python community that wants to neither give up nor rewrite their PyObject-laced libraries.
There are a couple of things you want to do (some of which overlap with the article): 1) Use a register-based VM (with a sliding and growing register file) instead of a stack-based VM. In theory you can make a stack-based VM fast with lots of macroinstructions that fuse smaller operations together, but it isn't worth it. 2) Use inline caching for method calls, property accesses, and primitive operations that do type…
Do you mean...group all the frequent operations together so they overlap on cache lines? It's hard to tell how much this would help, have you tried it?