Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Python Is Slow: Looking Under the Hood

jakevdp.github.io

Post reply on HN
Print view View original

111–120 of 156 posts

Re: Why Python Is Slow: Looking Under the Hood

#111

Post by ggchappell »

lispm wrote: post #20
bunderbunder wrote: post #15

Earlier quoted context omitted.

Might want to start by asking the core Python community. Check out the first sentence on wiki.python.org. Nowadays it's the norm for interpreted languages to be JIT-compiled. They still universally call themselves interpreted languages because the jitter is an implementation detail, and the use of one does nothing to negate the characteristics they're trying to advertise when describing themselves as interpreted lang…

There are several compiler-only Lisp implementations which can execute source from text. Nobody there would call them interpreter because of that. In Lisp we call them interpreter, when the implementation traverses the source code during execution. If it compiles the source to some byte-code or machine code, we call in Compiler. The incremental nature of a compiler, then does not make it that we would call it an Inte…

> There are several compiler-only Lisp implementations which can execute source from text.

Why not? I would.

Re: Why Python Is Slow: Looking Under the Hood

#112

Post by ggchappell »

jeffreyrogers wrote: post #17
ggchappell wrote: post #10

> 2. Python is interpreted rather than compiled. Can we stop saying things like this? Virtually all Python is compiled, as part of the interpretation process. There is a valid point here, of course. The point is that the top Python compilers do not compile to native code. So say that. Words have meanings. Use them correctly.

There are three primary methods of running code: interpretation, compilation to object code, and running on a VM/JIT. Obviously python is interpreted by any meaningful definition of interpretation and thus it is not compiled in the typically sense of the word (at least not in the implementation everyone uses). The reason human language is so expressive is because we can leave out a lot of context and formalism that i…

> It's pedantic to expect a writer to hedge against every possible interpretation, when their focus should be on communicating clearly in the first place.

Our discipline (computer programming) is, perhaps more than any other, one in which precision of meaning is of paramount importance. I expect people to hedge. Especially when it's easy: "Python is not usually compiled to native code."

In any case, since Python is virtually always compiled, saying it's not is simply false. That's hardly clear communication, I think.

> Obviously python is interpreted by any meaningful definition of interpretation ....

Now, here I'm not sure what you mean. You've already distinguished interpretation from Just-In-Time compilation to Virtual Machine code. Since the latter is pretty much always what happens with Python, would you then say that Python is not interepreted? (Serious question!)

Me, I don't have a problem with these concepts overlapping (and I don't see why many people do). Python is interpreted. The usual mechanism for this is JIT compilation to virtual-machine code, which is then -- in the case of CPython -- interpreted as-is. In contrast, something like "C" is traditionally Ahead-Of-Time (AOT) compiled to native code. Interpreted: no. Compiled: yes. And then there are languages whose interpreters do not involve a compilation step -- although they are becoming rare.

Re: Why Python Is Slow: Looking Under the Hood

#113

Post by rdtsc »

Cyther606 wrote: post #102
csirac2 wrote: post #91

Earlier quoted context omitted.

I don't know why I always have to chip in on "perl is unreadable lol" comments, but over the last 8 years apart from a steady trickle of C coding here and there the bulk of my dayjob has moved around from C/Verilog, then I discovered Ruby, then Perl/R/Python, to full-time Python now. It's true that unsupervised, weak coders using perl turn out worse code than in other languages. But it really doesn't take much to pro…

Check out Nim. import rdstdin, strutils let time24 = readLineFromStdin("Enter a 24-hour time: ").split(':').map(parseInt) hours24 = time24[0] minutes24 = time24[1] flights: array[8, tuple[since: int, depart: string, arrive: string]] = [(480, "8:00 a.m.", "10:16 a.m."), (583, "9:43 a.m.", "11:52 a.m."), (679, "11:19 a.m.", "1:31 p.m."), (767, "12:47 p.m.", "3:00 p.m."), (840, "2:00 p.m.", "4:08 p.m."), (945, "3:45 p.m…

If I had to guess speed is pretty close to C, right.

Nim is pretty awesome and has been around for a while. I wish one of big entities out there Google, Mozilla, Microsoft, Apple would have adopted Nim and ran with it instead of inventing their own langauges.

Re: Why Python Is Slow: Looking Under the Hood

#114

Post by csirac2 »

fidotron wrote: post #108
csirac2 wrote: post #91

Earlier quoted context omitted.

I don't know why I always have to chip in on "perl is unreadable lol" comments, but over the last 8 years apart from a steady trickle of C coding here and there the bulk of my dayjob has moved around from C/Verilog, then I discovered Ruby, then Perl/R/Python, to full-time Python now. It's true that unsupervised, weak coders using perl turn out worse code than in other languages. But it really doesn't take much to pro…

Last time I used Perl it went badly, but it was a typical hacked up mess. For someone mainly in the Python/Java/C++ world is Moose something worth looking at as a mind expansion exercise? Your description makes it sound appealing. And, dare I ask, what about Perl 6?

I want to write something concise and coherent, but it's not happening today :) Instead, assuming you've read the teasers in the Moose manual [1] I'd recommend this [2] interesting comparison of how horizontal code re-use can be achieved in Java, Ruby, PHP and Perl+Moose. There's more philosophical/winding essays on Moose from Chromatic, for example at [3].

Roles/traits/method modifiers/MOP/composability etc. are all great things for getting more reusability... however (and this might sound stupid) the thing that saddens me most when writing Python code is when I find myself adding a bunch of asserts or adding program logic "manually" in situations where I'd normally specify that sort of thing declaratively in a Moose class definition or by referring to a centrally managed Type or delegate stuff through attribute features.

On the other hand, I'm barely into year 2 of full-time Python dev, so perhaps I've yet to find the idiomatic way of doing Python things I used to take for granted in Moose.

Regarding Perl 6, I don't know much about it, except that the original authors of Moose had some inspiration from it.

Is picking up Perl+Moose mind-expanding? It was for me, but I feel that what Moose gave me in Perl was a bit of a band-aid over the fact that it's such a malleable open-ended dynamic language. So as a C++/Java programmer this aspect might not be so enlightening to you, except to see how Moose achieves it in a pretty painless way that I think is very nice and idiomatic for a dynamic language (with the caveats that brings). To put it another way: it gives Perl some of the great benefits of properly declaring things up-front, without the boilerplate/inflexibility pain that I perceive the Java ecosystem's bureaucracy to be (I haven't touched Java for 10 years, so take that with a grain of salt).

If you want to explore some Moose-ish kinds of things within Python, check out [4] (there's another Moose clone in Python that's similarly inactive, sadly) and [5] (Enthought's stuff is perhaps a bit too heavy and incomplete to be the "Moose of the python world", but it gives you a good idea of some of the nice patterns possible when you think outside of the core Python OO featureset).

[1] http://search.cpan.org/dist/Moose/lib/Moose/Manual.pod [2] http://radar.oreilly.com/2014/01/horizontal-reuse-an-alterna... [3] http://modernperlbooks.com/mt/2009/05/perl-roles-versus-inte... [4] https://github.com/frasertweedale/elk [5] http://code.enthought.com/projects/traits/

Re: Why Python Is Slow: Looking Under the Hood

#115

Post by chrisseaton »

vidarh wrote: post #110
kyllo wrote: post #85

Earlier quoted context omitted.

Runtime metaprogramming. In Python you can create new classes and add methods to existing classes at runtime. If you AOT-compile everything to assembly, you have to take away those runtime features, and then it's not Python anymore.

No, you don't. You "just" need slower fallbacks that does metadata lookup for things that are not statically decidable. Consider how many people end up implement their own introspectable C object models.

I think the very real risk here is that you think you're adding those slower fallback features just for edge cases, but in reality they get used in the majority of cases.

I believe that some code paths critical for performance in Rails use metaprogramming, for example.

I did an experiment in this paper http://dl.acm.org/citation.cfm?id=2647517&dl=ACM&coll=DL&CFI... where I showed that just be rearranging some code you can confuse the Rubinius object allocator into using a slow path for instance variables. That gives you both slow performance, and worse than that, unpredictable performance - the programmer doesn't understand what they've done wrong to trigger that fallback performance.

Re: Why Python Is Slow: Looking Under the Hood

#116

Post by vlisivka »

haberman wrote: post #93
cleaver wrote: post #77

Earlier quoted context omitted.

For a general-purpose problem, Perl won't be particularly fast. For a Perl-type problem (scanning and parsing big files), Perl is very fast. Doing a Perl-type problem in a general-purpose language would be considerably slower. However, Python or others will perform much better in the "can I read my own code six months later" benchmark.

> For a Perl-type problem (scanning and parsing big files), Perl is very fast. I think it's a matter of what you're comparing it to. Compared to using Perl for a general-purpose problem, Perl for scanning/parsing is fast. Compared to scanning/parsing with C, Perl is not fast. $ ruby -e '1.upto(1000000) { |n| puts "This is line number #{n}" }' > file $ time perl -ne 'print if /number 12345/' I gave Perl every possible…

You are comparing two interpreters. One is slow, because it is written in C, second is fast, because it is written in C.

Re: Why Python Is Slow: Looking Under the Hood

#117

Post by cityhall »

madengr wrote: post #80

Please forgive my ignorance, but why can't Python just be compiled into assembly like C? The compiler should be able sift through the code and see that "x=1" is an integer, or converts to a float when "x=x+1.0". If I'm doing something like counting from 1 to 10 and summing the count, why can't this be compiled to run the same speed as C? Obviously since it's interpreted, but when I'm done with development, why can't…

Cython, numba, and other projects do this for a subset of the language, but you lose the metaprogramming that makes things like SQLAlchemy possible. You also have to treat most user objects as hashmaps and constantly look up their fields since members can be added or redefined at any time.

Re: Why Python Is Slow: Looking Under the Hood

#118

Post by rhgraysonii »

One of my favorite topics lately is Python optimization. A few weeks ago FB put out a release on how they had done fast randomized SVD and in the implementations section they mentinoned using intels kernel math libraries for the BLAS libraries. Largely the result of this is many more functions are F-Contiguous rather than C-contiguous (in linear algebra, fortan obviously wins here). I had done something similar on a much smaller scale about 4 months ago. https://medium.com/@_devbob/from-0-to-warp-speed-b780a2bc36c...

Does anyone else have some good bits or blog entries on these sorts of inner workings and optimizations?

Re: Why Python Is Slow: Looking Under the Hood

#119

Post by simula67 »

fidotron wrote: post #108
csirac2 wrote: post #91

Earlier quoted context omitted.

I don't know why I always have to chip in on "perl is unreadable lol" comments, but over the last 8 years apart from a steady trickle of C coding here and there the bulk of my dayjob has moved around from C/Verilog, then I discovered Ruby, then Perl/R/Python, to full-time Python now. It's true that unsupervised, weak coders using perl turn out worse code than in other languages. But it really doesn't take much to pro…

Last time I used Perl it went badly, but it was a typical hacked up mess. For someone mainly in the Python/Java/C++ world is Moose something worth looking at as a mind expansion exercise? Your description makes it sound appealing. And, dare I ask, what about Perl 6?

> And, dare I ask, what about Perl 6?

As an ardent Perl lover, I asked this question myself. Shameless self-promotion yada yada :

http://simula67.wordpress.com/2014/03/20/perl-6-evaluation/

Re: Why Python Is Slow: Looking Under the Hood

#120

Post by visarga »

Igglyboo wrote: post #107
haberman wrote: post #93

Earlier quoted context omitted.

> For a Perl-type problem (scanning and parsing big files), Perl is very fast. I think it's a matter of what you're comparing it to. Compared to using Perl for a general-purpose problem, Perl for scanning/parsing is fast. Compared to scanning/parsing with C, Perl is not fast. $ ruby -e '1.upto(1000000) { |n| puts "This is line number #{n}" }' > file $ time perl -ne 'print if /number 12345/' I gave Perl every possible…

I think the takeaway is that perl is "fast enough" for perl type problems. Writing a complicated file parser in C would be a nightmare.

When I have a one-time computation job that takes an hour to write and two hours to run in Perl, but in C takes 10 hours to write and half an hour to run, then Perl is faster than C.

And these one-time/rare/short jobs are much more frequent than intense, high throughput C code like the nginix web server or the node javascript interpreter.

Post reply on HN