Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Should Compile Your Python and Here’s Why

glyph.twistedmatrix.com

Post reply on HN
Print view View original

51–60 of 109 posts

Re: You Should Compile Your Python and Here’s Why

#51

Post by mule1 »

0xbadcafebee wrote: post #38
mule1 wrote: post #28

Not to be shit-eating, but I never understood why people use python instead of Go other than for ML teams. I program mostly in Java and Go so would love a perspective.

Probably for the same reasons people use Java and Go instead of C. It's just more convenient sometimes. But sometimes not. In terms of building a team at a company? In my experience it's been arbitrary. Manager / Lead has experience in language X, decides to hire people who also know it. Or all the other teams are already using language X. Or Manager / Lead has heard "X language is good at Y" and decides to go with t…

Yeah I get that, the same reason why I question some of the teams I've been on choice of JS frameworks, it's just what the developers were familiar with.

Go compiles so fast and reads cleaner imo, that I never saw the benefit of python. I always thought debugging python was clunky, could be unfamiliarity with interpreted languages. Also a few times I needed to use an ODBC driver with python left a bad taste in my mouth. But overall I get what you're saying, my new team writes all of their lambdas in python so I'm going to have to learn.

Re: You Should Compile Your Python and Here’s Why

#52

Post by linsomniac »

lorenzfx wrote: post #39

Does anyone have any personal experience in applying mypyc on type-hinted, but otherwise not specially optimized python code? I'm especially interested in what kind of performance speed-ups could be achieved.

glyph's article goes into this case and finds a 25% speed improvement in that case.

Re: You Should Compile Your Python and Here’s Why

#53

Post by linsomniac »

mjburgess wrote: post #21

I have rarely read a tech article this long without giving over to skim reading till the bottom. The tone and empathy with the target audience ('s thought process) is on-point.

glyph is ridiculously good at the intersection of smart, funny, empathetic.

Re: You Should Compile Your Python and Here’s Why

#54

Post by adgjlsfhk1 »

VWWHFSfQ wrote: post #44
pharmakom wrote: post #37

Earlier quoted context omitted.

I don’t always know how fast will be fast enough when I start a project and the risk of needing a rewrite really puts me off Python.

This is premature optimization right

Yes and no. One of the biggest disadvantages of python is that the way people typically optimize python code is to 1. rewrite in numpy 2. rewrite in Cython (or other python compiler) 3. rewrite in C++

The problem with this workflow is that it means you end up rewriting large chunks of your code over several years in systems that have fairly different idioms. If you suspect you might in the future be performance bottle-necked, writing your code in a faster (but still productive) language to start can be a lot better because you can then improve performance more incrementally without multiple full rewrites.

Re: You Should Compile Your Python and Here’s Why

#55

Post by detaro »

marcosdumay wrote: post #40
FrenchAmerican wrote: post #29

I'm learning Python. Do someone knows a resource to learn the well-known tricks for "fast" Python? A book, a website, a cheat sheet or even a MOOC or part of a MOOC?

I would advise against optimizing your Python programs for speed. Fighting against a language's nature never leads to good results, it's much easier and better to use a faster language if (or where) you need the speed. This is even more relevant if you are still learning the language. Focus on learning it, and leave arcane always changing implementation details for after you know it well.

Or rather, focus on the big things. Algorithmic complexity etc.

Also for all languages where you want to do performance work: Learn to use a profiler, so you can find the things that matter. And then selectively look what you can improve there. Even in Python, some hacks or slightly unergonomic patterns in a really hot loop can be worth a lot.

Re: You Should Compile Your Python and Here’s Why

#56

Post by js2 »

joelthelion wrote: post #48
1vuio0pswjnm7 wrote: post #14

"Python is Slow, And That's Fine, Because It's Fast Enough" At least this is honest. No matter what the script does, no matter how fast the libraries, the Python intepreter has a slow startup time. On multiple occasions I have seen people commenting on HN argue that Python is not slow. I think for these commenters Python is "fast enough". For others, like me, it may not be "fast enough". IOW, the question is not whet…

> the Python intepreter has a slow startup time. This really depends on your perspective. Sure, it's a lot slower than running a native binary. But it's still fast enough for interactive tools that you run often. If you compare that to java tools, for example, which take seconds if not dozens of seconds to start (gradle, I'm looking at you!), python is far better.

> (gradle, I'm looking at you!)

gradle has pretty much the worst ergonomics of any developer tool I've ever used[1], so that's a pretty low bar.

[1] e.g. "Something went wrong. Re-run with -debug or -info. Now here's 5000 lines of useless stack traces that won't help at all." Nevermind an ecosystem of plugins where it's not clear where the DSL ends and API starts, and makes it virtually impossible to locate the actual source of an error message, if you get one at all.

Re: You Should Compile Your Python and Here’s Why

#57

Post by im3w1l »

adgjlsfhk1 wrote: post #54
VWWHFSfQ wrote: post #44

Earlier quoted context omitted.

This is premature optimization right

Yes and no. One of the biggest disadvantages of python is that the way people typically optimize python code is to 1. rewrite in numpy 2. rewrite in Cython (or other python compiler) 3. rewrite in C++ The problem with this workflow is that it means you end up rewriting large chunks of your code over several years in systems that have fairly different idioms. If you suspect you might in the future be performance bottl…

I think the idea is to rewrite only the hot part in c++. If there isn't a small hot part, then yes python is a poor choice.

Re: You Should Compile Your Python and Here’s Why

#58

Post by js2 »

sgtnoodle wrote: post #11

I've been writing some python the last few days. Today I did something terrible. I like to believe it was out of wisdom rather than ignorance. I needed to do some nuanced bit manipulations, and so I built up a string of ascii 0's and 1's, sliced the string, and then converted back to integers. I worked on a codebase many years ago when I was an intern. That code would read frames off a CAN bus in a car. For some reas…

> Today I did something terrible. I like to believe it was out of wisdom rather than ignorance. I needed to do some nuanced bit manipulations, and so I built up a string of ascii 0's and 1's, sliced the string, and then converted back to integers.

Similar, in awk:

https://gist.github.com/jaysoffian/e41ca479d70e60efe59fded93...

Re: You Should Compile Your Python and Here’s Why

#59

Post by Sinidir »

taneq wrote: post #25
sgtnoodle wrote: post #11

I've been writing some python the last few days. Today I did something terrible. I like to believe it was out of wisdom rather than ignorance. I needed to do some nuanced bit manipulations, and so I built up a string of ascii 0's and 1's, sliced the string, and then converted back to integers. I worked on a codebase many years ago when I was an intern. That code would read frames off a CAN bus in a car. For some reas…

Reminds me of that Tao of Programming koan (if I remember it rightly) that went something like: The novice, in his frustration, struck the side of his computer. The master walked over and asked what he was doing. The novice exclaimed, "my computer is not working and I do not know why!" The master admonished him, saying "you cannot solve the problem by striking the computer without knowing what is wrong." Then the mas…

That is hilarious.

Re: You Should Compile Your Python and Here’s Why

#60

Post by 8organicbits »

mule1 wrote: post #28

Not to be shit-eating, but I never understood why people use python instead of Go other than for ML teams. I program mostly in Java and Go so would love a perspective.

For what it's worth, go is still "new" in my book. I'm seeing 2012 for the first 1.0 public release. In contrast, we've had python since 1991. There's a large number of programmers who haven't had a chance to be exposed to go. I have used it, and enjoyed it. I'd expect, but haven't confirmed, that python has a larger collection of libs (especially long tail).

Although, looking now I'm seeing a short support window for go releaI. That can hurt uptake.

Like go 1.18 was released 2022-03-15 and will EOL Q1 2023.

Lots of projects want long term support so that churn isn't desired. I'm curious how teams handle that.

Post reply on HN