Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

You Should Compile Your Python and Here’s Why

glyph.twistedmatrix.com

Post reply on HN
Print view View original

101–109 of 109 posts

Re: You Should Compile Your Python and Here’s Why

#101

Post by bcrl »

fgeiger wrote: post #8

Frankly, I would find it much easier to write the naive C version than this "moderately" optimized Python version in the article although I am much more familiar with Python than with C.

And if he'd applied the same optimizations to the C version, it would have trounced the optimized Python version. I really don't like it when authors unfairly penalize their "not preferred" solution -- it makes me feel like the entire time spent reading the article was nothing more than listening to pointless chest thumping by the author. The whole article was barely about the actual compiling of Python code.

Re: You Should Compile Your Python and Here’s Why

#102

Post by mumblemumble »

js2 wrote: post #56
joelthelion wrote: post #48

Earlier quoted context omitted.

> the Python intepreter has a slow startup time. This really depends on your perspective. Sure, it's a lot slower than running a native binary. But it's still fast enough for interactive tools that you run often. If you compare that to java tools, for example, which take seconds if not dozens of seconds to start (gradle, I'm looking at you!), python is far better.

> (gradle, I'm looking at you!) gradle has pretty much the worst ergonomics of any developer tool I've ever used[1], so that's a pretty low bar. [1] e.g. "Something went wrong. Re-run with -debug or -info. Now here's 5000 lines of useless stack traces that won't help at all." Nevermind an ecosystem of plugins where it's not clear where the DSL ends and API starts, and makes it virtually impossible to locate the actua…

This time last year, I was deep into writing Gradle plugins at work.

Gradle's entire architecture is the best argument I have ever seen for why you should never use a big blob of shared mutable state when all you really needed to do was pass discrete values up and down the call stack.

Compare with Bazel. I appreciate why people bounce off of many of its strictures. But when you just look at the architecture and what it takes to develop for it, it's amazing how much easier it is to grok, while achieving a similar level of flexibility.

Re: You Should Compile Your Python and Here’s Why

#103

Post by mumblemumble »

dralley wrote: post #88
bhandziuk wrote: post #83

Earlier quoted context omitted.

god I love Hg over git. I don't understand why git is more popular.

It's pretty easy to understand. Github.

I suspect that, back in the day, Bitbucket (which was originally hg-only) might have won out over GitHub if Git hadn't also had the huge publicity boost that comes with being written by Linus.

Re: You Should Compile Your Python and Here’s Why

#104

Post by mumblemumble »

teaearlgraycold wrote: post #74

Honestly for a language that’s supposed to be simple and easy to use Python seems really complicated and hard to understand. Anything big or performance critical should probably not be written in Python - and yet people are doing it. And then we end up with a house of cards with this project built on top of the very shaky foundation of mypy.

It's because Python is both simple and easy to use, and complicated and hard to understand.

It's a Janus of a programming language. There are lots of complexities that alternately tie software engineers up in knots and let them go into raptures of blissful hackery. But they generally aren't all that visible in the face it presents to data scientists and devops.

And it's hard to make sweeping statements about "big or performance critical". A couple times in the past few years years I've seen or been involved in projects that successfully replaced big Java applications with Python implementations that were 1% the SLOC and had better performance. They were special situations, to be sure, though more so, I think, in terms of explaining the performance improvement than the cost savings.

Re: You Should Compile Your Python and Here’s Why

#105

Post by BiteCode_dev »

ciupicri wrote: post #99
BiteCode_dev wrote: post #67

Earlier quoted context omitted.

There are plenty of things for which python is not slow, but people using python don't know a lot about programming, and end up with a slow result. A couple of advises: - the right algo will go a long, long way. - know your data structures. E.G: assigning to a slice is ridiculously fast (even while unpacking), memory views may save a lot on byte heavy workloads, heapq and deque are underrated, etc. Also check out htt…

How do you pre-allocate lists and dicts?

l = [None] * size

Re: You Should Compile Your Python and Here’s Why

#106

Post by byteface »

lorenzfx wrote: post #39

Does anyone have any personal experience in applying mypyc on type-hinted, but otherwise not specially optimized python code? I'm especially interested in what kind of performance speed-ups could be achieved.

After reading the article last night. I spent the whole evening about 5 hours getting one of my DOM libraries to compile with mypyc. It's a hacky codebase with little to no type hinting. I had to rewrite a fair abit to appease mypy and in most cases just used 'Any'. It's still got runtime issues and is buggy but I got at least a 2x speed increase on rendering a single node... https://github.com/byteface/htmlx/blob/mypyc/benchmarks.md

However I was unable to compile from my mac, could only compile using linux. but that could be as I'm using older version of dev tools.

Re: You Should Compile Your Python and Here’s Why

#107

Post by taneq »

bitwize wrote: post #30
taneq wrote: post #25

Earlier quoted context omitted.

Reminds me of that Tao of Programming koan (if I remember it rightly) that went something like: The novice, in his frustration, struck the side of his computer. The master walked over and asked what he was doing. The novice exclaimed, "my computer is not working and I do not know why!" The master admonished him, saying "you cannot solve the problem by striking the computer without knowing what is wrong." Then the mas…

It was one of the AI koans in Jargon: > A novice was trying to fix a broken Lisp machine by turning the power off and on. > Knight, seeing what the student was doing, spoke sternly: “You cannot fix a machine by just power-cycling it with no understanding of what is going wrong.” > Knight turned the machine off and on. > The machine worked.

Yes! Ah well, I got the general gist of it right still. It's been a very long time since I read it. :)

Re: You Should Compile Your Python and Here’s Why

#108

Post by BenjiWiebe »

bsder wrote: post #100
fgeiger wrote: post #8

Frankly, I would find it much easier to write the naive C version than this "moderately" optimized Python version in the article although I am much more familiar with Python than with C.

I really hate these kinds of articles because they don't DO anything useful in the sample programs. Grab a web page and parse it for some data. Suddenly, C looks like trash. The network latency negates any speed advantage from C and the fact that C has to manually manage memory everywhere makes your C code look like a dumpster fire relative to the Python code. You can always pick a task that favors one language or th…

Good point. I love C, but I can't imagine doing web scraping in it. I pick up Ruby for that.

Re: You Should Compile Your Python and Here’s Why

#109

Post by wheelerof4te »

Instead of making up some arcane ways to hack Python, try using builtin data structures and functions more.

I see a bunch of Python code that is just a massive dump of custom classes on top of classes, inheriting other classes.

The amount of pointer chasing to get to the actual primitives can be mind-bogling. Add that to a big nested loop and you've got a performance tragedy.

Post reply on HN