Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

201–210 of 336 posts

Re: Python’s Weak Performance Matters

#201

Post by commandlinefan »

skywhopper wrote: post #198

The quoted argument that "easy to write but slow languages are better because programmer time is far more costly than CPU speed" was pretty common, and I honestly think correct, 10-15 years ago. But things have changed. CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed re…

I've been "premature optimization is the root of all evil"ed in more code reviews than I can count - yet when people complain about software, they first complain that it's ugly and then they complain that it's slow. Users don't care what it's written in.

Re: Python’s Weak Performance Matters

#202

Post by urschrei »

loeg wrote: post #89
myopicgoat wrote: post #80

Earlier quoted context omitted.

From 4 years ago... a lot has changed since. Consider this recent post for example: https://tk3369.wordpress.com/2018/02/04/an-updated-analysis-...

The linked article doesn't attempt to refute the significant claims from Dan's article (which has an update from a year later, so 2015, at the bottom): 1. That the language is (was) undertested, and as a result, full of easy to run into bugs 2. The language makes it easy to ignore errors 3. APIs are inconsistent 4. The head branch isn't kept build-clean (i.e., often it fails to build) 5. Code is often undocumented, w…

It's really shocking to me that years later, nobody has addressed the most serious issue (community interaction) Dan brought up. Just reading the archived mailing list thread I could not believe the suspicion, and later outright hostility towards Dan's comments. It obviously dawned on a couple of core contributors, as can be seen by the sheepish attempts to walk things back, but this sort of interaction _with the project leadership_ is a giant red flag about how they view their community and users. If someone like Dan can be treated in that way publicly (and far worse privately, according to his own account), things are going to be so much worse for other people.

Re: Python’s Weak Performance Matters

#203

Post by lmm »

gameswithgo wrote: post #196
passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

I would argue that performance always matters, and that Python is never the right tool for the job in an absolute sense. Python may be the right tool for the job given the options we have today but there is no reason we cannot have a language exactly as nice to use as Python is, but that also provides good performance. Languages like Nim or F# approximate that ideal, for instance. And while I realize there are high p…

> It is a failure of our community that we allow languages to proliferate while remaining slow. This is bad because allowing slow tools to become popular means people create slow things, which wastes other peoples time and energy.

Make it work, then make it work right, then make it work fast. I mean yes, a lot of things are slower than they should be, but the level of outright correctness bugs in software today is mindblowing. So while replacing our tools with faster tools should be something we do in the long term, I'd put a higher priority on lowering defect rates and making it easier to produce working software.

Re: Python’s Weak Performance Matters

#204

Post by theli0nheart »

skywhopper wrote: post #198

The quoted argument that "easy to write but slow languages are better because programmer time is far more costly than CPU speed" was pretty common, and I honestly think correct, 10-15 years ago. But things have changed. CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed re…

> CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed really is more expensive than programmer time, if you are running that code at a big enough scale.

When you're starting a startup, scaling out your application to hundreds, thousands, or millions of servers isn't something you're going to do right off the bat, and more likely, that's never going to happen, no matter how successful your company is. The number of companies operating at that sort of scale can be counted on one hand. Most startups can run on a few boxes.

For those situations (which probably pertains to most new projects outside of big companies), programmer time is indeed, still the most important and costly input. If you're a startup and your 100x more performant Go code takes an extra 3-6 months to write, and a competitor beats you to market, no one is going to care how much faster your runtime performance is on the CPU, especially if you're a web app, when CPU time is probably should be last on the list of items which could lead to slow application performance to end users.

100x difference in CPU time is nothing compared to the 1000x loss in a cache hit or a 10000x disk read. I'd love to see an example where the CPU difference outweighs any influence from disk or memory.

Re: Python’s Weak Performance Matters

#205

Post by joshuamorton »

luispedrocoelho wrote: post #112
joshuamorton wrote: post #58

Earlier quoted context omitted.

Right, and my point is that you could probably 1. Have gotten similar performance boosts elsewhere, meaning that you wouldn't have needed to refactor this function in the first place (although the implication of a 10000x speedup means that may not be true, although I can absolutely see the potential for 100x speedups in this code, depending on exactly what the input data is) 2. Its likely that there are much more nat…

1. No, that function was the bottleneck, by far, and I can tell you that >10,000x was what we got between the initial version and the final one. 2. I don't care about faster at this point. The function is fast enough. Maybe there is some magic incantation of pandas that will be readable and compute the same values, but I will believe it when I see it. What I thought was more idiomatic was much slower. I think this is…

1. you don't get 10000x speedups by changing languages. It's likely that this optimization would be necessary in any case.

2. You don't care about improving the code, but you did care enough to write an article saying that the language didn't fit your needs without actually doing the due diligence to check and see if the language fit your needs. That's the part that gets me.

Re: Python’s Weak Performance Matters

#206

Post by Rotareti »

Erwin wrote: post #151
Rotareti wrote: post #118

I use a lot of Python for web stuff and I haven't been in a situation where Python itself was the performance bottleneck. I always thought that when you run into a situation where Python is the bottleneck, you replace the critical bits with something like C/C++/Rust. Following this approach, you would get the best of both worlds: rapid proof-of-concept / time-to-marked with the option to improve performance critical…

It requires that your code is architected so performance critical sections of Python can move into C etc. Let's say that your code creates a complex object tree from some configuration settings, and executes Python methods and code from all over it, using heavy OO. That is difficult to move to C++ as your performance is spent on Python bookkeeping -- you are calling methods and thus looking things up in dictionary, y…

Thanks for the insights!

> Let's say that your code creates a complex object tree from some configuration settings, and executes Python methods and code from all over it, using heavy OO.

Luckily, this does not apply to the codebases I'm working on, which are all quite functional (no classes, no inheritance, pure functions exclusively, immutable data types, etc.) I have the feeling that this will not strike me that hard. If you rely on pure functions you have all the application state that you need for the function on the parameters and you pass all new state back through `return`. I guess all I had to do is convert the types once for the C function call (from Python to C) and once for the `return` (from C to Python)?

Re: Python’s Weak Performance Matters

#207

Post by NoahTheDuke »

bzbarsky wrote: post #200
dorfsmay wrote: post #183

Earlier quoted context omitted.

• the language is easy, very attractive for non-CS people • the language is consistent (everything is an object, or a pointer to an object rather, you can only pass by pointer, scope and namespace that make sense all the time, etc...) • you can learn it gradually, you can start using it even if you know only 10% of the language • Amazing documentation. The official tutorial is easy to read, and by the time you are th…

Scope and namespace that make sense all the time? for foo in bar: pass # Why is "foo" in scope here???

Because Python doesn't have block scoping, only function scoping.

Re: Python’s Weak Performance Matters

#208

Post by joshuamorton »

Sean1708 wrote: post #188
joshuamorton wrote: post #25

Earlier quoted context omitted.

While I'm more of a pythonist than a C-ist, hearing "1000x speedup" and "line for line" to me implies that you aren't writing idiomatic python. Idiomatic python is (often) faster than not, and (often) more difficult to translate to lower level languages. As a simple example, list-comprehensions are faster than loops, and can't be line for line translated into C++.

> list-comprehensions are faster than loops Are they really though? In my experience you'll gain a couple of percent because you're getting rid of that call to append, but that's hardly the orders of magnitude OP was looking for.

Yes, I wasn't saying they would fix this problem (they won't), but making the broader point that in general, more idiomatic/pleasant looking code is faster. This is neither of those, and so my hunch is that there are improvements that can be made to both form and function.

Re: Python’s Weak Performance Matters

#209

Post by theli0nheart »

gameswithgo wrote: post #196
passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

I would argue that performance always matters, and that Python is never the right tool for the job in an absolute sense. Python may be the right tool for the job given the options we have today but there is no reason we cannot have a language exactly as nice to use as Python is, but that also provides good performance. Languages like Nim or F# approximate that ideal, for instance. And while I realize there are high p…

You're conflating Python the language and Python the default runtime implementation (CPython). PyPy, a Python JIT compiler, has shown you can have an incredibly fast Python implementation. In some cases, it's faster than C.

Maybe this is what you're looking for. :)

http://speed.pypy.org

Re: Python’s Weak Performance Matters

#210

Post by pleasecalllater »

hidenotslide wrote: post #179
pleasecalllater wrote: post #146

Earlier quoted context omitted.

Well, I just wanted to use pandas to load a 4GB csv file. After using 32GB of my RAM, and 4GB of swap I gave up. I've just loaded all that data to Postgres, and made a couple of queries. This way I stopped using pandas at all.

I hardly use pandas at this point besides read_csv, which is very good once you know the syntax for parsing strings/dates, skipping rows, dropping columns, etc. After that I usually just keep the numpy array since all I need is floats. I guess the index groupby stuff is cool, but I never really needed it. Postgres is fine but if you're just doing numerics it doesn't help much.

It helps with having smaller RAM requirement. And I have the group by, and materilized indices, which helps a lot to preserve huge modified datasets.
Post reply on HN