Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

311–320 of 336 posts

Re: Python’s Weak Performance Matters

#311

Post by [deleted] »

bbatha wrote: post #272
halbritt wrote: post #265

Earlier quoted context omitted.

To your point 1: This is the reason that AsyncIO exists in Python 3.6 and IIRC Facebook is pushing really, really hard to adopt it.

Absolutely, but its arguably 10 years too late. Node has already eaten python's lunch in the server space. To get your existing python code working on async you may basically need a full rewrite including any libraries that you pulled in. At that point most companies ask themselves if python is really the right language to conduct a rewrite in.

[deleted]

Re: Python’s Weak Performance Matters

#312

Post by MR4D »

theli0nheart wrote: post #266
blub wrote: post #222

Earlier quoted context omitted.

Python and CPython are the same thing for the majority of Python developers, just like the Oracle JVM is a synonym of Java. Yes, there are other exotic runtimes, but only one official one and that's what people will use.

I get that, and I agree, but my point was that if you're writing Python and finding that your code is too slow, it's far easier to just drop in a faster runtime than it is to rewrite your entire existing codebase in a new language.

Not if you deploy your code to environments you don’t fully control.

Re: Python’s Weak Performance Matters

#313

Post by weberc2 »

theli0nheart wrote: post #204
skywhopper wrote: post #198

The quoted argument that "easy to write but slow languages are better because programmer time is far more costly than CPU speed" was pretty common, and I honestly think correct, 10-15 years ago. But things have changed. CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed re…

> CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed really is more expensive than programmer time, if you are running that code at a big enough scale. When you're starting a startup, scaling out your application to hundreds, thousands, or millions of servers isn't somethi…

I write Python for my day job, but I often prototype in Go because it's so much easier. Also, regarding I/O, Python does nothing during I/O unless you take care to write async code and only call async libraries. Go does async by default and truly outclasses Python at IO bound tasks. For CPU bound tasks, Go is only 2 orders of magnitude faster. For I/O bound tasks, Go is about 4-6 orders of magnitude faster unless Python is carefully optimized.

Re: Python’s Weak Performance Matters

#314

Post by proto-n »

luispedrocoelho wrote: post #185
proto-n wrote: post #140

Earlier quoted context omitted.

I've also found that loops with dictionary (or set) lookups are a pain point in python performance. However, this example strikes me as a pretty-obvious pandas use-case: interesting = set(line.strip() for line in open('interesting.txt')) total=0 for c in chunks: # im lazy to actually write it df = pd.read_csv('data.txt', sep='\t', skiprows=c.start, nrows=c.length, names=['id','val']) total += df['val'][df['id'].isin(…

`isin` is worse in terms of performance as it does linear iteration of the array. Reading in chunks is not bad (and you can just use `chunksize=...` as a parameter to `read_csv`), but pandas `read_csv` is not so efficient either. Furthemore, even replacing `isin` with something like `df['id'].map(interesting.__contains__)` still is pretty slow. Btw, deleting `interesting` (when it goes out of scope) might take hours(…

Ok, I said I wasn't sure about the implementation, so I looked it up. In fact `isin` uses either hash tables or np.in1d (for larger sets, since according to pandas authors it is faster after a certain threshold). See https://github.com/pandas-dev/pandas/blob/master/pandas/core...

Re: Python’s Weak Performance Matters

#315

Post by lmm »

blub wrote: post #277
lmm wrote: post #237

Earlier quoted context omitted.

> Does anyone believe that other industries think like that? Software is very unlike other industries. > For any complex project, making it work and fast only at the end will either result in massive cost overruns or an outright canceled project. Citation needed, because my experience is the exact opposite: projects that put effort into making it fast from the first never get off the ground.

Software is different, but it doesn't bend space and time. Redesign costs can be significantly cheaper, but they are still present, and if the project is complex enough can be just as high as in physical world project catastrophes. I assume you have heard of various software projects which cost millions and overshot their budget by 50%, 100%, 200%, etc... Now regarding the make -> work -> fast cliché: I tend to have…

> Redesign costs can be significantly cheaper, but they are still present, and if the project is complex enough can be just as high as in physical world project catastrophes. I assume you have heard of various software projects which cost millions and overshot their budget by 50%, 100%, 200%, etc...

Indeed; in my experience such failures tend to be caused by too much rather than too little design and architecture up front.

> How do those of us that like to complain about "premature optimisation" whenever performance is discussed design software? Because based on HN discussions it looks like code & pray.

Don't try to design at the start when you don't know anything; let the design emerge. Do the simplest thing that might possibly work; 90% of the time it does work, the other 10% of the time you learn about another constraint. Get the simplest use case working, then iteratively expand to the full functionality. Refactor continuously and fearlessly (and adopt whatever testing/verification practices you need to make it fearless), driven by the changes that you need to make.

It's easy to mock, but it works much better than trying to design as a separate activity.

Re: Python’s Weak Performance Matters

#316

Post by rhaps0dy »

ChrisRackauckas wrote: post #288
ptype wrote: post #106

I love Python. The language is a joy, the eco system is fantastic. But yes, let’s be honest, if you can not vectorise your code it is slow, and I think that will be its downfall eventually. I’m excited about Julia, I hope it gains popularity and the eco system grows. Until then, and in particular until the data frames story can compete with pandas, it Python with Cython for me, but I’d rather skip the Cython if it wa…

I'm using Julia and loving it. I've built a bunch of differential equation solvers which routinely outperform the classic C++/Fortran codes. I started out without "software development" experience but Julia and its community got me up to speed and helped me build something quite unique. Now Julia is the only language that has the numerical libraries I need to do my research. In fact, the whole library story in Python…

>If you're doing something which is actually new, like PhD methods research, you need to be writing a lot of stuff from scratch.

Sometimes you also reuse a lot of stuff, it depends. For machine learning in particular, almost everything uses some sort of gradient-based optimisation algorithm. In these cases, it is very useful to have an automatic differentiation library. My coworker said Julia has about 3, IIRC, and until an amalgamation of them is merged into the standard library Julia isn't completely ready.

Re: Python’s Weak Performance Matters

#317

Post by theli0nheart »

MR4D wrote: post #312
theli0nheart wrote: post #266

Earlier quoted context omitted.

I get that, and I agree, but my point was that if you're writing Python and finding that your code is too slow, it's far easier to just drop in a faster runtime than it is to rewrite your entire existing codebase in a new language.

Not if you deploy your code to environments you don’t fully control.

Well, then you're out of luck. :)

Re: Python’s Weak Performance Matters

#318

Post by blub »

brians wrote: post #279
blub wrote: post #228

Earlier quoted context omitted.

This "make it work" adage makes no sense at all when scrutinized. Does anyone believe that other industries think like that? First let's invent a washing machine that washes, but occasionally sets clothes on fire and takes 24h for a washing cycle. Then we'll redesign it so it doesn't set things on fire, and finally redesign it yet again to finish in 2h. It's incredibly wasteful. For any complex project, making it wor…

Yeah. Speed is the dominant cost in software, but other industries treat other costs similarly. To pick another darling, look at Tesla: the Roadster is expensive, flammable, suitable only for enthusiasts. The S is generally useful, but too expansive. Still a narrow market. The E is their first general purpose product. That’s normal. Same deal with Apple II, Mac, iPhone. Same deal with ether, coarse general anesthetic…

All of the things you mentioned worked and had good enough performance. There was no crappy version of the iPhone which ran out of battery in 1h and took seconds to refresh when scrolling.

So obviously they thought about the performance aspect from the start.

Re: Python’s Weak Performance Matters

#319

Post by weberc2 »

DrJosiah wrote: post #240
gameswithgo wrote: post #196

Earlier quoted context omitted.

I would argue that performance always matters, and that Python is never the right tool for the job in an absolute sense. Python may be the right tool for the job given the options we have today but there is no reason we cannot have a language exactly as nice to use as Python is, but that also provides good performance. Languages like Nim or F# approximate that ideal, for instance. And while I realize there are high p…

As a guy who just spent the past summer doing (his 19th year of) Python, (6rd year of) Cython(/Pyrex), and (8th year of) C, picking around PyPy, and writing a couple transpilers; the tools are incredible. And I think you're talking out your butt here. Speed is a matter of choice and a bit of manual tooling. You seem like a Golang advocate; great. I hope it continues to work for you. Python solves 95% of problems toda…

I don't buy these arguments. You hit on a pathological case in Go where Python managed to be faster. That's not remotely typical, and even the "just use C/Cython/Numpy" is a huge oversell--there are some problems that are amenable to dropping into a lower level language, but many more are not. At best, it's just very, very hard to make Python as fast as unoptimized Go.

Besides performance, CGo and tooling seem to make up the bulk of your criticisms. I agree that Python's C interop story is strictly better than Go's, but this seems like throwing the baby out with the bathwater for a huge suite of applications. I also strongly disagree that Python's tooling is better; I think your 20 years of experience with it has biased you against the frustrations the rest of us have when we try to pick it up. By contrast, I've found Go's tooling to be superb--profiling, testing, benchmarking, documentation, etc all included out of the box. There are still holes (like debugging), but I daresay Go's tooling is categorically better than Python's.

Re: Python’s Weak Performance Matters

#320

Post by dagw »

blub wrote: post #228
lmm wrote: post #203

Earlier quoted context omitted.

> It is a failure of our community that we allow languages to proliferate while remaining slow. This is bad because allowing slow tools to become popular means people create slow things, which wastes other peoples time and energy. Make it work, then make it work right, then make it work fast. I mean yes, a lot of things are slower than they should be, but the level of outright correctness bugs in software today is mi…

This "make it work" adage makes no sense at all when scrutinized. Does anyone believe that other industries think like that? First let's invent a washing machine that washes, but occasionally sets clothes on fire and takes 24h for a washing cycle. Then we'll redesign it so it doesn't set things on fire, and finally redesign it yet again to finish in 2h. It's incredibly wasteful. For any complex project, making it wor…

"Make it work, make it right, make it fast" should primarily be applied in the context of individual functions and modules not whole complex projects. Don't start micro-optimizing your function for speed before you've gotten it to return the correct result. Also with the additional caveat that you should stop at make it work right until you're at least fairly sure that that function is actually important for overall performance.
Post reply on HN