Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Python’s Weak Performance Matters

metarabbit.wordpress.com

Post reply on HN
Print view View original

191–200 of 336 posts

Re: Python’s Weak Performance Matters

#191

Post by kiriliponi »

VHRanger wrote: post #171
fxmc wrote: post #103

Earlier quoted context omitted.

Have you given dask a try? It gives you out-of-core arrays with numpy semantics and distributed computing.

Dask doesn't solve that problem since it's a wrapper around pandas functions. If you can't make the core pandas code decently fast, dask won't save you.

dask.dataframe might not help but dask.distributed could in that case.

I've had success using it on non vanilla stuff (i.e. code that could not get converted to play natively with numpy/pandas structures)

As a bonus, the nice profiling tools (built within dask) have also helped me improve the performance of the code.

See https://distributed.readthedocs.io/en/latest/

Re: Python’s Weak Performance Matters

#192

Post by make3 »

autokad wrote: post #152
scribu wrote: post #149

Earlier quoted context omitted.

I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).

i dont seem to follow. if you are doing data science, all the bottle necked stuff will be running in numpy or pyspark. Choosing python over R, SPSS, Matlab usually doesnt come down to which one is faster, and R as far as i know is at least not vastly superior in speed.

or a cuda wrapper

Re: Python’s Weak Performance Matters

#193

Post by a3n »

wolf550e wrote: post #190
a3n wrote: post #189

CPython is obviously the way it is on purpose, by design, and quite successful. Yet I still find it ironic that this "slow" interpreter is written in C, the go to, general purpose, "low level and a half" fast language, and that "C" is right there in the name. Not knowing anything else, I might expect a project with "C" at the front of the name would be at least fast-ish, and that the naming was intended to signal tha…

What? No. The c in cpython was not meant to signal performance. The c was added after alternative implementations were created to mean "the original/reference implementation". Python's competitors perl ruby and php are likewise interpreters implemented in c and are equally slow. Anyway, what alternative did they have to implement a bytecode interpreter? C is portable and fast enough, especially with computed goto ext…

> The c in cpython was not meant to signal performance.

I thought it was clear that I understood that: "Not knowing anything else, I might expect ..." I guess not.

Re: Python’s Weak Performance Matters

#194

Post by [deleted] »

pcx wrote: post #78
munro wrote: post #66

> At the same time, data keeps getting bigger and computers come with more and more cores (which Python cannot easily take advantage of), while single-core performance is only slowly getting better. Thus, Python is a worse and worse solution, performance-wise. PySpark is makes it really easy to take advantage of multiple cores & machines. Most operations I want to do to my data I can find in PySpark's pyspark.sql.fun…

This is true, but using a Distributed system like Spark itself adds a ton of complexity in having to understand and manage it. If one can do something with a set of stateless processes, even if it's more performant, I feel it's a bad idea to use a distributed system instead. Not always, but a good majority of cases that I've seen. I've seen projects where Celery would be enough but instead they chose to use Spark/Sto…

[deleted]

Re: Python’s Weak Performance Matters

#195

Post by snissn »

jerf wrote: post #162
b0rsuk wrote: post #156

Is writing extensions a lost art? I read a few blog posts about speeding up Python and Ruby with Rust extensions. This should enable rewriting only the slow parts. Later, you could replace more of it if needed. Is writing extensions so very problematic in practice? I know Go has runtime issues making it not very good for mixing with other languages, so it often encourages rewriting the whole application in it.

Extensions aren't a total solution, though, which people often sell them as. You have an impedance mismatch between Python and C code, because Python has all of its objects packed in a way that is very strange to C, so you end up essentially deserializing all objects into C, then back out into Python, in a very expensive and allocation-heavy (on both sides) conversion. If you can set up your computation in Python and…

Disagreeing with you and agreeing with the parent, it sounds like a lost art... Numpy doesn't pack and unpack python data structures, it just uses C structures. Python extensions I've written just use C/C++ data types and only occasionally passes python native types back to python. Python is amazing for developer productivity, but the methods are a bit opaque.

Re: Python’s Weak Performance Matters

#196

Post by gameswithgo »

passive wrote: post #137

This is a weird article at this point in time. The question it addresses: "Does Python's performance matter?" Has always had the answer: "Sometimes, and you have options for those cases." The OP found a "sometimes", and he's using one of those options. In this case, he's got Python for prototyping and glue, with Haskell improving performance. This is as it should be. I don't know of any Python advocates who say it's…

I would argue that performance always matters, and that Python is never the right tool for the job in an absolute sense.

Python may be the right tool for the job given the options we have today but there is no reason we cannot have a language exactly as nice to use as Python is, but that also provides good performance. Languages like Nim or F# approximate that ideal, for instance. And while I realize there are high perf variants of Python, these should be the primary standard, and only path. The slow path shouldn't exist.

It is a failure of our community that we allow languages to proliferate while remaining slow. This is bad because allowing slow tools to become popular means people create slow things, which wastes other peoples time and energy.

Electron becoming a standard way to make cross platform desktop apps is another example. Someone is not wrong to choose electron for that job, but we the software community are wrong to have let something that inefficient become the easiest way to do that job.

You cannot simply dismiss this issue by saying "Well don't use the slow software you don't like then" as many of these things become de-facto standards that you cannot avoid. Your place of work may require Microsoft Teams as the chat software, and now you are using a huge % of your laptops ram and battery for simple text transmission. Atom becomes the popular target for language plugins and ends up the only usable way to get good IDE features for your language, and you suffer the performance hit for it.

We can do better!

Re: Python’s Weak Performance Matters

#197

Post by eesmith »

indubitable wrote: post #184
eesmith wrote: post #133

Earlier quoted context omitted.

The link you gave is indeed to view the source. It is not, however, an open source license, as characterized by the OSI or DFSG, nor free software as characterized by FSF. Here is the license - https://referencesource.microsoft.com/license.html . It contains obviously non-open source clauses like: > "Reference use" means use of the software within your company as a reference, in read only form, for the sole purposes…

From my perspective here, you're shifting the goal posts. In particular this conversation began with you apparently thinking that Mono lacked full implementations of ASP/ADO/etc. And I think that might justify the original comment that Linux implementations have been treated as second class citizens. I say might since ADO/ASP are not really part of .NET, but rather independent projects built on top of .NET. But in an…

You asked for possible reasons for why you are being downvoted for your reply to the following comment by woolvalley: "It has been for the longest time been a closed source MSFT only thing. It wasn't open source and running on linux was a second class citizen. Not sure if it is still a second class citizen."

Your reply was: "Why do you think it's been closed source with minimal cross platform effort?"

I don't think you can argue that I'm shifting the claim to the topic of being insufficiently permissive when that was part of the original thread.

The license[1] that you pointed to is not open source. It says "You may not use or distribute this Software or any derivative works in any form for commercial purposes".

Again, just because the source code is available, that doesn't mean it's open source. There are few that would agree with you that the early Microsoft licenses you pointed to meet the usual criteria for "Open Source" as it's used in the industry.

Your argument seems to be that things have changed and we should forget about the history. As woolvalley's comments were made in the past tense ("wasn't open source", "was a second class citizen"), I think you could have made a better reply along the lines of "a lot has changed since you last looked into C#", rather than the more accusative approach you did which challenges a viewpoint which seems to be historically justified.

I find the idea that "4 years is a very long time in software" to be laughable.

I've been using Python since the late 1990s, and am still going through the Python 2->3 upgrade cycle. My primary development environment of a Unix-like terminal environment and emacs would be recognizable to people in the 1980s. I've been developing and selling my software product for 8 years, with essentially the same core codebase.

Or, going down that rabbit hole, Alan Kay is famous for arguing that there is little in the way of significant new inventions in computing since 1980. See https://stackoverflow.com/questions/432922/significant-new-i... .

Re: Python’s Weak Performance Matters

#198

Post by skywhopper »

The quoted argument that "easy to write but slow languages are better because programmer time is far more costly than CPU speed" was pretty common, and I honestly think correct, 10-15 years ago. But things have changed.

CPU performance long ago hit physical limits, and more and more we are scaling out applications across hundreds, thousands, or millions of servers. We've passed the inflection point where CPU speed really is more expensive than programmer time, if you are running that code at a big enough scale.

Add in containerization and cloud VM platforms where the tradeoffs of space and performance versus money start to become very clear. Add in better, safer languages like Rust and Go for writing high-performance code. And today if you can spend three times the programming time writing in a faster and more efficient language, and it runs 100x as fast in a tenth the memory footprint, you are talking about massive overall cost savings.

Re: Python’s Weak Performance Matters

#199

Post by michaelsbradley »

autokad wrote: post #152
scribu wrote: post #149

Earlier quoted context omitted.

I think the article makes a lot more sense if you consider it in the context of "Python for data science". In the last few years, there's been a lot of hype about replacing other number crunching solutions (R, SPSS, even Matlab) with the Python ecosystem of tools (Pandas, SciPy, etc.).

i dont seem to follow. if you are doing data science, all the bottle necked stuff will be running in numpy or pyspark. Choosing python over R, SPSS, Matlab usually doesnt come down to which one is faster, and R as far as i know is at least not vastly superior in speed.

As with Python, the fast libraries written for R are usually implemented with something else under the hood. Take the data.table library, for example:

https://github.com/Rdatatable/data.table

It's wicked fast for many kinds of tasks, but its R API is just a thin layer on top of C.

Re: Python’s Weak Performance Matters

#200

Post by bzbarsky »

dorfsmay wrote: post #183
dTal wrote: post #144

Earlier quoted context omitted.

It can't be the only reason, or how would it ever have attracted such an ecosystem in the first place? Especially with its performance characteristics, and lack of corporate backing. No, Python was invented at a time when its closest competitor was Perl - and you need only compare typical Perl with typical Python to appreciate that Python really was a usability revelation. But that was nearly 30 years ago. I do think…

• the language is easy, very attractive for non-CS people • the language is consistent (everything is an object, or a pointer to an object rather, you can only pass by pointer, scope and namespace that make sense all the time, etc...) • you can learn it gradually, you can start using it even if you know only 10% of the language • Amazing documentation. The official tutorial is easy to read, and by the time you are th…

Scope and namespace that make sense all the time?

  for foo in bar:
    pass
  # Why is "foo" in scope here???
Post reply on HN