Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Python with Guido van Rossum

softwareatscale.dev

Post reply on HN
Print view View original

21–30 of 251 posts

Re: Faster Python with Guido van Rossum

#21

Post by stonemetal12 »

Qem wrote: post #6
amelius wrote: post #3

Python needs to go multicore, like OCaml. And add a modern concurrent garbage collector. One of the difficulties is of course to not break existing C extensions.

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

The multiprocessing module is a hack. It has so much overhead that it is not worth the effort in all but the most embarrassingly parallel situations.

Re: Faster Python with Guido van Rossum

#22

Post by ctur »

nicolaslem wrote: post #7
ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

Isn't it that maintaining large codebases in the long run is just hard? What language is not a trap?

Fair, maybe it's always a matter of picking the trap you want :) If you are optimizing for 0->1, Python is great. But if you want code to live and evolve over time with multiple authors, or ever will care about runtime performance, it's a dead end almost from the first line.

But sometimes the productivity benefit is all that matters and you accept you may have to throw it all out later (or invest insanely in making it work)... it's all about making an informed decision.

Re: Faster Python with Guido van Rossum

#23

Post by nicolaslem »

Qem wrote: post #6
amelius wrote: post #3

Python needs to go multicore, like OCaml. And add a modern concurrent garbage collector. One of the difficulties is of course to not break existing C extensions.

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

One thing that is often overlooked regarding Python multiprocessing is that depending on the version of Python and the OS, the processes are not launched the same way.

For example MacOS cannot use fork while it was (is?) the default on Linux.

Re: Faster Python with Guido van Rossum

#24

Post by ctur »

mrfusion wrote: post #15
ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

You may be right but I’m convinced most problems don’t need to scale.

Absolutely true.

Re: Faster Python with Guido van Rossum

#25

Post by formerly_proven »

Qem wrote: post #6
amelius wrote: post #3

Python needs to go multicore, like OCaml. And add a modern concurrent garbage collector. One of the difficulties is of course to not break existing C extensions.

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

Multiprocessing is pretty heavyweight, everything needs to be serialized, you can't share Python objects directly or with good performance across processes, there is (fairly new) shared memory support, but it's just a bag of bytes. It's not particularly good on the cross-platform angle, while it does work on Windows, the startup time for each process is like starting a fresh interpreter, because that's exactly what it is doing.

Multiprocessing can work if you have all your logic in Python already, and there is no notable shared state between the processes, and you don't need to coordinate much at all. So pretty much just the very simplest problems in concurrency. Everything else might technically be possible, but will likely be very slow and annoying.

Re: Faster Python with Guido van Rossum

#27

Post by qsort »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

Pretty much everything you're saying matches my experience, so, playing the devil's advocate:

- The typing argument is impossible to argue against, but isn't it the same problem you'd have with any other dynamic language? How is, say, Javascript any better? I don't see this as something that's specifically Python's fault, rather, yet another argument as to why starting a large project in 2021 without seriously considering static types is bordering professional malpractice.

- As for the GIL thing, I'm with you that it's never going to be lifted in any meaningful way, but wouldn't the subinterpreters idea along the lines of Ruby's ractors fix most of the pain in most cases?

Also, why are you skeptical about "the 5x plan"? I don't really have an opinion about it, but I'm interested in hearing any input.

Re: Faster Python with Guido van Rossum

#28

Post by tristor »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

This mirrors my experience as well. At one (at the time major) cloud provider we had to internally re-implement 100% API / behavior compatible Java applications for the publicly available Python applications for our OpenStack deployment, due to performance and scaling requirements. OpenStack Foundation required all projects to be only written in Python, which I think was a major factor in why OpenStack adoption was never significant. Scaling and deploying it was an absolute nightmare using public code.

Anything done at significant enough scale /must/ be multi-threaded to take full advantage of modern hardware which makes languages which don't provide a reasonable threading model effectively non-starters or at best a toy language for internal prototyping. I shifted from Python to Ruby, simply because Ruby provided a realistic way to interface with real threads, and then to Go, and haven't looked back. Simple applications are massively more performant in Go than in Python, and it's not just due to typing and compilation.

I love how simple Python is to learn and how it brings more people into the fold in approaching solving complex problems, but at the end of the day you should treat it like runnable pseudocode to get shared understanding so you can implement in a real language.

Re: Faster Python with Guido van Rossum

#29

Post by Lucasoato »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language, like Javascript...

... but I think you're forgetting the real motivations that drive people to use Python: it's easy to learn, it's very readable, there's a large (and not so much toxic) community behind, it's versatile. No one is saying that it will replace every other language, but it still can find its space even among large projects.

Re: Faster Python with Guido van Rossum

#30

Post by j1elo »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

I'm a C++ dev and was taught Python as a tool for prototyping and validation of ideas, not as a language for final implementations. Later with years I've read endless histories about the "traps" you mention, so I understood the notion I was taught wasn't without its merits.

So I still look suspiciously to all mid to large projects what are written in Python. It takes some serious effort and good engineering to maintain good quality, and still the tooling is not there for error detection, like in other better typed languages would be. I wouldn't feel confident relying on such project for a critical infrastructure piece. The idea should have been validated quickly with Python, then implemented in a more appropriate language.

EDIT: Obviously that's just my possibly wrong opinion. There are huge projects out there proving otherwise, but the issues discussed here are _very_ real.

Post reply on HN