Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Python with Guido van Rossum

softwareatscale.dev

Post reply on HN
Print view View original

141–150 of 251 posts

Re: Faster Python with Guido van Rossum

#141

Post by biztos »

coldtea wrote: post #105
ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

> It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... If you're a millionaire by that time, or have a strong established business, then it doesn't matter, it's a nice problem to have. Have your engineers have a go at it. See: Dropbox, Disqus, AirBnB, and others... And if you…

A third possibility is you’re a team in a larger company, and then it could matter a lot.

I wrote a few things in what I’m pretty sure was correct, idiomatic, best-practices Python 3 and almost immediately I could see some familiar problems from my Perl days looming on the horizon, especially around testing, packaging and maintainability.

I enjoyed writing Python code and for the right kind of startup or hobby project I might use it in the future, but there is so much black-boxed weirdness in the ecosystem, so much inconsistency in paradigms, and so much temptation to “cheat” that I’d definitely think twice before committing a team to it. And I didn’t even get as far as the performance issues.

On the other hand, of course, you can opportunistically “move fast and break things” within a larger org too. But I worry that if you get too committed to Python, you will be unreasonably dependent on your lead engineers sticking around.

Re: Faster Python with Guido van Rossum

#142

Post by BiteCode_dev »

CraigJPerry wrote: post #138

A question i’ve often wondered about is how much slower could you make python before users would leave it behind? Python is often 1-2 orders of magnitude slower than java. That really is a lot yet Python and Java are about comparable in adoption.

That's because:

- speeds matters of that level matters only for a limited number of use cases. Most programming use cases are fine with python speed. They were fine, in fact, with it 20 years ago, before we got that hardware speed.

- people are starting to complain about speed in python only because of all the data science going up. Suddenly here speed matters. Before, you had to get to google level of infra to find use cases where python was too slow.

- dev are expensive. Hardware is cheap. If you have a slow website, paying $200 dollar more on your server is not a big deal. But taking 3 more month dev that features with your $300k pro is another thing entirely.

Re: Faster Python with Guido van Rossum

#143

Post by BiteCode_dev »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

A trap for your dream world were you suddenly get google size ?

Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now.

It is still maintained by one single person, and he is not a professional dev.

And I'd say it's already very rare to reach that size in any project, in any company. Hell my last 3 paid projects as a freelancer have less than 20 concurrent users. And that's professional. That's people's real life.

2 out of 3 IT projects fail, no code is even shipped. I would worry about the scaling later. Once you do have the problem, pay the price for more hardware or a rewrite, you made it!

And if you do know for sure you will have the problem (you already work for a GAFAM), then not choosing python is ok, it's not a trap, it's a trade off.

But for the vast majority of us out there, having an easy, solid, battle tested and clean language with a huge ecosystem is worth a 10000 times more than some scaling potential in 5 years down the road.

I have seen so many big C++ or Java project fail I don't associate any language with big and failure anyway. I've seen devs arguing about the purity of this in Haskell, or that in Ocaml, and nothing gets to the end user because it's never perfect, or it's a toy, and doesn't handle IRL.

But what powers most companies ? Rail apps that you won't upgrade ever, but are still running. Spaghetti PHP code and wordpress that just won't dies. Horrible SAP scripts, VBA macros and excel sheets that actually deal with your real data.

Because they shipped. They did the job.

Re: Faster Python with Guido van Rossum

#144

Post by dragonwriter »

Lucasoato wrote: post #29
ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language, like Javascript... ... but I think you're forgetting the real motivations that drive people to use Python: it's easy to learn, it's very readable, there's a large (and not so much toxic) community behind, it's versatile. No one is saying that it will replace every other language, but it still can find its space eve…

> Ok, let's admit it, writing bad Python code is very easy, as it is in every dinamically typed language

Writing bad code is easy in most statically typed languages too.

Re: Faster Python with Guido van Rossum

#145

Post by hardolaf »

kjksf wrote: post #129
nickjj wrote: post #72

Earlier quoted context omitted.

Sure, Rust too for some of the desktop app components. But the takeaway is you only need to do that when you're operating at mega scale and actually hit these bottlenecks. In a ton of cases you'll be completely fine serving a few million monthly page views of your SAAS app on a single $20-40 a month server using Flask, Django or whatever Python web framework you prefer. Performance will be really good too. Just talki…

I'm a solo dev. I write all my backend code in Go (used Python before Go existed) because I can deploy Go executable on the cheapest render.com server and it runs using fraction of the available resources (~50 MB out of 512 MB available and literally 0.01% CPU use). If I used Python (or Ruby, node or even Java) I would likely have to go higher (and pay more) because those languages are not only slow, they are also me…

I gave Go a try recently and it was a quite pleasant experience. I don't see why I wouldn't use it for greenfield work over Python.

Re: Faster Python with Guido van Rossum

#146

Post by zohch »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

> I've seen the curve and the pain it's caused, and would never use it for any code that needs to be performant or actively developed on the multi-month or year timescale

Typed python is safer and more maintainable than Go. Slow as a dog, but it's really a hard choice when choosing performance and a bad type system with a half baked language vs a good language with a good type system that is a bit of a drag on performance.

And you really don't have to go all in typing. I have worked on gradually typing code bases and it pays off from day one almost.

Re: Faster Python with Guido van Rossum

#147

Post by njharman »

ctur wrote: post #4

I have become increasingly convinced Python as a language is a "trap" for any use of notable scale, be the scale about number of developers, codebase size, or performance requirements. It's a great 0->1 language and great at simple glue, but eventually you hit a wall and have to keep investing larger and larger amounts of people or computing resources to get continued returns... all due to fundamental design decision…

The GIL has already been removed in at least Jython and PyPy.

PyPy is 4x faster (on benchmarks), no xp with Jython.

Re: Faster Python with Guido van Rossum

#148

Post by throwaway894345 »

joconde wrote: post #132
throwaway894345 wrote: post #36

Earlier quoted context omitted.

> Adding types after-the-fact can certainly be painful, but at least it's not an all-or-nothing choice, we can opt in to types for selected parts of a codebase. One of the benefits of typing (that is rarely discussed) is that it deters people from writing code whose type would otherwise be crazy (e.g., "if someone passes the string 'foo' in for a parameter, then the return type is a string, otherwise it's a bytestrin…

If a person like that is able to harm the team, I'd argue that it's a management problem. Lazy coding is only an issue when you allow people to be lazy...

I think you could make the same argument about code styling or really just about anything. If you had sufficiently quality coworkers, these things probably wouldn't be an issue, but it's a lot easier to just implement the technical solution (a code formatter or type checker).

Re: Faster Python with Guido van Rossum

#149

Post by dragonwriter »

azeirah wrote: post #47
qsort wrote: post #27

Earlier quoted context omitted.

Pretty much everything you're saying matches my experience, so, playing the devil's advocate: - The typing argument is impossible to argue against, but isn't it the same problem you'd have with any other dynamic language? How is, say, Javascript any better? I don't see this as something that's specifically Python's fault, rather, yet another argument as to why starting a large project in 2021 without seriously consid…

For your first point, I don't think a language such as JavaScript -is- much better, which is why Typescript was invented as an alternative for larger codebases/organizations

TypeScript:JS is a lot like mypy:Python; the only real difference is that Python adapted so that mypy (originally envisioned as its own language) code is also valid Python as well as the reverse, obviating the need for a compilation step to run mypy in the Python runtime (whereas TS has to compile to JS to run); also, mypy has mypyc to compile (currently, only a subset of) mypy code to something more efficient than normal python.

Re: Faster Python with Guido van Rossum

#150

Post by zohch »

mpdehaan2 wrote: post #97
Qem wrote: post #6

Earlier quoted context omitted.

Given Python has the multiprocessing module, I always get confused when people talk about Python lack of support for multicore. What are the shortcomings of the multiprocessing module, that cause people to disregard it?

I am probably the most commercially successful user of the multiprocessing module :) It's basically fine anywhere you need a function call that you can dispatch out to like 50 or 500 workers on a queue and then do something after that returns, but any shared memory or IPC between the workers is up to you. Python is also fine for webserving because most web servers pre-fork workers or whatever, so this doesn't come in…

> any shared memory or IPC between the workers is up to you.

There is this:

- https://docs.python.org/3/library/multiprocessing.shared_mem...

- https://docs.python.org/3/library/queue.html

- https://docs.python.org/3.8/library/multiprocessing.html#mul...

- https://docs.python.org/3.8/library/multiprocessing.html#mul...

> It's harder if you want to do something different where you want threaded workflows with synchronized/protected like constructs that folks might be familiar with from say Java.

There is this:

- https://docs.python.org/3/library/threading.html - which you can use as a context manager (i.e. `with lock`)

Post reply on HN