Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

61–70 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#61

Post by dekhn »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Over quite some time I've become convinced multiprocessing module is better than an optional GIL removal.

It may leave many useful bits on the table (compared to pure multithreaded coding, like C++/pthreads) but I've still been able to get it to scale my application performance (CPU-bound, large-memory) to the number of cores of even large boxes (96+ vCPUs). IIRC the future/concurrent library was key to being productive.

20 years ago I would said different, as at the time, IronPython demonstrated a real alternative to CPython that was faster, and fully multitrhreaded (including the container classes).

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#62

Post by acdha »

charrondev wrote: post #30
LexiMax wrote: post #21

Earlier quoted context omitted.

In a past life I hacked on PHP for a living, and in the time it took Python 2 to ride off into the sunset, PHP got two major migrations under its belt in 5.2 to 5.3, and then again 5.6 to 7.0. It was amazing to see the contrast between the two languages. PHP gave you plenty of reasons to upgrade, and the amount of incompatible breaking changes was kept to a minimum, often paired with a way to easily shim older code t…

I’ve taken an application codebase from PHP 5.3 to 8.2 now and it was relatively easy the whole way. The real key to minimize the pain was writing effective integration tests with high coverage. We didn’t have a good test suite to start but once we added some utilities to easily call our various endpoints (and internal API client if you will) and make assertions about the coverage came quickly. Popular frameworks lik…

> The real key to minimize the pain was writing effective integration tests with high coverage

I think this makes it really hard to do comparisons: I’ve done Python 2 to 3 migrations which took an hour or two because the code had tests and was well-maintained, and PHP migrations which were painful slogs without tests and sloppy code (“is this ignored error new or something we should have fixed in the 2000s?”). Most developers don’t have enough data points to say whether the experience they had was due to the language or the culture.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#63

Post by oivey »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

You have to manually set up shared memory with its own API that has its own limitations, right? I thought some seamless integration was a new feature, but AFAICT, transfers between multiprocesses still leads to things being pickled and copied. Am I wrong?

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#64

Post by maxnoe »

cvnmalk wrote: post #28

Which, except for optparse, was all on the front page yesterday. So optparse is deprecated. More work I guess apart from auditing extensions for threading. Life is great in the Python treadmill.

IIRC, optparse was going to be be removed in 3.5(?) but outcry was large .

It has a depreciation warning in the docs since 3.2.

It was in the "please just use argparse instead" state for a long time, this "just" adds an actual code warning.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#65

Post by viraptor »

kroolik wrote: post #41
threatripper wrote: post #35

Earlier quoted context omitted.

You have a single big data structure that can't be shared easily between multiple processes. Can't you use multiprocessing with that? Maybe mapping the data structure to a file and mmapping that in multiple processes? Maybe wrapping the whole thing in database instead of just using one huge nested dictionary? To me multi-threading sounds so much less painful than all the alternatives that I could imagine. Just adding…

One annoying part with multiprocessing in Python is that you could abuse the COW mechanism to save on loading time when forking. But Python stores ref counters together with objects so every single read will bust your COW cache. Now, you wanted it simple, but got to fight with the memory model of a language that wasn't designed with performance in mind, for programs whose focus wasn't performance.

There's gc.freeze for that now https://docs.python.org/3/library/gc.html#gc.freeze

If you load something big before forking workers, there's no CoW issue with that big structure anymore.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#66

Post by brightball »

antod wrote: post #46
brightball wrote: post #36

Earlier quoted context omitted.

This is actually one of the reasons I was drawn to Ruby over Python. Ruby also has the GIL but jRuby is an excellent option when needed.

I wonder what lead to JRuby attracting support while Jython not? I know the Jython creator went on to other things (was it eg IronPython for dotnet?). I suppose it was the inverse with dotnet - eg IronPython surviving while IronRuby seems dead. Is it just down to corporate sponsorship?

JRuby has been pretty actively maintained for about 15 years and had a big release this year.

It’s an impressive project.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#67

Post by __d »

jeremycarter wrote: post #19
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.

Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic.

This is absolutely an expected outcome.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#68

Post by bafe »

quickthrower2 wrote: post #60
bafe wrote: post #55

Earlier quoted context omitted.

Not likely, particularly if you depend on modules written (partly) in C like numpy/scipy etc

I just did some searching around PyPy and that seems to be the case. IronPython is out of support now but the looks of it. Which is a shame. I heard of it 10 years ago, but assumed it was some "Microsoftized Python" and not at all a compatible thing :-)

The same happened to jython which is all but dead and stuck forever at python 2.7

If you stick to "pure Python" there's a larger chance you can use any python runtime and be able to run your code

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#69

Post by cypress66 »

__d wrote: post #67
jeremycarter wrote: post #19

Earlier quoted context omitted.

Similar experience. Even with multi process and threads python is slow, very slow. Java, Go and .NET all provide a very performant out of box experience.

Python is both an interpreter, and quite dynamic. Both of these lead to lower performance when compared to less dynamic, compiled solutions. All of Java, Go, and .NET are compiled and (much) less dynamic. This is absolutely an expected outcome.

Node is pretty performant for anything IO related, not compiled and reasonably dynamic.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#70

Post by qbasic_forever »

wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

Are you just reading from this data structure? If so I wouldn't do any locking or threading, I'd just use asyncio to serve up read requests to the data and it should scale quite well. Multithreading/processing is best for CPU limited workloads but this sounds like you're really just IO-bound (limited by the very high IO of reading from that data structure in memory).

If you're allowing writes to the shared data structure... I'd ask myself am I using the right tool for the job. A proper database server like postgres will handle concurrent writers much, much better than you could code up hastily. And it will handle failures, backups, storage, security, configuration, etc. far better than an ad hoc solution.

Post reply on HN