Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's up, Python? The GIL removed, a new compiler, optparse deprecated

bitecode.dev

Post reply on HN
Print view View original

201–210 of 302 posts

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#201

Post by coldtea »

felipetrz wrote: post #170
coldtea wrote: post #86

Earlier quoted context omitted.

> Use the python multiprocessing module. If you've already written it with the multithreading module, it is a drop in replacement. Your data structure will live in shared memory Only if it can be immutable. So it can't be shared and changed by multiple processes as needed (with synchronization). And even if you can have it mostly immutable, if you need to refresh it (e.g. after some time read a newer large file from…

For this use case it would be better to put the data in a shared SQLite database than relying on multiprocessing CoW. Even accessing objects from the shared memory would cause the reference counter to increment and the data would be copied, causing a memory usage explosion.

>For this use case it would be better to put the data in a shared SQLite database than relying on multiprocessing CoW

In Python yes. In Java you could take advantage of shared memory and get spared the overhead of SQLite.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#202

Post by nico_h »

The tenses in the headline and the article are very iffy.

It’s more like there will be work to remove the GIL that will start after a particular PEP will be approved.

The main reason we are still writing some stuff in java at our shop is because parallel processing sucks with multiprocessing, and trivial in java.

I look forward to a future where it as trivial in or even simpler in python.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#203

Post by dotnet00 »

9dev wrote: post #181
dotnet00 wrote: post #172

Earlier quoted context omitted.

Yes, this is even more the case in languages that are popular with more "applied" programming audiences, like scientific computing. Telling them "no you should be using this complicated DBMS" (or whatever other acronym) is not productive. It tends to get them exceptionally mad because their concern isn't the ideal way to write the code and architect the system, they simply want to write just enough code to continue t…

This stance always rubbed me the wrong way a bit. Effectively, code is one of the tools a researcher uses to do their work. As soon as their work interacts with other people, for example when publishing a purportedly reproducible study or supplying novel algorithms to developers, they have a responsibility to deliver proper work that can be used and understood by other people. This is something we expect of every oth…

I think data scientists tend to have a lot of overlap with computer people so expectations for them may be a bit higher, my experience comes mainly from physicists.

Reproducible, documented and bug free is fine, they care plenty about those things too, the issue is the "no you're doing it the wrong way, use this entirely different technology instead" being based almost entirely on ideological reasons.

If we take C multithreading as an example, with my superivising scientist, multithreading is fine, he's willing to put some time into learning how it works because it's valuable and has had a stable interface backed by a reliable body for a while now. But if tomorrow you came up to him and insisted that doing multithreading was wrong without a solid technical reason (eg actual bugs and an explanation of how the only way to fix it is to dump the existing code and spend a few months redesigning) you'd get shot down.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#204

Post by the8472 »

nwallin wrote: post #53
wmwmwm wrote: post #14

Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Every time I’ve done a quick implementation in Python of a service that then became popular (within a firm, so 100s or 1000s of clients) I’ve often ended up having to rewrite in Java so I can throw more threads at servicing the requests (often CPU heavy). I may have missed somethin…

> I may have missed something but I couldn’t figure out how to get the multi-threaded performance out of Python Multiprocessing. The answer is to use the python multiprocessing module, or to spin up multiple processes behind wsgi or whatever. > Historically I’ve written several services that load up some big datastructure (10s or 100s of GB), then expose an HTTP API on top of it. Use the python multiprocessing module…

Loading 100GB into RAM and then calling fork() is just painting a giant OOM Killer target on your back. It'll work until something breaks the CoWs or the parent gets restarted while some forks still linger or other fun things like that.

Threads make it transparent to the OS that this memory really must be shared between compute tasks.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#205

Post by kzrdude »

wtetzner wrote: post #142
dragonwriter wrote: post #136

Earlier quoted context omitted.

Because at the time of the 2-3 migration, parallelism wasn’t viewed as being as important as it is today.

Is that really true? We already had multicore machines, and Herb Sutter's "The free lunch is over" article had been published for years by then.

Python is used much more widely and for more data tasks now than it was then, I think. Besides, we've all slowly adapted to using parallelism everywhere, it was not overnight.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#206

Post by kzrdude »

nickcw wrote: post #74
cvnmalk wrote: post #28

Which, except for optparse, was all on the front page yesterday. So optparse is deprecated. More work I guess apart from auditing extensions for threading. Life is great in the Python treadmill.

I've tried to love argparse but it is so complicated. I always have to read the docs each time I use it. getopt has its own brutal simplicity.

I think argparse works fine. What worries me is that it's also "soft-deprecated", because devs have said it should get no further development. I hope it stays around, because I use it by default as a no-dependencies solution that I know how it works.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#207

Post by rightbyte »

wodenokoto wrote: post #140

When it was an in dev project, I felt the consensus on HN was that it was amazing work and a shame that it looked like the steering committee wouldn’t adopt it. Now they have and everyone seems to hate it.

It is probably language design enthusiasts push all these backwards incompatibilities into Python because they are not the users of the language.

They are a different group from those having their code broken in a never ending incompatibility churn.

Well atleast it gives us jobs ...

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#208

Post by Joeboy »

quickthrower2 wrote: post #151
doctoboggan wrote: post #148

Earlier quoted context omitted.

I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main ver…

Each library maintainer (probably mostly volunteers) has to decide whether to put effort into making their code thread safe. Clearly it won't be 100% of libraries that "upgrade". Then on top of that, they know their effort might be for nothing if the decision is made to keep Python GIL-only all along (one of the possible 3 outcomes at the end of the 5 years: ["gil", "nogil", "both supported").

> Clearly it won't be 100% of libraries that "upgrade".

I'm wondering how many libraries with binary extensions are actually in common use. Like, maybe 90% of python projects use a subset of a few hundred such packages?

That's a hassle if you maintain one of those packages, and will be a bit disappointing if in 5 years' time you're still depending on GIL-reliant packages.

But it's nothing like the chaos of the python 2-3 changes, where ~100% of python files in every package and end-user project had to be fixed.

I only learned about this this morning though, it's very possible I'm missing something. A lot of the concerns people are raising look a bit overblown to me.

I take the point that after so many abortive GIL removal attempts, it's harder to be confident this one will happen. But having the go-ahead from the steering council seems like a good indicator this one has traction.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#209

Post by kzrdude »

doctoboggan wrote: post #148
quickthrower2 wrote: post #145

Earlier quoted context omitted.

But you need to pick your horse. In 5 years time, Python will either be GIL or no GIL, and it is hard to tell which. It might be a setting (which might be more ideal). If you assume nogil, you need to choose dependencies that support that. You may need to trade off: eschew dependencies that aren't looking like they will be nogil compatible by the deadline. You are stuck on Python 3.18 maintenance branch or whatever,…

I don’t really understand this. Unless I am missing something you should always pick the “no GIL” version as that will work with or without a GIL. Thread safe No GIL code would be totally fine to run on python compiled with the GIL with zero modifications. Because of this I don’t expect there to be multiple versions of any library. Once a library does the (admittedly heavy) lift to no GIL it will just be the main ver…

Current plan says there has to be separate builds per module, as if it is an ABI break. Would be much better if it could be combined into one build. Hopefully necessity triggers some invention here.

Re: What's up, Python? The GIL removed, a new compiler, optparse deprecated

#210

Post by pjmlp »

tsimionescu wrote: post #163
mike_ivanov wrote: post #75

Earlier quoted context omitted.

"absolutely an expected outcome." Good day. Is it the right time to talk to you about Common Lisp?

To be fair, if you use CL in a similarly dynamic way as Python (don't compile anything, don't add any declarations etc) it won't be that much faster. You'll get some boost out of the stdlib stuff being compiled already, but otherwise it will incur similar performance penalties.

We can add Smalltalk, SELF, Dylan, JavaScript into the discussion then.
Post reply on HN