Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Intent to approve PEP 703: making the GIL optional

discuss.python.org

Post reply on HN
Print view View original

331–340 of 513 posts

Re: Intent to approve PEP 703: making the GIL optional

#331

Post by lozenge »

Dylan16807 wrote: post #311
lozenge wrote: post #300

Earlier quoted context omitted.

You have to recompile your C extensions for the nogil build. Consider an application where one thread is calling xs.append from a C extension and another is calling xs.pop and xs[-1] on the same list object xs. In the nogil build these operations need to use a fine-grained lock on xs, and in the gil build these are thread-safe due to each thread holding the GIL when it does these operations. On top of that, some of t…

> However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution Yes, that's what they were asking about. Why have two versions for a mode switch. Everything you explained before that is irrelevant, I'm afraid. > but I'm expecting this to be slower than running the gil build in GIL mode. That could be the answer to their qu…

1. The "stable ABI" is broken on nogil, the selling point of the stable ABI was "add this C preprocessor flag and your extension will work on all future CPython versions, after paying a speed penalty". This is useful for eg closed source extensions. If the nogil build was the only build available, these extensions would require recompilation.

2. There are a lot of users that want fast Python and a lot of effort was put in to optimise it. I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance, for a feature (nogil) which most won't be able to use due to using C extensions which haven't had nogil-supporting code changes.

Re: Intent to approve PEP 703: making the GIL optional

#332

Post by miraculixx »

AlphaSite wrote: post #237
commonlisp94 wrote: post #53

Earlier quoted context omitted.

Exactly. I think a lot of the negativity about GIL comes from a misunderstanding about forking processes. If python is being used as a scripting language, and spawning other tools, you're already getting free multi-core. A similar misunderstanding exists about SQLite and concurrency.. but that's a topic for another time.

Forking had a ton of its own downsides, it’s not a free lunch either, from poor ergonomics to communications overhead it works well for somethings and very poorly for others.

Yes, the same is true for free threading. Yet people assume free threading is free concurrency and that's the problem.

Re: Intent to approve PEP 703: making the GIL optional

#333

Post by miraculixx »

usrbinbash wrote: post #277
jmount wrote: post #46

Why would you even want a no-GIL Python? Java and C showed how much more effort it takes to maintain slower thread safe code for no real benefit. Parallelize at the fork level or at the isolated numeric library level.

> Parallelize at the fork level IPC is a PITA, and orchestrating processes is even worse. > or at the isolated numeric library level Not everything I want to parallelise in python runs in numpy. Simple example: WebService Backends. I have a 64 core server running a Werkzeug/Gunicorn application. The Service is mostly doing CPU bound tasks (data aggregation and analysis), so asyncio is pointless. What happens is, it r…

You would be well served with arena based (shared) object allocation and a gil per thread model. No need to draw everyone else into this.

Re: Intent to approve PEP 703: making the GIL optional

#334

Post by Dylan16807 »

lozenge wrote: post #331
Dylan16807 wrote: post #311

Earlier quoted context omitted.

> However, it looks like the nogil build will be able to run in "GIL mode" for maximum compatibility, including switching to GIL mode partway through execution Yes, that's what they were asking about. Why have two versions for a mode switch. Everything you explained before that is irrelevant, I'm afraid. > but I'm expecting this to be slower than running the gil build in GIL mode. That could be the answer to their qu…

1. The "stable ABI" is broken on nogil, the selling point of the stable ABI was "add this C preprocessor flag and your extension will work on all future CPython versions, after paying a speed penalty". This is useful for eg closed source extensions. If the nogil build was the only build available, these extensions would require recompilation. 2. There are a lot of users that want fast Python and a lot of effort was p…

> The "stable ABI" is broken on nogil, the selling point of the stable ABI was "add this C preprocessor flag and your extension will work on all future CPython versions, after paying a speed penalty". This is useful for eg closed source extensions. If the nogil build was the only build available, these extensions would require recompilation.

It looks for a flag, if it doesn't see it then it turns the GIL on.

Where is the need for recompilation?

> I think the core team wouldn't want to release and expect everybody to use a version which regresses in performance

But you're just guessing that it's slower in GIL mode, aren't you?

Re: Intent to approve PEP 703: making the GIL optional

#335

Post by merb »

lozenge wrote: post #306
slt2021 wrote: post #73

Earlier quoted context omitted.

I would argue that if you have large concurrency and shared complex state - you better off use kafka and redis/memcached as a shared state - and design proper fan-out. This design scales much better for systems that will eventually overgrow one big machine. the No-GIL pytohn will be of no use, when you need to deploy your app across 100s machines. I understand people want to take advantage of all cores etc, but at la…

PostgreSQL shows how far you can get with a single big box and using multiple cores and shared memory. It's incredibly powerful and the vast majority of applications never have data big enough to warrant "100s of machines".

and even they are looking into changing its shared memory and process model to work and make it multi-threaded: https://www.postgresql.org/message-id/flat/31cc6df9-53fe-3cd...

probably a way more problematic effort than no gil python tough and it might fail

Re: Intent to approve PEP 703: making the GIL optional

#336

Post by [deleted] »

majewsky wrote: post #328
Dylan16807 wrote: post #305

Earlier quoted context omitted.

Oh please. Even if we ignore that any question can have implications, they directly said "someone has to say it". They were explicitly suggesting something by asking that question.

They were suggesting that this is a question that is going to be on people's minds, which is very different from suggesting that the answer is yes.

[deleted]

Re: Intent to approve PEP 703: making the GIL optional

#337

Post by duckmysick »

pdntspa wrote: post #197
c-hendricks wrote: post #190

Earlier quoted context omitted.

GP would probably be better if they didn't default to snark / derision by calling it cat hearding.

But the python 2->3 migration has been exactly that

No cats were harmed in the migration from Python 2 to 3.

Programmers and project managers on the other hand...

Re: Intent to approve PEP 703: making the GIL optional

#338

Post by Dylan16807 »

majewsky wrote: post #328
Dylan16807 wrote: post #305

Earlier quoted context omitted.

Oh please. Even if we ignore that any question can have implications, they directly said "someone has to say it". They were explicitly suggesting something by asking that question.

They were suggesting that this is a question that is going to be on people's minds, which is very different from suggesting that the answer is yes.

All I can say is that the phrasing looks quite biased to me and it would have been easy to not phrase it that way.

Subtext is hard to prove, but that doesn't make it not real.

Re: Intent to approve PEP 703: making the GIL optional

#340

Post by grumpyprole »

bitcharmer wrote: post #262
soulbadguy wrote: post #162

Earlier quoted context omitted.

There are a lot of good things about java (and the jvm by some extend), but good concurency is not one of them.

This is not true. JSR-133 was one of the first and most prominent implementation of memory model among general purpose languages. You don't seem to know enough about the platform and it's history to make definite statements like that, imo.

JSR-133 was quite a few years after Java's release. But even after these changes, many problems remained. For example, standard library classes were often not thread safe out-of-the-box, even something seemingly innocuous like a date formatter class would malfunction in a threaded context. The rules around thread safety were also very complex, especially with regard to constructing objects. IMHO, it did not compare well with other languages around the same time built with concurrency in mind, for example Erlang. I was once very fortunate to meet Joe Armstrong and we briefly discussed Java, suffice to say he was not impressed by it.
Post reply on HN