Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

271–280 of 366 posts

Re: A viable solution for Python concurrency

#271

Post by rich_sasha »

lucb1e wrote: post #160
rich_sasha wrote: post #7

This is some of the best news I read in a while! Multiprocessing sort of works but it’s really sucky.

I actually have a great experience using that rather than dealing with concurrency hell. My use case is typically brute forcing this or that, for example a quick implementation to crack a key on some ctf challenge, or a proof of concept to crack a session token for a customer demo. Just spawn a few processes, each gets 1/nth of the work, not a big deal. But I could see how, if you want to have (e.g.) sound and UI ren…

For me the main thing is, I cannot parallelise easily a loop mid-function. I need to make a pool, separate out the loop body into a separate top-level function, and also deal with multiprocessing quirks (like processes dying semi-randomly).

It feels quite heavy and clunky, is my issue.

Re: A viable solution for Python concurrency

#272

Post by shauns »

justusw wrote: post #233
BiteCode_dev wrote: post #76

Earlier quoted context omitted.

I used them both extensively, and here are the main reasons I can think of: - The event loop in JS is invisible and implicit. V8 proved it can be done without paying a cost for it, and in fact most real life python projects are using uvloop because it's faster than asyncio default loop. JS dev don't think of the loop at all, because it's always been there. They don't have to chose a loop, or thinking about its lifecy…

I’m not an expert on Node.js but I was always under the impression that you couldn’t write await outside of an async function. Has that changed recently?

You are right that you can only await in an async function, but you can call an async function anywhere and get back a Promise.

await-ing that promise is often what you want to do but there are other ways to do work once the promise has resolved.

Re: A viable solution for Python concurrency

#275

Post by birdyrooster »

ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

I don’t see what is so weird about it. The syntax is simple and straightforward. It’s how I wrote a ton of Python and it works great as long as you know what blocks and what doesn’t. Using aio_multiprocess it’s easy to saturate all the cores on a machine using the same basic syntax. It’s lovely really but still Python is too slow compared to golang so I rarely use it.

Re: A viable solution for Python concurrency

#276

Post by qstwall »

It is hard to see the point of a GIL removal that will destabilize the C extension ecosystem for probably a decade again:

C extensions can already start as many threads as they like. Threaded pure Python, even if GIL-less, is still slow, so what is the point?

The whole point of Python (before asyncio, pattern matching etc.) was being simple and having a nice C interface. If that continues to erode, people will (and should) look at other languages. C++ for example is pretty Pythonic these days, Java does not have these problems, Erlang (while slow) was written for concurrency from scratch, etc.

Re: A viable solution for Python concurrency

#277

Post by smetj »

btown wrote: post #97
ferdowsi wrote: post #11

If this effort succeeds (and I hope it does) now Python developers will need to contend with the event-loop albatross of asyncio and all of its weird complexity. In an alternate Python timeline, asyncio was not introduced into the Python standard library, and instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work.

> instead we got a natively supported, robust, easy-to-use concurrency paradigm built around green/virtual threading that accommodates both IO and CPU bound work Minus the "natively supported" part, we have this today in http://www.gevent.org/ ! It's so, so empowering to be able to access the entire historical body of work of synchronous-I/O Python libraries, and with a single monkey patch cause every I/O operation,…

Gevent is the best thing which ever happened within the async world of Python. It's just great to work with. Rock solid. I have high IO production software running for years with gevent as the workhorse, copes with high load, no maintenance, brilliant.

Then came Asyncio, which I personally disliked for the simple reason it became so popular that everybody thinks it's necessary to write an asyncio version of their library/module. The result is it now tears everything in 2. Asyncio code and non-asyncio code ...

Everything like Asyncio, Curio, Trio are impressive and interesting but their interfaces are way too involved for the intended result and it should never be like that. (Gevent suffered less from that).

The only way forward is proper thread support without the GIL and therefor I applaud this effort.

Re: A viable solution for Python concurrency

#278

Post by qstwall »

BeetleB wrote: post #143
throwaway81523 wrote: post #59

Earlier quoted context omitted.

Yes, I have a big sense of tragedy about Python 3. Python should run on something like (or maybe the actual) Erlang BEAM with lightweight isolated processes. All my threaded Python code is written using that style anyway (threads communicating through synchronized queues) and I've almost never needed traditional shared mutable objects. Maybe completely never, but I'm not sure about a certain program any more. Added:…

You are likely being downvoted because most claims about the pain of a Python 3 transition are inflated/hyperbole. It took less than a day to migrate all my code to Python 3. And by "less than a day" I mean "less than 2 hours". Granted, bigger projects would take longer, but saying stuff like "10+ years of pain" is ridiculous. Probably less than 1% of projects had serious issues with the migration. We just hear of a…

It took several years to migrate Twisted or NumPy. The 10 years is the cumulative pain of having 30% of packages not migrated yet. It took at least 7 years to have 90% migrated.

Re: A viable solution for Python concurrency

#279

Post by qstwall »

SoylentOrange wrote: post #231
cletus wrote: post #226

This feels like Schrodinger's Cake to me (you know, having it and eating it too). > ... the first of which is called "biased reference counts" ... With this scheme, the reference count in each object is split in two, with one "local" count for the owner (creator) of the object and a shared count for all other threads. Since the owner has exclusive access to its count, increments and decrements can be done with fast,…

> My personal view is that the world has largely moved on from dynamically typed languages for anything non-trivial or that isn't essentially plumbing. For good reason. Of course people will bring up Javascript but it has a captive market as being the only thing that'll universally run code in a browser and the likes of TypeScript can ease that burden anyway. This is not correct. There is a lot of data analysis code…

It will still be slow if the threaded code is in pure Python.

Re: A viable solution for Python concurrency

#280

Post by masklinn »

arc-in-space wrote: post #207
omnicognate wrote: post #183

Earlier quoted context omitted.

I like the GIL and would prefer it not be removed, regardless of any impact on performance. It's a powerful assumption for both python code and extensions to be able to make that only one thread will be executing in the interpreter at a time. Knowing it, you can do a lot of things with a much lower cognitive burden. I tend to think through a problem initially in a non-concurrent way and then think "ok, but there's co…

I'm confused. The article mentions that Python's data structures would be made thread-safe, which sounds like there is no intention to change concurrency semantics. How does the presence of the GIL help? Can you elaborate what would be lost by removing it?

A C extension which does not release the gil does not have to worry about concurrent execution and can trivially work in global space.
Post reply on HN