Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A viable solution for Python concurrency

lwn.net

Post reply on HN
Print view View original

361–366 of 366 posts

Re: A viable solution for Python concurrency

#361

Post by globular-toast »

da39a3ee wrote: post #305
globular-toast wrote: post #181

Earlier quoted context omitted.

> Programs that just divide up work across processes are much easier to write without introducing obscure bugs due to the lack of atomicity. You often don't even need to do this yourself. GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? I agree with you that threads are talked about way more than they should be. It's like all programmers learn this one simple rule: to be f…

> GNU parallel is the way to go for dividing work up amongst CPU cores. Why reinvent the wheel? We’re not talking about writing scripts to run on your laptop. We’re talking about code written for production applications. Deploying GNU parallel to production nodes / containers would be a major change to production systems that may not be feasible and even if it is would come with a high cost in terms of added complexi…

I used to use GNU parallel to run big data tasks on supercomputers. There's nothing special about "production". It's all just computers.

Re: A viable solution for Python concurrency

#362

Post by rfoo »

the__alchemist wrote: post #356
rfoo wrote: post #344

Earlier quoted context omitted.

> Let Python excel at its core competencies Python is notably very popular in two communities: web developer and scientific computing. The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline…

Numpy/Scipy are great for prototyping, but IMO code the performant / concurrent stuff in C/C++/Rust/Julia once your prototype is done. Use the right tool for the job instead of trying to bend the wrong one.

Try to convince those "Data Scientists" to learn Julia/Rust or spend another ten years for learning how to write non-crashy C++ then.

Also, IMO Python is the right tool for glueing optimized C++ implementation of various linalg algos together. And this is exactly how we use Python. And we still have to workaround GIL. Yes, it is that bad.

Glue codes could be slow, but they must scale. They should not unnecessarily contend for a stupid lock. GIL is really a scalability bug and I do acknowledge it is a hard-to-fix one due to the world's legacy codebases depends on it. However when someone applies an absurd amount of computer science (the design OP posted is literally based on state-of-the-art PL research), you should at least be sincerely curious why people are being so serious about it.

Re: A viable solution for Python concurrency

#363

Post by rfoo »

detaro wrote: post #354
rfoo wrote: post #346

Earlier quoted context omitted.

Previous GIL removal attempts hurts single thread performance and it isn't that scalable, so people are usually by default dismissive. Most of Python codes depend on subtle details of CPython internal. For example sometimes it is just convenient to assume GIL exists (i.e. simplifies concurrency codes because "you know there are at most one thread running").

I haven't seen any appetite to even consider solutions that break the promises of the GIL and make currently atomic things non-atomic, so that second argument seems weird.

You may be surprised by this, then: https://mail.python.org/archives/list/python-dev@python.org/...

Re: A viable solution for Python concurrency

#364

Post by thehappypm »

the__alchemist wrote: post #356
rfoo wrote: post #344

Earlier quoted context omitted.

> Let Python excel at its core competencies Python is notably very popular in two communities: web developer and scientific computing. The former usually yell loudly every time someone propose to remove GIL. Meanwhile everyone in the scientific computing community had to learn how to workaround GIL which absolutely sucks and sometimes just impossible. (e.g. I have a mostly memory-bandwidth-bound data loading pipeline…

Numpy/Scipy are great for prototyping, but IMO code the performant / concurrent stuff in C/C++/Rust/Julia once your prototype is done. Use the right tool for the job instead of trying to bend the wrong one.

In actual production, corporate environments, performance isn't necessarily the most important thing. Maintainability is often much more important. Sure, a Rust system may be faster than Python, but when your lead engineer quits and your backup is on vacation, and nobody else even knows how the hell the code works, you're going to reach for something like Python or Java.

Re: A viable solution for Python concurrency

#365

Post by rtpg »

OOPMan wrote: post #335
BeFlatXIII wrote: post #332

Earlier quoted context omitted.

Could you tell us more about the "packing in Python timezones"? I'm curious to learn more (and it may be of tangential interest to one of my projects).

They just bundled the timezone database in with the language. It's not particularly groundbreaking...

not groundbreaking, but still useful and lets packages just assume that this stuff is available (well not just yet, but soon).

Useful stuff for lib maintainers.

Re: A viable solution for Python concurrency

#366

Post by masklinn »

BiteCode_dev wrote: post #349
masklinn wrote: post #296

Earlier quoted context omitted.

> Python is my favorite language, and I can live with the explicit loop, but explicit scheduling is ridiculous. Just run the damn coroutine You can't "just run the damn coroutine", that's not what coroutines are . But that does point to the mistake of Python, at least from a UX perspective: coroutines are more efficient but they're also a lot more prone to use errors, especially in dynamically typed languages. Async…

I know, but we could make coroutine creates with async have a special marker that is different to yield and auto schedule.

That’s what a task is.
Post reply on HN