An easy way to concurrency and parallelism with Python stdlib
1–10 of 70 posts
Re: An easy way to concurrency and parallelism with Python stdlib
#2Does not seem exactly like an easy way to me. Not super hard, surely, but not "easy". More like "moderately easy to do and a bit annoying to implement".
Probably 20% of the effort shown in this post could have been expended to just write something very similar in Golang, and it would have taken less time, too. Because the way I see it this is trying to emulate futures / promises (and it looks like it's succeeding, at least on the surface). That can spiral out of comfortable maintainable code territory pretty quickly.
But especially for something as trivial as a crawler, I don't see the appeal of Python. You got a good deal of languages with lower friction for doing parallel stuff nowadays (Golang, Elixir, Rust if you want to cry a bit, hell, even Lua has some parallel libraries nowadays, Zig, Nim...).
Re: An easy way to concurrency and parallelism with Python stdlib
#3Does not seem exactly like an easy way to me. Not super hard, surely, but not "easy". More like "moderately easy to do and a bit annoying to implement". Probably 20% of the effort shown in this post could have been expended to just write something very similar in Golang, and it would have taken less time, too. Because the way I see it this is trying to emulate futures / promises (and it looks like it's succeeding, at…
If you already know Python, the advice in this article is certainly a lot easier and more actionable than "just learn Go or Rust or Zig instead".
Re: An easy way to concurrency and parallelism with Python stdlib
#4Does not seem exactly like an easy way to me. Not super hard, surely, but not "easy". More like "moderately easy to do and a bit annoying to implement". Probably 20% of the effort shown in this post could have been expended to just write something very similar in Golang, and it would have taken less time, too. Because the way I see it this is trying to emulate futures / promises (and it looks like it's succeeding, at…
If you already know Python, the advice in this article is certainly a lot easier and more actionable than "just learn Go or Rust or Zig instead".
Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it.
It happened with me and many other former colleagues.
Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.
Re: An easy way to concurrency and parallelism with Python stdlib
#5Earlier quoted context omitted.
If you already know Python, the advice in this article is certainly a lot easier and more actionable than "just learn Go or Rust or Zig instead".
Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it. It happened with me and many other former colleagues. Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.
The point of the article is a handful of lines. The rest is accoutrement like the URL list and timing code. But sure, if
tasks = {}
for url in URLs:
future = executor.submit(fetch_url, url)
tasks[future] = url
tasks = {executor.submit(fetch_url, url): url for url in URLs}Re: An easy way to concurrency and parallelism with Python stdlib
#6The easiest and modern way is simply to use asyncio...
Re: An easy way to concurrency and parallelism with Python stdlib
#7Earlier quoted context omitted.
If you already know Python, the advice in this article is certainly a lot easier and more actionable than "just learn Go or Rust or Zig instead".
Certainly. My point is that if you need to write that much code and/or do that much research, at one point the effort of doing it in another language will be less than to keep insisting on using a tool that's not designed for it. It happened with me and many other former colleagues. Though obviously, everyone decides for themselves when does that point come -- or if it comes at all.
> and/or do that much research,
Is reading the official docs section on concurrency lots of research?
Re: An easy way to concurrency and parallelism with Python stdlib
#8[dead]
Re: An easy way to concurrency and parallelism with Python stdlib
#9The easiest and modern way is simply to use asyncio...
Which is concurrent but not parallel.
Re: An easy way to concurrency and parallelism with Python stdlib
#10Is there a way to add tasks with independent timeouts using only the Python stdlib? I was reading a piece of code yesterday that had `pebble` as dependency and it looked like it was only needed for the `pool.schedule(..., timeout=1)`.