31–40 of 95 posts
Yet another asyncio tutorial that shows you to run a few sleep tasks concurrently. Can we finally get one that shows how to do real stuff such like socket programming, wrapping non-async-compatible libraries and separating cpu-intensive blocking tasks to awaitable threads?
I’m still bummed that Python took this direction. Maybe introducing new keywords into the language for event loop concurrency was Python’s way of satisfying “explicit is better than implicit” but i can’s shake the feeling that callback passing and generator coroutines are a fad that is complex enough to occupy the imagination of a generation of programmers while offering little benefit compared to green threads.
Python is my language of first choice, but I must say that I am not that thrilled how this multithreading ended up. There are many tutorials about the topic promising to explain how it works, usually in the form of "simple introduction". But when one tries to implement something production-ready, with correct error handling etc., things starts to complicate pretty quickly; at least that was my experience. I don't wan…
I am curious what considerations your company had before switching some projects to Go. Python multithreading has been an issue for us as well. While asyncio looked good in the tutorials, gevent was much easier to work with. However, we still face multiple issues moving our celery workers to gevent and I am not sure if there is a better production friendly alternative for celery-gevent in python.
> Concurrency is like having two threads running on a single core CPU.
> Parallelism is like having two threads running simultaneously on different cores
> It is important to note that parallelism implies concurrency but not the other way round.
Aurgh! I don't think this attempted definition-by-simile is helpful, or even somewhat correct.
I much prefer yosefk's way of framing things:
> > concurrent (noun): Archaic. a rival or competitor.
> > Two lines that do not intersect are called parallel lines.
...
> Computation vs event handling
> With event handling systems such as vending machines, telephony, web servers and banks, concurrency is inherent to the problem – you must resolve inevitable conflicts between unpredictable requests. Parallelism is a part of the solution - it speeds things up, but the root of the problem is concurrency.
> With computational systems such as gift boxes, graphics, computer vision and scientific computing, concurrency is not a part of the problem – you compute an output from inputs known in advance, without any external events. Parallelism is where the problems start – it speeds things up, but it can introduce bugs.
...
> concurrency is dealing with inevitable timing-related conflicts, parallelism is avoiding unnecessary conflicts
yosefk's whole essay about this is great: https://yosefk.com/blog/parallelism-and-concurrency-need-dif...
Earlier quoted context omitted.
The cooperating part of this concurrency model is the complicated part. Consider how you would go about making an orm like sqlalchemy cooperate. Now you have to access properties like this: name = await account.user.name since a lookup may have to occur. This is extremely unnatural and would be better if you could just avoid writing await yet still depend on it being concurrent without blocking your event loop. The f…
I think this just means that a sqlalchemy style ORM doesn't fit the model. If you had an ORM where the calls which could call cause database queries were distinct from calls which just looked up local properties, then this would work fine...
await session.commit()
user.name # BlahError: Object not loaded
# correct
await session.commit()
await user.refresh()
user.name
Another thing that might not be completely obvious, but sessions and their objects (session×objects = transaction state) are never shared between threads, similarly it would be unwise to share them between different asynchronous tasks.
Earlier quoted context omitted.
Isn't Python's async/await syntax an implementation of green threads? I mean using await is almost exactly the cooperative scheduling idea. The article may use Futures and callbacks but you can just as easily do something like: result = await fake_network_request('one')
They're sort of similar, and you can probably get the same work done in either system, but I think real threading (green or otherwise), may leave you with less cognitive load. Spawning a thread may be complex, and thinking about how the threads are scheduled is often complex, but what each thread does can be very simple -- and you don't have to think about 'long running things need to be futured/awaited', you just do…
And that's how it starts, and in the end it's New Year's Eve and you're somehow, again, debugging a deadlock.
> green threads
yes please
Earlier quoted context omitted.
I am curious what considerations your company had before switching some projects to Go. Python multithreading has been an issue for us as well. While asyncio looked good in the tutorials, gevent was much easier to work with. However, we still face multiple issues moving our celery workers to gevent and I am not sure if there is a better production friendly alternative for celery-gevent in python.
we love gevent as well. we use rq instead of celery . Try that instead ( http://python-rq.org/docs/workers/ )
Wasn't there a language where every call was async? Instead of async ... A/returing Future[A] it did/would return A from method calls. If it didn't exist, one can imagine one. A.x = 3 would be wrapped in A.map(_.x = 3) etc. So you write code that would be executed when you finally await a value. No more red/blue world. Would probably need coroutines instead of threads for executing.
quoting the article: > Concurrency is like having two threads running on a single core CPU. > Parallelism is like having two threads running simultaneously on different cores > It is important to note that parallelism implies concurrency but not the other way round. Aurgh! I don't think this attempted definition-by-simile is helpful, or even somewhat correct. I much prefer yosefk's way of framing things: > > concurre…
>A parallel program is one that uses a multiplicity of computational hardware ....
>concurrency is a program-structuring technique in which there are multiple threads of control...
(a pdf can readily be found with your favorite search engine for the full extract :) ).
I would much prefer to see a precise, rigorous definition and then examples (or eg and then defn is also acceptable), instead of just a list of examples. Examples help you understand a rigorous statement. But, if you only give a hand waving explanation for something, I think it just creates more confusion in the end, as you never know exactly what is correct. It's leaving it open for ambiguity.