Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Think Python, 3rd Edition

allendowney.github.io

Post reply on HN
Print view View original

41–50 of 126 posts

Re: Think Python, 3rd Edition

#41

Post by andai »

bmitc wrote: post #35

How does one do proper concurrency in Python? As in separate "processes" running with encapsulation, fault tolerance, etc. For example, how does one bundle up a TCP client in a "process" such that the process handles failed connections, broken connections, retrieving data at periodic intervals (relatively fast), receiving "messages" from other "processes", etc.? There's Ray, Pyro, Pykka, Celery, multiprocessing, asyn…

Something something libuv?

Re: Think Python, 3rd Edition

#45

Post by da39a3ee »

bmitc wrote: post #35

How does one do proper concurrency in Python? As in separate "processes" running with encapsulation, fault tolerance, etc. For example, how does one bundle up a TCP client in a "process" such that the process handles failed connections, broken connections, retrieving data at periodic intervals (relatively fast), receiving "messages" from other "processes", etc.? There's Ray, Pyro, Pykka, Celery, multiprocessing, asyn…

Can you give an example of what you consider a good solution in a different language? That will help us see exactly what you're looking for and how/whether it might be achieved in Python.

Re: Think Python, 3rd Edition

#46

Post by Siecje »

bmitc wrote: post #35

How does one do proper concurrency in Python? As in separate "processes" running with encapsulation, fault tolerance, etc. For example, how does one bundle up a TCP client in a "process" such that the process handles failed connections, broken connections, retrieving data at periodic intervals (relatively fast), receiving "messages" from other "processes", etc.? There's Ray, Pyro, Pykka, Celery, multiprocessing, asyn…

Use asyncio. Bind to a socket, set blocking False, then asyncio.run(on_new_connection(sock)).

Inside that coroutine get the loop and await loop.sock_accept(sock)

And then asyncio.create_task(on_connection_data(connection)).

The only gotcha is you need to keep a reference to that task so it doesn't get garbage collected.

Re: Think Python, 3rd Edition

#47

Post by peruvian »

Apologies for changing the subject, but aside from real world experience (which I have and am getting at work), is there a resource of similar quality for more intermediate/advanced Python programmers? I always feel like there's a big chunk of the language or stdlib I do not know.

Re: Think Python, 3rd Edition

#48

Post by jononor »

bmitc wrote: post #35

How does one do proper concurrency in Python? As in separate "processes" running with encapsulation, fault tolerance, etc. For example, how does one bundle up a TCP client in a "process" such that the process handles failed connections, broken connections, retrieving data at periodic intervals (relatively fast), receiving "messages" from other "processes", etc.? There's Ray, Pyro, Pykka, Celery, multiprocessing, asyn…

If you are I/O bound then asyncio or good old fashioned gevent will do great. If you are CPU bound, then use multiprocessing. If you need to accept "jobs" from elsewhere, use RabbitMQ (with or without Celery). If you have mixed CPU/IO workload that fits the worker pattern, then you would do all 3. At the top level you have a RabbitMQ consumer, fetching jobs from a remote queue and then putting these into a multiprocessing queue processed by N=~cpucount processes. And each of these use asyncio/gevent to do their work.

Re: Think Python, 3rd Edition

#50

Post by jjice »

peruvian wrote: post #47

Apologies for changing the subject, but aside from real world experience (which I have and am getting at work), is there a resource of similar quality for more intermediate/advanced Python programmers? I always feel like there's a big chunk of the language or stdlib I do not know.

I enjoyed Effective Python. It's a "tips" style book with a good handful of recommendations with use cases and applications.
Post reply on HN