Nice, I learned about a couple new things. I take it these can all be implemented with the lock/semaphore primitives?
How to Make Python Wait
11–20 of 33 posts
Re: How to Make Python Wait
#12Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…
What if you are waiting on something that is not your Python code?
What about this?
Re: How to Make Python Wait
#13Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…
What if you are waiting on something that is not your Python code? What about this? https://github.com/rwarren/SystemEvent
You put the call in a function where you wait for it, and pass it to the thread pool.
Or you use asyncio for network based stuff.
Re: How to Make Python Wait
#14Earlier quoted context omitted.
What if you are waiting on something that is not your Python code? What about this? https://github.com/rwarren/SystemEvent
You put the call in a function where you wait for it, and pass it to the thread pool. Or you use asyncio for network based stuff.
Only one dependency "posix_ipc >= 1.0.0".
Re: How to Make Python Wait
#15Or just create a Queue(), give it as a parameter to the long calculation so it can store the result in it, and in the main thread just do q.get(). There are probably a dozen other synchronization primitives you can use, but this one is very versatile and you only need to keep one API in your head. Also, this approach somewhat mimics the concept of channels in Go.
Re: How to Make Python Wait
#16Earlier quoted context omitted.
What if you are waiting on something that is not your Python code? What about this? https://github.com/rwarren/SystemEvent
You put the call in a function where you wait for it, and pass it to the thread pool. Or you use asyncio for network based stuff.
> Or you use asyncio for network based stuff.
Only if you really need to. For 99% of network needs, using pool executors is simpler and easier than asyncio and it's one less only-sort-of-useful thing to have to learn.
Re: How to Make Python Wait
#17Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…
This example still involves a lot of manual work. It's often times even easier.
from concurrent.futures import ProcessPoolExecutor
import string
def hello() -> int:
seconds = random.randint(0, 5)
print(f'Hi {seconds}s')
time.sleep(seconds)
print(f'Bye {seconds}s')
return seconds
# Don't forget this for processes, or you'll get in trouble
if __name__ == "__main__":
inputs = list(string.printable)
results = []
# You can sub out ProcessPool with ThreadPool.
with ProcessPoolExecutor() as executor:
results += executor.map(hello, inputs)
[print(s) for s in results]Re: How to Make Python Wait
#18Earlier quoted context omitted.
You put the call in a function where you wait for it, and pass it to the thread pool. Or you use asyncio for network based stuff.
> Or you use asyncio for network based stuff. Only if you really need to. For 99% of network needs, using pool executors is simpler and easier than asyncio and it's one less only-sort-of-useful thing to have to learn.
We are getting there. asyncio in 3.7 is now quite ok to use.
Plus, I'm working with andrew sveltov on a new API for aiohttp, so that you can do:
def main():
response = await aiohttp.get(url)
asyncio.run(main())
The biggest problem is that I have yet seen a tutorial that explains properly how to use asyncio.
They all talk about the loop, and future, etc..
First, they all should tell you that asyncio should be used only with Python 3.7+. Don't even bother before. Not that it's not possible, but I've done it, and it's not worth the trouble.
Then, all tutorials should mention wait() or gather(), which are the most important functions of the whole framework. It kills me to never see those explained.
With just that knowledge, you can script happily at least as easily as with the pools I just had show case.
Now, I really hope that we are going to get trio's nurseries imported in the stdlib. yury selivanov is working on it from uvloop, so I got good hopes.
I did a proof of concept as an asyncio lib and it works decently, but having a standard would be much, much better.
Re: How to Make Python Wait
#19Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…
This example still involves a lot of manual work. It's often times even easier. from concurrent.futures import ProcessPoolExecutor import string def hello() -> int: seconds = random.randint(0, 5) print(f'Hi {seconds}s') time.sleep(seconds) print(f'Bye {seconds}s') return seconds # Don't forget this for processes, or you'll get in trouble if __name__ == "__main__": inputs = list(string.printable) results = [] # You ca…
"with" is always a good idea indeed.
But be careful, map() and submit() + as_completed() don't have the same effect.
The first one will give you the result in the insertion order, while the later give you the results in the order they are completed.
Re: How to Make Python Wait
#20The idea of waiting with a progress indicator has a large bug, on an 8-bit or 16-bit machine you cannot read or write atomically from a progress variable. I guess the code works because of the interpreter lock that cripples python but it's very bad hygeine in all languages.