Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to Make Python Wait

blog.miguelgrinberg.com

Post reply on HN
Print view View original

21–30 of 33 posts

Re: How to Make Python Wait

#21

Post by pyjonista »

I often find myself overwhelmed thinking about concurrency/parallelism in Python: time.sleep, concurrent.futures, multiprocessing, threads, queues, asyncio, uvloop, tornado, twisted, events, coroutines, curio, trio, select, gevent, eventlet...

What is a good solution that the majority of us simple humans with limited time and resources should pursue? I want to write software that competes with similar solutions written in Go. Does that exist? Is there one solution that it's safe to adopt and generally recommended by the community? I used to think that asyncio was the solution to that but I was unable to fully understand its API. Please help!

Re: How to Make Python Wait

#22

Post by vxNsr »

BiteCode_dev wrote: post #8

Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…

Quintessential HN, top comment totally disapproves the posted article. Thanks. I was just having an issue with this and was excited to see some gains with the link here and then to see there's an even better way is terrific!

If you wouldn't mind going into a little more detail about what you're doing I'd really appreciate it!

Re: How to Make Python Wait

#23

Post by [deleted] »

ZeroCool2u wrote: post #17
BiteCode_dev wrote: post #8

Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…

This example still involves a lot of manual work. It's often times even easier. from concurrent.futures import ProcessPoolExecutor import string def hello() -> int: seconds = random.randint(0, 5) print(f'Hi {seconds}s') time.sleep(seconds) print(f'Bye {seconds}s') return seconds # Don't forget this for processes, or you'll get in trouble if __name__ == "__main__": inputs = list(string.printable) results = [] # You ca…

[deleted]

Re: How to Make Python Wait

#24

Post by beering »

williamDafoe wrote: post #20

The idea of waiting with a progress indicator has a large bug, on an 8-bit or 16-bit machine you cannot read or write atomically from a progress variable. I guess the code works because of the interpreter lock that cripples python but it's very bad hygeine in all languages.

Can you elaborate 1) why you can't r/w atomatically from a progress variable (even if it's a byte?), 2) why are we writing Python code for 8-bit or 16-bit machines, and 3) why is it bad to take advantage of our language's guarantees on execution, e.g. with Python and JavaScript?

Re: How to Make Python Wait

#25

Post by hangonhn »

amelius wrote: post #15

Or just create a Queue(), give it as a parameter to the long calculation so it can store the result in it, and in the main thread just do q.get(). There are probably a dozen other synchronization primitives you can use, but this one is very versatile and you only need to keep one API in your head. Also, this approach somewhat mimics the concept of channels in Go.

This is among the best concurrent programming advices I've ever been given and have given out. Use a queue and a pool of workers. It makes your synchronization so much easier and simpler.

Re: How to Make Python Wait

#26

Post by BiteCode_dev »

vxNsr wrote: post #22
BiteCode_dev wrote: post #8

Don't follow this blog post advice. Manually dealing with threads and processes is useful if you want to build a framework or a very complex workflow. But chances are you just want to run stuff concurrently, in the background. In that case (which is most people case), you really want to use one of the stdlib pools: it takes care of sync, serialization, communication with queues, worker life cycle, task distribution,…

Quintessential HN, top comment totally disapproves the posted article. Thanks. I was just having an issue with this and was excited to see some gains with the link here and then to see there's an even better way is terrific! If you wouldn't mind going into a little more detail about what you're doing I'd really appreciate it!

> If you wouldn't mind going into a little more detail about what you're doing I'd really appreciate it!

What do you want to know ?

Re: How to Make Python Wait

#27

Post by saagarjha »

beering wrote: post #24
williamDafoe wrote: post #20

The idea of waiting with a progress indicator has a large bug, on an 8-bit or 16-bit machine you cannot read or write atomically from a progress variable. I guess the code works because of the interpreter lock that cripples python but it's very bad hygeine in all languages.

Can you elaborate 1) why you can't r/w atomatically from a progress variable (even if it's a byte?), 2) why are we writing Python code for 8-bit or 16-bit machines, and 3) why is it bad to take advantage of our language's guarantees on execution, e.g. with Python and JavaScript?

Not every processor supports atomic memory accesses.

Re: How to Make Python Wait

#30

Post by rickycook »

pyjonista wrote: post #21

I often find myself overwhelmed thinking about concurrency/parallelism in Python: time.sleep, concurrent.futures, multiprocessing, threads, queues, asyncio, uvloop, tornado, twisted, events, coroutines, curio, trio, select, gevent, eventlet... What is a good solution that the majority of us simple humans with limited time and resources should pursue? I want to write software that competes with similar solutions writt…

i’d say that multiprocessing probably covers 90% of “do it in the background” tasks, and asyncio covers 90% of async networking tasks

futures, uvloop, tornado, twisted, coros, gevent, etc are kinda just all related to, or do similar to asyncio

threading is kinda not as useful as you might like in python because of the GIL (simplistically, assume that python can only do 1 thing at once regardless of having multiple threads available or not until you know why that’s not always the case)

Post reply on HN