I prefer using a purpose-built tool like GNU parallel. Parallel's only purpose is to run things in parallel and collect the results together. The advantage is you only have to learn to use it once, rather than learn to do this again and again in all the different languages/tools you might use.
Neat Parallel Output in Python
11–20 of 47 posts
Re: Neat Parallel Output in Python
#12interesting footer link https://notbyai.fyi/
Re: Neat Parallel Output in Python
#13clever! will have to see if this works with tqdm progress bars, has anyone tried that?
Re: Neat Parallel Output in Python
#14I've got the impression that in practice, what keeps developers from letting trivially parallelizable tasks run in parallel is a) the overhead of dealing with poor parallelization primitives and b) the difficulty in properly showing the status of parallel invocations.
Having good features to support this in standard libraries would go a long way to incentivizing devs to actually parallelize.
Re: Neat Parallel Output in Python
#15Cool! Out of interest, I tried to do this in Java :)
https://github.com/cloverthe/java-parallel-output/blob/maste...
here is video: https://streamable.com/yxm05v
Re: Neat Parallel Output in Python
#16You probably don't want to run this gist directly. Looks like a risk of a runaway process creation, depending on the platform. The process creation code should be guarded by `if __name__ == "__main__"`.
Re: Neat Parallel Output in Python
#17clever! will have to see if this works with tqdm progress bars, has anyone tried that?
tqdm has a position parameter which allows offsetting concurrent progress bars. It should work automatically for intra-process concurrency anyway. I don’t know if it works correctly with multiple processes though.
Re: Neat Parallel Output in Python
#18I prefer using a purpose-built tool like GNU parallel. Parallel's only purpose is to run things in parallel and collect the results together. The advantage is you only have to learn to use it once, rather than learn to do this again and again in all the different languages/tools you might use.
I can’t help but notice you have not explained how to handle progress reporting in gnu parallel.
Also do you really use parallel from software trying to parallelise its internal workload? Note that in tfa the workers are an implementation detail of a wider program.
Re: Neat Parallel Output in Python
#19clever! will have to see if this works with tqdm progress bars, has anyone tried that?
tqdm has a position parameter which allows offsetting concurrent progress bars. It should work automatically for intra-process concurrency anyway. I don’t know if it works correctly with multiple processes though.
yeah my code needs to use multiprocessing, which does not play nice with tqdm. thanks for the tip about positions though, that helped me search more effectively and came up with two promising comments. unmerged / require some workarounds, but might just work:
https://github.com/tqdm/tqdm/issues/1000#issuecomment-184208... https://github.com/tqdm/tqdm/issues/811#issuecomment-1368850...
Re: Neat Parallel Output in Python
#20One could use the dialog command, for which there is a "dialog" python module to show a list of progress bars.
https://pythondialog.sourceforge.io/images/screenshots/mixed...
