Neat Parallel Output in Python
bernsteinbear.com
Neat Parallel Output in Python
1–10 of 47 posts
Re: Neat Parallel Output in Python
#2This is indeed neat, but I would be surprised if there wasn't already a library for such multi-line functionality.
Re: Neat Parallel Output in Python
#3I like the self-built approach especially for the learning value.
If you’re using this in a CLI tool you’re writing in Python you might be using the library rich anyway, which provides this functionality as well including some extra features.
Re: Neat Parallel Output in Python
#4cool call me when they got one for go lol
Re: Neat Parallel Output in Python
#5Having the workers acquire the lock and update the terminal themselves seems like it would cause lock contention.
An alternative would be to have only the main process do the updating and have the workers message it about progress, using a queue.
Re: Neat Parallel Output in Python
#6The provided code doesn't seem self-contained, as if it would run just like that without modification. For example, where does 'num_lines' come from?
Edit: never mind. Ignore that. There is a link on the page which I overlooked, with a more complete example. https://gist.githubusercontent.com/tekknolagi/4bee494a6e4483...
Cool stuff. Now I'm eager to find a way to make this work for multiple tqdm progress bars, running in parallel.
Re: Neat Parallel Output in Python
#7I like the self-built approach especially for the learning value. If you’re using this in a CLI tool you’re writing in Python you might be using the library rich anyway, which provides this functionality as well including some extra features. https://rich.readthedocs.io/en/stable/progress.html
What I don't like about rich is that, dependencies and all, its installed size comes out to around 20 MB. 9 MB of that is due to its dependency on pygments for syntax highlighting, which a lot of people probably don't even want/need.
If anyone knows of a smaller, more focused library providing something similar to rich's Live Display functionality, I'd appreciate it.
Re: Neat Parallel Output in Python
#8I like the self-built approach especially for the learning value. If you’re using this in a CLI tool you’re writing in Python you might be using the library rich anyway, which provides this functionality as well including some extra features. https://rich.readthedocs.io/en/stable/progress.html
What I don't like about rich is that, dependencies and all, its installed size comes out to around 20 MB. 9 MB of that is due to its dependency on pygments for syntax highlighting, which a lot of people probably don't even want/need. If anyone knows of a smaller, more focused library providing something similar to rich's Live Display functionality, I'd appreciate it.
There is an open issue [1] on GitHub to make it more modular and get rid of markdown and syntax highlighting but I have no hope for rich to get more minimal.
Re: Neat Parallel Output in Python
#9I like the self-built approach especially for the learning value. If you’re using this in a CLI tool you’re writing in Python you might be using the library rich anyway, which provides this functionality as well including some extra features. https://rich.readthedocs.io/en/stable/progress.html
What I don't like about rich is that, dependencies and all, its installed size comes out to around 20 MB. 9 MB of that is due to its dependency on pygments for syntax highlighting, which a lot of people probably don't even want/need. If anyone knows of a smaller, more focused library providing something similar to rich's Live Display functionality, I'd appreciate it.
It’s a bit obvious, but you could contribute to make pygments an optional dependency, and then not install that. Or you fork it outright.
Re: Neat Parallel Output in Python
#10You could also just send output to separate files by re-opening stdout/stderr and use https://www.vanheusden.com/multitail/index.html (or GNU screen or tmux or whatever) to multiplex a terminal in a more organized way. This also solves an "open problem" in the article of stray prints.
If you really want to share one terminal / stdout but also prevent timing-based record splitting, you could also send outputs to FIFOs / named pipes and then have a simple record boundary honoring merge program like https://github.com/c-blake/bu/blob/main/funnel.nim with doc at https://github.com/c-blake/bu/blob/main/doc/funnel.md As long as "record format" is shared (e.g. newline-terminated) this can also solve the stray print problem.