31–40 of 51 posts
It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?
It's open source, be the change you want to see in the world
PV (Pipe Viewer) – add a progress bar to most command-line programs - https://news.ycombinator.com/item?id=23826845 - July 2020 (2 comments)
A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=8761094 - Dec 2014 (1 comment)
Pipe Viewer - https://news.ycombinator.com/item?id=5942115 - June 2013 (1 comment)
Pipe Viewer - https://news.ycombinator.com/item?id=4020026 - May 2012 (26 comments)
A Unix Utility You Should Know About: Pipe Viewer - https://news.ycombinator.com/item?id=462244 - Feb 2009 (63 comments)
It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?
i've consistently lost and found this tool over and over again for over 20 years
As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2.…
IIRC pv uses splice internally and simply tells the kernel to mive pipe buffers from one pipe to the other, so it is very unlikely to be a bottleneck.
It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?
it's instantaneous, but the -T (transfer buffer % full display) is sometimes useful for that. (0% full -> source limited, 100% full -> sink limited)
Earlier quoted context omitted.
Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`
A similar trick that's nice is piping tar through ssh. Handy if you don't have rsync or something better around. Even handy for one file, since it preserves permissions, etc. tar -cf - some/dir | ssh remote 'cd /place/to/go && tar -xvf -'
As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2.…
As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2.…
1) this gets it out of the pipeline. 2) the program gets to have the named arguments. 3) pv's out put is on a separate terminal. 4) your job never needs to know.
Downside: it only sees the currently open files, so it doesn't work well for batch jobs. Still, it's handy to see which file it's on, and how fast the progress is.
Also, for rsync: "--info=progress2 --no-i-r" will show you the progress for a whole job.
As a person who runs a lot of ETL-like commands at work, I never find myself using pv(1). I love the idea of it, but for the commands I most want to measure progress of, they always seem to be either: 1. things where I'd be paranoid about pv(1) itself becoming the bottleneck in the pipeline — e.g. dd(1) of large disks where I've explicitly set a large blocksize and set conv=idirect/odirect, to optimize throughput. 2.…
For rsync to get reliable global progress there is --no-i-r --info=progress2 . --no-i-r adds a bit of upfront work, but it's well worth it IMO.