Pipe Viewer

#1

Post by 0x45696e6172 » Tue, Oct 18, 2022, 9:15 AM UTC

Re: Pipe Viewer

#2

Post by trabant00 » Wed, Oct 19, 2022, 2:46 PM UTC

I've used it mostly to measure events per second with something like:

  tail -f /some/log | grep something | pv -lr > /dev/null
  or
  tcpdump expression | pv -lr > /dev/null

Re: Pipe Viewer

#3

Post by JayGuerette » Wed, Oct 19, 2022, 2:46 PM UTC

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth:

pv -L 200K bigfile.iso'

Complete with a progress bar, speed, and ETA.

Re: Pipe Viewer

#4

Post by est » Wed, Oct 19, 2022, 2:50 PM UTC

pv was the tool when I discovered sometimes the VPS have only 10Gbps memory copy speed.

Re: Pipe Viewer

#5

Post by smcl » Wed, Oct 19, 2022, 2:50 PM UTC

JayGuerette wrote: post #3

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K bigfile.iso' Complete with a progress bar, speed, and ETA.

Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`

Re: Pipe Viewer

#6

Post by cwillu » Wed, Oct 19, 2022, 2:51 PM UTC

pv -d $(pidof xz):1 is great for when you realize too late that something is slow enough that you want a progress indication, and definitely do not want to restart from scratch.

Re: Pipe Viewer

#7

Post by xuhu » Wed, Oct 19, 2022, 3:01 PM UTC

cwillu wrote: post #6

pv -d $(pidof xz):1 is great for when you realize too late that something is slow enough that you want a progress indication, and definitely do not want to restart from scratch.

How `pv -d` work ? Does it use perf probes or attach to the target PID ?

Re: Pipe Viewer

#8

Post by fbergen » Wed, Oct 19, 2022, 3:03 PM UTC

smcl wrote: post #5

JayGuerette wrote: post #3

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K bigfile.iso' Complete with a progress bar, speed, and ETA.

Oh damn that's neat I never thought to use `ssh` directly when transferring a file, I always used `scp bigfile.iso name@server.org:path/in/destination`

Also see `scp -l 200 bigfile.iso name@server.org:path/in/destination`

from man page:

-l limit

Limits the used bandwidth, specified in Kbit/s.

Re: Pipe Viewer

#9

Post by londons_explore » Wed, Oct 19, 2022, 3:11 PM UTC

It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed.

Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?

Re: Pipe Viewer

#10

Post by remram » Wed, Oct 19, 2022, 3:17 PM UTC

xuhu wrote: post #7

cwillu wrote: post #6

pv -d $(pidof xz):1 is great for when you realize too late that something is slow enough that you want a progress indication, and definitely do not want to restart from scratch.

How `pv -d` work ? Does it use perf probes or attach to the target PID ?

It finds the file using /proc//fd/ and watches its size grow. It doesn't work with pipes, devices, a file being overwritten (not appended to), or anything whose size doesn't grow.