Live data from Hacker News

Pipe Viewer

ivarch.com

41–50 of 51 posts

Re: Pipe Viewer

#41

pv is a great tool. One of it's lesser known features is throttling; transfer a file without dominating your bandwidth: pv -L 200K bigfile.iso' Complete with a progress bar, speed, and ETA.

Similarly, though useful less often these days, using -B/--buffer-size to increase the amount that it can buffer. If reading data from traditional hard drives, piping that data through some process, and writing the result back to the same drives, this option can increase throughput significantly by reducing head movements. It can help on other storage systems too, but usually not so much so.

Re: Pipe Viewer

#43
Funny timing as I recently, as of yesterday, found out about pv when searching for a way to view the progress of a zfs send and receive

Another utility I found out about is “progress” available at least on debian systems. It can monitor stuff like cp and mv without actually being used in the command

Re: Pipe Viewer

#44

It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?

It's open source, be the change you want to see in the world

Unfortunately not everyone is a developer.

Re: Pipe Viewer

#47

It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?

I have written a (free OSS) Linux tool called `psn` for this kind of stuff [1]. It samples each interesting thread's state from /proc/PID/task/TID/status,wchan,syscall etc and shows you a summary of which threads were blocked in which state, in which syscalls and where in the kernel were they stuck (wchan). It can be used with applications like pv, tar, dd, mysqld, httpd, etc.

[1] https://0x.tools/#linux-process-snapper

Re: Pipe Viewer

#49

It would be nice to indicate if the upstream or the downstream is the 'limiting' factor in speed. Ie. within pv, is it the reading the input stream or the writing the output stream that is blocking most of the time?

It's open source, be the change you want to see in the world

[deleted]

Re: Pipe Viewer

#50
post #10
post #7

Earlier quoted context omitted.

How `pv -d` work ? Does it use perf probes or attach to the target PID ?

It finds the file using /proc/ /fd/ and watches its size grow. It doesn't work with pipes, devices, a file being overwritten (not appended to), or anything whose size doesn't grow.

It does, even for reading (cat /dev/nvme0n1 > /dev/null). As per changelog:

1.5.2 - 10 February 2014 allow --watchfd to look at block devices

You can see the position in /proc//fdinfo/

Post reply on HN