Random and Useless Thoughts on Everything
Linux commands I never knew existed: Pipe Viewer
I have been using linux since about 1995, and it still surprises me how creative the people who contribute code and work on projects are. The other day I had to image a bunch of machines, they all needed to be 100% identical, so I figured I would just use “dd” and send to output over to each slave machine using netcat. For anyone who has ever done this before, it can be a bit hard to figure out how fast data is being sent over to the system being imaged. This is where Pipe Viewer came in handy. Quite simply this app reports the bitrate that the data is sent over a unix pipe. Brilliant.
It is not installed by default in debian, so the following will install the app.
user@slashzero-desktop:~$ sudo aptitude install pv
Once installed, passing the “–help” parameter will list the basic options.
user@slashzero-desktop:~$ pv --help
Usage: pv [OPTION] [FILE]...
Concatenate FILE(s), or standard input, to standard output,
with monitoring.
-p, --progress show progress bar
-t, --timer show elapsed time
-e, --eta show estimated time of arrival (completion)
-r, --rate show data transfer rate counter
-b, --bytes show number of bytes transferred
-f, --force output even if standard error is not a terminal
-n, --numeric output percentages, not visual information
-q, --quiet do not output any transfer information at all
-c, --cursor use cursor positioning escape sequences
-W, --wait display nothing until first byte transferred
-s, --size SIZE set estimated data size to SIZE bytes
-l, --line-mode count lines instead of bytes
-i, --interval SEC update every SEC seconds
-w, --width WIDTH assume terminal is WIDTH characters wide
-H, --height HEIGHT assume terminal is HEIGHT rows high
-N, --name NAME prefix visual information with NAME
-L, --rate-limit RATE limit transfer to RATE bytes per second
-B, --buffer-size BYTES use a buffer size of BYTES
-R, --remote PID update settings of process PID
-h, --help show this help and exit
-V, --version show version information and exit
Please report any bugs to Andrew Wood .
And here is a quick example, this command tar’s up a directory and send the output to gzip, showing a rate of 223MB/s:
user@slashzero-desktop:/home$ tar zcf - user | pv /bin/gzip > /tmp/backup.tar.gz
59.7kB 0:00:00 [ 223MB/s] [===========================================>] 100%
Incredibly useful stuff.
Thanks – that’s pretty useful. I’ve never seen or heard of that one.
I don’t get the purpose of piping your tar through gzip. Your tar command already gzips the archive. I realize that it is for the purpose of demonstration, but tar -zcf user /tmp/backup.tar.gz performs the same action, minus the pv stuff.
Frank: The purpose of piping your tar to gzip in this case is so that you can watch the progress with pv.
You’d probably get better performance if you weren’t gzipping it twice. tar cf – would suffice.
Thanks! I’d never heard of it, and could use something like that!
While pv certainly has uses, sending a USR1 signal to dd will cause it to dump a progress report including rate like so:
18335302+0 records in 18335302+0 records out 9387674624 bytes (9.4 GB) copied, 34.6279 seconds, 271 MB/s
That was my point Subie. He’s GZiping the file twice.
Not sure why, but it seems I need to use “tar cf – user |pv |/bin/gzip > backup.tar.gz” to get it work on Ubuntu 8.04 64-bit.
It’s a wrong syntax:
$ cat toto | cat tata | gzip > titi.gz
give the same result than
$ gzip titi.gz
“cat toto” will simply be ignored.
for doing what you expect, you need to do something like:
$ cd /home
$ size=`du -bs user | sed s/[[:space:]].*$//g`
$ tar zcf – user | pv -s $size | gzip > /tmp/backup.tar.gz
366MB 0:00:33 [ 11MB/s] [=================================>] 100%
And you will maybe get a “not so high” rate…
It’s a wrong syntax:
$ cat toto | cat tata | gzip > titi.gz
give the same result than
$ gzip < tata > titi.gz
“cat toto” will simply be ignored.
for doing what you expect, you need to do something like:
$ cd /home
$ size=`du -bs user | sed s/[[:space:]].*$//g`
$ tar zcf – user | pv -s $size | gzip > /tmp/backup.tar.gz
366MB 0:00:33 [ 11MB/s] [=================================>] 100%
And you will maybe get a “not so high” rate…
It’s a wrong syntax:
$ cat toto | cat tata | gzip > titi.gz
give the same result than
$ gzip < tata > titi.gz
“cat toto” will simply be ignored.
for doing what you expect, you need to do something like:
$ cd /home
$ size=`du -bs user | sed s/[[:space:]].*$//g`
$ tar cpf – user | pv -s $size | gzip > /tmp/backup.tar.gz
366MB 0:00:33 [ 11MB/s] [=================================>] 100%
And you will maybe get a “not so high” rate…
Yeah, I just used an example command and didn’t put a lot of thought into it, frankly I didn’t really expect anyone would ever see it, I will try and put more effort into the next entry.
Thanks!
single watch winder…
[...]Linux commands I never knew existed: Pipe Viewer | Slashzerø[...]…
Hi,
I would like to copy one file from my computer to another. Both are linux and I need the log file showing the time taken to copy the file. Please help me asap.
Thanks,