I need to copy a very large directory (talking in terabytes here) and want to monitor the progress.
I found that pv is a nice utility, but how can I use it for copying directories recursively? (pv src > dst doesn’t work because they are directories)
PS: Running on OS X Mountain Lion, pv was installed from Mac Ports
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
Use rsync --progress [SRC] [DST]
Do check the man rsync page because it has a lot of very useful options. -a for archive is a good start, but it depends on your exact requirements.
Copying through a pipe will unnecessarily slow down the copy process, especially if it is files based.
Method 2
You could use tar or pax or cpio:
mkdir -p dst && (cd src && tar cf - .) | pv -trb | (cd dst && tar xpf -)
Method 3
Tar.
tar -cf - /var/log/ | pv | tar -C . -x
Example:
# tar -cf - /var/log/ | pv | tar -C . -x tar: Removing leading `/' from member names 58MB 0:00:05 [ 2.2MB/s] [ <=>
Method 4
Here are some commands to copy directories with progress information.
If there are many small files:
cp -av sourcedir targetdir | pv -l -s filecount > logfile
This will report progress based on number of files that are copied.
You can redirect to /dev/null if you don’t need logfile.
Use the following command to get filecount:
find sourcedir | wc -l
If there are few huge files:
tar c sourcedir | pv -s size | tar x -C targetdir
This will report progress based on bytes that are copied.
targetdir must exist.
Use the following command to get size:
du -sh sourcedir
If you want to use rsync:
rsync -ai sourcedir/ targetdir/ | pv -l -s filecount > logfile
Get filecount as shown above.
If you are copying on the same system then rsync -a is practically the same as cp -a . The advantages of rsync is when you are copying over the network or if you are updating (or comparing) a previous copy.
See here for more details:
Method 5
I made a progress bar for rsync (in a wrapper):
rsyncy -a FROM/ TO
It looks like this:
More info on GitHub, install with pip3 install --user rsyncy
Method 6
You could do a du -b /directory/ on both source and destination while a normal copy command is in progress, and compare the two. This is just as effective and doesn’t slow the copy process by pushing it through a pipe.
Method 7
find source-dir -print0 | xargs -0 -I % pv % > dest-dir/%
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0