I am trying to set up rsync to synchronize my main web server to the remote server by adding newly generated file to the latter.
Here is the command that I use:
rsync -avh --update -e "ssh -i /path/to/thishost-rsync-key" <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="6715020a081302121402152715020a0813020f081413">[email protected]</a>:/foo/bar /foo/bar
But it seems that the web server actually transfers all files despite the ‘–update’ flag. I have tried different flag combinations (e.g. omitting ‘-a’ and using’-uv’ instead) but none helped. How can I modify the rsync command to send out only newly added files?
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
From man rsync:
--ignore-existing skip updating files that exist on receiver
--update does something slightly different, which is probably why you are getting unexpected results (see man rsync):
This forces rsync to skip any files which exist on the destination and have a modified time that is newer than the source file. (If an existing destination file has a modification time equal to the source file’s, it will be updated if the sizes are different.)
Method 2
In my case I had similar issues, having all files transferred instead of only the modified/new ones. I solved this by using parameters -t (instead of -a), and -P (equivalent to --partial --progress):
rsync -h -v -r -P -t source target
This transfers only new files, and files already existing but modified: -a does too much, like user and group ID sync, which in my case can not work as I have different users and groups on my source and target systems.
The parameters in detail:
-h: human readable numbers-v: verbose-r: recurse into directories-P:--partial(keep partially transferred files) +
--progress(show progress during transfer)-t: preserve modification times
Method 3
From my experience with rsync, a 1TB partition copying is too large to be efficient. It takes rsync forever to process it. Instead, do it by subdirectories. That is, run rsync for each main subdirectory. It goes a lot faster if it doesn’t have to juggle tens of thousands of files.
Method 4
To copy only new files you can use following command:
rsync -v -a --ignore-existing /backup/ /site/
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0