I’m trying to copy a batch of files with scp but it is very slow. This is an example with 10 files:
$ time scp cap_* <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="a1d4d2c4d3e1c9ced2d5">[email protected]</a>:~/dir cap_20151023T113018_704979707.png 100% 413KB 413.2KB/s 00:00 cap_20151023T113019_999990226.png 100% 413KB 412.6KB/s 00:00 cap_20151023T113020_649251955.png 100% 417KB 416.8KB/s 00:00 cap_20151023T113021_284028464.png 100% 417KB 416.8KB/s 00:00 cap_20151023T113021_927950468.png 100% 413KB 413.0KB/s 00:00 cap_20151023T113022_567641507.png 100% 413KB 413.1KB/s 00:00 cap_20151023T113023_203534753.png 100% 414KB 413.5KB/s 00:00 cap_20151023T113023_855350640.png 100% 412KB 411.7KB/s 00:00 cap_20151023T113024_496387641.png 100% 412KB 412.3KB/s 00:00 cap_20151023T113025_138012848.png 100% 414KB 413.8KB/s 00:00 cap_20151023T113025_778042791.png 100% 413KB 413.4KB/s 00:00 real 0m43.932s user 0m0.074s sys 0m0.030s
The strange thing is that the transfer rate is about 413KB/s and the file size is about 413KB so really it should transfer one file per second, however it’s taking about 4.3 seconds per file.
Any idea where this overhead comes from, and is there any way to make it faster?
Answers:
Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.
Method 1
You could use rsync (over ssh), which uses a single connection to transfer all the source files.
rsync -avP cap_* <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="2e5b5d4b5c6e46415d5a">[email protected]</a>:dir
If you don’t have rsync (and why not!?) you can use tar with ssh like this, which avoids creating a temporary file (these two alternatives are equivalent):
tar czf - cap_* | ssh <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="8afff9eff8cae2e5f9fe">[email protected]</a> tar xvzfC - dir tar cf - cap_* | gzip | ssh <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="f580869087b59d9a8681">[email protected]</a> 'cd dir && gzip -d | tar xvf -'
The rsync is to be preferred, all other things being equal, because it’s restartable in the event of an interruption.
Method 2
@wurtel’s comment is probably correct: there’s a lot of overhead establishing each connection. If you can fix that you’ll get faster transfers (and if you can’t, just use @roaima’s rsync workaround). I did an experiment transferring similar-sized files (head -c 417K /dev/urandom > foo.1 and made some copies of that file) to a host that takes a while to connect (HOST4) and one that responds very quickly (HOST1):
$ time ssh $HOST1 echo real 0m0.146s user 0m0.016s sys 0m0.008s $ time scp * $HOST1: foo.1 100% 417KB 417.0KB/s 00:00 foo.2 100% 417KB 417.0KB/s 00:00 foo.3 100% 417KB 417.0KB/s 00:00 foo.4 100% 417KB 417.0KB/s 00:00 foo.5 100% 417KB 417.0KB/s 00:00 real 0m0.337s user 0m0.032s sys 0m0.016s $ time ssh $HOST4 echo real 0m1.369s user 0m0.020s sys 0m0.016s $ time scp * $HOST4: foo.1 100% 417KB 417.0KB/s 00:00 foo.2 100% 417KB 417.0KB/s 00:00 foo.3 100% 417KB 417.0KB/s 00:00 foo.4 100% 417KB 417.0KB/s 00:00 foo.5 100% 417KB 417.0KB/s 00:00 real 0m6.489s user 0m0.052s sys 0m0.020s $
Method 3
It’s the negotiation of the transfer that takes time. In general, operations on n files of b bytes each takes much, much longer than a single operation on a single file of n * b bytes. This is also true e.g. for disk I/O.
If you look carefully you’ll see that the transfer rate in this case is size_of_the_file/secs.
To transfer files more efficiently, bundle them together with tar, then transfer the tarball:
tar cvf myarchive.tar cap_20151023T*.png
or, if you also want to compress the archive,
tar cvzf myarchive.tar.gz myfile*
Whether to compress or not depends on the file contents, eg. if they’re JPEGs or PNGs, compression won’t have any effect.
Method 4
I’ve used the technique described here (archived) which uses parallel gzip and netcat to quickly compress and copy data.
It boils down to:
# SOURCE: > tar -cf - /u02/databases/mydb/data_file-1.dbf | pigz | nc -l 8888 # TARGET: > nc <source host> 8888 | pigz -d | tar xf - -C /
This uses tar to gather up the file or files. Then uses pigz to get many cpu threads to compress and send the file, the network transmission is using netcat. On the receiving side, netcat listens then uncompresses (in parallel) and untars.
Method 5
Just had this issue doing a site-to-site transfer of a large mp4 file via scp. Was getting ~250KB/s. After disabling UDP flood protection (FP) on the destination firewall, the transfer rate increased to 6.5MB/s. When turning FP back on, the rate dropped back to ~250KB/s.
Sender: cygwin, Receiver: Fedora 20, Firewall Sophos UTM.
What does SSH use UDP for? @ superuser.com — It doesn’t directly from what I read.
In reviewing the firewall log, flood detection was occurring on both source & dest ports 4500 over the public IP addresses, not the private site-to-site internal VPN addresses. So it seems my issue is likely a NAT Traversal situation where the scp TCP data is ultimately encrypted and encapsulated in ESP & UDP packets, and consequently subject to FP. To remove scp from the equation, I ran a Windows file copy operation across the VPN and noticed similar performance to scp with and without FP enabled. Also ran an iperf test over TCP and noticed 2Mbits/sec with FP, and 55Mbits/sec without.
How Does NAT-T work with IPSec? @ cisco.com
Method 6
Another reason that scp is slower than it should be, especially on high bandwidth networks, is that it has statically defined internal flow control buffers which end up becoming network performance bottlenecks.
HPN-SSH is a patched version of OpenSSH which increases the size of these buffers. It makes a massive difference to scp transfer speed (see the charts on the site, but I also speak from personal experience). Of course, to get the benefits you need to install HPN-SSH on all your hosts but it’s well worth it if you regularly need to transfer large files around.
Method 7
Since this question is not that old and no one else referred to this solution I think it is appropriate since it pushes the bandwidth to the max limit (10MiB/s in my case) unlike scp that is at arount 250kb/s so that answers your question.
Actually the same 250kb/s with rsync – at least with the port specifier rclone -Avvp cap_* -e "ssh -p 1087 -i id_rsa" [email protected]:~/dir
Quoting a post to the openssh-unix-dev mailing list:
The scp protocol is outdated, inflexible and not readily fixed. Its authors recommend the use of more modern protocols like sftp and rsync for file transfer instead
The same syntax applies to sftp so instead of scp text.txt [email protected] it is now sftp text.txt [email protected](usage examples scp interchangable with sftp)
Also the recent version of OpenSSH should activate the daemon – at least in my case on an arch linux server but you might have to install an sftp package on other distributions.
One more working example with an ssh encryption file flag (id_rsa) and a non standard ssh port 1087 instead of 22 to safe you time fiddeling with the syntax:
sftp -P 1087 -i id_rsa <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="d1a4a2b4a391a2b4a3a7b4a3">[email protected]</a>:/home/user/Downloads/Video/*/*.mp4 /home/user/Videos/
Also your sftp might be limited to 800kb/s or ~1 Mbit/s.
You can check this with:
# sysctl -a | grep net.*rmem
and you can change the limits e.g. like this if they are too slow:
# sysctl -w net.ipv4.tcp_rmem='40960 873800 62914560' # sysctl -w net.core.rmem_max=8388608
Method 8
you can make online website in server
$ sudo apt-get apache # open port 80 in your server # copy your file in server to /var/www/html/ # if cant copy or symlinke its , use: $ sudo chwon $USER /var/www/html/
and download your files by wget or curl…
$ wget http://40.86.167.128/video.mp4
for me i got 100% speed in wifi 4mbit/s which meens my speed is 4mbit/s
and 25% for wifi 100mbit/s which meens 24mib/s not 100mbits
but rsyn or scp i got 1mbit/s for wifi 4mib/s
Method 9
you need to disable selinux, follow this link
https://linuxconfig.org/how-to-disable-enable-selinux-on-ubuntu-20-04-focal-fossa-linux
All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0