Speed up copying 1000000 small files

I have 1000000 4-20 kb files in a dir (You can generate similar files like this: seq 10000 | gzip > a; seq 1000000 | parallel --bar 'head -c{=$_=int(rand()*16)+4=}k a > {}')

. I need to copy that dir. But it seems I have to do a seek for each files so this takes quite a while.

Is there a way in which I can speed this up?

I am currently thinking that if I could get the disk blocks that these files occupy, I could sort those, merge the blocks that were close (given that sequential read is often faster than seeking) and read these blocks, so that they were in RAM cache (I have 32 GB RAM) before doing the copy.

But for that to work I need a way to identify which blocks the files are on.

I am using EXT4 on a magnetic device (i.e. not SSD).

Edit:

This ought to work but it does not:

ls |
parallel -IOO --pipe "sudo parallel -j100 hdparm --fibmap {}'|tail -n +5'" |
sort -nk 2 | 
perl -ane 'if($u+10000 < $F[1]) { print "$l ",($u-$l),"n"; $l=$F[1] } $u=$F[2]' |
sudo parallel --colsep ' ' dd if=/dev/sda1 skip={1} bs=512 count={2} '| cat >/dev/null'

When testing it on a big file it does not cache the file.

Edit2:

Here are some benchmarks. Cache was flushed (echo 3 >/proc/sys/vm/drop_caches) between each run. Measurements done with iostats -dkx 5.

rsync -Hav foo/ bar/: 1800 KB/s
cp -a foo/ bar/: 3600 KB/s
cat sort-by-inode | parallel -j1 -X cp foo/{} bar/: 5000 KB/s
cat sort-by-inode | shuf | parallel -j1 -X cp foo/{} bar/: 3000 KB/s
cat sort-by-inode | shuf | parallel -j10 -X cp foo/{} bar/: 7000 KB/s
cat sort-by-inode | parallel -j10 -X cp foo/{} bar/: 8000 KB/s
cat sort-by-inode | parallel -j100 -X cp foo/{} bar/: 9000 KB/s
cat sort-by-inode | parallel -j500 -X cp foo/{} bar/: 10000 KB/s

So what can we learn from that?

It seems sorting by inode is a good idea. But it seems parallelizing multiple cp boosts performance even further. It is worth stressing that the source foo/ is a magnetic disk, so this attacks the myth that parallelizing I/O to a single spindle will not speed up I/O: Parallelizing clearly and consistently speeds up the copying here.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Assuming that

• entries returned by readdir are not sorted by inode number
• reading files in inode order reduces the number of seek operations
• the content of most files are in the initial 8k allocation (an ext4 optimization) which also should yield less seek operations

you can try to speed up copying via copying files in inode order.

That means using something like this:

$ cd /mnt/src
$ ls -U -i | sort -k1,1 -n | cut -d' ' -f2- > ~/clist
$ xargs cp -t /mnt2/dst < ~/clist

Method 2

GNU tar – in the pax tradition – handles hardlinks on its own.

cd "$srcdir" ; tar --hard-dereference -cf - ./* |
    tar -C"${tgtdir}" -vxf -

That way you only have the two tar processes and you don’t need to keep invoking cp over and over again.

Method 3

On a similar vein to @maxschlepzig’s answer, you can parse the output of filefrag to sort files in the order that their first fragments appear on disk:

find . -maxdepth 1 -type f |
  xargs -d'n' filefrag -v |
  sed -n '
    /^   0:        0../ {
      s/^.{28}([0-9][0-9]*).*/1/
      h
      }
    / found$/ {
      s/:[^:]*$//
      H
      g
      s/n/ /p
      }' |
    sort -nk 1,1 |
    cut -d' ' -f 2- |
    cpio -p dest_dir

MMV with the above sed script, so be sure to test thoroughly.

Otherwise, whatever you do, filefrag (part of e2fsprogs) will be much faster to use than hdparm as it can take multiple file arguments. Just the overhead of running hdparm 1,000,000 times is going to add a lot of overhead.

Also it probably wouldn’t be so difficult to write a perl script (or C program), to a FIEMAP ioctl for each file, create a sorted array of the blocks that should be copied and the files the belong to and then to copy everything in order by reading the size of each block from the corresponding file (be careful not to run out of files descriptors though).

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0