Real-time file synchronization

Is there any tool available to sync files between two or more linux servers immediately after writing the file in the disk? The rsync command does not suit me on this, because if I set rsync in cron, the minimum time I can set is 1 minute, but I need it on real-time basis.

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

Haven’t used it myself but read about it recently. There is a daemon called lsyncd, which I presume does exactly what you need.

Read more about it HERE

Method 2

Realtime File synchronisation between multiple servers in multi master mode

There is a good tool called lsyncd to sync files between multiple servers on real time basis.
Here I have tried with two servers.

Hosts: Server1 and Server2

OS Used: CentOS 7

Install below packages on both the servers.

# yum install -y epel-release
# yum -y install lua lua-devel pkgconfig gcc asciidoc lsyncd

generate ssh-key in both the servers and add to authorized_keys file. [add public key of server1 to server2’s authorized_keys and public key of server2 to server1’s authorized_keys file]

Server1 configuration

Open /etc/lsyncd.conf and comment out the default configuration using -- at the beginning of the line and add below configuration to the file.

settings {
  logfile = "/var/log/lsyncd/lsyncd.log",
  delay = 1
}
sync {
  default.rsync,
  source="/home/test/public_html/",
  target="server2:/home/test/public_html/",
  rsync = {
    compress = true,
    acls = true,
    verbose = true,
    owner = true,
    group = true,
    perms = true,
    rsh = "/usr/bin/ssh -p 22 -o StrictHostKeyChecking=no"
  }
}

Change the target IP in target parameter.

You can change the delay parameter as per your requirement. Here it is set 1 second.

Now create the log directory.

# mkdir -p /var/log/lsyncd

Enable the lsyncd service to start automatically.

# systemctl enable lsyncd.service

Start the service.

# systemctl start lsyncd.service

Server2 Configuration

Follow the same configuration as Server1 and change the target IP.

Now the synchronisation is set.

You can check the activity from tailf /var/log/lsyncd/lsyncd.log

Thanks to MelBurslan for his suggestion.

Method 3

Syncthing might be an option. It’s extremely fast, transfer is encrypted and there are clients for multiple platforms. It uses “inotify” to sync changed files instantly.

Method 4

Inotify-tools

Provide an interface to inotify, consisting of:

inotifywait

This command simply blocks for inotify events, making it appropriate for use in shell scripts. It can watch any set of files and directories, and can recursively watch entire directory trees.

inotifywatch

This command collects filesystem usage statistics and outputs counts of each inotify event.

Method 5

You would need to approach this with a clustered filesystem type solution – a simple sync between the two machines won’t give you real-time response.

Method 6

“SIOS Protection Suite for Linux” can offer this, on AWS.

http://www.linuxclustering.net/2016/03/21/step-by-step-how-to-configure-a-linux-failover-cluster-in-amazon-ec2-without-shared-storage-amazon-aws-sanless-cluster/#InstallConfigureCluster

Method 7

Here is my working config, for this example i used /tmp/test both as a local and a distant directory.

I had an error that said ssh-askpass wasn’t installed:

sudo apt install ssh-askpass

I also needed to be using a non standard port for ssh.

settings {
    logfile = "/var/log/lsyncd/lsyncd.log",
    statusFile = "/var/log/lsyncd/lsyncd-status.log",
    statusInterval = 1
}

sync {
    default.rsyncssh,
    delete = false,
    source = "/tmp/test",
    rsync = {
    binary = "/usr/bin/rsync",
    archive = true,
    perms = true,
    owner = true,
    _extra = {"-a"},
    update = true,
    protect_args = true,
    sparse = true,
    rsh = "/usr/bin/ssh -l src_username -i /home/src_username/.ssh/id_rsa",
    compress = true
},
ssh = {
    port = 6765,
    extra = {"-i","/home/src_username/.ssh/id_rsa"}
    },
host = "<a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="46222f3532272832193335233428272b23062e29353228272b23683e3e3e">[email protected]</a>",
targetdir = "/tmp/test"
}

And after 4 hours of wandering:

Wed Feb  2 04:22:52 2022 Normal: --- Startup, daemonizing ---
Wed Feb  2 04:22:52 2022 Normal: recursive startup rsync: /tmp/test/ -> <a href="https://getridbug.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="36525f4542575842694345534458575b53765e59454258575b53184e4e4e">[email protected]</a>:/tmp/test/
Warning: Permanently added '[domain.xyz]:6765,[123.123.123.123]:6765' (ECDSA) to the list of known hosts.
Wed Feb  2 04:23:06 2022 Normal: Startup of "/tmp/test/" finished: 0
Wed Feb  2 04:24:13 2022 Normal: Calling rsync with filter-list of new/modified files/dirs 

sent 61 bytes  received 12 bytes  146.00 bytes/sec
total size is 57  speedup is 0.78
Wed Feb  2 04:24:13 2022 Normal: Finished a list after exitcode: 0

Method 8

I tried lsyncd but had multiple issues, the daemon wouldnt sync files after a certain time, would only sync on a service restart.

I wrote a sync daemon called Poni – written in Crystal, very easy to configure Yaml config, listens on inotify and rsyncs, can specify additional options, see readme

https://github.com/perfecto25/poni

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0