rsync remote synchronization

1, Introduction to rsync synchronization

It is an open source fast backup tool, which can mirror and synchronize the entire directory tree between different hosts, support incremental backup, maintain links and permissions, and adopt optimized synchronization algorithm to perform compression before transmission. Therefore, it is very suitable for remote backup, mirror server and other applications.

2, rsync real-time synchronization

1. Lack of periodic synchronization

  • The backup time is fixed, with obvious delay and poor real-time performance
  • When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary

2. Advantages of real-time synchronization

  • Once the original synchronization changes, start the backup immediately
  • As long as there is no change in the original synchronization, the backup will not be performed

3, About inotify

1. inotify mechanism of Linux kernel

  • Available from version 2.6.13
  • You can monitor changes in the file system and respond to notifications
  • Auxiliary software: inotify tools

4, rsync+inotify real-time synchronization

1. Adjust inotify kernel parameters

  • max_queue_events: monitors the size of the event queue
  • max_user_instances: the maximum number of monitored instances
  • max_user_watches: maximum number of monitored files per instance

2. Installing inotify tools AIDS

  • inotifywait: used for continuous monitoring and real-time output of results
  • inotifywatch: used for short-term monitoring, and the results will be released after the task is completed

3. Trigger rsync synchronization through inodifywait

  • Use while and read to continuously obtain monitoring results
  • Based on the results, you can make further judgment and decide what operation to perform

5, Experimental steps

1. Configure rsync source server

1) Turn off the firewall and determine whether rsync is installed

systemctl stop firewalld
setenforce 0

rpm -q rsync

2) Add / etc / Rsync Conf configuration file

vim /etc/rsyncd.conf				#Add the following configuration items
uid = nobody
gid = nobody
use chroot = yes										#Locked in the source directory
address =									#Listening address
port 873												#The listening port tcp/udp 873 can be viewed through cat /etc/services | grep rsync
log file = /var/log/rsyncd.log							#Log file location
pid file = /var/run/							#File location where the process ID is stored
hosts allow =							#Allowed client address
[hzh]												#Shared module name
path = /var/www/html									#The actual path of the source directory
comment = Document Root of
read only = yes											#Is it read-only
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z		#File types that are no longer compressed during synchronization
auth users = hzh									#Authorized accounts. Multiple accounts are separated by spaces
secrets file = /etc/rsyncd_users.db						#Data file for storing account information

###If anonymity is adopted, just remove the configuration items of "auth users" and "secrets file".

3) Create data files for backup accounts

vim /etc/rsyncd_users.db
hzh:abc123					#There is no need to establish a system user with the same name

chmod 600 /etc/rsyncd_users.db

4) Ensure that all users have access to the source directory / var/www/html

yum -y install httpd
chmod +r /var/www/html/
ls -ld /var/www/html/

5) Start rsync service program

rsync --daemon				#Start the rsync service and run it as an independent monitoring service (daemon) 

netstat -anpt | grep rsync

6) Turn off rsync service

kill $(cat /var/run/
rm -rf /var/run/

2. Initiator

Basic format: rsync [options] original location target location
1. Common options
-r: Recursive mode, including all files in the directory and subdirectory
-l: For symbolic link files, you can still copy them as symbolic link files
-v: Displays details of the synchronization process
-z: Compress when transferring files
-a: The archive mode preserves the permissions, attributes and other information of the file, which is equivalent to the combined option "- rlptgoD"
-p: Keep the permission mark of the file
-t: Time stamp of retention file
-g: Keep the group mark of the file (for super users only)
-o: Keep the ownership mark of the file (for super users only)
-H: Keep hard wired files
-A: Retain ACL attribute information
-D: Retention of special documents and other equipment
– delete: output a file that exists in the target location but not in the original location
– checksum: decide whether to skip the file based on the checksum (not the file size and modification time)
2. Synchronization method
1) Download the specified resources to the local / opt directory for backup.

	rsync -avz hzh@ /opt					#Password abc123


rsync -avz rsync://hzh@ /opt	

2) Interactive free formatting

echo "abc123" > /etc/server.pass
chmod 600 /etc/server.pass

crontab -e
30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass hzh@ /opt/


The inotify notification interface can be used to monitor various changes of the file system, such as file access, deletion, movement, modification and so on. Using this mechanism, it is very convenient to realize file change alarm and incremental backup, and respond in time to changes in directories or files.
The combination of inotify mechanism and rsync tool can realize triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; Otherwise, it is in a silent waiting state.
Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring, which is more suitable for uplink synchronization when applied in triggered backup.
1) Modify rsync source server configuration file

vim /etc/rsyncd.conf
read only = no											#Turn off read-only, and uplink synchronization can be written

kill $(cat /var/run/
rm -rf /var/run/

rsync --daemon	
netstat -anpt | grep rsync

chmod 777 /var/www/html/

2) Adjust inotify kernel parameters (executed on the client)
In the Linux kernel, the default inotify mechanism provides three control parameters: max_queue_events (monitor event queue, the default value is 16384), max_user_instances (the maximum number of monitored instances, the default value is 128), max_user_watches (the maximum number of monitoring files per instance, the default value is 8192). When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.

cat /proc/sys/fs/inotify/max_queued_events
cat /proc/sys/fs/inotify/max_user_instances
cat /proc/sys/fs/inotify/max_user_watches

vim /etc/sysctl.conf
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

sysctl -p

3) Install inotify tools (client)

Inotify tools need to be installed to use inotify mechanism to provide inotifywait and inotifywatch auxiliary tools.

notifywait: it can monitor various events such as modify, create, move, delete and attrib, and output results as soon as there is a change.

inotifywatch: it can be used to collect file system changes and output the summarized changes after the operation.

yum -y install gcc gcc-c++ make

tar zxvf inotify-tools-3.14.tar.gz -C /opt/

cd /opt/inotify-tools-3.14
make && make install

4) Execute the "inotifywait" command, and then send it to / var/www/html (client) at another terminal
Add and move files under the directory, and track the output results on the screen.
inotifywait -mrq -e modify,create,move,delete /var/www/html

#Option "- e": used to specify which events to monitor
#Option "- m": continuous monitoring
#Option "- r": indicates recursion of the entire directory
#Option "- q": simplify output information

5) Write trigger synchronization script at another terminal

vim /opt/
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ hzh@"
##while determines whether the monitoring record is received
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then

6) Empowerment

chmod +x /opt/
chmod 777 /var/www/html/
chmod +x /etc/rc.d/rc.local
echo '/opt/' >> /etc/rc.d/rc.local				#Add automatic execution after startup

Posted by mgs019 on Thu, 14 Apr 2022 05:45:05 +0930