1, Introduction to rsync synchronization
It is an open source fast backup tool, which can mirror and synchronize the entire directory tree between different hosts, support incremental backup, maintain links and permissions, and adopt optimized synchronization algorithm to perform compression before transmission. Therefore, it is very suitable for remote backup, mirror server and other applications.
2, rsync real-time synchronization
1. Lack of periodic synchronization
- The backup time is fixed, with obvious delay and poor real-time performance
- When the synchronization source does not change for a long time, intensive periodic tasks are unnecessary
2. Advantages of real-time synchronization
- Once the original synchronization changes, start the backup immediately
- As long as there is no change in the original synchronization, the backup will not be performed
3, About inotify
1. inotify mechanism of Linux kernel
- Available from version 2.6.13
- You can monitor changes in the file system and respond to notifications
- Auxiliary software: inotify tools
4, rsync+inotify real-time synchronization
1. Adjust inotify kernel parameters
- max_queue_events: monitors the size of the event queue
- max_user_instances: the maximum number of monitored instances
- max_user_watches: maximum number of monitored files per instance
2. Installing inotify tools AIDS
- inotifywait: used for continuous monitoring and real-time output of results
- inotifywatch: used for short-term monitoring, and the results will be released after the task is completed
3. Trigger rsync synchronization through inodifywait
- Use while and read to continuously obtain monitoring results
- Based on the results, you can make further judgment and decide what operation to perform
5, Experimental steps
1. Configure rsync source server
1) Turn off the firewall and determine whether rsync is installed
systemctl stop firewalld setenforce 0 rpm -q rsync
2) Add / etc / Rsync Conf configuration file
vim /etc/rsyncd.conf #Add the following configuration items uid = nobody gid = nobody use chroot = yes #Locked in the source directory address = 192.168.241.3 #Listening address port 873 #The listening port tcp/udp 873 can be viewed through cat /etc/services | grep rsync log file = /var/log/rsyncd.log #Log file location pid file = /var/run/rsyncd.pid #File location where the process ID is stored hosts allow = 192.168.241.0/24 #Allowed client address [hzh] #Shared module name path = /var/www/html #The actual path of the source directory comment = Document Root of www.hzh.com read only = yes #Is it read-only dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z #File types that are no longer compressed during synchronization auth users = hzh #Authorized accounts. Multiple accounts are separated by spaces secrets file = /etc/rsyncd_users.db #Data file for storing account information ###If anonymity is adopted, just remove the configuration items of "auth users" and "secrets file".
3) Create data files for backup accounts
vim /etc/rsyncd_users.db hzh:abc123 #There is no need to establish a system user with the same name chmod 600 /etc/rsyncd_users.db
4) Ensure that all users have access to the source directory / var/www/html
yum -y install httpd chmod +r /var/www/html/ ls -ld /var/www/html/
5) Start rsync service program
rsync --daemon #Start the rsync service and run it as an independent monitoring service (daemon) netstat -anpt | grep rsync
6) Turn off rsync service
kill $(cat /var/run/rsyncd.pid) perhaps rm -rf /var/run/rsyncd.pid
2. Initiator
Basic format: rsync [options] original location target location
1. Common options
-r: Recursive mode, including all files in the directory and subdirectory
-l: For symbolic link files, you can still copy them as symbolic link files
-v: Displays details of the synchronization process
-z: Compress when transferring files
-a: The archive mode preserves the permissions, attributes and other information of the file, which is equivalent to the combined option "- rlptgoD"
-p: Keep the permission mark of the file
-t: Time stamp of retention file
-g: Keep the group mark of the file (for super users only)
-o: Keep the ownership mark of the file (for super users only)
-H: Keep hard wired files
-A: Retain ACL attribute information
-D: Retention of special documents and other equipment
– delete: output a file that exists in the target location but not in the original location
– checksum: decide whether to skip the file based on the checksum (not the file size and modification time)
2. Synchronization method
1) Download the specified resources to the local / opt directory for backup.
rsync -avz hzh@192.168.241.3::hzh /opt #Password abc123
perhaps
rsync -avz rsync://hzh@192.168.241.3/hzh /opt
2) Interactive free formatting
echo "abc123" > /etc/server.pass chmod 600 /etc/server.pass crontab -e 30 22 * * * /usr/bin/rsync -az --delete --password-file=/etc/server.pass hzh@192.168.241.3::hzh /opt/
3,rsync+inotify
The inotify notification interface can be used to monitor various changes of the file system, such as file access, deletion, movement, modification and so on. Using this mechanism, it is very convenient to realize file change alarm and incremental backup, and respond in time to changes in directories or files.
The combination of inotify mechanism and rsync tool can realize triggered backup (real-time synchronization), that is, as long as the document in the original location changes, the incremental backup operation will be started immediately; Otherwise, it is in a silent waiting state.
Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring, which is more suitable for uplink synchronization when applied in triggered backup.
1) Modify rsync source server configuration file
vim /etc/rsyncd.conf ...... read only = no #Turn off read-only, and uplink synchronization can be written kill $(cat /var/run/rsyncd.pid) perhaps rm -rf /var/run/rsyncd.pid rsync --daemon netstat -anpt | grep rsync chmod 777 /var/www/html/
2) Adjust inotify kernel parameters (executed on the client)
In the Linux kernel, the default inotify mechanism provides three control parameters: max_queue_events (monitor event queue, the default value is 16384), max_user_instances (the maximum number of monitored instances, the default value is 128), max_user_watches (the maximum number of monitoring files per instance, the default value is 8192). When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.
cat /proc/sys/fs/inotify/max_queued_events cat /proc/sys/fs/inotify/max_user_instances cat /proc/sys/fs/inotify/max_user_watches vim /etc/sysctl.conf fs.inotify.max_queued_events = 16384 fs.inotify.max_user_instances = 1024 fs.inotify.max_user_watches = 1048576 sysctl -p
3) Install inotify tools (client)
Inotify tools need to be installed to use inotify mechanism to provide inotifywait and inotifywatch auxiliary tools.
notifywait: it can monitor various events such as modify, create, move, delete and attrib, and output results as soon as there is a change.
inotifywatch: it can be used to collect file system changes and output the summarized changes after the operation.
yum -y install gcc gcc-c++ make tar zxvf inotify-tools-3.14.tar.gz -C /opt/ cd /opt/inotify-tools-3.14 ./configure make && make install
4) Execute the "inotifywait" command, and then send it to / var/www/html (client) at another terminal
Add and move files under the directory, and track the output results on the screen.
inotifywait -mrq -e modify,create,move,delete /var/www/html
#Option "- e": used to specify which events to monitor
#Option "- m": continuous monitoring
#Option "- r": indicates recursion of the entire directory
#Option "- q": simplify output information
5) Write trigger synchronization script at another terminal
vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/server.pass /var/www/html/ hzh@192.168.241.3::hzh/"
$INOTIFY_CMD | while read DIRECTORY EVENT FILE
##while determines whether the monitoring record is received
do
if [ $(pgrep rsync | wc -l) -le 0 ] ; then
$RSYNC_CMD
fi
done
6) Empowerment
chmod +x /opt/inotify.sh chmod 777 /var/www/html/ chmod +x /etc/rc.d/rc.local echo '/opt/inotify.sh' >> /etc/rc.d/rc.local #Add automatic execution after startup
