rsync backup case
Client requirements
1. The client prepares the directory of backup in advance. The directory rules are as follows: / backup/nfs_172.16.1.31_2018-09-02
2. The client packs backup locally (system configuration file, application configuration, etc.) and copies it to / backup/nfs_172.16.1.31_2018-09-02
3. The client finally pushes the backed up data to the backup server
4. The client executes the script at 1 a.m. every day
5. The client keeps the data of the last 7 days locally to avoid wasting disk space
Server requirements
1. The server deploys rsync to receive the backup data pushed by the client
2. The server needs to check whether the data pushed by the client is complete every day
3. The server needs to notify the administrator of the verification results every day
4. The server only keeps the backup data for 6 months, and all the rest are deleted
Note: the backup directory of all servers must be / backup
1. The client puts the files to be backed up into the specified directory / backup/nfs_172.16.1.31_2018-09-02
2. The client uses the rsync command to push NFS once a day at 1 a.m_ 172.16.1.31_ 2018-09-05
3. The client can keep the data of the last 7 days
1, Host planning
host | IP | identity |
---|---|---|
backup | 172.16.1.41 | rsync server |
nfs | 172.16.1.31 | rsync client |
web01 | 172.161.7 | rsync client |
2, Study needs
customer demand
1.The backup directory prepared by the client in advance. The directory rules are as follows::**/backup/nfs_172.16.1.31_2018-09-02** #The client needs to prepare a directory [backup] to store backup files. Under this directory, a directory composed of [hostname _IP _time] is automatically generated 2.The client packs the backup locally(System configuration file, application configuration, etc)Copy to**/backup/nfs_172.16.1.31_2018-09-02** #The system configuration file and application configuration are uniformly defined here as [/ etc/passwd] [/ etc/fstab], packaged with * * tar * * command and copied to the directory just created 3.The client finally pushes the backed up data to the backup server #rsync push to backup server [backup] 4.The client executes the script at 1 a.m. every day #Just schedule the task 5.The client keeps the data of the last 7 days locally, Avoid wasting disk space #Delete the contents of the previous seven days
Server requirements
1.Server deployment rsync,It is used to receive backup data pushed by the client #Deploy rsync 2.The server needs to check whether the data pushed by the client is complete every day #md5sum can be used for verification 3.The server needs to notify the administrator of the verification results every day #Deploy mail system 4.The server only retains backup data for 6 months,Delete the rest #Delete the content of the previous 180 days
3, Client preparation
[on web01 and nfs]
Take nfs as an example, web01 can copy the script
Batch modification time. And execute the script to generate the data file
[root@nfs scripts]# for i in {1..30};do date -s 2021/2/$i;sh /scripts/client_backup.sh;done
Client script
1,establish/backup catalogue [root@nfs ~]# mkdir /backup 2,Create script storage directory [root@nfs ~]# mkdir /scripts 3,Create a script in the script directory [root@nfs scripts]# vim client_backup.sh #!/bin/bash #Date:2021-2-8 #Author:zxm #Warning: the command uses an absolute path PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin #Define variables Hostname=$(hostname) IP=$(ifconfig eth1 | awk 'NR==2 {print $2}') Date=$(date +%F) SRC=/backup DEST=${Hostname}_${IP}_${Date} #1. Create directory #mkdir -p $SRC/$DEST #There is no error in this way. If you need to back up twice on the same day, #Then the action of creating a directory has been operated twice, and a judgment is made here #If this folder exists, you don't have to create it #[- d $SRC/$DEST] judge whether the directory exists and | realize the function of logical or [see extended note content] [ -d $SRC/$DEST ] || mkdir -p $SRC/$DEST #2. Backup files #Here, we can also judge whether there are compressed files at one time. If the files to be backed up need to be backed up twice a day, and #If the content is changed, you cannot judge because the added content cannot be backed up #In the tar command, c is to create, z is to compress gz format, f is to force, and P is to solve this error = = > tar: removing leading ` / 'from member names [ -f $SRC/$DEST/sys.tar.gz ] || /usr/bin/tar czfP $SRC/$DEST/sys.tar.gz /etc/fstab /etc/passwd && \ [ -f $SRC/$DEST/other.tar.gz ] || /usr/bin/tar czfP $SRC/$DEST/other.tar.gz /scripts/client_backup.sh #3. Marking with md5sum #Similarly, if the script is executed only once a day and the content is not increased, there is no need to judge whether the value of md5sum has changed. Here are multiple tests on the same day to prevent the value from changing [ -f $SRC/$DEST/flag_$Date ] || /usr/bin/md5sum $SRC/$DEST/*.tar.gz > $SRC/$DEST/flag_$Date #4. Push backup files to backup server export RSYNC_PASSWORD=123456 rsync -avz $SRC/$DEST rsync_backup@172.16.1.41::backup #5. Keep the local data of the last 7 days find $SRC/ -type d -mtime +7|xargs rm -rf
Expand knowledge
&&operator format command1 && command2 &&The command on the left (command 1) returns true(That is, after 0 is returned and successfully executed,&&The command on the right (Command 2) can be executed; In other words, "if this command is executed successfully&&Then execute this command ". 1,Use between commands && Connect to realize the function of logic and. 2,Only in && The command on the left returns true (command returns value) $? == 0),&& The command on the right will be executed. 3,As long as one command returns false (command returns value) $? == 1),Later commands will not be executed. ======================================================================================================================= ||operator format command1 || command2 ||Then and&&contrary. If||Command on the left( command1)If the execution is not successful, execute||Command on the right( command2);Or in other words, "if this command fails||Then execute this command. 1,Use between commands || Connect and realize the function of logical or. 2,Only in || The command on the left returns false (command return value) $? == 1),|| The command on the right will be executed. This and c The logic or syntax functions in the language are the same, that is, to realize short-circuit logic or operation. 3,As long as one command returns true (command returns value) $? == 0),Later commands will not be executed. ----------------------------------------------------------------------------------------------------------------------- command mtime ,xargs use mtime Chang Yu find Use it together, and there are other commands[ atime,ctime,amin,cmin,mmin] Usage: find . {-atime/-ctime/-mtime/-amin/-cmin/-mmin} [-/+]num Parameter analysis: 1.First parameter“.",Represents the current directory. For other directories, you can enter absolute directory and relative directory location; 2.The second parameter is divided into two parts, preceded by letters a,c,m Operation type, followed by time Is the date, min Minutes (note that only time,min As a unit); 3.The third parameter is quantity, in which no sign indicates that it meets the quantity-Indicates that after the quantity is met, with+Indicates that the quantity meets the previous. Specific explanation: ======================================================================================================================= atmin: Visit time( access time),It refers to the time when the file was last read. It can be used touch Change the command to the current time; -atime<24 Hours> Search for files or directories that have been accessed at a specified time, in 24 hours. For example, the current time is May 18, 2016 14:10:00,Query may 18, 2016 00:00:00 By May 18, 2016 23:59:59 Accessed files. time Represents the date, and the time unit is day,The query statement is: find . -atime 0 (-amin (similarly) ======================================================================================================================= ctime: Change time( change time),It refers to the file itself (permission, group and location)......)The last time to be changed, the change action can make chmod,chgrp,mv wait; -ctime<24 Hours> Find the file or directory that was changed at the specified time, in 24 hours. For example, the current time is May 18, 2016 14:10:00,Query may 18, 2016 00:00:00 By May 18, 2016 23:59:59 A modified file. The query statement is: find . -ctime 0 ======================================================================================================================= mtime: Modification time( modify time),It refers to the time when the file content was last modified. The modification action can make echo Redirection vi wait; -mtime<24 Hours> Find files or directories that have been changed at a specified time, in 24 hours. For example, the current time is May 18, 2016 14:10:00,Query may 18, 2016 00:00:00 By May 18, 2016 23:59:59 A file whose contents have been modified. The query statement is: find . -mtime 0 ----------------------------------------------------------------------------------------------------------------------- xargs
4, Server preparation
Install rsync
1.install rsync(Backup servers are installed) [root@backup ~]# yum install -y rsync [root@backup ~]# rpm -qc rsync #Find the configuration file /etc/rsyncd.conf #Master profile /etc/sysconfig/rsyncd #option 2.Server configuration [root@backup ~]# vim /etc/rsyncd.conf uid = rsync gid = rsync port = 873 fake super = yes use chroot = no max connections = 200 timeout = 600 ignore errors read only = false list = false auth users = rsync_backup secrets file = /etc/rsync.passwd log file = /var/log/rsyncd.log [backup] comment = welcome to oldboyedu backup! path = /backup 3.Create user on server [root@backup ~]# useradd rsync -s /sbin/nologin -M 4.Server creates password file [root@backup ~]# vim /etc/rsync.passwd rsync_backup:123456 [root@backup ~]# echo "rsync_backup:123456" > /etc/rsync.passwd #to grant authorization [root@backup ~]# chmod 600 /etc/rsync.passwd 5.The server creates a real file directory [root@backup ~]# mkdir /backup [root@backup ~]# chown -R rsync.rsync /backup/ 6.Server start service [root@backup ~]# systemctl start rsyncd [root@backup ~]# systemctl enable rsyncd #Verify startup [root@backup ~]# netstat -lntp tcp 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 18673/rsync tcp6 0 0 :::873 :::* LISTEN 18673/rsync [root@backup ~]# ps -ef | grep rsync root 18673 1 0 17:01 ? 00:00:00 /usr/bin/rsync --daemon --no-detach root 18680 7850 0 17:02 pts/0 00:00:00 grep --color=auto rsync
Configure mail server
1.Configure mail [root@backup ~]# yum install mailx -y [root@backup ~]# vim /etc/mail.rc set from=123@qq.com set smtp=smtps://smtp.qq.com:465 set smtp-auth-user=123@qq.com set smtp-auth-password=xxxxxx #Generate authorization code set smtp-auth=login set ssl-verify=ignore set nss-config-dir=/etc/pki/nssdb/ 2.Verify whether the mail can be sent successfully [root@backup ~]# echo "test" | mail -s "hello" 123@hotmail.com
Server script
1,establish/backup catalogue [root@nfs ~]# mkdir /backup 2,Create script storage directory [root@nfs ~]# mkdir /scripts 3,Write server script [root@backup scripts]# vim server_backup.sh #!/bin/bash #Date:2021-2-8 #Author:zxm #Warning: the command uses an absolute path PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin #Define variables Date=$(date +%F) SRC=/backup #1. Use md5 for verification and save the verification results md5sum -c $SRC/*_$DATE/flag_$DATE > $SRC/Data_check_$DATE #2. Send the saved result file to the administrator mail -s "Rsync Backup $DATE" zxmabc@hotmail.com < $SRC/Data_check_$DATE #3. Keep the data of the last 180 days find $SRC/ -type d -mtime +180|xargs rm -rf
Finally, conduct the overall test:
1. Delete the entire directory of the client
2. Delete all contents under server / backup
3. Write scheduled task test [at 1:00 a.m. on the modified client and 5:00 a.m. on the server]
#client [root@nfs ~]# rm -rvf /backup/* [root@nfs ~]# crontab -e * 1 * * * sh /scripts/client_backup.sh #Server [root@backup ~]# rm -rvf /backup/* [root@backup ~]# crontab -e * 5 * * * sh /scripts/server_backup.sh
How to extend backup to multiple servers:
[root@web01 ~]# rsync -avz root@172.16.1.31:/server / [root@web01 ~]# sh /server/scripts/client_push_data.sh