Redis persistence strategy
I. persistence
1.1 what is persistence
All data of redis is saved in memory, and data updates will be asynchronously saved to the hard disk
1.2 implementation method of persistence
Snapshot: a completed backup of data at a certain time, -mysql of Dump -redis of RDB Write log: log any operation. To recover data, just run the log again -mysql of Binlog -Hhase of HLog -Redis of AOF
Second RDB
2.1 what is RDB
2.2 trigger mechanism - three main ways
''' save(synchronization) 1 Client execution save command---->redis Server---->Synchronous creation RDB Binary file 2 Will cause redis Very large amount of data 3 File strategy: if old RDB If it exists, it will replace the old one 4 Complexity o(n) ''' ''' bgsave(Asynchronous, Backgroud saving started) 1 Client execution save command---->redis Server---->Asynchronous creation RDB Binary file( fork Function to generate a child process( fork Will block reids),implement createRDB,Successful execution, return to reids Message) 2 Access at this time redis,The client will respond normally 3 File strategy: follow save Same if old RDB If it exists, it will replace the old one 4 Complexity o(n) ''' ''' Automatic (via configuration) to configure seconds changes save 900 1 save 300 10 save 60 10000 If 60 s Changed 1 in w Data, automatically generated rdb If 300 s 10 pieces of data are changed in the, which is automatically generated rdb If 900 s One piece of data is changed in and automatically generated rdb If any of the above three items is met, it will be generated automatically rdb,Internal use bgsave ''' #to configure: save 900 1 # Configure one save 300 10 # Configure one save 60 10000 # Configure one dbfilename dump.rdb # The name of the RDB file. The default is dump rdb dir ./ # rdb files exist in the current directory stop-writes-on-bgsave-error yes # If bgsave has an error, whether to stop writing or not. The default value is yes rdbcompression yes # Use compressed format rdbchecksum yes # Check and verify rdb files #Optimal configuration save 900 1 save 300 10 save 60 10000 dbfilename dump-${port}.rdb # With the port number as the file name, there may be many reids on one machine, which will not be disordered dir /bigdiskpath # Save the path to a large hard disk location directory stop-writes-on-bgsave-error yes # Stop with error rdbcompression yes # compress rdbchecksum yes # check
2.3 trigger mechanism - ways not to be ignored
1 Full replication # If save and bgsave are not executed and rdb policies are not added, rdb files will be generated. If master-slave replication is enabled, the master will automatically generate rdb files 2 debug reload # The debug level restart will not empty the data in the memory 3 shutdown save # Closing will start rdb generation
Triple A of
3.1 RDB issues
Time consuming, performance consuming:
Uncontrollable, data may be lost
3.2 AOF introduction
Every time the client writes a command, a log is recorded and put into the log file. If there is a downtime, the data can be completely recovered
3.3 three strategies of AOF
The log is not written directly to the hard disk, but is first placed in the buffer, which is written to the hard disk according to some strategies
always: redis – "buffer flushed by write command –" fsync each command to hard disk – "AOF file"
everysec (default): redis - "buffer flushed by write command" - fsync buffer to hard disk every second - "AOF file"
no:redis - "buffer flushed by write command" - determined by the operating system, the buffer fsync to the hard disk – "AOF file"
command | always | everysec | no |
---|---|---|---|
advantage | No data loss | fsync once per second, losing 1 second of data | Never mind |
shortcoming | The IO overhead is large, and the general sata disk has only a few hundred TPS | Lose 1 second data | Uncontrollable |
3.4 AOF rewrite
With the gradual writing of commands and the increase of concurrency, the AOF file will become larger and larger. This problem can be solved by AOF rewriting
Native AOF | AOF rewrite |
---|---|
set hello world set hello java set hello hehe incr counter incr counter rpush mylist a rpush mylist b rpush mylist c Expired data | set hello hehe set counter 2 rpush mylist a b c |
The essence is to optimize the expired, useless, repeated and optimizeable commands
This reduces disk usage and accelerates recovery
Implementation mode
bgrewriteaof:
The client sends the bgrewriteaof command to the server, and the server will start a fork process to complete AOF rewriting
AOF override configuration:
Configuration name | meaning |
---|---|
auto-aof-rewrite-min-size | AOF file rewriting requires size |
auto-aof-rewrite-percentage | AOF file growth rate |
Statistical name | meaning |
---|---|
aof_current_size | AOF current size in bytes |
aof_base_size | Size of AOF last started and rewritten (unit: bytes) |
Automatic trigger timing (two conditions are met at the same time):
aof_ Current_ Size > auto AOF rewrite min size: the current size is larger than the size required for rewriting
(aof_current_size-aof_base_size)/aof_ base_ Size > auto AOF rewrite percentage: (growth rate) the current size minus the size rewritten last time, except that the size rewritten last time is greater than the growth rate in the configuration
Rewrite process
to configure
appendonly yes # Set this option to yes to open appendfilename "appendonly-${port}.aof" #File save name appendfsync everysec # Adopt the second strategy dir /bigdiskpath # Storage path no-appendfsync-on-rewrite yes # Whether to perform aof append operation during aof rewriting, because aof rewriting consumes performance and disk consumption. There is a certain conflict between normal aof writing to disk, and the data during this period is allowed to be lost
IV. selection of RDB and AOF
4.1 comparison between RDB and aof
command | rdb | aof |
---|---|---|
boot priority | low | High (hang up and restart will load aof data) |
volume | Small | large |
Recovery speed | fast | slow |
Data security | Lost data | Determined by strategy |
Weight | heavy | light |
4.2 rdb best strategy
rdb is turned off when the master and slave operate
Centralized management: backup data by day and hour
Master slave configuration, open from node
4.3 aof best strategy
On: cache and storage are on in most cases,
aof rewriting centralized management
everysec: Policy refreshed per second
4.4 best strategy
Small slice: the maximum memory of each redis is 4g
Caching or storage: different strategies are used according to characteristics
Monitor hard disk, memory, network load, etc. from time to time
Enough memory