Redis persistence strategy

I. persistence

1.1 what is persistence

All data of redis is saved in memory, and data updates will be asynchronously saved to the hard disk

1.2 implementation method of persistence

Snapshot: a completed backup of data at a certain time,
	-mysql of Dump
    -redis of RDB
 Write log: log any operation. To recover data, just run the log again
	-mysql of Binlog
    -Hhase of HLog
    -Redis of AOF

Second RDB

2.1 what is RDB

2.2 trigger mechanism - three main ways

'''
save(synchronization)
1 Client execution save command---->redis Server---->Synchronous creation RDB Binary file
2 Will cause redis Very large amount of data
3 File strategy: if old RDB If it exists, it will replace the old one
4 Complexity o(n)
'''

'''
bgsave(Asynchronous, Backgroud saving started)

1 Client execution save command---->redis Server---->Asynchronous creation RDB Binary file( fork Function to generate a child process( fork Will block reids)，implement createRDB，Successful execution, return to reids Message)
2 Access at this time redis，The client will respond normally
3 File strategy: follow save Same if old RDB If it exists, it will replace the old one
4 Complexity o(n)
'''

'''
Automatic (via configuration)
to configure   seconds   changes
save   900        1
save   300        10
save   60         10000
 If 60 s Changed 1 in w Data, automatically generated rdb
 If 300 s 10 pieces of data are changed in the, which is automatically generated rdb
 If 900 s One piece of data is changed in and automatically generated rdb

If any of the above three items is met, it will be generated automatically rdb，Internal use bgsave
'''

#to configure:
save 900 1 # Configure one
save 300 10 # Configure one
save 60 10000 # Configure one
dbfilename dump.rdb  # The name of the RDB file. The default is dump rdb
dir ./ # rdb files exist in the current directory

stop-writes-on-bgsave-error yes # If bgsave has an error, whether to stop writing or not. The default value is yes
rdbcompression yes # Use compressed format
rdbchecksum yes # Check and verify rdb files

#Optimal configuration
save 900 1 
save 300 10 
save 60 10000 
dbfilename dump-${port}.rdb  # With the port number as the file name, there may be many reids on one machine, which will not be disordered
dir /bigdiskpath # Save the path to a large hard disk location directory
stop-writes-on-bgsave-error yes # Stop with error
rdbcompression yes # compress
rdbchecksum yes # check

2.3 trigger mechanism - ways not to be ignored

1 Full replication # If save and bgsave are not executed and rdb policies are not added, rdb files will be generated. If master-slave replication is enabled, the master will automatically generate rdb files
2 debug reload # The debug level restart will not empty the data in the memory
3 shutdown save # Closing will start rdb generation

Triple A of

3.1 RDB issues

Time consuming, performance consuming:

Uncontrollable, data may be lost

3.2 AOF introduction

Every time the client writes a command, a log is recorded and put into the log file. If there is a downtime, the data can be completely recovered

3.3 three strategies of AOF

The log is not written directly to the hard disk, but is first placed in the buffer, which is written to the hard disk according to some strategies

always: redis – "buffer flushed by write command –" fsync each command to hard disk – "AOF file"

everysec (default): redis - "buffer flushed by write command" - fsync buffer to hard disk every second - "AOF file"

no:redis - "buffer flushed by write command" - determined by the operating system, the buffer fsync to the hard disk – "AOF file"

command	always	everysec	no
advantage	No data loss	fsync once per second, losing 1 second of data	Never mind
shortcoming	The IO overhead is large, and the general sata disk has only a few hundred TPS	Lose 1 second data	Uncontrollable

3.4 AOF rewrite

With the gradual writing of commands and the increase of concurrency, the AOF file will become larger and larger. This problem can be solved by AOF rewriting

Native AOF	AOF rewrite
set hello world set hello java set hello hehe incr counter incr counter rpush mylist a rpush mylist b rpush mylist c Expired data	set hello hehe set counter 2 rpush mylist a b c

The essence is to optimize the expired, useless, repeated and optimizeable commands

This reduces disk usage and accelerates recovery

Implementation mode

bgrewriteaof:

The client sends the bgrewriteaof command to the server, and the server will start a fork process to complete AOF rewriting

AOF override configuration:

Configuration name	meaning
auto-aof-rewrite-min-size	AOF file rewriting requires size
auto-aof-rewrite-percentage	AOF file growth rate

Statistical name	meaning
aof_current_size	AOF current size in bytes
aof_base_size	Size of AOF last started and rewritten (unit: bytes)

Automatic trigger timing (two conditions are met at the same time):

aof_ Current_ Size > auto AOF rewrite min size: the current size is larger than the size required for rewriting

(aof_current_size-aof_base_size)/aof_ base_ Size > auto AOF rewrite percentage: (growth rate) the current size minus the size rewritten last time, except that the size rewritten last time is greater than the growth rate in the configuration

Rewrite process

to configure

appendonly yes # Set this option to yes to open
appendfilename "appendonly-${port}.aof" #File save name
appendfsync everysec # Adopt the second strategy
dir /bigdiskpath # Storage path
no-appendfsync-on-rewrite yes # Whether to perform aof append operation during aof rewriting, because aof rewriting consumes performance and disk consumption. There is a certain conflict between normal aof writing to disk, and the data during this period is allowed to be lost

IV. selection of RDB and AOF

4.1 comparison between RDB and aof

command	rdb	aof
boot priority	low	High (hang up and restart will load aof data)
volume	Small	large
Recovery speed	fast	slow
Data security	Lost data	Determined by strategy
Weight	heavy	light

4.2 rdb best strategy

rdb is turned off when the master and slave operate

Centralized management: backup data by day and hour

Master slave configuration, open from node

4.3 aof best strategy

On: cache and storage are on in most cases,

aof rewriting centralized management

everysec: Policy refreshed per second

4.4 best strategy

Small slice: the maximum memory of each redis is 4g

Caching or storage: different strategies are used according to characteristics

Monitor hard disk, memory, network load, etc. from time to time

Enough memory