1. What is a transaction?

In simple terms, a transaction is a series of operations performed by a single logical unit.

1.1, the four characteristics of transactions ACID

Transactions have the following four characteristics:

• 1. Atomicity: All operations that constitute a transaction must be a logical unit, either all or none of them.
• 2. Consistency: The state of the database must be stable or consistent before and after the transaction is executed. After A(1000) transfers 100 to B(200), the sum of A(900) and B(300) remains the same.
• 3. Isolation: Transactions are isolated from each other and do not affect each other.
• 4. Durability: After the transaction is successfully executed, the data must be written to the disk, and the data will not be lost after the shutdown and restart.

2. Transactions in Redis

Transactions in Redis are done through four commands: multi, exec, discard, and watch.

A single command in Redis is atomic, so what ensures a transaction is that multiple sets of commands are executed together.

The Redis command set is packaged together, and the same task is used to ensure that the commands are executed in an orderly and uninterrupted manner, thereby ensuring transactionality.

Redis is a weak transaction and does not support transaction rollback.

2.1. Transaction commands

Introduction to Transaction Commands

• 1, multi (open transaction)
  • Used to indicate the beginning of a transaction block, Redis will put subsequent commands into the queue one by one, and then use exec to execute the queue command atomically.
  • begin similar to mysql transaction
• 2. exec (commit transaction)
  • execute command queue
  • commit similar to mysql transaction
• 3, discard (clear the execution command)
  • Clear data from command queue
  • Similar to the rollback of the mysql transaction, but different from the rollback, here all the commands in the queue are directly cleared, so that they are not executed. So it's not a rollback. Just a purge.
• 4 , watch
  • Monitor a redis key If the key changes, watch can monitor it later. If in a transaction, a key that has been monitored is modified, the queue will be emptied at this time.
• 5,unwatch
  • Cancel listening for a redis key

transaction operation

# Ordinarily execute multiple commands
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set m_name zhangsan
QUEUED
127.0.0.1:6379> hmset m_set name zhangsan age 20 
QUEUED
127.0.0.1:6379> exec
1) OK
2) OK

# Clearing the queue before executing the command will result in unsuccessful transaction execution
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set m_name_1 lisi
QUEUED
127.0.0.1:6379> hmset m_set_1 name lisi age 21
QUEUED
# The clear queue command was executed before committing the transaction
127.0.0.1:6379> discard
OK
127.0.0.1:6379> exec
(error) ERR EXEC without MULTI

# Listening to a key and changing its value on another client before the transaction commits will also cause the transaction to fail
127.0.0.1:6379> set m_name_2 wangwu01
OK
127.0.0.1:6379> watch m_name_2
OK
127.0.0.1:6379> multi
OK
127.0.0.1:6379> set m_name_2 wangwu02
QUEUED
# After another client executes before exec, it will return nil here, that is, the queue is emptied, instead of successful execution
127.0.0.1:6379> exec
(nil)

# another client executes before exec
127.0.0.1:6379> set m_name_2 niuqi
OK

2.2, transaction mechanism analysis

We have always said before that Redis transaction commands are packaged and placed in a queue. So let's take a look at the data structure of the Redis client.

client data structure

typedef struct client {
    // Client unique ID
    uint64_t id;   
    // Client Status Indicates whether it is in a transaction
    uint64_t flags;         
    // transaction status
    multiState mstate;
    // ...and others will not be listed one by one
} client;

multiState transaction state data structure

typedef struct multiState {
    // The transaction queue is an array, in the order of first-in, first-out, the command that is enqueued first comes first, and the command that enters the queue last
    multiCmd *commands;     /* Array of MULTI commands */
    // Number of queued commands
    int count;              /* Total number of MULTI commands */
    // ...slightly
} multiState;

multiCmd transaction command data structure

/* Client MULTI/EXEC state */
typedef struct multiCmd {
    // parameters of the command
    robj **argv;
    // parameter length
    int argv_len;
    // Number of parameters
    int argc;
    // pointer to redis command
    struct redisCommand *cmd;
} multiCmd;

Redis transaction execution flow diagram

Analysis of Transaction Execution Process of Redis

• 1. At the beginning of the transaction, in the Client, there are attribute flags, which are used to indicate whether it is in the transaction. At this time, set flags=REDIS_MULTI
• 2. The Client stores commands in the transaction queue, except for some commands in the transaction itself (EXEC,DISCARD,WATCH,MULTI)
• 3. The client puts the command into multiCmd *commands, which is the command queue
• 4. The Redis client will send the exec command to the server and send the command queue to the server
• 5. After the server receives the command queue, it traverses and executes it at one time. If all executions are successful, the execution results are packaged and returned to the client at one time.
• 6. If the execution fails, set flags=REDIS_DIRTY_EXEC, end the loop, and return failure.

2.3. Analysis of monitoring mechanism

We know that Redis has an expires dictionary for key expiration events. Similarly, the monitored key also has a similar watched_keys dictionary. The key is the key to be monitored, and the value is a linked list that records all the clients that monitor this key.

The monitoring is to monitor whether the key is changed. If it is changed, the flags attribute of the client monitoring the key is set to REDIS_DIRTY_CAS.

The Redis client sends an exec command to the server, and the server determines the flags of the Redis client. If it is REDIS_DIRTY_CAS, the transaction queue is cleared.

redis listening mechanism diagram

redis listen key data structure

Go back and look at the watched_keys of the RedisDb class. It is indeed a dictionary. The data structure is as follows:

typedef struct redisDb {
    dict *dict;                 /* Store all key-value s */
    dict *expires;              /* The expiration time of the stored key */
    dict *blocking_keys;        /* blpop Store blocking key s and client objects*/
    dict *ready_keys;           /* After blocking, push, and respond to the blocked clients and key s */
    dict *watched_keys;         /* Store watch monitoring keys and client objects WATCHED keys for MULTI/EXEC CAS */
    int id;                     /* The ID of the database is 0-15, and the default redis has 16 databases */
    long long avg_ttl;          /* The average ttl(time in live) time of the storage object is used for statistics */
    unsigned long expires_cursor; /* Cursor of the active expire cycle. */
    list *defrag_later;         /* List of key names to attempt to defrag one by one, gradually. */
    clusterSlotToKeyMapping *slots_to_keys; /* Array of slots to keys. Only used in cluster mode (db 0). */
} redisDb;

2.4. Weak transactionality of Redis

Why is Redis said to be weakly transactional? Because if there is a syntax error in the redis transaction, all commands in the entire queue will be violently cleared directly.

# Set a value to test outside the transaction
127.0.0.1:6379> set m_err_1 test
OK
127.0.0.1:6379> get m_err_1 
"test"
# Open the transaction, modify the value, but other commands in the queue have syntax errors, and the entire transaction will be discard ed
127.0.0.1:6379> multi 
OK
127.0.0.1:6379> set m_err_1 test1
QUEUED
127.0.0.1:6379> sets m_err_1 test2
(error) ERR unknown command `sets`, with args beginning with: `m_err_1`, `test2`, 
127.0.0.1:6379> set m_err_1 test3
QUEUED
127.0.0.1:6379> exec
(error) EXECABORT Transaction discarded because of previous errors.

# get the value again
127.0.0.1:6379> get m_err_1
"test"

We found that if there is a syntax error in the command queue, it is to clear all the commands in the queue directly, not to roll back the transaction, but the syntax error can guarantee atomicity.

Let's look at some more, what if there is a type error? For example, after opening a transaction, set a key, first set it as a string, and then operate it as a list.

# open transaction
127.0.0.1:6379> multi 
OK
# set to string
127.0.0.1:6379> set m_err_1 test_type_1
QUEUED
# Insert two values ​​into the list
127.0.0.1:6379> lpush m_err_1 test_type_1 test_type_2
QUEUED
# implement
127.0.0.1:6379> exec
1) OK
2) (error) WRONGTYPE Operation against a key holding the wrong kind of valu
# Re-acquiring the value, we found that ours has been changed, obviously, the transaction execution failed.
127.0.0.1:6379> get m_err_1
"test_type_1"

Until now, we have determined that redis does not support transaction rollback. Because our transaction failed, but the command was executed successfully.

Weak Transaction Summary

• 1. Most transaction failures are due to syntax errors (rollbacks are supported) or type errors (rollbacks are not supported), and these two errors can be encountered in the re-development stage.
• 2. Redis ignores transaction rollback for performance.

So, is there no way for redis to guarantee atomicity? Of course, Redis' lua script is a supplement to weak transactions.

3. lua script in Redis

lua is a lightweight and compact scripting language written in standard C language and open in source code form. It is designed to be embedded in applications to provide flexible extension and customization functions for applications.

Lua application scenarios: game development, standalone application scripts, Web application scripts, extensions and database plugins.

OpenResty: A scalable Nginx-based Web platform, a third-party server that integrates lua modules on top of nginx.

OpenResty is a scalable Web platform implemented by extending Nginx through Lua. It integrates a large number of sophisticated Lua libraries, third-party modules and most of the dependencies. It is used to easily build dynamic Web applications, Web services and dynamic Web applications that can handle ultra-high concurrency (tens of millions of daily activities) and have extremely high scalability.
close. The function is similar to nginx, because it supports lua dynamic scripts, so it is more flexible, and can realize authentication, current limiting, shunting, and logging.
recording, grayscale publishing and other functions.

OpenResty extends the functions of nginx through Lua scripts to provide services such as load balancing, request routing, security authentication, service authentication, traffic control, and log monitoring.

Similar to Kong (Api Gateway), tengine (Ali)

3.1, Lua installation (Linux)

lua script download and install http://www.lua.org/download.html

lua script reference documentation: http://www.lua.org/manual/5.4/

# curl direct download
curl -R -O http://www.lua.org/ftp/lua-5.4.4.tar.gz
# decompress
tar zxf lua-5.4.4.tar.gz
# enter, directory
cd lua-5.4.4
# Compile and install
make all test

write lua script

Write a lua script test.lua, just define a local variable and print it out.

local name = "zhangsan"

print("name:",name)

execute lua script

[root@VM-0-5-centos ~]# lua test.lua 
name:	zhangsan

3.2. Using Lua in Redis

Since Redis 2.6, the built-in lua compiler can be used to evaluate lua scripts using the EVAL command.

The script command is atomic. When the Redis server executes the script command again, it does not allow new commands to be executed (it will block and no longer accept commands). ,

EVAL command

By executing the eval command of redis, you can run a lua script.

EVAL script numkeys key [key ...] arg [arg ...]

EVAL command description

• 1. script: It is a Lua script program, which will be run in the context of the Redis server. This script does not (and should not) be defined as a Lua function.
• 2. numkeys: Specifies the number of key name parameters.
• 3. key [key ...]: Starting from the third parameter of EVAL, numkeys keys (keys) are used to indicate which Redis keys (keys) are used in the script. These key name parameters can be found in Lua Through the global variable KEYS array, access with 1 as the base address ( KEYS[1] , KEYS[2] , and so on)
• 4. arg [arg ...]: It can be accessed in Lua through the global variable ARGV array. The form of access is similar to the KEYS variable (ARGV[1], ARGV[2], etc.)

Simply put, it is

eval lua Script Fragment Number of Parameters(Assuming the number of parameters=2)  Parameter 1 Parameter 2 Parameter 1 value Parameter 2 value

EVAL command execution

# Executing a lua script is to return the incoming parameters and corresponding values
127.0.0.1:6379> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 name age zhangsan 20
1) "name"
2) "age"
3) "zhangsan"
4) "20"

Calling redis in lua script

We have reached how to accept and return parameters, so how to call redis in lua script?

• 1,redis.call
  • The return value is the return value of the redis command execution
  • If there is an error, return an error message and do not continue execution
• 2,redis.pcall
  • The return value is the return value of the redis command execution
  • If there is an error, log the error message and continue to execute

In fact, redis.call will throw the exception, and redis.pcall will catch the exception and will not throw it.

The lua script calls redis to set the value

# Use redis.call to set the value
127.0.0.1:6379> eval "return redis.call('set',KEYS[1],ARGV[1])" 1 eval_01 001
OK
127.0.0.1:6379> get eval_01
"001"

EVALSHA command

The previous eval command sends the contents of the script itself once every time, so that the script is compiled every time.

Redis provides a caching mechanism, so the script is not recompiled every time, and in some scenarios, the bandwidth consumed by the script transfer may be unnecessary.

In order to reduce the consumption of bandwidth, Redis implements the evaklsha command, which has the same function as eval, except that the first parameter it accepts is not the script, but the SHA1 checksum (sum) of the script.

So how to get the value of this SHA1, you need to mention the Script command.

• 1. SCRIPT FLUSH: Clear all script caches.
• 2. SCRIPT EXISTS: According to the given script checksum, check whether the specified script exists in the script cache.
• 3. SCRIPT LOAD: Loads a script into the script cache, returns the SHA1 digest, but does not run it immediately.
• 4. SCRIPT KILL: Kill the currently running script

Execute the evalsha command

# Use script load to load the script content into the cache and return the value of sha
127.0.0.1:6379> script load "return redis.call('set',KEYS[1],ARGV[1])"
"c686f316aaf1eb01d5a4de1b0b63cd233010e63d"
# Execute using evalsha and the value of the returned sha + the number of parameters, parameter names and values
127.0.0.1:6379> evalsha c686f316aaf1eb01d5a4de1b0b63cd233010e63d 1 eval_02 002
OK
# get results
127.0.0.1:6379> get eval_02
"002"

We all wrote the script in the code line above. Can we write the script content in xxx.lua and execute it directly? Of course it is possible.

Run external lua scripts with redis-cli

Write an external script test2.lua, set the value to redis.

# The content of the script is to set a value
return redis.call('set',KEYS[1],ARGV[1])

# For the execution result, you can use ./redis-cli -h 127.0.0.1 -p 6379 to specify redis ip, port, etc.
root@62ddf68b878d:/data# redis-cli --eval /data/test2.lua eval_03 , test03       
OK

Using Redis to integrate lua scripts is mainly to ensure that the performance is the atomicity of transactions, because the transaction function of redis is indeed a bit poor!

4. Redis script replication

If Redis has master-slave replication enabled, how does the script replicate from the master server to the slave server?

First of all, there are two modes of script replication in redis, script propagation mode and command propagation mode.

When master-slave is turned on and AOF persistence is turned on.

4.1. Script propagation mode

In fact, what script is executed by the master server, what kind of script is executed by the slave server. But if there is a current event, random functions etc will cause the difference.

The master server executes the command

# Execute multiple redis commands and return
127.0.0.1:6379> eval "local result1 = redis.call('set',KEYS[1],ARGV[1]); local result2 = redis.call('set',KEYS[2],ARGV[2]); return {result1, result2}" 2 eval_test_01 eval_test_02 0001 0002
1) OK
2) OK
127.0.0.1:6379> get eval_test_01
"0001"
127.0.0.1:6379> get eval_test_02
"0002"

then the master will send the exact same eval command to the slave:

eval "local result1 = redis.call('set',KEYS[1],ARGV[1]); local result2 = redis.call('set',KEYS[2],ARGV[2]); return {result1, result2}" 2 eval_test_01 eval_test_02 0001 0002

Note: Scripts executed in this mode cannot have time, internal state, random functions, etc. Executing the same script and parameters must have the same effect. In Redis5, it is also in the same transaction.

4.2, command propagation mode

The master server in command propagation mode will wrap all write commands generated by executing the script with transactions, and then copy the transactions to the AOF file and the slave server.

Because the command propagation mode replicates the write commands and not the script itself, even if the script itself contains time, internal state, random functions, etc., the write commands replicated by the master to all slaves are still the same.

In order to enable command propagation mode, the user needs to call the following function in the script before performing any write operation with the script:

redis.replicate_commands()

redis.replicate_commands() is only valid for scripts that call this function: after executing the current script using command propagation mode, the server will automatically switch back to the default script propagation mode.

execute script

eval "redis.replicate_commands();local result1 = redis.call('set',KEYS[1],ARGV[1]); local result2 = redis.call('set',KEYS[2],ARGV[2]); return {result1, result2}" 2 eval_test_03 eval_test_04 0003 0004

appendonly.aof file content

*1
$5
MULTI
*3
$3
set
$12
eval_test_03
$4
0003
*3
$3
set
$12
eval_test_04
$4
0004
*1
$4
EXEC

As you can see, the command executed by our script is executed in a transaction.

In the same way, the master server only needs to send these commands to the slave server to achieve master-slave script data synchronization.

5. Redis pipeline/transaction/script

• 1. The pipeline is actually a one-time execution of a batch of commands, which does not guarantee atomicity. The commands are independent and belong to stateless operations (that is, ordinary batch processing).
• 2. Transactions and scripts are atomic, but transactions are weakly atomic, while lua scripts are strongly atomic.
• 3. The lua script can use the lua language to write more complex logic.
• 4. The atomicity of lua scripts is stronger than transactions. During the execution of scripts, other clients or any other scripts or commands cannot be executed. Therefore, the execution event of the lua script should be as short as possible, otherwise it will cause redis to block and cannot do other work.

6. Summary

Redis transactions are weak transactions, and the performance of multiple commands to open transactions together is relatively low, and atomicity cannot be guaranteed. So the lua script is a supplement to it, it is mainly to ensure the atomicity of redis.

For example, in some businesses (interface Api idempotent design, token generation, (take out the token and determine whether it exists, this is not an atomic operation)) we need to obtain a key and determine whether the key exists. You can use lua script to achieve.

There are still many places where we need to ensure atomicity of multiple command operations of redis. In this case, lua script may be the best choice.

7. Related articles

I also wrote other related articles on Redis, if you are interested, you can click to view it!

• <<Redis master-slave replication + sentinel election mechanism analysis>>
• <<Redis Weak Transactionality and Lua Script Atomic Analysis>>
• <<Analysis of Redis persistence mechanism>>
• <<Analysis of event processing mechanism of Redis>>
• <<How do Redis client and server communicate? >>
• <<Analysis of Redis elimination mechanism>>
• <<Analysis of the underlying data structure of Redis>>
• <<8 data types of Redis, what scenarios are used? >>
• << What is cache? What are the categories of cache? What is the cost of using it? >>
• <<6 common usage scenarios for caching>>