How to ensure the performance and security of Redis database? It's enough to read this Redis performance test and security optimization configuration guide

Contents of this chapter
0x00 Redis performance indicator monitoring

(1) Performance index

1. Basic activity indicator: basic activity
2. Performance index:
3. Memory index: memory
4. Persistence indicator
5. Error indicator: error
6. Description of other indicators

(2) Performance test tool

1. Redis benchmark command
2.redisbench tool
3.rdb memory analysis tool

(3) Benchmarking practices

3.1 single instance redis test in k8s

0x01 Redis security optimization

1.Security

Non privileged operation
file right
Interface binding
Change default service port
Authentication configuration
Disable specific commands
Logging
Prevent string escape and NoSQL injection
Protect against attacks triggered by carefully selected inputs from external clients
Firewall restricted access
It is forbidden to store sensitive plaintext data in redis
Redis security configuration summary example

2.Performance Optimization

Key optimization items
Redis performance optimization summary example

Pre knowledge learning supplement
1. Introduction and installation of redis database foundation- https://blog.weiyigeek.top/20...

2. Introduction and use of basic data types of redis database- https://blog.weiyigeek.top/20...

3. Principle introduction and master-slave configuration of redis basic operation and maintenance- https://blog.weiyigeek.top/20...

4. Sentinel and cluster installation and configuration of redis basic operation and maintenance- https://blog.weiyigeek.top/20...

5. Installation and configuration of redis basic operation and maintenance in K8S- https://blog.weiyigeek.top/20...

7. Actual combat of redis database disaster recovery and backup enterprise- https://blog.weiyigeek.top/20...

8.Redis database client operation practice and pit entry and exit- https://blog.weiyigeek.top/20...

0x00 Redis performance indicator monitoring

(1) Performance index

Common indicators and parameters of Redis server:

redis-cli -a password info > redis-Performance.txt # We can export the info related information of redis server to a file
  # 2.clients:
  # 3.memory: 
  # 4.persistence: 
  # 5.stats: General Statistics
  # 6.Replication: 
  # 7.CPU: CPU usage
  # 8.cluster: 
  # 9.Keypass: key value pair statistics quantity information
  
10.20.172.108:6379> info  # (1) Interactive viewing of redis server information
# Environment parameters for Server operation
redis_version:6.2.5
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:cb556a016f8668d
redis_mode:standalone
os:Linux 5.11.0-25-generic x86_64
arch_bits:64
multiplexing_api:epoll
atomicvar_api:c11-builtin
gcc_version:9.3.0
process_id:187640
process_supervised:no
run_id:97838216d4fe0de4739e7814b5a2e1d0d32d0982
tcp_port:6379
server_time_usec:1630241617439942
uptime_in_seconds:10930
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:2851665
executable:/opt/databases/redis-6.2.5/src/./redis-server
config_file:/home/weiyigeek/redis/6379/redis-6379.conf
io_threads_active:0

# Clients client related information
connected_clients:7
cluster_connections:0
maxclients:10000
client_recent_max_input_buffer:32
client_recent_max_output_buffer:0
blocked_clients:0
tracking_clients:0
clients_in_timeout_table:0

# Memory server running memory statistics
used_memory:2050432
used_memory_human:1.96M
used_memory_rss:5140480
used_memory_rss_human:4.90M
used_memory_peak:2253512
used_memory_peak_human:2.15M
used_memory_peak_perc:90.99%
used_memory_overhead:1982152
used_memory_startup:810376
used_memory_dataset:68280
used_memory_dataset_perc:5.51%
allocator_allocated:2204376
allocator_active:2555904
allocator_resident:5230592
total_system_memory:12442619904
total_system_memory_human:11.59G
used_memory_lua:37888
used_memory_lua_human:37.00K
used_memory_scripts:0
used_memory_scripts_human:0B
number_of_cached_scripts:0
maxmemory:0
maxmemory_human:0B
maxmemory_policy:noeviction
allocator_frag_ratio:1.16
allocator_frag_bytes:351528
allocator_rss_ratio:2.05
allocator_rss_bytes:2674688
rss_overhead_ratio:0.98
rss_overhead_bytes:-90112
mem_fragmentation_ratio:2.59
mem_fragmentation_bytes:3153776
mem_not_counted_for_evict:124
mem_replication_backlog:1048576
mem_clients_slaves:0
mem_clients_normal:123000
mem_aof_buffer:128
mem_allocator:jemalloc-5.1.0
active_defrag_running:0
lazyfree_pending_objects:0
lazyfreed_objects:0

# Persistence persistent data related information
loading:0
current_cow_size:0
current_cow_size_age:0
current_fork_perc:0.00
current_save_keys_processed:0
current_save_keys_total:0
rdb_changes_since_last_save:3
rdb_bgsave_in_progress:0
rdb_last_save_time:1630230687
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:-1
rdb_current_bgsave_time_sec:-1
rdb_last_cow_size:0
aof_enabled:1
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:0
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok
aof_last_cow_size:462848
module_fork_in_progress:0
module_fork_last_cow_size:0
aof_current_size:150
aof_base_size:92
aof_pending_rewrite:0
aof_buffer_length:0
aof_rewrite_buffer_length:0
aof_pending_bio_fsync:0
aof_delayed_fsync:0

# Stats general statistics information
total_connections_received:25
total_commands_processed:50482
instantaneous_ops_per_sec:4
total_net_input_bytes:2758703
total_net_output_bytes:22330756
instantaneous_input_kbps:0.23
instantaneous_output_kbps:0.55
rejected_connections:0
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:0
expired_stale_perc:0.00
expired_time_cap_reached_count:0
expire_cycle_cpu_milliseconds:0
evicted_keys:0
keyspace_hits:1
keyspace_misses:0
pubsub_channels:1
pubsub_patterns:0
latest_fork_usec:310
total_forks:1
migrate_cached_sockets:0
slave_expires_tracked_keys:0
active_defrag_hits:0
active_defrag_misses:0
active_defrag_key_hits:0
active_defrag_key_misses:0
tracking_total_keys:0
tracking_total_items:0
tracking_total_prefixes:0
unexpected_error_replies:0
total_error_replies:8
dump_payload_sanitizations:0
total_reads_processed:48899
total_writes_processed:97139
io_threaded_reads_processed:0
io_threaded_writes_processed:0

# Replication master-slave related indicator information
role:master
connected_slaves:0
master_failover_state:no-failover
master_replid:32da05299a5a36de431b4c05122f7d2b93eca169
master_replid2:3e15749ad586d60bd0d1c93854f6f719a22316ce
master_repl_offset:8915
second_repl_offset:829
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:1
repl_backlog_histlen:8915

# CPU processor mode occupancy information
used_cpu_sys:12.012184
used_cpu_user:9.453505
used_cpu_sys_children:0.002158
used_cpu_user_children:0.000000
used_cpu_sys_main_thread:11.802969
used_cpu_user_main_thread:9.286577

# Modules module loading

# Errorstats error status information
errorstat_NOAUTH:count=6
errorstat_READONLY:count=1
errorstat_WRONGPASS:count=1

# Cluster cluster information
cluster_enabled:0

# Keyspace library id and key quantity related information
db0:keys=1,expires=0,avg_ttl=0


1. Basic activity indicator: basic activity

NameDescription
connected_clientsNumber of client connections
conected_lavesNumber of slave
master_last_io_seconds_agoThe number of seconds since the last master-slave interaction
keyspaceTotal number of key values in the database
grep -En "connected_clients|conected_laves|master_last_io_seconds_ago|keyspace" redis-Performance.txt
27:connected_clients:7 # # Number of client connections
28:connected_slaves:1  # # Number of slave connections
128:keyspace_hits:1
129:keyspace_misses:0


2. Performance index:

NameDescription
latencyRedis response time to a request
instantaneous_ops_per_secAverage total number of requests processed per second
hi rate(calculated)Calculated hit rate
grep -En "latency|instantaneous_ops_per_sec|hi rate" redis-Performance.txt
114:instantaneous_ops_per_sec:3


3. Memory index: memory

NameDescription
used_memoryMemory used
mem_fragmentation_ratioMemory fragmentation rate
evicted_keysMaximum number of key s removed due to memory limitation
blocked_clientsClients blocked due to BLPOP,BRPOP,or BRPOPLPUSH
grep -En "used_memory|mem_fragmentation_ratio|evicted_keys|blocked_clients" redis-Performance.txt
32:blocked_clients:0
37:used_memory:2050432
38:used_memory_human:1.96M      # # The total amount of memory allocated by the memory allocator from the operating system
39:used_memory_rss:5234688     
40:used_memory_rss_human:4.99M  # # The memory occupation seen by the operating system and the memory occupation seen by the top command
41:used_memory_peak:2253512
42:used_memory_peak_human:2.15M # # Peak of redis memory consumption
43:used_memory_peak_perc:90.99%
44:used_memory_overhead:1982152
45:used_memory_startup:810376
46:used_memory_dataset:68280
47:used_memory_dataset_perc:5.51%
53:used_memory_lua:37888
54:used_memory_lua_human:37.00K #  # Amount of memory occupied by lua script engine
55:used_memory_scripts:0
56:used_memory_scripts_human:0B
67:mem_fragmentation_ratio:2.63
127:evicted_keys:0

4. Persistence indicator

NameDescription
rdb_last_save_timeTimestamp of the last persistent save disk
rdb_changes_sice_last_saveThe number of changes to the database since the last persistence
grep -En "rdb_last_save_time|rdb_changes_sice_last_save" redis-Performance.txt
88:rdb_last_save_time:1630230687
89:rdb_changes_since_last_save:0   # The number of changes to the database since the last persistence

5. Error indicator: error

NameDescription
rejected_connectionsThe number of connections rejected due to maxclient limit reached
keyspace_missesNumber of key value lookup failures (misses)
master_link_down_since_secondsDuration of master-slave disconnection (in seconds)
grep -En "rejected_connections|master_link_down_since_seconds" redis-Performance.txt
9:master_link_down_since_seconds:10937
119:rejected_connections:0
keyspace_misses:0    # The number of times the key value search fails (misses). It may be attacked by the hacker

<br/>

6. Description of other indicators

# 1. If the copy backlog buffer is set too small, the instructions in it will be overwritten and the offset will not be found, thus triggering full synchronization
repl_backlog_size: 1048576

# 2. By viewing sync_ partial_ The err variable determines whether the backlog buffer needs to be expanded. It indicates the number of master-slave semi synchronous replication failures
sync_partial_err:1


(2) Performance test tool

1. Redis benchmark command

Description: redis performance test is realized by executing multiple commands at the same time, which are executed under the redis directory;

Official website reference: https://redis.io/topics/bench...

Syntax parameters:

Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests]> [-k <boolean>]

# Parameter Description:
 -h <hostname>      Server hostname (default 127.0.0.1)
 -p <port>          Server port (default 6379)
 -s <socket>        Server socket (overrides host and port)
 -a <password>      Password for Redis Auth
 -c <clients>       Number of parallel connections (default 50)
 -n <requests>      Total number of requests (default 100000)
 -d <size>          Data size of SET/GET value in bytes (default 2)
 --dbnum <db>       SELECT the specified db number (default 0)
 -k <boolean>       1=keep alive 0=reconnect (default 1)
 -r <keyspacelen>   right SET/GET/INCR Use random keys to SADD Use random valueuse this option to extend strings within parameters_rand_int _uu)，The specified range of this parameter is 0 to keyspacelen-1 12 digits between.
 -P <numreq>        Pipeline <numreq> requests. Default 1 (no pipeline).
 -q                 Quiet. Just show query/sec values
 --csv              Output in CSV format
 -l                 Loop. Run the tests forever
 -t <tests>         Only run the comma separated list of tests. The test names are the same as the ones produced as output.
 -I                 Idle mode. Just open N idle connections and wait.


Basic instance:

# (1) 10000 requests are executed at the same time to test the performance (all default tests). The - q parameter allows the result to display only the number of requests executed per second
$ ./redis-benchmark -h 127.0.0.1 -p 6379 -t set,lpush -n 10000 -q
$ ./redis-benchmark -n 10000  -q 
   # PING_INLINE: 41493.78 requests per second
   # PING_BULK: 44843.05 requests per second
   # SET: 42194.09 requests per second
   # GET: 44052.86 requests per second
   # INCR: 43290.04 requests per second
   # LPUSH: 42194.09 requests per second
   # RPUSH: 42372.88 requests per second
   # LPOP: 42194.09 requests per second
   # RPOP: 42194.09 requests per second
   # SADD: 43668.12 requests per second
   # HSET: 42372.88 requests per second
   # SPOP: 44843.05 requests per second
   # LPUSH (needed to benchmark LRANGE): 42553.19 requests per second
   # LRANGE_100 (first 100 elements): 21367.52 requests per second
   # LRANGE_300 (first 300 elements): 9451.80 requests per second
   # LRANGE_500 (first 450 elements): 6807.35 requests per second
   # LRANGE_600 (first 600 elements): 5350.46 requests per second
   # MSET (10 keys): 36363.64 requests per second

# (2) Run the test of the specified project. For example, we require only the SET and LPUSH commands to be run in quiet mode
$ redis-benchmark -t set,lpush -n 100000 -q
  # SET: 74239.05 requests per second
  # LPUSH: 79239.30 requests per second


# (3) Specify the eval script command for benchmarking
$ redis-benchmark -n 100000 -q script load "redis.call('set','foo','bar')"
  # script load redis.call('set','foo','bar'): 69881.20 requests per second

# (4) Select the size of the key space. By default, the benchmark runs for a single key, and we can usually simulate a more real workload space by using large keys.
# For example, if I want to run 1 million SET operations and use a random key for each operation among 100000 possible keys,
$ redis-cli flushall
$ redis-benchmark -t set -r 100000 -n 1000000

# (5) By default, each client only sends the next command when receiving the reply from the previous command. Redis supports pipeline, so it can send multiple commands at one time, which can be imagined as parallel.
# For example: run the benchmark in MacBook Air 11 "using a pipeline of 16 commands
redis-benchmark -n 1000000 -t set,get -P 16 -q
  # SET: 403063.28 requests per second
  # GET: 508388.41 requests per second

# (6) Benchmarking using Unix domain sockets
$ numactl -C 6 ./redis-benchmark -q -n 100000 -s /tmp/redis.sock -d 256

# (7) Using TCP loopback
$ numactl -C 6 ./redis-benchmark -q -n 100000 -d 256


Benchmark comparison between Redis and Memcached in memory databases:

#!/bin/bash

# BIN=./redis-benchmark
BIN=./mc-benchmark
payload=32
iterations=100000
keyspace=100000

for clients in 1 5 10 20 30 40 50 60 70 80 90 100 200 300
do
    SPEED=0
    for dummy in 0 1 2
    do
        S=$($BIN -n $iterations -r $keyspace -d $payload -c $clients | grep 'per second' | tail -1 | cut -f 1 -d'.')
        if [ $(($S > $SPEED)) != "0" ]
        then
            SPEED=$S
        fi
    done
    echo "$clients $SPEED"
done

Finally, the following are the results in graphical form generated using gnuplot:


Factors affecting benchmarking

  • 1) Workload (number of connected clients)
  • 2) Different versions of Redis

  • 3) Physical configuration of servers providing services (disk, network, CPU, memory). On multi CPU slot servers, Redis performance depends on NUMA configuration and process location.

  • The test was completed by 50 clients executing 2 million requests at the same time.
  • Use the loopback interface to perform the test.
  • The test is performed using a key space of 1 million keys.

  • The test is executed with and without pipelines (16 command pipelines).

    Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipeline) / (without pipeline)

    $ ./ Redis benchmark - R 1000000 - N 2000000 - t get, set, lpush, lpop - p 16 -q# better than no pipeline.
    SET: 552028.75 requests per second
    GET: 707463.75 requests per second
    LPUSH: 767459.75 requests per second
    LPOP: 770119.38 requests per second

    $ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
    SET: 122556.53 requests per second
    GET: 123601.76 requests per second
    LPUSH: 136752.14 requests per second
    LPOP: 132424.03 requests per second

  • 4) Compared with using the same hardware without virtualization, Redis runs slower on the VM (Redis is recommended as the preferred physical machine)
  • 5) Depending on the platform, unix domain sockets can achieve about 50% more throughput than TCP/IP loopback (for example, on Linux).
  • 6) Compared with TCP/IP loopback, the performance advantage of Unix domain socket tends to decrease when a large number of pipelines (i.e. long pipelines) are used.
  • 7) When using Ethernet network to access Redis, when the data size remains below the Ethernet packet size (about 1500 bytes), the pipelined aggregation command is particularly effective. When processing 10 byte, 100 byte or 1000 byte queries, it will produce almost the same throughput.


2.redisbench tool

Description: the officially recommended redis benchmark has no way to specify the cluster mode during cluster benchmark testing. Redis & redis cluster benchmark tool is introduced here to facilitate the processing of cluster benchmark testing. Download address: https://github.com/panjiang/r...


redisbench features

  • Developed and built with Golang
  • redis single instance can be tested
  • redis cluster can be tested
  • Multiple cores can be used
  • It supports running on multiple machines at the same time and is used to test large redis clusters (the same machine hardware is required)


Format syntax:

./redisbench -h
-a string   #Redis instance address or Cluster addresses. IP:PORT[,IP:PORT]
-c int      #Clients number for concurrence (default 1)
-cluster    #true: cluster mode, false: instance mode
-d int      #Data size in bytes (default 1000)
-ma string  #addresses for run multiple testers at the same time
-mo int     #the order current tester is in multiple testers
-n int      #Testing times at every client (default 1)


Basic example:

# Test single instance mode
./redisbench -a 127.0.0.1:6379 -c 200 -n 20000 -d 3

# Test cluster
./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3

# Use multiple test nodes
./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 1 &
./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 2 &
./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 3

Tips: the test results will be automatically printed out: request value, request time, TPS is not actually demonstrated here. Interested friends can download the test by themselves.

<br/>

3.rdb memory analysis tool

Description: RDR is a tool for parsing redis rdbfile. Compared with redis RDB tools, RDR is implemented by golang and is faster.

  • Analyze which Key value in Redis memory occupies the most memory
  • It is analyzed that the Key at the beginning of Redis memory occupies the most, which is conducive to memory optimization
  • Redis Key value is displayed in Dashboard, which is more intuitive

Installation and download address: https://github.com/xueqiu/rdr...

matters needing attention:

  • 1. Add the executable permission Chmod + X RDR before using Linux and windows_ linux

Basic syntax:

# RDR parameter interpretation
show Web page display rdbfile Statistics for
keys from rdbfile Get all key
help help

Basic instance (take Linux as an example):

#1. Analyze and count the usage proportion of various types in multiple redis RDBS 
./rdr-linux keys *.rdb
SEARCH:PROJECTS_BY_ID:68
SEARCH:PROJECTS_BY_ID:64
SEARCH:PROJECTS_BY_PARENTID_LIST:0
SEARCH:PROJECTS_BY_PARENTID_LIST:64
SEARCH:PROJECTS_BY_PARENTID_LIST:66
SEARCH:PROJECTS_BY_PARENTID_LIST:68
SEARCH:PROJECTS_BY_ID:66
SEARCH:PROJECTSINFO_BY_PARENTID_LIST:64
ALLOW:SEARCH_BY_SFZH:230103197805153637

./rdr-linux dump dump.rdb
{"CurrentInstance": "dump.rdb",
"LargestKeyPrefixes": {
   "list": [
   {
      "Type": "list",
      "Key": "site",
      "Bytes": 144,
      "Num": 1
   }
],


#2. The web page displays the analysis results
./rdr-linux show -p 8080 *.rdb
start parsing...
parse dump.rdb  done
parsing finished, please access http://{$IP}:8080


(3) Benchmarking practices

3.1 single instance redis test in k8s

Environmental description:

# Host node running the Pod
osImage: Ubuntu 20.04.1 LTS
kernelVersion: 5.4.0-42-generic
kubeProxyVersion: v1.19.6
kubeletVersion: v1.19.6
containerRuntimeVersion: docker://19.3.14

$ Server Relevant information
redis_version:6.2.5
os:Linux 5.4.0-42-generic x86_64
arch_bits:64
gcc_version:10.3.1
process_id:1
process_supervised:no
tcp_port:6379
hz:10
configured_hz:10
lru_clock:3627023
io_threads_active:0

$ redis Configured memory limit
maxmemory:1073741824
maxmemory_human:1.00G
maxmemory_policy:volatile-lru

$ cpu Relevant information
Model name: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz
 Physics CPU number: 1
 logic CPU number: 4
CPU Number of cores: 4

<br/>

Benchmark:

# Test 1 Execute the set command and get command 10 million times, in which the size of each read is 256 bytes, and the number of requesting clients is 50 by default.
~$ redis-benchmark -h 10.102.39.181 -a 123456 -d 256 -t set,get -n 10000000 -q                            
SET: 42445.36 requests per second
GET: 49504.70 requests per second
# -Pod resource peak during test
very 1.0s: kubectl top pod -n database redis-cm-0  Tue Sep  7 20:36:50 2021
NAME         CPU(cores)   MEMORY(bytes)
redis-cm-0   848          18Mi

# Test 2 Similarly, execute the set command and get command for 10 million times, in which the size of each read is 256 bytes. The only difference is that pipelined-P 16 is used for testing (it can be seen that the number of set and get requests per second is significantly increased).
~$ redis-benchmark -h 10.102.39.181 -a 123456 -d 256 -t set,get -n 10000000 -P 16 -q                            
SET: 96019.98 requests per second
GET: 316575.91 requests per second

# -Peak Pod resources during test
very 1.0s: kubectl top pod -n database redis-cm-0  Tue Sep  7 20:46:50 2021
NAME         CPU(cores)   MEMORY(bytes)
redis-cm-0   457m         337Mi

<br/>

Practical test

  • 1) Insert 100000 data through shell pipe and redis pipe for comparison

    ## Shell-pipe.sh
#!/bin/bash
echo "start time: $(date +%s)"
for ((i=0;i<100000;i++));do
echo -en "helloworld-redis-${i}" | redis-cli -h 10.102.39.181 --no-auth-warning -a 123456 -x set username${i} >> ok.txt
done
echo "Completion time: $(date +%s)"

## redis-pipe.sh
#!/bin/bash
echo "start time: $(date +%s)"
python3 redis-set.py >> set-command.txt
cat set-command.txt | redis-cli -h 10.102.39.181 -a 123456 --no-auth-warning --pipe
echo "Completion time: $(date +%s)"

## redis-set.py
tee redis-set.py <<'EOF'
#!/usr/bin/python
for i in range(100000):
print('set name'+str(i)+' helloworld-redis-'+str(i))
EOF

  • 2) The execution efficiency of script insertion is recorded by using the time command

    ## Insert data in redis pipe mode
~/k8s/benchmark$ time ./redis-pipe.sh
# Start time: 1631020862
# All data transferred. Waiting for the last reply...
# Last reply received from server.
# errors: 0, replies: 100000
# Completion time: 1630862

# real    0m0.466s
# user    0m0.126s
# sys     0m0.035s

# Keyspace
db0:keys=100000,expires=0,avg_ttl=0

## Insert data in shell pipe mode
~/k8s/benchmark$ time ./shell-pip.sh
# Start time: 1631021312
# Completion time: 1631021921
# real    10m9.265s # Total time spent from the beginning to the end of the program (including CPU).
# user    3m44.411s # The program itself and the time of calling the subprocess.
# sys     1m55.435s # The execution time of the system call called by the program itself or indirectly.
# Keyspace
db0:keys=200000,expires=0,avg_ttl=0

Tips: it can be seen from the above results that the total time consumption of real of the two methods is very different. The efficiency of redis pipe method is not an order of magnitude compared with that of ordinary shell pipe method, so try to use redis pipe method to submit data in the development program.

# In order to facilitate the follow-up demonstration, grip inserted 80W pieces of data into the database in about 4s.
start time: 1631022423
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 800000
 Completion time: 1631022427
real    0m4.023s
user    0m0.885s
sys     0m0.187s

0x01 Redis security optimization

1.Security

Description: Redis provides access control, code security issues, and selecting attacks that can be triggered from outside by malicious input. Our operation and maintenance personnel need to configure them accordingly to improve security.

Non privileged operation

Description: redis can run without root permission. It is recommended to run it as a non privileged redis user only for this purpose. This way can prevent the possibility of CONFIG SET/GET directory and other similar runtime configuration instructions to the greatest extent..

#It is necessary to set up a separate redis account. redis crackit takes advantage of the root user feature to reset authorized_keys.  First create a redis account, and then start it through this account.
$ useradd redis
$ setsid sudo -u redis redis-server /etc/redis.conf
$ ps -elf|grep redis   #You can see that the redis user started
  # 4 S root      9048     1  0  80   0 - 59753 poll_s 19:43 ?        00:00:00 sudo -u redis redis-server /etc redis.conf
  # 4 S redis     9049  9048  0  80   0 - 38471 ep_pol 19:43 ?        00:00:00 redis-server


file right

Description: because the redis password is stored in clear text in the configuration file, we need to restrict the access permission of the redis file directory. For example, set the permission of the redis home directory to 700(rwx ------), if redis The conf configuration file is independent of the redis home directory, and the permission is modified to 600(rw -------).

# file right
chmod 700 /opt/redis/redis-5.0.4/
chmod 600 /etc/redis.conf

# Owner and group
chown redis:redis /etc/redis.conf
chown redis:redis /opt/redis/redis-5.0.4/

<br/>

Interface binding

Description: except for the trusted clients in the network, everyone should refuse to access the Redis port. Therefore, the server running Redis should only be directly accessed by the computer using Redis to implement the application.

If the server has two network interfaces (one area a and one area B), if only the machine access in area a is required, it will only be bound to the network interface in area A. if the server itself accesses, it will only be bound to the local loopback interface.

# Through Redis Add the following line to the conf file to bind Redis to a single interface:
bind 127.0.0.1 192.168.1.200

Tips: note that you can bind IPV6 in addition to IPV4


Change default service port

Description: in addition to specifying the binding interface, we can also change the default redis service port to prevent hackers from scanning and detecting redis services.

# Change the default service disconnect from 6379 to 63791
port 63791

<br/>

Authentication configuration

Description: it is very necessary to set an authentication password for Redis server. The following is a summary of several ways Redis configures password authentication:

Operation process:

# 1. Through redis Conf file. Redis needs to be restarted after this method is modified.
vim /etc/redis.conf
requirepass WeiyiGeek  # WeiyiGeek is the authentication password
masterauth  WeiyiGeek  # Configure the authentication password of the master node. Note that if the master is configured with a password, the slave must also configure the corresponding password parameters, otherwise it cannot be copied normally

# 2. Configure through the command line. This method has the advantage of no need to restart Redis.
redis 127.0.0.1:6379[1]> config set requirepass my_redis  
OK  
redis 127.0.0.1:6379[1]> config get requirepass  
1) "requirepass" 
2) "my_redis"

<br/>

Log in to Redis server with password authentication:

# Method 1: password plaintext will be recorded in the system command execution history (extremely not recommended / unsafe)
redis-cli -h 127.0.0.1 -p 6379 -a WeiyiGeek

# Mode 2: interactive configuration
redis-cli -h 127.0.0.1 -p 6379
redis 127.0.0.1:6379> auth WeiyiGeek # OK

Note: the AUTH command, like all other Redis commands, is sent in an unencrypted manner, so it cannot prevent attackers who have sufficient access to the network to eavesdrop. Therefore, for highly sensitive data, it is recommended to configure TLS support (Redis optionally supports TLS on all communication channels) to encrypt data and command transmission.

<br/>

Disable specific commands

Description: we can disable the commands in Redis or rename them to unpredictable names, so that ordinary clients are limited to a specified set of commands. For example, the vulnerability uses the two commands of config/save to complete the attack.

Since redis has no user permission restrictions, it is recommended to disable or rename dangerous commands using the rename configuration item, so that attackers who do not know the renaming rules cannot execute such commands FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, DEBUG, EVAL

For example, ordinary users may not be able to call the Redis CONFIG command to change the instance configuration, but the system providing and deleting instances should be able to do so.

# redis.conf configuration file
# Mode 1 The config / flush command is renamed to an unpredictable name
rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c53

# Mode 2 Disable it completely (or any other command) by renaming it to an empty string
rename-command CONFIG ""
rename-command FLUSHALL  ""

Tips: note that the redis server service needs to be restarted after configuration.


Logging

Description: create an access (or Debug) log for redis (set according to requirements). When establishing redis honeypot, if there is an attack attempt, it will find and monitor redis security status in time, and can monitor cmdstat_* Indicator information alarm;

# Execute info commandstats to see the number of times the command is executed, the CPU time consumed by the command (unit: ms), and the average CPU time consumed by executing each command (unit: ms)
cmdstat_get:calls=2,usec=15,usec_per_call=7.50
cmdstat_select:calls=1,usec=9,usec_per_call=9.00
cmdstat_keys:calls=4,usec=1948,usec_per_call=487.00
cmdstat_auth:calls=3123,usec=8291,usec_per_call=2.65

Logging configuration:

logfile "/usr/local/redis/redis.log" #Log file storage directory
loglevel verbose  #Record access information


Prevent string escape and NoSQL injection

Description: Redis protocol does not have the concept of string escape, so it is generally impossible to inject using ordinary client libraries, but it is possible to construct malicious scripts through Lua scripts executed by EVAL and EVALSHA commands.

> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first second

Solution: applications should avoid using strings obtained from untrusted sources to form the body of Lua scripts.

Tips: evalsha evaluates cached server-side scripts through its SHA1 summary. Scripts are cached on the server side using the SCRIPT LOAD command. This command is the same as EVAL in other respects.

<br/>

Protect against attacks triggered by carefully selected inputs from external clients

Description: it is possible for attackers to construct malicious data structures and insert them into Redis database, which may trigger the ill conditioned (worst-case) algorithm complexity of data structures implemented inside Redis.

For example, an attacker can provide a set of strings known to be hashed into the same bucket to the hash through a Web form, so as to turn the expected time (average time) of O(1) into the worst case of O(N), consume more CPU than expected, and eventually lead to a denial of service.

Solution: in order to prevent this specific attack, Redis uses the pseudo-random seed of each execution for the hash function.

<br/>

Firewall restricted access

Description: the security configuration was made for the redis server service level earlier. Here, restrictions are made for the network level to allow access only to the specified IP address. The advantage of configuring a firewall on the host is to prevent east-west traffic in the same network segment.

Firewall setting command on Linux system:

iptables -A INPUT -s x.x.x.x -p tcp --dport 6379 -j ACCEPT  #If you need access from other machines or set the slave mode, remember to add the corresponding firewall settings (Centos6)
firewall-cmd --add-rich-rule="rule family="ipv4" source address="x.x.x.x" port protocol="tcp" port="6379" accept" --permanent  #(Centos7)

System firewall setting command on Windows:

New-NetFirewallRule -Name "redis-server-access" -DisplayName "redis-server" -Description "redis-server Client access firewall rules" -Direction Inbound -LocalPort 6379 -RemoteAddress x.x.x.x -Protocol TCP -Action Allow -Enabled True
Get-NetFirewallRule -Name "redis-server-access"  | Format-Table


It is forbidden to store sensitive plaintext data in redis

Description: Redis is designed to provide high-performance KV services. At least at present, it is relatively weak in access control and data persistence. Therefore, from the application level, it is not recommended to use Redis to store sensitive information, such as authentication passwords.


Redis security configuration summary example

Security configuration example:

# Configuration file VIM / etc / redis / redis conf
# 1. Operate the trusted intranet and try to avoid public network access (if there are other fixed IP addresses in the intranet, a firewall needs to be set)
bind 127.0.0.1 

# 2. Bind the network interface monitored by redis (multiple IPS can be bound at the same time through the redis configuration item bind), and change 6379 to another port (or use unix pipeline for data management)
port 63791  

# 3. Enable redis password authentication and set high complexity password settings. Due to the high query efficiency, auth commands can be processed more than 10w times per second (so the intensity needs to be increased)
# echo -e "weiyigeek"|sha256sum 
requirepass 097575a79efcd7ea7b1efa2bcda78a4fc7cbd0820736b2f2708e72c3d21f8b61

# 4. Log file storage directory and record redis access information.
# debug (a lot of information, useful for development/testing)
# verbose (many rarely useful info, but not a mess like the debug level)
# Notice (modestly verbose, what you want in production probability) default
# warning (only very important / critical messages are logged)
logfile "/usr/local/redis/redis.log" 
loglevel verbose  

# 5. Protection mode is enabled by default. (no) it should only be disabled if
# -Are you sure you want clients of other hosts to connect to Redis
# -Even if authentication is not configured, there is no specific set of interfaces
# -Use the "bind" instruction to explicitly list.
protected-mode yes

# 6. Rename special commands (as required)
# `FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, DEBUG, EVAL`
rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52
rename-command FLUSHDB b840fc02d524045429941cc15f59e41cb7be6c53
rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c54
rename-command EVAL b840fc02d524045429941cc15f59e41cb7be6c55
rename-command DEBUG b840fc02d524045429941cc15f59e41cb7be6c56
rename-command SHUTDOWN b840fc02d524045429941cc15f59e41cb7be6c7


2.Performance Optimization

Description: Redis development and operation and maintenance personnel pay more attention to some configuration optimization of Redis itself, such as AOF and RDB configuration optimization, data structure configuration optimization, etc., but they don't know or care about whether the operating system needs to do some configuration optimization for Redis. However, facts have proved that a good system operation configuration can escort the good operation of Redis services.

Key optimization items

  • Step 1.vm.overcommit_memory best practices
    Such logs may appear when Redis is started, and then figure out what is overcommit?
    Description: the Linux operating system replies yes to most requests for memory in order to run more programs. Because memory will not be used immediately after applying for memory, this technology is called overcommit.

    # If Redis has the above log at startup, it means ` VM overcommit_ Memory = 0 `, Redis prompts to set it to 1.
# WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the 
command 'sysctl -w vm.overcommit_memory=1' for this to take effect.

# Note: the available memory in this article represents the sum of physical memory and swap.
  • Redis suggests setting this value to 1 so that fork can execute successfully under low memory (set a reasonable maxmemory to ensure that the machine has 20% ~ 30% idle memory).

  • Centralized management of aof rewriting and rdb bgsave.

    Tips : In the log Background save Represents bgsave and bgrewriteaof，What should the operating system do if the currently available memory is insufficient fork. If vm.overcommit_memory=0，Means that if there is no available memory, the memory application fails, corresponding to Redis namely fork Execution failed at Redis Your log will appear:`Cannot allocate memory `

<br/>

  • Step 2.vm.swapniess best practices
    Description: swap is important for the operating system. When the physical memory is insufficient, you can swap out some memory pages to solve the urgent problem. However, there is no free lunch in the world. Swap space is provided by hard disk. For applications requiring high concurrency and high throughput, disk IO will usually become a system bottleneck. In Linux, it is not necessary to wait until all physical memory is used up before using swap. The system parameter swppiness determines the tendency of the operating system to use swap. The value range of swap is 0 ~ 100. The larger the value of swap, the higher the probability that the operating system may use swap, and the lower the value of swap, indicating that the operating system prefers to use physical memory.

If Linux > 3.5, VM Swapniess = 1 (prefer swap to OOM killer), otherwise VM Swapniess = 0 (prefer OOM killer to swap) to achieve the following two goals:

1. Make Redis fast enough when there is enough physical memory.
2. Avoid the death of Redis when the physical memory is insufficient (if Redis is currently highly available, death is better than blocking).

Operation and maintenance tips: OOM(Out Of Memory) killer mechanism means that when the Linux operating system finds that there is insufficient available memory, it forces to kill some user processes (non kernel processes) to ensure that the system has enough available memory for allocation.

<br/>

  • Step 3.kernel.mm.transparent_hugepage.enabled best practices
    Redis may see the following logs when starting:

    WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

Tips: from the prompt, Redis suggests modifying the related configuration of transparent tiger pages (THP). The Linux kernel adds the transparent tiger pages (THP) feature in the 2.6.38 kernel, which supports the allocation of large memory pages (2MB), which is enabled by default. When enabled, the speed of fork child processes can be reduced, but after fork, each memory page changes from 4KB to 2MB, which will greatly increase the memory consumption of the parent process during rewriting. At the same time, the copy memory page unit caused by each write command is enlarged by 512 times, which will slow down the execution time of write operations and lead to a large number of slow queries of write operations.

Therefore, it is recommended to disable this feature in the Redis log. The disabling method is as follows: echo never > / sys / kernel / mm / transparent_ hugepage/enabled

<br/>

  • Step 4.Transparent Huge Pages
    Redis may see the following log when starting: warning you have transparent tiger pages (THP) support enabled in your kernel This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_ hugepage/enabled' as root, and add it to your /etc/rc. local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.

From the prompt, Redis suggests modifying the relevant configuration of transparent tiger pages (THP). The Linux kernel adds the transparent tiger pages (THP) feature in the 2.6.38 kernel, which supports the allocation of large memory pages (2MB), which is enabled by default. When enabled, the speed of fork child processes can be reduced, but after fork, each memory page changes from 4KB to 2MB, which will greatly increase the memory consumption of the parent process during rewriting. At the same time, the copy memory page unit caused by each write command is enlarged by 512 times, which will slow down the execution time of write operations and lead to a large number of slow queries of write operations. For example, a simple incr command will also appear in slow queries. Therefore, it is recommended to disable this feature in Redis logs. The disabling methods are as follows:

# The THP configuration is still effective after the machine is restarted
tee -a /etc/rc.local <<'EOF'
echo never >  /sys/kernel/mm/transparent_hugepage/enabled
EOF

<br/>

  • Step 5. Optimized configuration of oom killer
    The OOM killer will selectively kill the user process when the available memory is insufficient. It will set a weight for each user process. The higher the weight, the higher the probability of being "attacked". On the contrary, the lower the probability. The weight of each process is stored in / proc / {progress_id} / oom_ In score, this value is controlled by / proc/{progress_id}/oom_adj control, OOM_ The minimum value of adj is different in different Linux versions. You can refer to oom in Linux source code H (from - 15 to - 17)

When OOM_ When adj is set to the minimum value, the process will not be killed by OOM killer. The setting method is as follows:

# command
echo {value} > /proc/${process_id}/oom_adj

# script
for redis_pid in $(pgrep -f "redis-server")
do
  echo -17 > /proc/${redis_pid}/oom_adj
done

<br/>

  • Step 6. Set the number of open files, handles and the maximum number of processes per user
    Description: the following parameters are mainly set to the maximum number of Linux file handles that can be used by a single process to avoid abnormal error reporting under high concurrency. Suggestions mentioned officially in Redis

    # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
First line: Redis It is suggested that open files At least set it to 10032, so how does this 10032 come from? because maxclients The default is 10000. These are used to handle client connections. In addition, Redis Up to 32 file descriptors will be used internally, so 10032 here = 10000 + 32. 

# Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.
Second line: Redis Cannot open files Set to 10032 because it does not have permission to set.

# Current maximum open files is 4096. Maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit –n'.
Line 3: current system open files It's 4096, so maxclients Set to 4096-32=4064 If you want to set higher maxclients，Please use ulimit -n To set. From the above three lines of log analysis, we can see that open files Restricted priority ratio maxclients Big.

    terms of settlement:

    # temporary
ulimit –Sn 10032
# permanent
  • soft nofile 10032
  • hard nofile 10032
  • soft nproc 65535

  • hard nproc 65535
    EOF

<br/>

  • Step 7.TCP backlog log queue optimization
    Description: the default TCP backlog of Redis is 511, which can be adjusted by modifying the configuration of TCP backlog. If the TCP backlog of Linux is less than the TCP backlog set by Redis, the following logs will be seen when Redis is started:

    # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.

    resolvent:

    # see
cat /proc/sys/net/core/somaxconn
128

# modify
echo 511 > /proc/sys/net/core/somaxconn

<br/>

  • Step 8. Ensure the consistency of Redis server clock
    Description: we know that Redis Sentinel and Redis Cluster, which require multiple Redis instances, may involve multiple servers. Although Redis does not have strict requirements on the clocks of multiple servers, if the clocks of the servers of multiple Redis instances are inconsistent, it is very difficult to troubleshoot the logs of some abnormal situations, such as the failover of Redis Cluster. If the log times are different, it will cause great trouble for us to troubleshoot the problems (Note: it will not affect the cluster function, and the cluster nodes depend on their own clocks). Generally, NTP service will be provided in the company to provide standard time service, so as to achieve the effect of correcting the clock

For example, the NTP service is synchronized once every hour

0 * * * * /usr/sbin/ntpdate ntp.xx.com > /dev/null 2>&1


Redis performance optimization summary example

System optimization configuration

# -Set memory allocation policy
sudo sysctl -w vm.overcommit_memory=1

# -Try to use physical memory (fast) for kernel version greater than > = 3 X (prefer swap to OOM killer)
sudo sysctl -w vm.swapniess=1

# -Disable THP feature to reduce memory consumption
echo never > /sys/kernel/mm/transparent_hugepage/enabled

# -OOM killer feature optimization
for redis_pid in $(pgrep -f "redis-server")
do
  echo -17 > /proc/${redis_pid}/oom_adj
done

# -Set the number of open files, handles and the maximum number of processes per user
tee etc/security/limits.conf <<'EOF'
*  soft    nofile          10032
*  hard    nofile          10032
*  soft    nproc           65535
*  hard    nproc           65535
EOF

# -SYN queue length setting this parameter can accommodate more networks waiting to connect.
echo 511 > /proc/sys/net/core/somaxconn
sudo sysctl -w net.ipv4.tcp_max_syn_backlog=2048 

# -Synchronize time every hour
0 * * * * /usr/sbin/ntpdate ntp.xx.com > /dev/null 2>&1


Application configuration optimization

# Maximum number of client connections (it needs to be adjusted according to the actual situation, which is related to the open files of the system, and the number value is open files(10032) - 32)
maxclients 10000

# Key items of cluster configuration optimization
# Cluster timeout. If this time is set too small, the operation of reselecting the Master may be caused by network fluctuations
cluster-node-timeout 5000
# After writing, the master node must be synchronized to a slave to prevent data loss (the requirement is that its slave node must be > = 1)
min‐replicas‐to‐write 1


Optimization in application

# (1) Query execution time refers to the time spent on executing a query command, excluding IO operations such as client response (talking) and sending reply
redis> SLOWLOG LEN   # Manage the slow logs of redis and view the current number of logs 
redis> SLOWLOG RESET # Clear the slowlog, and the upper LEN becomes 0

# (2) Disconnect
# List all connected clients
redis 127.0.0.1:6379> CLIENT LIST
addr=127.0.0.1:43501 fd=5 age=10 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client
# Kill the connection of the current client
redis 127.0.0.1:6379> CLIENT KILL 127.0.0.1:43501
OK

Original link: https://blog.weiyigeek.top/20...

It's not easy to write this article. If you think this article is good, please give this column [point a praise, put a coin, collect a collection, close a note, and turn a hair] (the five great feelings in the world), which will affirm me. Thank you!.

Source of this article My Blog site or WeiyiGeek public account as well as My BiliBili column (please mail me for technical exchange and friend chain exchange). Thank you for your support! (๑′ᴗ‵๑) ❤
Welcome all like-minded friends to study and exchange together. If the article is wrong, please leave your valuable knowledge and suggestions through email[ master#weiyigeek.top ]Contact me!

Tags: security Operation & Maintenance Redis

Posted by paulytrick on Thu, 14 Apr 2022 10:54:30 +0930

Popular Tags

©2023 Programs Wiki