Contents of this chapter
0x00 Redis performance indicator monitoring
(1) Performance index
1. Basic activity indicator: basic activity
2. Performance index:
3. Memory index: memory
4. Persistence indicator
5. Error indicator: error
6. Description of other indicators
(2) Performance test tool
1. Redis benchmark command
2.redisbench tool
3.rdb memory analysis tool
(3) Benchmarking practices
3.1 single instance redis test in k8s
0x01 Redis security optimization
1.Security
Non privileged operation
file right
Interface binding
Change default service port
Authentication configuration
Disable specific commands
Logging
Prevent string escape and NoSQL injection
Protect against attacks triggered by carefully selected inputs from external clients
Firewall restricted access
It is forbidden to store sensitive plaintext data in redis
Redis security configuration summary example
2.Performance Optimization
Key optimization items
Redis performance optimization summary example
Pre knowledge learning supplement
1. Introduction and installation of redis database foundation- https://blog.weiyigeek.top/20...
2. Introduction and use of basic data types of redis database- https://blog.weiyigeek.top/20...
3. Principle introduction and master-slave configuration of redis basic operation and maintenance- https://blog.weiyigeek.top/20...
4. Sentinel and cluster installation and configuration of redis basic operation and maintenance- https://blog.weiyigeek.top/20...
5. Installation and configuration of redis basic operation and maintenance in K8S- https://blog.weiyigeek.top/20...
7. Actual combat of redis database disaster recovery and backup enterprise- https://blog.weiyigeek.top/20...
8.Redis database client operation practice and pit entry and exit- https://blog.weiyigeek.top/20...
0x00 Redis performance indicator monitoring
(1) Performance index
Common indicators and parameters of Redis server:
redis-cli -a password info > redis-Performance.txt # We can export the info related information of redis server to a file # 2.clients: # 3.memory: # 4.persistence: # 5.stats: General Statistics # 6.Replication: # 7.CPU: CPU usage # 8.cluster: # 9.Keypass: key value pair statistics quantity information 10.20.172.108:6379> info # (1) Interactive viewing of redis server information # Environment parameters for Server operation redis_version:6.2.5 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:cb556a016f8668d redis_mode:standalone os:Linux 5.11.0-25-generic x86_64 arch_bits:64 multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:9.3.0 process_id:187640 process_supervised:no run_id:97838216d4fe0de4739e7814b5a2e1d0d32d0982 tcp_port:6379 server_time_usec:1630241617439942 uptime_in_seconds:10930 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:2851665 executable:/opt/databases/redis-6.2.5/src/./redis-server config_file:/home/weiyigeek/redis/6379/redis-6379.conf io_threads_active:0 # Clients client related information connected_clients:7 cluster_connections:0 maxclients:10000 client_recent_max_input_buffer:32 client_recent_max_output_buffer:0 blocked_clients:0 tracking_clients:0 clients_in_timeout_table:0 # Memory server running memory statistics used_memory:2050432 used_memory_human:1.96M used_memory_rss:5140480 used_memory_rss_human:4.90M used_memory_peak:2253512 used_memory_peak_human:2.15M used_memory_peak_perc:90.99% used_memory_overhead:1982152 used_memory_startup:810376 used_memory_dataset:68280 used_memory_dataset_perc:5.51% allocator_allocated:2204376 allocator_active:2555904 allocator_resident:5230592 total_system_memory:12442619904 total_system_memory_human:11.59G used_memory_lua:37888 used_memory_lua_human:37.00K used_memory_scripts:0 used_memory_scripts_human:0B number_of_cached_scripts:0 maxmemory:0 maxmemory_human:0B maxmemory_policy:noeviction allocator_frag_ratio:1.16 allocator_frag_bytes:351528 allocator_rss_ratio:2.05 allocator_rss_bytes:2674688 rss_overhead_ratio:0.98 rss_overhead_bytes:-90112 mem_fragmentation_ratio:2.59 mem_fragmentation_bytes:3153776 mem_not_counted_for_evict:124 mem_replication_backlog:1048576 mem_clients_slaves:0 mem_clients_normal:123000 mem_aof_buffer:128 mem_allocator:jemalloc-5.1.0 active_defrag_running:0 lazyfree_pending_objects:0 lazyfreed_objects:0 # Persistence persistent data related information loading:0 current_cow_size:0 current_cow_size_age:0 current_fork_perc:0.00 current_save_keys_processed:0 current_save_keys_total:0 rdb_changes_since_last_save:3 rdb_bgsave_in_progress:0 rdb_last_save_time:1630230687 rdb_last_bgsave_status:ok rdb_last_bgsave_time_sec:-1 rdb_current_bgsave_time_sec:-1 rdb_last_cow_size:0 aof_enabled:1 aof_rewrite_in_progress:0 aof_rewrite_scheduled:0 aof_last_rewrite_time_sec:0 aof_current_rewrite_time_sec:-1 aof_last_bgrewrite_status:ok aof_last_write_status:ok aof_last_cow_size:462848 module_fork_in_progress:0 module_fork_last_cow_size:0 aof_current_size:150 aof_base_size:92 aof_pending_rewrite:0 aof_buffer_length:0 aof_rewrite_buffer_length:0 aof_pending_bio_fsync:0 aof_delayed_fsync:0 # Stats general statistics information total_connections_received:25 total_commands_processed:50482 instantaneous_ops_per_sec:4 total_net_input_bytes:2758703 total_net_output_bytes:22330756 instantaneous_input_kbps:0.23 instantaneous_output_kbps:0.55 rejected_connections:0 sync_full:0 sync_partial_ok:0 sync_partial_err:0 expired_keys:0 expired_stale_perc:0.00 expired_time_cap_reached_count:0 expire_cycle_cpu_milliseconds:0 evicted_keys:0 keyspace_hits:1 keyspace_misses:0 pubsub_channels:1 pubsub_patterns:0 latest_fork_usec:310 total_forks:1 migrate_cached_sockets:0 slave_expires_tracked_keys:0 active_defrag_hits:0 active_defrag_misses:0 active_defrag_key_hits:0 active_defrag_key_misses:0 tracking_total_keys:0 tracking_total_items:0 tracking_total_prefixes:0 unexpected_error_replies:0 total_error_replies:8 dump_payload_sanitizations:0 total_reads_processed:48899 total_writes_processed:97139 io_threaded_reads_processed:0 io_threaded_writes_processed:0 # Replication master-slave related indicator information role:master connected_slaves:0 master_failover_state:no-failover master_replid:32da05299a5a36de431b4c05122f7d2b93eca169 master_replid2:3e15749ad586d60bd0d1c93854f6f719a22316ce master_repl_offset:8915 second_repl_offset:829 repl_backlog_active:1 repl_backlog_size:1048576 repl_backlog_first_byte_offset:1 repl_backlog_histlen:8915 # CPU processor mode occupancy information used_cpu_sys:12.012184 used_cpu_user:9.453505 used_cpu_sys_children:0.002158 used_cpu_user_children:0.000000 used_cpu_sys_main_thread:11.802969 used_cpu_user_main_thread:9.286577 # Modules module loading # Errorstats error status information errorstat_NOAUTH:count=6 errorstat_READONLY:count=1 errorstat_WRONGPASS:count=1 # Cluster cluster information cluster_enabled:0 # Keyspace library id and key quantity related information db0:keys=1,expires=0,avg_ttl=0
1. Basic activity indicator: basic activity
Name | Description |
---|---|
connected_clients | Number of client connections |
conected_laves | Number of slave |
master_last_io_seconds_ago | The number of seconds since the last master-slave interaction |
keyspace | Total number of key values in the database |
grep -En "connected_clients|conected_laves|master_last_io_seconds_ago|keyspace" redis-Performance.txt 27:connected_clients:7 # # Number of client connections 28:connected_slaves:1 # # Number of slave connections 128:keyspace_hits:1 129:keyspace_misses:0
2. Performance index:
Name | Description |
---|---|
latency | Redis response time to a request |
instantaneous_ops_per_sec | Average total number of requests processed per second |
hi rate(calculated) | Calculated hit rate |
grep -En "latency|instantaneous_ops_per_sec|hi rate" redis-Performance.txt 114:instantaneous_ops_per_sec:3
3. Memory index: memory
Name | Description |
---|---|
used_memory | Memory used |
mem_fragmentation_ratio | Memory fragmentation rate |
evicted_keys | Maximum number of key s removed due to memory limitation |
blocked_clients | Clients blocked due to BLPOP,BRPOP,or BRPOPLPUSH |
grep -En "used_memory|mem_fragmentation_ratio|evicted_keys|blocked_clients" redis-Performance.txt 32:blocked_clients:0 37:used_memory:2050432 38:used_memory_human:1.96M # # The total amount of memory allocated by the memory allocator from the operating system 39:used_memory_rss:5234688 40:used_memory_rss_human:4.99M # # The memory occupation seen by the operating system and the memory occupation seen by the top command 41:used_memory_peak:2253512 42:used_memory_peak_human:2.15M # # Peak of redis memory consumption 43:used_memory_peak_perc:90.99% 44:used_memory_overhead:1982152 45:used_memory_startup:810376 46:used_memory_dataset:68280 47:used_memory_dataset_perc:5.51% 53:used_memory_lua:37888 54:used_memory_lua_human:37.00K # # Amount of memory occupied by lua script engine 55:used_memory_scripts:0 56:used_memory_scripts_human:0B 67:mem_fragmentation_ratio:2.63 127:evicted_keys:0
4. Persistence indicator
Name | Description |
---|---|
rdb_last_save_time | Timestamp of the last persistent save disk |
rdb_changes_sice_last_save | The number of changes to the database since the last persistence |
grep -En "rdb_last_save_time|rdb_changes_sice_last_save" redis-Performance.txt 88:rdb_last_save_time:1630230687 89:rdb_changes_since_last_save:0 # The number of changes to the database since the last persistence
5. Error indicator: error
Name | Description |
---|---|
rejected_connections | The number of connections rejected due to maxclient limit reached |
keyspace_misses | Number of key value lookup failures (misses) |
master_link_down_since_seconds | Duration of master-slave disconnection (in seconds) |
grep -En "rejected_connections|master_link_down_since_seconds" redis-Performance.txt 9:master_link_down_since_seconds:10937 119:rejected_connections:0 keyspace_misses:0 # The number of times the key value search fails (misses). It may be attacked by the hacker
<br/>
6. Description of other indicators
# 1. If the copy backlog buffer is set too small, the instructions in it will be overwritten and the offset will not be found, thus triggering full synchronization repl_backlog_size: 1048576 # 2. By viewing sync_ partial_ The err variable determines whether the backlog buffer needs to be expanded. It indicates the number of master-slave semi synchronous replication failures sync_partial_err:1
(2) Performance test tool
1. Redis benchmark command
Description: redis performance test is realized by executing multiple commands at the same time, which are executed under the redis directory;
Official website reference: https://redis.io/topics/bench...
Syntax parameters:
Usage: redis-benchmark [-h <host>] [-p <port>] [-c <clients>] [-n <requests]> [-k <boolean>] # Parameter Description: -h <hostname> Server hostname (default 127.0.0.1) -p <port> Server port (default 6379) -s <socket> Server socket (overrides host and port) -a <password> Password for Redis Auth -c <clients> Number of parallel connections (default 50) -n <requests> Total number of requests (default 100000) -d <size> Data size of SET/GET value in bytes (default 2) --dbnum <db> SELECT the specified db number (default 0) -k <boolean> 1=keep alive 0=reconnect (default 1) -r <keyspacelen> right SET/GET/INCR Use random keys to SADD Use random valueuse this option to extend strings within parameters_rand_int _uu),The specified range of this parameter is 0 to keyspacelen-1 12 digits between. -P <numreq> Pipeline <numreq> requests. Default 1 (no pipeline). -q Quiet. Just show query/sec values --csv Output in CSV format -l Loop. Run the tests forever -t <tests> Only run the comma separated list of tests. The test names are the same as the ones produced as output. -I Idle mode. Just open N idle connections and wait.
Basic instance:
# (1) 10000 requests are executed at the same time to test the performance (all default tests). The - q parameter allows the result to display only the number of requests executed per second $ ./redis-benchmark -h 127.0.0.1 -p 6379 -t set,lpush -n 10000 -q $ ./redis-benchmark -n 10000 -q # PING_INLINE: 41493.78 requests per second # PING_BULK: 44843.05 requests per second # SET: 42194.09 requests per second # GET: 44052.86 requests per second # INCR: 43290.04 requests per second # LPUSH: 42194.09 requests per second # RPUSH: 42372.88 requests per second # LPOP: 42194.09 requests per second # RPOP: 42194.09 requests per second # SADD: 43668.12 requests per second # HSET: 42372.88 requests per second # SPOP: 44843.05 requests per second # LPUSH (needed to benchmark LRANGE): 42553.19 requests per second # LRANGE_100 (first 100 elements): 21367.52 requests per second # LRANGE_300 (first 300 elements): 9451.80 requests per second # LRANGE_500 (first 450 elements): 6807.35 requests per second # LRANGE_600 (first 600 elements): 5350.46 requests per second # MSET (10 keys): 36363.64 requests per second # (2) Run the test of the specified project. For example, we require only the SET and LPUSH commands to be run in quiet mode $ redis-benchmark -t set,lpush -n 100000 -q # SET: 74239.05 requests per second # LPUSH: 79239.30 requests per second # (3) Specify the eval script command for benchmarking $ redis-benchmark -n 100000 -q script load "redis.call('set','foo','bar')" # script load redis.call('set','foo','bar'): 69881.20 requests per second # (4) Select the size of the key space. By default, the benchmark runs for a single key, and we can usually simulate a more real workload space by using large keys. # For example, if I want to run 1 million SET operations and use a random key for each operation among 100000 possible keys, $ redis-cli flushall $ redis-benchmark -t set -r 100000 -n 1000000 # (5) By default, each client only sends the next command when receiving the reply from the previous command. Redis supports pipeline, so it can send multiple commands at one time, which can be imagined as parallel. # For example: run the benchmark in MacBook Air 11 "using a pipeline of 16 commands redis-benchmark -n 1000000 -t set,get -P 16 -q # SET: 403063.28 requests per second # GET: 508388.41 requests per second # (6) Benchmarking using Unix domain sockets $ numactl -C 6 ./redis-benchmark -q -n 100000 -s /tmp/redis.sock -d 256 # (7) Using TCP loopback $ numactl -C 6 ./redis-benchmark -q -n 100000 -d 256
Benchmark comparison between Redis and Memcached in memory databases:
#!/bin/bash # BIN=./redis-benchmark BIN=./mc-benchmark payload=32 iterations=100000 keyspace=100000 for clients in 1 5 10 20 30 40 50 60 70 80 90 100 200 300 do SPEED=0 for dummy in 0 1 2 do S=$($BIN -n $iterations -r $keyspace -d $payload -c $clients | grep 'per second' | tail -1 | cut -f 1 -d'.') if [ $(($S > $SPEED)) != "0" ] then SPEED=$S fi done echo "$clients $SPEED" done
Finally, the following are the results in graphical form generated using gnuplot:
Factors affecting benchmarking
- 1) Workload (number of connected clients)
- 2) Different versions of Redis
3) Physical configuration of servers providing services (disk, network, CPU, memory). On multi CPU slot servers, Redis performance depends on NUMA configuration and process location.
- The test was completed by 50 clients executing 2 million requests at the same time.
- Use the loopback interface to perform the test.
- The test is performed using a key space of 1 million keys.
The test is executed with and without pipelines (16 command pipelines).
Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (with pipeline) / (without pipeline)
$ ./ Redis benchmark - R 1000000 - N 2000000 - t get, set, lpush, lpop - p 16 -q# better than no pipeline.
SET: 552028.75 requests per second
GET: 707463.75 requests per second
LPUSH: 767459.75 requests per second
LPOP: 770119.38 requests per second$ ./redis-benchmark -r 1000000 -n 2000000 -t get,set,lpush,lpop -q
SET: 122556.53 requests per second
GET: 123601.76 requests per second
LPUSH: 136752.14 requests per second
LPOP: 132424.03 requests per second- 4) Compared with using the same hardware without virtualization, Redis runs slower on the VM (Redis is recommended as the preferred physical machine)
- 5) Depending on the platform, unix domain sockets can achieve about 50% more throughput than TCP/IP loopback (for example, on Linux).
- 6) Compared with TCP/IP loopback, the performance advantage of Unix domain socket tends to decrease when a large number of pipelines (i.e. long pipelines) are used.
- 7) When using Ethernet network to access Redis, when the data size remains below the Ethernet packet size (about 1500 bytes), the pipelined aggregation command is particularly effective. When processing 10 byte, 100 byte or 1000 byte queries, it will produce almost the same throughput.
2.redisbench tool
Description: the officially recommended redis benchmark has no way to specify the cluster mode during cluster benchmark testing. Redis & redis cluster benchmark tool is introduced here to facilitate the processing of cluster benchmark testing. Download address: https://github.com/panjiang/r...
redisbench features
- Developed and built with Golang
- redis single instance can be tested
- redis cluster can be tested
- Multiple cores can be used
- It supports running on multiple machines at the same time and is used to test large redis clusters (the same machine hardware is required)
Format syntax:
./redisbench -h -a string #Redis instance address or Cluster addresses. IP:PORT[,IP:PORT] -c int #Clients number for concurrence (default 1) -cluster #true: cluster mode, false: instance mode -d int #Data size in bytes (default 1000) -ma string #addresses for run multiple testers at the same time -mo int #the order current tester is in multiple testers -n int #Testing times at every client (default 1)
Basic example:
# Test single instance mode ./redisbench -a 127.0.0.1:6379 -c 200 -n 20000 -d 3 # Test cluster ./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 # Use multiple test nodes ./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 1 & ./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 2 & ./redisbench -cluster=true -a 192.168.1.11:6379,192.168.1.11:6380 -c 500 -n 2000 -d 3 -ma 192.168.1.11:9001,192.168.1.11:9002,192.168.1.11:9003 -mo 3
Tips: the test results will be automatically printed out: request value, request time, TPS is not actually demonstrated here. Interested friends can download the test by themselves.
<br/>
3.rdb memory analysis tool
Description: RDR is a tool for parsing redis rdbfile. Compared with redis RDB tools, RDR is implemented by golang and is faster.
- Analyze which Key value in Redis memory occupies the most memory
- It is analyzed that the Key at the beginning of Redis memory occupies the most, which is conducive to memory optimization
- Redis Key value is displayed in Dashboard, which is more intuitive
Installation and download address: https://github.com/xueqiu/rdr...
matters needing attention:
- 1. Add the executable permission Chmod + X RDR before using Linux and windows_ linux
Basic syntax:
# RDR parameter interpretation show Web page display rdbfile Statistics for keys from rdbfile Get all key help help
Basic instance (take Linux as an example):
#1. Analyze and count the usage proportion of various types in multiple redis RDBS ./rdr-linux keys *.rdb SEARCH:PROJECTS_BY_ID:68 SEARCH:PROJECTS_BY_ID:64 SEARCH:PROJECTS_BY_PARENTID_LIST:0 SEARCH:PROJECTS_BY_PARENTID_LIST:64 SEARCH:PROJECTS_BY_PARENTID_LIST:66 SEARCH:PROJECTS_BY_PARENTID_LIST:68 SEARCH:PROJECTS_BY_ID:66 SEARCH:PROJECTSINFO_BY_PARENTID_LIST:64 ALLOW:SEARCH_BY_SFZH:230103197805153637 ./rdr-linux dump dump.rdb {"CurrentInstance": "dump.rdb", "LargestKeyPrefixes": { "list": [ { "Type": "list", "Key": "site", "Bytes": 144, "Num": 1 } ], #2. The web page displays the analysis results ./rdr-linux show -p 8080 *.rdb start parsing... parse dump.rdb done parsing finished, please access http://{$IP}:8080
(3) Benchmarking practices
3.1 single instance redis test in k8s
Environmental description:
# Host node running the Pod osImage: Ubuntu 20.04.1 LTS kernelVersion: 5.4.0-42-generic kubeProxyVersion: v1.19.6 kubeletVersion: v1.19.6 containerRuntimeVersion: docker://19.3.14 $ Server Relevant information redis_version:6.2.5 os:Linux 5.4.0-42-generic x86_64 arch_bits:64 gcc_version:10.3.1 process_id:1 process_supervised:no tcp_port:6379 hz:10 configured_hz:10 lru_clock:3627023 io_threads_active:0 $ redis Configured memory limit maxmemory:1073741824 maxmemory_human:1.00G maxmemory_policy:volatile-lru $ cpu Relevant information Model name: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz Physics CPU number: 1 logic CPU number: 4 CPU Number of cores: 4
<br/>
Benchmark:
# Test 1 Execute the set command and get command 10 million times, in which the size of each read is 256 bytes, and the number of requesting clients is 50 by default. ~$ redis-benchmark -h 10.102.39.181 -a 123456 -d 256 -t set,get -n 10000000 -q SET: 42445.36 requests per second GET: 49504.70 requests per second # -Pod resource peak during test very 1.0s: kubectl top pod -n database redis-cm-0 Tue Sep 7 20:36:50 2021 NAME CPU(cores) MEMORY(bytes) redis-cm-0 848 18Mi # Test 2 Similarly, execute the set command and get command for 10 million times, in which the size of each read is 256 bytes. The only difference is that pipelined-P 16 is used for testing (it can be seen that the number of set and get requests per second is significantly increased). ~$ redis-benchmark -h 10.102.39.181 -a 123456 -d 256 -t set,get -n 10000000 -P 16 -q SET: 96019.98 requests per second GET: 316575.91 requests per second # -Peak Pod resources during test very 1.0s: kubectl top pod -n database redis-cm-0 Tue Sep 7 20:46:50 2021 NAME CPU(cores) MEMORY(bytes) redis-cm-0 457m 337Mi
<br/>
Practical test
1) Insert 100000 data through shell pipe and redis pipe for comparison
## Shell-pipe.sh #!/bin/bash echo "start time: $(date +%s)" for ((i=0;i<100000;i++));do echo -en "helloworld-redis-${i}" | redis-cli -h 10.102.39.181 --no-auth-warning -a 123456 -x set username${i} >> ok.txt done echo "Completion time: $(date +%s)" ## redis-pipe.sh #!/bin/bash echo "start time: $(date +%s)" python3 redis-set.py >> set-command.txt cat set-command.txt | redis-cli -h 10.102.39.181 -a 123456 --no-auth-warning --pipe echo "Completion time: $(date +%s)" ## redis-set.py tee redis-set.py <<'EOF' #!/usr/bin/python for i in range(100000): print('set name'+str(i)+' helloworld-redis-'+str(i)) EOF
2) The execution efficiency of script insertion is recorded by using the time command
## Insert data in redis pipe mode ~/k8s/benchmark$ time ./redis-pipe.sh # Start time: 1631020862 # All data transferred. Waiting for the last reply... # Last reply received from server. # errors: 0, replies: 100000 # Completion time: 1630862 # real 0m0.466s # user 0m0.126s # sys 0m0.035s # Keyspace db0:keys=100000,expires=0,avg_ttl=0 ## Insert data in shell pipe mode ~/k8s/benchmark$ time ./shell-pip.sh # Start time: 1631021312 # Completion time: 1631021921 # real 10m9.265s # Total time spent from the beginning to the end of the program (including CPU). # user 3m44.411s # The program itself and the time of calling the subprocess. # sys 1m55.435s # The execution time of the system call called by the program itself or indirectly. # Keyspace db0:keys=200000,expires=0,avg_ttl=0
Tips: it can be seen from the above results that the total time consumption of real of the two methods is very different. The efficiency of redis pipe method is not an order of magnitude compared with that of ordinary shell pipe method, so try to use redis pipe method to submit data in the development program.
# In order to facilitate the follow-up demonstration, grip inserted 80W pieces of data into the database in about 4s. start time: 1631022423 All data transferred. Waiting for the last reply... Last reply received from server. errors: 0, replies: 800000 Completion time: 1631022427 real 0m4.023s user 0m0.885s sys 0m0.187s
0x01 Redis security optimization
1.Security
Description: Redis provides access control, code security issues, and selecting attacks that can be triggered from outside by malicious input. Our operation and maintenance personnel need to configure them accordingly to improve security.
Non privileged operation
Description: redis can run without root permission. It is recommended to run it as a non privileged redis user only for this purpose. This way can prevent the possibility of CONFIG SET/GET directory and other similar runtime configuration instructions to the greatest extent..
#It is necessary to set up a separate redis account. redis crackit takes advantage of the root user feature to reset authorized_keys. First create a redis account, and then start it through this account. $ useradd redis $ setsid sudo -u redis redis-server /etc/redis.conf $ ps -elf|grep redis #You can see that the redis user started # 4 S root 9048 1 0 80 0 - 59753 poll_s 19:43 ? 00:00:00 sudo -u redis redis-server /etc redis.conf # 4 S redis 9049 9048 0 80 0 - 38471 ep_pol 19:43 ? 00:00:00 redis-server
file right
Description: because the redis password is stored in clear text in the configuration file, we need to restrict the access permission of the redis file directory. For example, set the permission of the redis home directory to 700(rwx ------), if redis The conf configuration file is independent of the redis home directory, and the permission is modified to 600(rw -------).
# file right chmod 700 /opt/redis/redis-5.0.4/ chmod 600 /etc/redis.conf # Owner and group chown redis:redis /etc/redis.conf chown redis:redis /opt/redis/redis-5.0.4/
<br/>
Interface binding
Description: except for the trusted clients in the network, everyone should refuse to access the Redis port. Therefore, the server running Redis should only be directly accessed by the computer using Redis to implement the application.
If the server has two network interfaces (one area a and one area B), if only the machine access in area a is required, it will only be bound to the network interface in area A. if the server itself accesses, it will only be bound to the local loopback interface.
# Through Redis Add the following line to the conf file to bind Redis to a single interface: bind 127.0.0.1 192.168.1.200
Tips: note that you can bind IPV6 in addition to IPV4
Change default service port
Description: in addition to specifying the binding interface, we can also change the default redis service port to prevent hackers from scanning and detecting redis services.
# Change the default service disconnect from 6379 to 63791 port 63791
<br/>
Authentication configuration
Description: it is very necessary to set an authentication password for Redis server. The following is a summary of several ways Redis configures password authentication:
Operation process:
# 1. Through redis Conf file. Redis needs to be restarted after this method is modified. vim /etc/redis.conf requirepass WeiyiGeek # WeiyiGeek is the authentication password masterauth WeiyiGeek # Configure the authentication password of the master node. Note that if the master is configured with a password, the slave must also configure the corresponding password parameters, otherwise it cannot be copied normally # 2. Configure through the command line. This method has the advantage of no need to restart Redis. redis 127.0.0.1:6379[1]> config set requirepass my_redis OK redis 127.0.0.1:6379[1]> config get requirepass 1) "requirepass" 2) "my_redis"
<br/>
Log in to Redis server with password authentication:
# Method 1: password plaintext will be recorded in the system command execution history (extremely not recommended / unsafe) redis-cli -h 127.0.0.1 -p 6379 -a WeiyiGeek # Mode 2: interactive configuration redis-cli -h 127.0.0.1 -p 6379 redis 127.0.0.1:6379> auth WeiyiGeek # OK
Note: the AUTH command, like all other Redis commands, is sent in an unencrypted manner, so it cannot prevent attackers who have sufficient access to the network to eavesdrop. Therefore, for highly sensitive data, it is recommended to configure TLS support (Redis optionally supports TLS on all communication channels) to encrypt data and command transmission.
<br/>
Disable specific commands
Description: we can disable the commands in Redis or rename them to unpredictable names, so that ordinary clients are limited to a specified set of commands. For example, the vulnerability uses the two commands of config/save to complete the attack.
Since redis has no user permission restrictions, it is recommended to disable or rename dangerous commands using the rename configuration item, so that attackers who do not know the renaming rules cannot execute such commands FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, DEBUG, EVAL
For example, ordinary users may not be able to call the Redis CONFIG command to change the instance configuration, but the system providing and deleting instances should be able to do so.
# redis.conf configuration file # Mode 1 The config / flush command is renamed to an unpredictable name rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52 rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c53 # Mode 2 Disable it completely (or any other command) by renaming it to an empty string rename-command CONFIG "" rename-command FLUSHALL ""
Tips: note that the redis server service needs to be restarted after configuration.
Logging
Description: create an access (or Debug) log for redis (set according to requirements). When establishing redis honeypot, if there is an attack attempt, it will find and monitor redis security status in time, and can monitor cmdstat_* Indicator information alarm;
# Execute info commandstats to see the number of times the command is executed, the CPU time consumed by the command (unit: ms), and the average CPU time consumed by executing each command (unit: ms) cmdstat_get:calls=2,usec=15,usec_per_call=7.50 cmdstat_select:calls=1,usec=9,usec_per_call=9.00 cmdstat_keys:calls=4,usec=1948,usec_per_call=487.00 cmdstat_auth:calls=3123,usec=8291,usec_per_call=2.65
Logging configuration:
logfile "/usr/local/redis/redis.log" #Log file storage directory loglevel verbose #Record access information
Prevent string escape and NoSQL injection
Description: Redis protocol does not have the concept of string escape, so it is generally impossible to inject using ordinary client libraries, but it is possible to construct malicious scripts through Lua scripts executed by EVAL and EVALSHA commands.
> eval "return {KEYS[1],KEYS[2],ARGV[1],ARGV[2]}" 2 key1 key2 first second
Solution: applications should avoid using strings obtained from untrusted sources to form the body of Lua scripts.
Tips: evalsha evaluates cached server-side scripts through its SHA1 summary. Scripts are cached on the server side using the SCRIPT LOAD command. This command is the same as EVAL in other respects.
<br/>
Protect against attacks triggered by carefully selected inputs from external clients
Description: it is possible for attackers to construct malicious data structures and insert them into Redis database, which may trigger the ill conditioned (worst-case) algorithm complexity of data structures implemented inside Redis.
For example, an attacker can provide a set of strings known to be hashed into the same bucket to the hash through a Web form, so as to turn the expected time (average time) of O(1) into the worst case of O(N), consume more CPU than expected, and eventually lead to a denial of service.
Solution: in order to prevent this specific attack, Redis uses the pseudo-random seed of each execution for the hash function.
<br/>
Firewall restricted access
Description: the security configuration was made for the redis server service level earlier. Here, restrictions are made for the network level to allow access only to the specified IP address. The advantage of configuring a firewall on the host is to prevent east-west traffic in the same network segment.
Firewall setting command on Linux system:
iptables -A INPUT -s x.x.x.x -p tcp --dport 6379 -j ACCEPT #If you need access from other machines or set the slave mode, remember to add the corresponding firewall settings (Centos6) firewall-cmd --add-rich-rule="rule family="ipv4" source address="x.x.x.x" port protocol="tcp" port="6379" accept" --permanent #(Centos7)
System firewall setting command on Windows:
New-NetFirewallRule -Name "redis-server-access" -DisplayName "redis-server" -Description "redis-server Client access firewall rules" -Direction Inbound -LocalPort 6379 -RemoteAddress x.x.x.x -Protocol TCP -Action Allow -Enabled True Get-NetFirewallRule -Name "redis-server-access" | Format-Table
It is forbidden to store sensitive plaintext data in redis
Description: Redis is designed to provide high-performance KV services. At least at present, it is relatively weak in access control and data persistence. Therefore, from the application level, it is not recommended to use Redis to store sensitive information, such as authentication passwords.
Redis security configuration summary example
Security configuration example:
# Configuration file VIM / etc / redis / redis conf # 1. Operate the trusted intranet and try to avoid public network access (if there are other fixed IP addresses in the intranet, a firewall needs to be set) bind 127.0.0.1 # 2. Bind the network interface monitored by redis (multiple IPS can be bound at the same time through the redis configuration item bind), and change 6379 to another port (or use unix pipeline for data management) port 63791 # 3. Enable redis password authentication and set high complexity password settings. Due to the high query efficiency, auth commands can be processed more than 10w times per second (so the intensity needs to be increased) # echo -e "weiyigeek"|sha256sum requirepass 097575a79efcd7ea7b1efa2bcda78a4fc7cbd0820736b2f2708e72c3d21f8b61 # 4. Log file storage directory and record redis access information. # debug (a lot of information, useful for development/testing) # verbose (many rarely useful info, but not a mess like the debug level) # Notice (modestly verbose, what you want in production probability) default # warning (only very important / critical messages are logged) logfile "/usr/local/redis/redis.log" loglevel verbose # 5. Protection mode is enabled by default. (no) it should only be disabled if # -Are you sure you want clients of other hosts to connect to Redis # -Even if authentication is not configured, there is no specific set of interfaces # -Use the "bind" instruction to explicitly list. protected-mode yes # 6. Rename special commands (as required) # `FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, DEBUG, EVAL` rename-command CONFIG b840fc02d524045429941cc15f59e41cb7be6c52 rename-command FLUSHDB b840fc02d524045429941cc15f59e41cb7be6c53 rename-command FLUSHALL b840fc02d524045429941cc15f59e41cb7be6c54 rename-command EVAL b840fc02d524045429941cc15f59e41cb7be6c55 rename-command DEBUG b840fc02d524045429941cc15f59e41cb7be6c56 rename-command SHUTDOWN b840fc02d524045429941cc15f59e41cb7be6c7
2.Performance Optimization
Description: Redis development and operation and maintenance personnel pay more attention to some configuration optimization of Redis itself, such as AOF and RDB configuration optimization, data structure configuration optimization, etc., but they don't know or care about whether the operating system needs to do some configuration optimization for Redis. However, facts have proved that a good system operation configuration can escort the good operation of Redis services.
Key optimization items
Step 1.vm.overcommit_memory best practices
Such logs may appear when Redis is started, and then figure out what is overcommit?
Description: the Linux operating system replies yes to most requests for memory in order to run more programs. Because memory will not be used immediately after applying for memory, this technology is called overcommit.# If Redis has the above log at startup, it means ` VM overcommit_ Memory = 0 `, Redis prompts to set it to 1. # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl -w vm.overcommit_memory=1' for this to take effect. # Note: the available memory in this article represents the sum of physical memory and swap.
- Redis suggests setting this value to 1 so that fork can execute successfully under low memory (set a reasonable maxmemory to ensure that the machine has 20% ~ 30% idle memory).
Centralized management of aof rewriting and rdb bgsave.
Tips : In the log Background save Represents bgsave and bgrewriteaof,What should the operating system do if the currently available memory is insufficient fork. If vm.overcommit_memory=0,Means that if there is no available memory, the memory application fails, corresponding to Redis namely fork Execution failed at Redis Your log will appear:`Cannot allocate memory `
<br/>
- Step 2.vm.swapniess best practices
Description: swap is important for the operating system. When the physical memory is insufficient, you can swap out some memory pages to solve the urgent problem. However, there is no free lunch in the world. Swap space is provided by hard disk. For applications requiring high concurrency and high throughput, disk IO will usually become a system bottleneck. In Linux, it is not necessary to wait until all physical memory is used up before using swap. The system parameter swppiness determines the tendency of the operating system to use swap. The value range of swap is 0 ~ 100. The larger the value of swap, the higher the probability that the operating system may use swap, and the lower the value of swap, indicating that the operating system prefers to use physical memory.
If Linux > 3.5, VM Swapniess = 1 (prefer swap to OOM killer), otherwise VM Swapniess = 0 (prefer OOM killer to swap) to achieve the following two goals:
1. Make Redis fast enough when there is enough physical memory.
2. Avoid the death of Redis when the physical memory is insufficient (if Redis is currently highly available, death is better than blocking).
Operation and maintenance tips: OOM(Out Of Memory) killer mechanism means that when the Linux operating system finds that there is insufficient available memory, it forces to kill some user processes (non kernel processes) to ensure that the system has enough available memory for allocation.
<br/>
Step 3.kernel.mm.transparent_hugepage.enabled best practices
Redis may see the following logs when starting:WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
Tips: from the prompt, Redis suggests modifying the related configuration of transparent tiger pages (THP). The Linux kernel adds the transparent tiger pages (THP) feature in the 2.6.38 kernel, which supports the allocation of large memory pages (2MB), which is enabled by default. When enabled, the speed of fork child processes can be reduced, but after fork, each memory page changes from 4KB to 2MB, which will greatly increase the memory consumption of the parent process during rewriting. At the same time, the copy memory page unit caused by each write command is enlarged by 512 times, which will slow down the execution time of write operations and lead to a large number of slow queries of write operations.
Therefore, it is recommended to disable this feature in the Redis log. The disabling method is as follows: echo never > / sys / kernel / mm / transparent_ hugepage/enabled
<br/>
- Step 4.Transparent Huge Pages
Redis may see the following log when starting: warning you have transparent tiger pages (THP) support enabled in your kernel This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_ hugepage/enabled' as root, and add it to your /etc/rc. local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
From the prompt, Redis suggests modifying the relevant configuration of transparent tiger pages (THP). The Linux kernel adds the transparent tiger pages (THP) feature in the 2.6.38 kernel, which supports the allocation of large memory pages (2MB), which is enabled by default. When enabled, the speed of fork child processes can be reduced, but after fork, each memory page changes from 4KB to 2MB, which will greatly increase the memory consumption of the parent process during rewriting. At the same time, the copy memory page unit caused by each write command is enlarged by 512 times, which will slow down the execution time of write operations and lead to a large number of slow queries of write operations. For example, a simple incr command will also appear in slow queries. Therefore, it is recommended to disable this feature in Redis logs. The disabling methods are as follows:
# The THP configuration is still effective after the machine is restarted tee -a /etc/rc.local <<'EOF' echo never > /sys/kernel/mm/transparent_hugepage/enabled EOF
<br/>
- Step 5. Optimized configuration of oom killer
The OOM killer will selectively kill the user process when the available memory is insufficient. It will set a weight for each user process. The higher the weight, the higher the probability of being "attacked". On the contrary, the lower the probability. The weight of each process is stored in / proc / {progress_id} / oom_ In score, this value is controlled by / proc/{progress_id}/oom_adj control, OOM_ The minimum value of adj is different in different Linux versions. You can refer to oom in Linux source code H (from - 15 to - 17)
When OOM_ When adj is set to the minimum value, the process will not be killed by OOM killer. The setting method is as follows:
# command echo {value} > /proc/${process_id}/oom_adj # script for redis_pid in $(pgrep -f "redis-server") do echo -17 > /proc/${redis_pid}/oom_adj done
<br/>
Step 6. Set the number of open files, handles and the maximum number of processes per user
Description: the following parameters are mainly set to the maximum number of Linux file handles that can be used by a single process to avoid abnormal error reporting under high concurrency. Suggestions mentioned officially in Redis# You requested maxclients of 10000 requiring at least 10032 max file descriptors. First line: Redis It is suggested that open files At least set it to 10032, so how does this 10032 come from? because maxclients The default is 10000. These are used to handle client connections. In addition, Redis Up to 32 file descriptors will be used internally, so 10032 here = 10000 + 32. # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted. Second line: Redis Cannot open files Set to 10032 because it does not have permission to set. # Current maximum open files is 4096. Maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit –n'. Line 3: current system open files It's 4096, so maxclients Set to 4096-32=4064 If you want to set higher maxclients,Please use ulimit -n To set. From the above three lines of log analysis, we can see that open files Restricted priority ratio maxclients Big.
terms of settlement:
# temporary ulimit –Sn 10032 # permanent
- soft nofile 10032
- hard nofile 10032
- soft nproc 65535
hard nproc 65535
EOF
<br/>
Step 7.TCP backlog log queue optimization
Description: the default TCP backlog of Redis is 511, which can be adjusted by modifying the configuration of TCP backlog. If the TCP backlog of Linux is less than the TCP backlog set by Redis, the following logs will be seen when Redis is started:# WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
resolvent:
# see cat /proc/sys/net/core/somaxconn 128 # modify echo 511 > /proc/sys/net/core/somaxconn
<br/>
- Step 8. Ensure the consistency of Redis server clock
Description: we know that Redis Sentinel and Redis Cluster, which require multiple Redis instances, may involve multiple servers. Although Redis does not have strict requirements on the clocks of multiple servers, if the clocks of the servers of multiple Redis instances are inconsistent, it is very difficult to troubleshoot the logs of some abnormal situations, such as the failover of Redis Cluster. If the log times are different, it will cause great trouble for us to troubleshoot the problems (Note: it will not affect the cluster function, and the cluster nodes depend on their own clocks). Generally, NTP service will be provided in the company to provide standard time service, so as to achieve the effect of correcting the clock
For example, the NTP service is synchronized once every hour
0 * * * * /usr/sbin/ntpdate ntp.xx.com > /dev/null 2>&1
Redis performance optimization summary example
System optimization configuration
# -Set memory allocation policy sudo sysctl -w vm.overcommit_memory=1 # -Try to use physical memory (fast) for kernel version greater than > = 3 X (prefer swap to OOM killer) sudo sysctl -w vm.swapniess=1 # -Disable THP feature to reduce memory consumption echo never > /sys/kernel/mm/transparent_hugepage/enabled # -OOM killer feature optimization for redis_pid in $(pgrep -f "redis-server") do echo -17 > /proc/${redis_pid}/oom_adj done # -Set the number of open files, handles and the maximum number of processes per user tee etc/security/limits.conf <<'EOF' * soft nofile 10032 * hard nofile 10032 * soft nproc 65535 * hard nproc 65535 EOF # -SYN queue length setting this parameter can accommodate more networks waiting to connect. echo 511 > /proc/sys/net/core/somaxconn sudo sysctl -w net.ipv4.tcp_max_syn_backlog=2048 # -Synchronize time every hour 0 * * * * /usr/sbin/ntpdate ntp.xx.com > /dev/null 2>&1
Application configuration optimization
# Maximum number of client connections (it needs to be adjusted according to the actual situation, which is related to the open files of the system, and the number value is open files(10032) - 32) maxclients 10000 # Key items of cluster configuration optimization # Cluster timeout. If this time is set too small, the operation of reselecting the Master may be caused by network fluctuations cluster-node-timeout 5000 # After writing, the master node must be synchronized to a slave to prevent data loss (the requirement is that its slave node must be > = 1) min‐replicas‐to‐write 1
Optimization in application
# (1) Query execution time refers to the time spent on executing a query command, excluding IO operations such as client response (talking) and sending reply redis> SLOWLOG LEN # Manage the slow logs of redis and view the current number of logs redis> SLOWLOG RESET # Clear the slowlog, and the upper LEN becomes 0 # (2) Disconnect # List all connected clients redis 127.0.0.1:6379> CLIENT LIST addr=127.0.0.1:43501 fd=5 age=10 idle=0 flags=N db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=32768 obl=0 oll=0 omem=0 events=r cmd=client # Kill the connection of the current client redis 127.0.0.1:6379> CLIENT KILL 127.0.0.1:43501 OK
Original link: https://blog.weiyigeek.top/20...
It's not easy to write this article. If you think this article is good, please give this column [point a praise, put a coin, collect a collection, close a note, and turn a hair] (the five great feelings in the world), which will affirm me. Thank you!.
Source of this article My Blog site or WeiyiGeek public account as well as My BiliBili column (please mail me for technical exchange and friend chain exchange). Thank you for your support! (๑′ᴗ‵๑) ❤
Welcome all like-minded friends to study and exchange together. If the article is wrong, please leave your valuable knowledge and suggestions through email[ master#weiyigeek.top ]Contact me!