Redis主节点内存占用过高
0. 基本情况
Redis采用集群模式，560个主节点，主从比为1:1，单台机器上为16个节点。info memory观察到主节点A单个Redis内存used_memory_rss_human为9.2G（设置的maxmemory_human为6G）超过最大值设置。
1.查看客户端连接情况
[root@ A ~]# redis-cli -p 9001 info clients
# Clients
connected_clients:4705
client_longest_output_list:204928
client_biggest_input_buf:0
blocked_clients:0
通过client_longest_output_list发现有大量的输出排队，进一步观察所有的客户端均存在这个情况。
在A机器上刷客户端输出list：
CLIENT_A:9001# Clients
client_longest_output_list:491335
CLIENT_A:9002# Clients
client_longest_output_list:520840
CLIENT_A:9003# Clients
client_longest_output_list:486498
CLIENT_A:9004# Clients
client_longest_output_list:485387
CLIENT_A:9005# Clients
client_longest_output_list:480000
CLIENT_A:9006# Clients
client_longest_output_list:537211
CLIENT_A:9007# Clients
client_longest_output_list:487594
CLIENT_A:9008# Clients
client_longest_output_list:490037
CLIENT_A:9009# Clients
client_longest_output_list:478734
CLIENT_A:9010# Clients
client_longest_output_list:524717
CLIENT_A:9011# Clients
client_longest_output_list:487200
CLIENT_A:9012# Clients
client_longest_output_list:491687
CLIENT_A:9013# Clients
client_longest_output_list:483900
CLIENT_A:9014# Clients
client_longest_output_list:524557
CLIENT_A:9015# Clients
client_longest_output_list:497967
CLIENT_A:9016# Clients
client_longest_output_list:484250
2.输出内存占用不为0 的内容
[root@ A redis-node]# redis-cli -p 9001 client list |grep -v "omem=0"
id=843 addr=client_B:53475 fd=2248 name= age=582784 idle=0 flags=S db=0 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=456078 omem=7446059139 events=rw cmd=replconf
通过cmd可知是主从同步占用大量输出内存
3.检查统计值
有问题Client A的从client_B
[root@localhost 9001]# redis-cli -p 9001
127.0.0.1:9001> info stats
# Stats
total_connections_received:2
total_commands_processed:25
instantaneous_ops_per_sec:0
total_net_input_bytes:1280055129
total_net_output_bytes:36461
instantaneous_input_kbps:192.22
instantaneous_output_kbps:0.00
rejected_connections:0
正常的机器C的从Client D
[root@CLIENT D redis-node]# redis-cli -p 9001
127.0.0.1:9001> info Stats
# Stats
total_connections_received:10
total_commands_processed:23220212769
instantaneous_ops_per_sec:18281
total_net_input_bytes:3405817622262
total_net_output_bytes:59530419
instantaneous_input_kbps:2882.54
instantaneous_output_kbps:0.03
rejected_connections:0
输入速度差了10倍，其他配置均一样，怀疑是网络问题。
4.检查网络时延
机器A的从client_B
[root@A redis-node]# traceroute client_B
traceroute to client_B (client_B), 30 hops max, 60 byte packets
1 *.*.*.254 0.752 ms 0.809 ms 0.900 ms
2 *.*.*.133 2.164 ms 2.170 ms 2.157 ms
3 * * *
4 * * *
5 *.*.*.2 163.542 ms 163.940 ms 164.181 ms
6 client_B (client_B) 160.995 ms 156.202 ms 158.603 ms
Clinet C的从Client D
[root@SHJX-YSP-GKTZCQ-SEV-128-35 ~]# traceroute Client D
traceroute to Client D (Client D), 30 hops max, 60 byte packets
1 *.*.*.254 0.732 ms 0.799 ms 0.897 ms
2 *.*.*.133 3.147 ms 3.182 ms 3.168 ms
3 * * *
4 * * *
5 *.*.*.2 5.537 ms 5.788 ms 5.961 ms
6 Client D (Client D) 1.953 ms 1.960 ms 1.956 ms
测试机器A到机器D性能
[root@ 42 9001]# traceroute Client D
traceroute to Client D (Client D), 30 hops max, 60 byte packets
1 *.*.*.254 (10.180.128.254) 0.738 ms 0.800 ms 0.889 ms
2 *.*.*.133 (10.180.176.133) 2.156 ms 2.190 ms 2.177 ms
3 * * *
4 * * *
5 *.*.*.2 5.768 ms 5.942 ms 6.114 ms
6 Client D (Client D) 1.940 ms 1.964 ms 1.948 ms
说明是机器A到client_B网络有问题
5.解决办法
更换从的机器到client_B到clinet E
启动后机器A上9001 内存使用情况
127.0.0.1:9001> info memory
# Memory
used_memory:2771776520
used_memory_human:2.58G
used_memory_rss:3403390976
used_memory_rss_human:3.17G
used_memory_peak:18154232496
used_memory_peak_human:16.91G
used_memory_peak_perc:15.27%
used_memory_overhead:758586508
used_memory_startup:8105584
used_memory_dataset:2013190012
used_memory_dataset_perc:72.84%
total_system_memory:540783263744
total_system_memory_human:503.64G
used_memory_lua:37888
used_memory_lua_human:37.00K
maxmemory:6442450944
maxmemory_human:6.00G
maxmemory_policy:volatile-ttl
mem_fragmentation_ratio:1.23
mem_allocator:jemalloc-4.0.3
active_defrag_running:0
lazyfree_pending_objects:0
127.0.0.1:9001>
6.解决后观察
Client E内存同步情况
127.0.0.1:9001> info stats
# Stats
total_connections_received:37
total_commands_processed:5318696
instantaneous_ops_per_sec:19310
total_net_input_bytes:1711079396
total_net_output_bytes:4954958
instantaneous_input_kbps:2967.88
instantaneous_output_kbps:70.17
rejected_connections:0
sync_full:0
修改后机器A上的9001 输出
[root@A redis-node]# redis-cli -p 9001 info clients
# Clients
connected_clients:4695
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:
7.总结：
在redis设置同样而内存使用不同时，应先查看info命令对比正常机器的输出不同，优先关注Stats的输出情况以及本地的redis.log日志
