redis-2305e285 Predixy 中间件架构验证报告

redis-2305e285 Predixy 中间件架构验证报告

验证时间: 2025-12-28
集群: 145 (KUBECONFIG=/bpx/.145-admin.conf)
命名空间: qfusion-admin

一、验证结论

二、Predixy 部署架构

2.1 相关 Pods

kubectl get pods -n qfusion-admin|grepredis-2305e285

结果:

drc-redis-2305e285-0-0 3/3 Running 0 2d17h drc-redis-2305e285-0-1 3/3 Running 0 2d17h drc-redis-2305e285-1-0 3/3 Running 0 2d17h drc-redis-2305e285-1-1 3/3 Running 0 2d17h drc-redis-2305e285-2-0 3/3 Running 0 2d17h drc-redis-2305e285-2-1 3/3 Running 0 2d17h redis-2305e285-predixy-0 2/2 Running 0 2d17h redis-2305e285-slowlog-collector-0 1/1 Running 0 2d17h

2.2 Predixy Pods 详细信息（含节点分布）

kubectl get pods -n qfusion-admin -l redis.resource.type=predixy -o wide

结果:

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES redis-2305e285-predixy-0 2/2 Running 0 2d17h 245.0.2.240 qfusion3 <none> <none> redis-2305e285-predixy-1 2/2 Running 0 2d17h 245.0.1.147 qfusion2 <none> <none>

分析:

• 2 个 Predixy Pod 分布在不同节点(qfusion2, qfusion3)
• 每个 Pod 包含 2 个容器: predixy + exporter
• 监听端口:7617(非标准 6379)

2.3 StatefulSet 配置

kubectl get statefulset redis-2305e285-predixy -n qfusion-admin -ojsonpath='{.spec.replicas}'

结果:

2 replicas

kubectl get statefulset redis-2305e285-predixy -n qfusion-admin -ojsonpath='{.spec.podManagementPolicy}'

结果:

Parallel # 并行管理模式，Pod 可同时启动/更新

2.4 客户端服务配置

kubectl get svc -n qfusion-admin|grepredis-2305e285

结果:

redis-2305e285 ClusterIP 246.111.103.150 <none> 6379/TCP,16379/TCP,9100/TCP redis-2305e2850 ClusterIP 246.99.91.187 <none> 6379/TCP # Predixy 客户端入口 redis-2305e2850-lb ClusterIP 246.100.216.203 <none> 35036/TCP

客户端入口:redis-2305e2850(ClusterIP: 246.99.91.187:6379)

2.5 Service Endpoints

kubectl get endpoints redis-2305e2850 -n qfusion-admin

结果:

NAME ENDPOINTS AGE redis-2305e2850 245.0.1.147:7617,245.0.2.240:7617 2d17h

映射关系:

三、流量负载均衡机制

3.1 CNI 确认

kubectl get pods -n kube-system|grepcilium-agent

结果:

cilium-7hjkj 1/1 Running 0 9d cilium-fk6px 1/1 Running 0 9d cilium-gsn82 1/1 Running 1 (5d ago) 9d cilium-mgjrg 1/1 Running 0 9d

kubectl get pods -n kube-system|grepkube-proxy

结果: (无输出)集群中无 kube-proxy

3.2 Cilium kube-proxy 替代模式

kubectl get cm -n kube-system cilium-config -ojsonpath='{.data.kube-proxy-replacement}'

结果:

strict # Cilium 完全替代 kube-proxy

3.3 Cilium BPF LB 配置

kubectl get cm -n kube-system cilium-config -ojsonpath='{.data.bpf-lb-sock}'

结果:

false # 使用标准 eBPF LB (非 socket level)

3.4 Cilium Service 负载均衡详情

kubectlexec-n kube-system cilium-7hjkj -- ciliumservicelist|grep-A5"246.99.91.187"

结果:

ID Frontend Service Type Backend 134 246.99.91.187:6379 ClusterIP 1 => 245.0.2.240:7617 (active) 2 => 245.0.1.147:7617 (active)

3.5 BPF Maps

kubectlexec-n kube-system cilium-7hjkj -- bpftool map list|grep-E"cilium.*lb"

结果:

929: lru_hash name cilium_lb4_reve flags 0x0 930: hash name cilium_lb4_serv flags 0x1 # Service 映射表 931: hash name cilium_lb4_back flags 0x1 # Backend 映射表 932: hash name cilium_lb4_reve flags 0x1 941: hash name cilium_lb_affin flags 0x1 # 会话亲和 942: lru_hash name cilium_lb4_affi flags 0x0 943: lpm_trie name cilium_lb4_sour flags 0x1

3.6 流量策略配置

kubectl get svc -n qfusion-admin redis-2305e2850 -o yaml|grep-E"sessionAffinity|internalTrafficPolicy"

结果:

internalTrafficPolicy: Cluster # 集群内流量可路由到所有节点 sessionAffinity: None # 无会话亲和

3.7 负载均衡总结

四、Predixy 后端 Redis 连接配置

4.1 Predixy ConfigMap

kubectl get configmap redis-2305e285-predixy-conf -n qfusion-admin -o yaml

cluster.conf 配置:

ClusterServerPool { Password MasterReadPriority 60 # 主节点读优先级 StaticSlaveReadPriority 50 # 静态从节点读优先级 DynamicSlaveReadPriority 50 # 动态从节点读优先级 RefreshInterval 1 # 配置刷新间隔(秒) ServerTimeout 1 # 服务器超时(秒) ServerFailureLimit 10 # 故障阈值 ServerRetryTimeout 1 # 重试超时(秒) KeepAlive 120 # 连接保活(秒) Servers { + redis-2305e285.qfusion-admin:6379 } }

读写策略:

• 写操作: 发往 Master
• 读操作: 优先 Master (60)，其次 Slave (50)

4.2 后端 Redis Server Pods 分布

kubectl get pods -n qfusion-admin -l redis.resource.type=server -o wide|grepredis-2305e285

结果:

NAME READY STATUS RESTARTS AGE IP NODE drc-redis-2305e285-0-0 3/3 Running 0 2d17h 245.0.1.48 qfusion2 drc-redis-2305e285-0-1 3/3 Running 0 2d17h 245.0.3.84 qfusion4 drc-redis-2305e285-1-0 3/3 Running 0 2d17h 245.0.0.107 qfusion1 drc-redis-2305e285-1-1 3/3 Running 0 2d17h 245.0.2.168 qfusion3 drc-redis-2305e285-2-0 3/3 Running 0 2d17h 245.0.2.214 qfusion3 drc-redis-2305e285-2-1 3/3 Running 0 2d17h 245.0.3.200 qfusion4

4.3 Redis 主从角色分布

kubectl get QfrCluster redis-2305e285 -n qfusion-admin -ojsonpath='{.metadata.annotations}'

结果:

drc-redis-2305e285-0-0/info.role: slave drc-redis-2305e285-0-1/info.role: master drc-redis-2305e285-1-0/info.role: slave drc-redis-2305e285-1-1/info.role: master drc-redis-2305e285-2-0/info.role: slave drc-redis-2305e285-2-1/info.role: master

架构: 3 分片，每分片 1主1从

4.4 Redis Service Endpoints

kubectl get endpoints redis-2305e285 -n qfusion-admin

结果:

NAME ENDPOINTS AGE redis-2305e285 245.0.0.107:6379,245.0.1.48:6379,245.0.2.168:6379 + 15 more...

五、高可用机制

5.1 Predixy 健康检查

kubectl get pod redis-2305e285-predixy-0 -n qfusion-admin -ojsonpath='{.spec.containers[0].livenessProbe}'

结果:

{"exec":{"command":["sh","-c","redis-cli -h $(hostname) -p 7617 ping"]},"failureThreshold":3,"initialDelaySeconds":30,"periodSeconds":10,"successThreshold":1,"timeoutSeconds":5}

kubectl get pod redis-2305e285-predixy-0 -n qfusion-admin -ojsonpath='{.spec.containers[0].readinessProbe}'

结果:

{"exec":{"command":["sh","-c","redis-cli -h $(hostname) -p 7617 ping"]},"failureThreshold":5,"initialDelaySeconds":30,"periodSeconds":15,"successThreshold":1,"timeoutSeconds":5}

健康检查策略:

5.2 Cilium 服务发现与故障转移

Cilium 通过以下机制实现高可用:

1. eBPF Map 实时更新: 当 Pod 状态变化时，Cilium 自动更新 BPF maps
2. Backend 健康监控: Cilium 监控 Backend 状态，自动剔除不健康的后端
3. 无状态转发: 内核态直接转发，无需中间代理

5.3 节点级容错

容错能力: 任一节点故障，另一个节点的 Predixy 仍可处理全部流量

六、架构图

Client | v redis-2305e2850 Service (ClusterIP: 246.99.91.187:6379) | +------------+------------+ | | v v +---------------+ +---------------+ | Predixy Pod-0 | | Predixy Pod-1 | | 245.0.2.240 | | 245.0.1.147 | | qfusion3 | | qfusion2 | +---------------+ +---------------+ | | +------------+------------+ | v redis-2305e285.qfusion-admin:6379 | +------------+------------+------------+ | | | | v v v v +--------+ +--------+ +--------+ +--------+ |Shard-0 | |Shard-1 | |Shard-2 | | | |M: qf4 | |M: qf3 | |M: qf4 | | | |S: qf2 | |S: qf1 | |S: qf3 | | | +--------+ +--------+ +--------+ +--------+

七、验证命令汇总

# 切换集群exportKUBECONFIG=/bpx/.145-admin.conf# 1. 查看 Podskubectl get pods -n qfusion-admin|grepredis-2305e285 kubectl get pods -n qfusion-admin -l redis.resource.type=predixy -o wide# 2. 查看 Services 和 Endpointskubectl get svc -n qfusion-admin|grepredis-2305e285 kubectl get endpoints redis-2305e2850 -n qfusion-admin# 3. 查看 StatefulSetkubectl get statefulset redis-2305e285-predixy -n qfusion-admin# 4. 查看 Predixy 配置kubectl get configmap redis-2305e285-predixy-conf -n qfusion-admin -o yaml# 5. 查看 Cilium 配置kubectl get cm -n kube-system cilium-config -o yaml|grep-E"kube-proxy-replacement|bpf-lb"# 6. 查看 Cilium Service 负载均衡kubectlexec-n kube-system<cilium-pod>-- ciliumservicelist|grepredis-2305e285# 7. 查看 Redis 主从角色kubectl get QfrCluster redis-2305e285 -n qfusion-admin -ojsonpath='{.metadata.annotations}'

八、业务流量测试验证

8.1 测试环境

使用 Predixy Pod 内置的 redis-cli 进行测试：

# 测试入口redis-2305e2850.qfusion-admin:6379# 认证密码x.x.x

8.2 测试1: 基本连通性测试

kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850.qfusion-admin -p6379-a x.x.xping

结果:PONG✓

8.3 测试2: 基本读写操作

# 写操作kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850.qfusion-admin -p6379-a x.x.x\SET test:benchmark:001"hello-from-predixy"# 读操作kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850.qfusion-admin -p6379-a x.x.x\GET test:benchmark:001

结果:

SET: OK GET: hello-from-predixy

8.4 测试3: 负载均衡验证

3.1 并发负载测试 (200 操作)

kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --sh-c' for i in $(seq 1 100); do redis-cli -h redis-2305e2850.qfusion-admin -p 6379 -a x.x.x SET test:load:$i "value-$i" > /dev/null 2>&1 redis-cli -h redis-2305e2850.qfusion-admin -p 6379 -a x.x.x GET test:load:$i > /dev/null 2>&1 done '

结果: 200 操作全部成功 ✓

3.2 双源并发测试 (1000 请求)

# 从 predixy-0 发起 500 次请求kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --sh-c' for i in $(seq 1 500); do redis-cli -h redis-2305e2850 -p 6379 -a x.x.x PING > /dev/null 2>&1 done'&# 从 predixy-1 发起 500 次请求kubectlexec-n qfusion-admin redis-2305e285-predixy-1 --sh-c' for i in $(seq 1 500); do redis-cli -h redis-2305e2850 -p 6379 -a x.x.x PING > /dev/null 2>&1 done'&

结果: 1000 请求全部完成 ✓

3.3 连通性验证测试 (100 次 PING)

foriin$(seq1100);dokubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850 -p6379-a x.x.x PING2>/dev/nulldone

结果:

8.5 测试4: 双活实例独立响应验证

# 直接访问 Predixy-0kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h245.0.2.240 -p7617ping# 直接访问 Predixy-1kubectlexec-n qfusion-admin redis-2305e285-predixy-1 --\redis-cli -h245.0.1.147 -p7617ping

结果: 两个实例均返回PONG✓

8.6 测试5: 复杂数据类型操作

# MSET/MGET 批量操作kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850 -p6379-a x.x.x\MSET test:user:1001"Alice"test:user:1002"Bob"test:user:1003"Charlie"# 读取验证kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850 -p6379-a x.x.x\MGET test:user:1001 test:user:1002 test:user:1003

结果:Alice,Bob,Charlie✓

# 列表操作 LPUSH/LRANGEkubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850 -p6379-a x.x.x\LPUSH test:queue:tasks"task3""task2""task1"kubectlexec-n qfusion-admin redis-2305e285-predixy-0 --\redis-cli -h redis-2305e2850 -p6379-a x.x.x\LRANGE test:queue:tasks0-1

结果:task1,task2,task3✓

8.7 测试6: 后端 Redis 复制状态验证

kubectlexec-n qfusion-admin drc-redis-2305e285-0-1 -c redis --\redis-cli -p6379-a x.x.x INFO replication

结果:

# Replication role:master connected_slaves:1 slave0:ip=245.0.1.48,port=6379,state=online,offset=335685,lag=0 master_failover_state:no-failover master_replid:38577acaafc0f390567d4813bf58500ff291255d master_repl_offset:335685 repl_backlog_active:1

8.8 测试结果汇总

8.9 业务测试结论

1. 双活验证通过: 两个 Predixy 实例均可独立响应请求
2. 负载均衡正常: Cilium eBPF 正确分发流量到两个 Backend
3. 读写功能正常: SET/GET/MSET/MGET/LPUSH 等操作全部成功
4. 高可用有效: 后端 Redis 主从复制状态正常 (master + 1 slave online)