Red Hat OpenShift 4 的 etcd 之幾個你應該要知道的事之二

承襲上文 Red Hat OpenShift 4 的 etcd 之幾個你應該要知道的事之一,繼續針對硬碟的部分進行調整,畢竟 etcd 對硬碟 IOPS 相當要求,建議用 SSD 比較好

沒意外的,開場分享個 Deploying OpenShift 4.4 to VMware vSphere 7,整體上比較重要的是利用 VMware vSphere 7 裡面的 DRS (Dynamic Resource Scheduler) 及 NSX-T 的部分,之後有機會再說

etcd 節點硬碟調教

ionice: set or get process I/O scheduling class and priority

這邊使用 ionice (1) - Linux Man Pages 來完成這件事,剛好 CoreOS 內建有支援這個指令 XD,差點就要包一個 Container 而且要蠻高的權限 (需要 root) 去定期執行了

Before

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ~]$ pgrep etcd
740369
740441

[[email protected] ~]$ ps uax |grep 740369
root 740369 7.6 8.4 9672968 1382392 ? Ssl 17:05 4:25 etcd --initial-advertise-peer-urls=https://10.0.97.3:2380 --cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-control-plane-2.crt --key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-serving/etcd-serving-control-plane-2.key --trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-serving-ca/ca-bundle.crt --client-cert-auth=true --peer-cert-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-control-plane-2.crt --peer-key-file=/etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-control-plane-2.key --peer-trusted-ca-file=/etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt --peer-client-cert-auth=true --advertise-client-urls=https://10.0.97.3:2379 --listen-client-urls=https://0.0.0.0:2379 --listen-peer-urls=https://0.0.0.0:2380 --listen-metrics-urls=https://0.0.0.0:9978
core 901685 0.0 0.0 12760 1076 pts/0 S+ 18:03 0:00 grep --color=auto 740369

[[email protected] ~]$ ps uax |grep 740441
root 740441 0.9 0.1 124764 25288 ? Ssl 17:05 0:32 etcd grpc-proxy start --endpoints https://10.0.97.3:9978 --metrics-addr https://0.0.0.0:9979 --listen-addr 127.0.0.1:9977 --key /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-control-plane-2.key --key-file /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-control-plane-2.key --cert /etc/kubernetes/static-pod-certs/secrets/etcd-all-peer/etcd-peer-control-plane-2.crt --cert-file /etc/kubernetes/static-pod-certs/secrets/etcd-all-serving-metrics/etcd-serving-metrics-control-plane-2.crt --cacert /etc/kubernetes/static-pod-certs/configmaps/etcd-peer-client-ca/ca-bundle.crt --trusted-ca-file /etc/kubernetes/static-pod-certs/configmaps/etcd-metrics-proxy-serving-ca/ca-bundle.crt
core 902165 0.0 0.0 12760 1060 pts/0 S+ 18:03 0:00 grep --color=auto 740441

# Get priority and class
[[email protected] ~]$ sudo ionice -p 740369 740441
none: prio 4
none: prio 0

Tuning

我們可以直接登入 CoreOS 後下 ionice 對特定 PID 和硬碟處理權限拉高,可以搭配 ansible 來針對 master node 執行

1
2
# Set priority and class
[[email protected] ~]$ sudo ionice -c2 -n0 -p `pgrep etcd`
  • -c2: best-effort
  • -n0: the highest priority level

After

1
2
3
4
5
6
7
[[email protected] ~]$ pgrep etcd
740369
740441

[[email protected] ~]$ sudo ionice -p 740369 740441
best-effort: prio 0
best-effort: prio 0

後話

恩…反正裝 etcd 的那個節點之硬碟選用高 IOPS,例如用 SSD,效率和穩定性會提升不少

References

Comments