openshift11 minutes read (About 1638 words)

Red Hat OpenShift 4 DNS & NTP & PTP 網路套餐

最近雖然受到 COVID-19 影響導致有些國外專案進度有點落後，但就 OpenShift 4 功能本身不知道為啥成長幅度變的蠻快的…有時候想跟也很難跟上，只能透過影片來了解，然後 Red Hat 本家不知道哪根筋不對了，跑去 Twitch (!!??) 開了一個頻道 RedHatOpenShift Twitch，三不五時就開直播上演下圖的狀況，相當有事

走馬看花第 8 天

CoreDNS 做 DNS Forwarder

CoreDNS: DNS and Service Discovery

其實 OpenShift 內建有 DNS 服務，主要目的是要來解析 Service IP <-> DNS，所採用的技術是使用 CoreDNS。關於 CoreDNS ，主要是提供雲原生的 DNS 服務和服務發現的解決方案，有興趣想參考效能比較的話，可以看 Scaling CoreDNS in Kubernetes Clusters，目前已於 2019/01 時從 CNCF 專案海中畢業。

如果你仔細觀察 CoreDNS 在 OpenShift 上的運作狀況可以發現他是從 openshift-dns-operator 部署，並於每一個節點採用 DaemonSet 的形式運行 CoreDNS

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# Operator Information
$ oc get deployment/dns-operator -n openshift-dns-operator
NAME           READY   UP-TO-DATE   AVAILABLE   AGE
dns-operator   1/1     1            1           113m

$ oc get clusteroperators/dns
NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
dns    4.3.27    True        False         False      108m

# Project Information
$ oc get all -n openshift-dns -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP            NODE                                              NOMINATED NODE   READINESS GATES
pod/dns-default-9vq2j   2/2     Running   0          84m   10.129.0.10   ip-10-0-146-166.ap-southeast-1.compute.internal   <none>           <none>
pod/dns-default-g2bps   2/2     Running   0          84m   10.130.0.9    ip-10-0-167-32.ap-southeast-1.compute.internal    <none>           <none>
pod/dns-default-mg7cx   2/2     Running   0          84m   10.128.0.13   ip-10-0-129-186.ap-southeast-1.compute.internal   <none>           <none>
pod/dns-default-sxzcn   2/2     Running   0          75m   10.128.2.3    ip-10-0-132-54.ap-southeast-1.compute.internal    <none>           <none>
pod/dns-default-wj97g   2/2     Running   0          75m   10.131.0.3    ip-10-0-154-36.ap-southeast-1.compute.internal    <none>           <none>

NAME                  TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)                  AGE
service/dns-default   ClusterIP   172.30.0.10   <none>        53/UDP,53/TCP,9153/TCP   83m

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/dns-default   5         5         5       5            5           kubernetes.io/os=linux   83m

倘若你想要透過 VPN 去連線至其他雲裡面的服務，那這樣會有需要特別多加幾條 DNS Forwarder 的能力在集群裡面，遵照 DNS Operator in OpenShift Container Platform

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
# Default Setting
$ oc get configmap/dns-default -n openshift-dns -o yaml
apiVersion: v1
data:
  Corefile: |
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 30
        reload
    }
kind: ConfigMap
metadata:
  labels:
    dns.operator.openshift.io/owning-dns: default
  name: dns-default
  namespace: openshift-dns

# 新增 DNS Forwarder
$ cat << EOF >> apply-dns-forwarder.yaml
apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  servers:
  - name: pichuang-server-farm
    zones:
      - pichuang.internal
    forwardPlugin:
      upstreams:
        - 192.168.77.100
        - 192.168.77.99:5533
  - name: redhat-server-farm
    zones:
      - rhtw.private
      - rhglobal.public
    forwardPlugin:
      upstreams:
        - 10.99.1.100
        - 172.88.2.99:5533
EOF

# 執行變更
$ oc apply -f apply-dns-forwarder.yaml

# 檢查 CoreDNS 設定
$ oc get configmap/dns-default -n openshift-dns -o yaml
apiVersion: v1
data:
  Corefile: |
    # pichuang-server-farm
    pichuang.internal:5353 {
        forward . 192.168.77.100 192.168.77.99:5533
    }
    # redhat-server-farm
    rhtw.private:5353 rhglobal.public:5353 {
        forward . 10.99.1.100 172.88.2.99:5533
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            upstream
            fallthrough in-addr.arpa ip6.arpa
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            policy sequential
        }
        cache 30
        reload
    }
kind: ConfigMap
metadata:
  labels:
    dns.operator.openshift.io/owning-dns: default
  name: dns-default
  namespace: openshift-dns

時間設定 Chrony

打從 RHEL 7 之後，Red Hat 官方因為安全及效能上的考量，建議從 ntpd 轉換成使用 chronyd，實際上也是蠻推薦多用 chronyd 當作主要的 NTP Server/Client，就連 OpenShift 預載的 CoreOS 內部也是以 Chronyd 為校時基準，參考 NTP configuration - OpenShift Tips，以下是校時的操作

  1. 先檢查預設的狀態，隨便挑一台用 oc debug node/<node name> 進去觀察時間
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
$ oc debug node/ip-10-0-129-186.ap-southeast-1.compute.internal
Starting pod/ip-10-0-129-186ap-southeast-1computeinternal-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.129.186
If you don't see a command prompt, try pressing enter.

sh-4.2# chroot /host
sh-4.4# cat /etc/chrony.conf
sh-4.4#  chronyc tracking
Reference ID    : DABA0324 (time1.maxonline.com.sg)
Stratum         : 2
Ref time (UTC)  : Tue Jul 07 08:12:11 2020
System time     : 0.000018082 seconds fast of NTP time
Last offset     : +0.000020540 seconds
RMS offset      : 0.000040942 seconds
Frequency       : 1.571 ppm slow
Residual freq   : +0.007 ppm
Skew            : 0.394 ppm
Root delay      : 0.002178520 seconds
Root dispersion : 0.000406400 seconds
Update interval : 64.2 seconds
Leap status     : Normal

  1. 準備 chrony.conf，然後進行 Base64 編碼

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    $ cat << EOF >> chrony.conf
    pool 0.tw.pool.ntp.org iburst
    pool 3.tw.pool.ntp.org iburst
    driftfile /var/lib/chrony/drift
    makestep 1.0 3
    rtcsync
    keyfile /etc/chrony.keys
    leapsectz right/UTC
    logdir /var/log/chrony
    EOF

    $ cat chrony.conf | python3 -m base64 | xargs echo -n
    cG9vbCAwLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKcG9vbCAzLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmtleWZpbGUgL2V0Yy9jaHJvbnkua2V5cwpsZWFwc2VjdHogcmlnaHQvVVRDCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkK

  2. 準備 2 個聲明檔，如果你在 machine config pool 還有多新增 infra 的話，也要特別寫一個，原則上就是一個角色要有一個檔案就是了

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    $ cat << EOF > 98-worker-chrony.yaml
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 98-worker-chrony-configuration
    spec:
      config:
        ignition:
          version: 2.2.0
        storage:
          files:
          - contents:
              source: data:text/plain;charset=utf-8;base64,cG9vbCAwLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKcG9vbCAzLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmtleWZpbGUgL2V0Yy9jaHJvbnkua2V5cwpsZWFwc2VjdHogcmlnaHQvVVRDCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkK
            verification: {}
            filesystem: root
            mode: 420
            path: /etc/chrony.conf
    EOF

    $ cat << EOF > 98-master-chrony.yaml
    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: master
      name: 98-master-chrony-configuration
    spec:
      config:
        ignition:
          version: 2.2.0
        storage:
          files:
          - contents:
              source: data:text/plain;charset=utf-8;base64,cG9vbCAwLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKcG9vbCAzLnR3LnBvb2wubnRwLm9yZyBpYnVyc3QKZHJpZnRmaWxlIC92YXIvbGliL2Nocm9ueS9kcmlmdAptYWtlc3RlcCAxLjAgMwpydGNzeW5jCmtleWZpbGUgL2V0Yy9jaHJvbnkua2V5cwpsZWFwc2VjdHogcmlnaHQvVVRDCmxvZ2RpciAvdmFyL2xvZy9jaHJvbnkK
            verification: {}
            filesystem: root
            mode: 420
            path: /etc/chrony.conf
    EOF

    $ oc apply -f 98-master-chrony.yaml
    machineconfig.machineconfiguration.openshift.io/98-masters-chrony-configuration created

    $ oc apply -f 98-worker-chrony.yaml
    machineconfig.machineconfiguration.openshift.io/98-workers-chrony-configuration created

精確時間協定 PTP

想不到吧! OpenShift 4 有內建支援 IEEE 1588 (Precision Time Protocol, PTP) 協定，來支援硬體級別的精確時間同步，當然要使用這個技術需要仰賴硬體設備上的支援，包含像是網卡以及交換機，OpenShift 4 是將 Linux PTP 透過 PTP Operator 安裝於所有節點之上，透過內建自動能力查詢機制來協助將具備 PTP 能力之網卡設定其相關參數。其實主要這個功能是為了 O-RAN W6 裡面有定義條目 5.5.1 Cloud Platform Time Synchronization Architecture 而做的，裡面描述如下

The Time Sync deployment architecture which is described below relies on usage of Precision Time Protocol (PTP) IEEE 1588-2008 (a.k.a. IEEE 1588 Version 2) to synchronize clocks throughout the Edge Cloud site.

其實安裝相當簡單，主要是透過 Operator Framework 從頭打到底，如果有興趣想看過程的可以參考該 GitHub - pichuang/redhat-ptp-demo

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
$ oc get all
NAME                                READY   STATUS    RESTARTS   AGE
pod/linuxptp-daemon-2z2sx           1/1     Running   0          19m
pod/linuxptp-daemon-nllxn           1/1     Running   0          19m
pod/linuxptp-daemon-phkwp           1/1     Running   0          19m
pod/linuxptp-daemon-v7w4s           1/1     Running   0          19m
pod/linuxptp-daemon-xh5fp           1/1     Running   0          19m
pod/ptp-operator-575797775d-7bcdf   1/1     Running   0          19m

NAME                          TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/ptp-monitor-service   ClusterIP   None         <none>        9091/TCP   19m

NAME                             DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/linuxptp-daemon   5         5         5       5            5           kubernetes.io/os=linux   19m

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ptp-operator   1/1     1            1           19m

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/ptp-operator-575797775d   1         1         1       19m

哦對了，我身上沒有支援 PTP 的網卡，所以看不出來效果，帶有錢的大大找我去研究相關使用方式 XD

Appendix

Environment Information

  • OpenShift 4.3.27
  • Kubernetes Version: v1.16.2+d6d3cff
#

Comments

Follow

Categories

Recent

Archives

Tags

12apps1
agile1
annual1
ansible9
automation2
aws1
azure1
blogger1
bonding1
centos1
ceph1
chaos engineering1
cicada1
cloud1
cnf1
cni2
cnv1
container11
contrail1
coreos3
dac1
dgx1
digitaltransformation1
docker1
docker-compose1
driver2
easyrsa3
edge computing2
etcd2
event2
flexswitch1
gcp1
github1
gpu1
harden81
headless1
helm1
homecloud3
iac2
images1
infra12
intel5
interview1
juniper1
k9s1
kanban1
kubeinvaders1
kubernetes4
kubevirt1
kvm1
libvirt1
linux8
misc3
multus-cni1
netadmin1
netdevops2
network2
networking4
nfv2
ngc1
nmcli1
nsx-t1
nvidia1
oas1
ofdpa3
onl1
onos1
ons1
openapi3
openflow1
openshift29
openshift37
openshift420
openssl1
openstack1
operator2
oracle1
ovs1
ovs-ofctl2
performance1
pki4
pods1
qcow1
queue1
redhat16
registry1
restful1
rhel1
router1
sdn6
sds1
security1
serial1
services1
sff1
skopeo1
smart start2
snaproute1
ssl2
storage2
swagger1
switch1
teaming1
technical writer1
telco4
terraform1
tesla1
tower1
troubleshooting1
tscc1
vmware3
web-console1
weblogic1
win71
ztp1
×