in container

OpenShift Origin Advanced Installation Guideline

Updated: June 30, 2018

Objective

Following the OpenShift Origin 3.9 – Advanced Installation guideline, I’ll build OpenShift Origin for testing environment. Please DO NOT use it in production environment

In the design architecture, I prepare 6 VMs, include 1 master node, 1 load balancer node, 2 infra nodes, and 2 app nodes.

Architecture

Environments

  • master1.pichuang.local
    • 192.168.77.31
    • Fedora 21
  • node1.pichuang.local
    • 192.168.77.32
    • Fedora 21
  • node2.pichuang.local
    • 192.168.77.33
    • Fedora 21
  • lb.pichuang.local
    • 192.168.77.34
    • Fedora 21
  • infra1.pichuang.local
    • 192.168.77.35
    • Fedora 21
  • infra2.pichuang.local
    • 192.168.77.36
    • Fedora 21
  1. Each of these nodes should be provisioned with an SSH public key which can be used to access all nodes from the Ansible
    • ssh-copy-id is your good friend
  2. Do not disable selinux
  3. Suggest use root during installation processes

Processes

Set-up DNS

In pichuang.local zone,

infra2.pichuang.local.  86400   A   192.168.77.36
infra1.pichuang.local.  86400   A   192.168.77.35
apps.pichuang.local.    86400   NS  ns.apps.pichuang.local.
lb.pichuang.local.  86400   A   192.168.77.34
ocp-public.pichuang.local.  86400   CNAME   master1.pichuang.local.
ocp-internal.pichuang.local.    86400   CNAME   master1.pichuang.local.
node2.pichuang.local.   86400   A   192.168.77.33
node1.pichuang.local.   86400   A   192.168.77.32
master1.pichuang.local. 86400   A   192.168.77.31
pichuang.local. NS  ns.pichuang.local.
ns.pichuang.local.  A   192.168.77.100

In apps.pichuang.local zone,

*.apps.pichuang.local.  86400   A   192.168.77.35
*.apps.pichuang.local.  86400   A   192.168.77.36
apps.pichuang.local.    NS  ns.apps.pichuang.local.
ns.apps.pichuang.local. A   192.168.77.100
  • Prepare a DNS wildcard (*.apps.pichuang.local) and point to IP of infra nodes (192.168.77.35-36) for setup round-robin DNS

Ansible Inventory for OpenShift

cat << EOF > /etc/ansible/hosts
# Create an OSEv3 group that contains the masters, nodes, and etcd groups
[OSEv3:children]
masters
nodes
etcd
lb

# Set variables common for all OSEv3 hosts
[OSEv3:vars]
ansible_ssh_user=root

# Use OpenShift Origin
openshift_deployment_type=origin
openshift_master_cluster_method=native

# openshift_master_cluster_hostname and openshift_master_cluster_public_hostname must resolve to lb
openshift_master_cluster_hostname=ocp-internal.pichuang.local
openshift_master_cluster_public_hostname=ocp-public.pichuang.local

# You may set up DNS wildcard for point to infra nodes.
openshift_master_default_subdomain=apps.pichuang.local

# For testing environment only, skip all necessary system check
openshift_disable_check=memory_availability,disk_availability

# enable ntp on masters to ensure proper failover
openshift_clock_enabled=true

# Use firewalld instead of default iptables.
os_firewall_use_firewalld=true

# uncomment the following to enable htpasswd authentication; defaults to DenyAllPasswordIdentityProvider
openshift_master_identity_providers=[{'name': 'htpasswd_auth', 'login': 'true', 'challenge': 'true', 'kind': 'HTPasswdPasswordIdentityProvider', 'filename': '/etc/origin/master/htpasswd'}]

# host group for masters
[masters]
master1.pichuang.local

# host group for lb
# Only pass request for "openshift_master_cluster_hostname" and "openshift_master_cluster_public_hostname" instead of "openshift_master_default_subdomain" by default
[lb]
lb.pichuang.local

# host group for etcd
# Can be point to master node
[etcd]
master1.pichuang.local

# host group for nodes, includes region info
# It will install router on the nodes if the labels is region: infra
[nodes]
master1.pichuang.local
node1.pichuang.local openshift_node_labels="{'region': 'primary', 'zone': 'east'}"
node2.pichuang.local openshift_node_labels="{'region': 'primary', 'zone': 'west'}"
infra1.pichuang.local openshift_node_labels="{'region': 'infra', 'zone': 'default'}"
infra2.pichuang.local openshift_node_labels="{'region': 'infra', 'zone': 'default'}"

EOF

All-in-one installation

dnf install -y ansible pyOpenSSL python-cryptography python-lxml
git clone https://github.com/openshift/openshift-ansible
cd openshift-ansible
ansible-playbook -i /etc/ansible/hosts playbooks/prerequisites.yml
ansible-playbook -i /etc/ansible/hosts playbooks/deploy_cluster.yml

Create Account and Access OpenShift Origin Web console

htpasswd /etc/origin/master/htpasswd pichuang
oadm policy add-cluster-role-to-user cluster-admin pichuang
oc login -u pichuang -n default

Verifying OpenShift Origin Installation

  1. Verify that the master is started and nodes are registered and reporting in Ready status, but not including lb.pichuang.local because it is out-of-scope
# oc get nodes
NAME                       STATUS    ROLES     AGE       VERSION
master1.pichuang.local     Ready     master    1m        v1.9.1+xxx
node1.pichuang.local       Ready     compute   1m        v1.9.1+xxx
node2.pichuang.local       Ready     compute   1m        v1.9.1+xxx
infra1.pichuang.local      Ready     compute   1m        v1.9.1+xxx
infra2.pichuang.local      Ready     compute   1m        v1.9.1+xxx
  1. The web console would be found at https://ocp-internal.pichuang.local:8443/console
  2. Verify that the load balancer would be found at http://lb.pichuang.local:9000

Conclusion

  1. 初期規劃時間會比實際部署需要更多的討論時間,因為依據不同環境有些服務可以拆成獨立的機器或者是合成一台
  2. 針對 DNS Wildcard 的部分建議是採用 Load Balancer 來取代,而不建議採用本篇 DNS round-robin 作法,因為其中一台 Router 死了,DNS Server 並不會知道,還是會隨機分配流量
  3. etcd 依據 OpenShift 3.9 官方安裝指南,建議另外安裝獨立機器來使用,但這裡為求精簡測試,把 etcd 的功能放在 master 上
  4. 其實 Infra node 本身也是 node 的一種,會稱作 Infra node,主要是上面會多一個標籤 region: infra,而這標籤會讓該主機安裝 Router 在上面,可至 default project 查詢
  5. 從外的所有查詢流量 (*.apps.pichuang.local) 都會經過 Router,而它會協助轉導至 Service 底下的 Pods 來做處理
  6. OpenShift Ansible Playbook 所部署出來的 lb server,其實僅有服務 atomic-openshift-api,並不包含外部資料流量 (*.apps.pichuang.local),可由 lb.pichuang.local 裡的 /etc/haproxy/haproxy.cfg 獲得該設定

References