kubeadm 安装 kubernetes 1.16.1集群

Author	Date	Last Update
liliming	2019-06-09	2019-11-17

一、环境准备

1.1 实验环境

环境	版本
CentOS	7.7.1908
Docker	18.09.7
Kubernetes	1.16.1
Helm	3.0
Kernel	3.10.0-1062.1.2.el7.x86_64
CPU	2
Cgroup Driver	systemd
Network	Calico
serviceSubnet	10.1.0.0/16
podSubnet	10.2.0.0/16
apiserver	apiserver.yoho8
ingress-controller	ingress-nginx

1.2 主机规划

Hostname	IP
kube-node001	10.0.0.11
kube-node002	10.0.0.12
kube-node003	10.0.0.13

1.3 配置`hosts`文件

$ cat <<EOF | tee -a /etc/hosts
10.0.0.11 kube-node001 apiserver.yoho8
10.0.0.12 kube-node002
10.0.0.13 kube-node003
EOF

1.4 禁用防火墙和`selinux`

systemctl stop firewalld
systemctl disable firewalld

setenforce 0
sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config

1.5 由于开启内核 ipv4 转发需要加载 br_netfilter 模块，所以加载下该模块：

$ modprobe br_netfilter

1.6 创建`/etc/sysctl.d/k8s.conf`文件，添加如下内容：

如果以下配置在/etc/sysctl.conf已经存在，那么修改就行

$ cat <<EOF> /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
vm.swappiness = 0
EOF

$ sysctl -p /etc/sysctl.d/k8s.conf

bridge-nf

使得 netfilter 可以对 Linux 网桥上的 IPv4/ARP/IPv6 包过滤。比如，设置net.bridge.bridge-nf-call-iptables＝1后，二层的网桥在转发包时也会被 iptables的 FORWARD 规则所过滤。常用的选项包括：

net.bridge.bridge-nf-call-arptables：是否在 arptables 的 FORWARD 中过滤网桥的 ARP 包
net.bridge.bridge-nf-call-ip6tables：是否在 ip6tables 链中过滤 IPv6 包
net.bridge.bridge-nf-call-iptables：是否在 iptables 链中过滤 IPv4 包
net.bridge.bridge-nf-filter-vlan-tagged：是否在 iptables/arptables 中过滤打了 vlan 标签的包。

1.7 关闭 swap

swap 严重影响k8s性能，所以关闭

swapoff -a
sed -i 's/.*swap.*/#&/' /etc/fstab

1.7 配置`kube-proxy`开启`ipvs`的前置条件

由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

执行命令

$ cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

检查

lsmod | grep -e ip_vs -e nf_conntrack_ipv4

安装ipset查看ipvs的代理规则，安装管理工具ipvsadm

yum install -y ipset ipvsadm

如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。

二、安装`docker & kubernetes`

2.1 安装Docker

2.1.1 安装docker的yum源

yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

2.1.2 安装指定版本的docker

注意k8s兼容性,一般最新的k8s只能兼容前几个版本的Docker

yum makecache fast

yum install -y docker-ce-18.09.7 docker-ce-cli-18.09.7 containerd.io

2.1.3 修改配置

修改Docker的Cgroup system 为 Systemd
配置镜像加速
其他一些优化

由于默认情况下 kubelet 使用的 cgroupdriver 是 systemd，所以需要保持 docker 和kubelet 的 cgroupdriver 一致，我们这里修改 docker 的 cgroupdriver=systemd。如果不修改 docker 则需要修改 kubelet 的启动配置，需要保证两者一致。

mkdir /etc/docker -p
tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors": [
        "https://dockerhub.azk8s.cn",
        "https://reg-mirror.qiniu.com",
        "https://registry.docker-cn.com"
    ],
    "exec-opts": ["native.cgroupdriver=systemd"],
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "100m",
        "max-file": "10"
    },

    "storage-driver": "overlay2",
    "live-restore": true
}
EOF

2.1.4 重启Docker

systemctl daemon-reload
systemctl enable docker && systemctl start docker

2.2 安装Kubernetes

2.2.1 添加阿里云kubernetes源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

2.2.2 下载`kubectl` `kubeadm` `kubelet`

yum install -y kubelet-1.16.1 kubeadm-1.16.1 kubectl-1.16.1

2.2.3 重启docker和kubernetes

systemctl daemon-reload && systemctl restart docker
systemctl enable kubelet && systemctl start kubelet

2.2.4 kubectl 命令自动补全

yum install -y bash-completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc

==以上操作在所有节点执行==

或者执行一键脚本

curl -fsSL https://gitee.com/llmgo/shell/raw/master/kubernetes.sh | bash

三、初始化kubernetes集群

3.1 配置kubeadm-config集群初始化文件

cat <<EOF > ./kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.1
imageRepository: registry.aliyuncs.com/google_containers
controlPlaneEndpoint: "apiserver.yoho8:6443"
networking:
  serviceSubnet: "10.1.0.0/16"
  podSubnet: "10.2.0.0/16"
  dnsDomain: "cluster.local"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
EOF

serviceSubnet: Service的IP段
podSubnet: Pod的IP段
controlPlaneEndpoint: 主节点IP或者域名
imageRepository: 指定镜像地址，默认gcr用不了，所以指定阿里云
mode: 指定kube-proxy模式为ipvs(默认是iptables)

3.1 开始初始化

kubeadm init --config=kubeadm-config.yaml --upload-certs

如果看到如下输入代表初始化成功：

...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join apiserver.yoho8:6443 --token ye6lec.w890f6fbdhi0pv90 \
    --discovery-token-ca-cert-hash sha256:5e0b1fb0f0523f7e2e5d29f4a1f70be0a2f00b4ca91c0401747cdb6094fd923e \
    --control-plane --certificate-key 75cae1eb13c0f6eff5035707abc481362e6027690e5c2e6516e24a53e998106e

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join apiserver.yoho8:6443 --token ye6lec.w890f6fbdhi0pv90 \
    --discovery-token-ca-cert-hash sha256:5e0b1fb0f0523f7e2e5d29f4a1f70be0a2f00b4ca91c0401747cdb6094fd923e

按照提示执行执行如下指令

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

其他节点加入

在kube-node002和kube-node003执行

kubeadm join apiserver.yoho8:6443 --token ye6lec.w890f6fbdhi0pv90 \
    --discovery-token-ca-cert-hash sha256:5e0b1fb0f0523f7e2e5d29f4a1f70be0a2f00b4ca91c0401747cdb6094fd923e

# 上面信息的最后就是node节点加入的命令，如果是控制节点就是上面的那条join命令。如果忘记保存了也没事，执行下面的这条命令，不过有效期只有两小时。

$ kubeadm token create --print-join-command
kubeadm join apiserver.yoho8:6443 --token ch5cb3.qr6q2ad9ge1h6z7o     --discovery-token-ca-cert-hash sha256:5e0b1fb0f0523f7e2e5d29f4a1f70be0a2f00b4ca91c0401747cdb6094fd923e

去除master的taint

# 去除taint
kubectl taint nodes kube-node001 node-role.kubernetes.io/master-

# 添加taint（NoSchedule 以后不会将Pod调度到这个节点，但是已经在运行的没有影响）
kubectl taint node kube-node001 node-role.kubernetes.io/master=:NoSchedule

如果集群初始化遇到问题，可以使用下面的命令进行清理

kubeadm reset
rm -rf /var/lib/cni/
rm -rf ~/.kube
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
ipvsadm --clear

3.3 安装Network (二选一即可)

这里我们选择calico

3.3.1 安装flannel

export POD_SUBNET=10.2.0.0/16
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
sed -i "s#10.244.0.0/16#${POD_SUBNET}#" kube-flannel.yml
sed -i 's/quay.io/quay.azk8s.cn/g' kube-flannel.yml
kubectl apply -f  kube-flannel.yml

Github地址：https://github.com/coreos/flannel

3.3.2 安装calico

export POD_SUBNET=10.2.0.0/16
wget https://docs.projectcalico.org/v3.10/manifests/calico.yaml
sed -i "s#192.168.0.0/16#${POD_SUBNET}#" calico.yaml
kubectl apply -f calico.yaml

参考文档：https://docs.projectcalico.org/v3.9/getting-started/kubernetes/

3.4 检查集群情况

耐心等待，可能会需要一点时间

$ kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-7b9dcdcc5-tphr9   1/1     Running   0          24m
kube-system   calico-node-kw2k6                         1/1     Running   0          15m
kube-system   calico-node-pnzks                         1/1     Running   0          24m
kube-system   calico-node-zgr2x                         1/1     Running   0          12s
kube-system   coredns-58cc8c89f4-ll9mn                  1/1     Running   0          28m
kube-system   coredns-58cc8c89f4-p7ckb                  1/1     Running   0          28m
kube-system   etcd-kube-node001                         1/1     Running   0          27m
kube-system   kube-apiserver-kube-node001               1/1     Running   0          27m
kube-system   kube-controller-manager-kube-node001      1/1     Running   0          27m
kube-system   kube-proxy-jtjv4                          1/1     Running   0          14m
kube-system   kube-proxy-md7f5                          1/1     Running   0          15m
kube-system   kube-proxy-tjrd4                          1/1     Running   0          28m
kube-system   kube-scheduler-kube-node001               1/1     Running   0          27m

$ kubectl get nodes
NAME           STATUS   ROLES    AGE     VERSION
kube-node001   Ready    master   18m     v1.16.1
kube-node002   Ready    <none>   5m31s   v1.16.1
kube-node003   Ready    <none>   4m44s   v1.16.1

检查ipvs是否成功开启

$ kubectl logs kube-proxy-jtjv4 -n kube-system
I1117 11:14:14.892257       1 node.go:135] Successfully retrieved node IP: 10.0.83.42
I1117 11:14:14.892304       1 server_others.go:176] Using ipvs Proxier.
W1117 11:14:14.892489       1 proxier.go:420] IPVS scheduler not specified, use rr by default
..

日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。

测试DNS

# 启动 busybox plus
kubectl run -it --rm --restart=Never --image=radial/busyboxplus:curl --generator=run-pod/v1 curl

# 进入后执行nslookup kubernetes.default确认解析正常
[ root@curl-66959f6557-dfztk:/ ]$ nslookup kubernetes.default
Server:    10.1.0.10
Address 1: 10.1.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.1.0.1 kubernetes.default.svc.cluster.local

3.5 如何从集群中移除Node

移除kube-node002

在master执行

$ kubectl drain kube-node002 --delete-local-data --force --ignore-daemonsets
$ kubectl delete nodes kube-node002

在kube-node002执行

$ kubeadm reset
$ rm -rf /var/lib/cni/

3.6 如何给Node加上Roles标签

查看节点的标签

$ kubectl get nodes --show-labels
NAME           STATUS   ROLES    AGE   VERSION   LABELS
kube-node001   Ready    master   54m   v1.16.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kube-node001,kubernetes.io/os=linux,node-role.kubernetes.io/master=
kube-node002   Ready    <none>   41m   v1.16.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kube-node002,kubernetes.io/os=linux
kube-node003   Ready    <none>   40m   v1.16.1   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=kube-node003,kubernetes.io/os=linux

增加标签

$ kubectl label nodes kube-node002 node-role.kubernetes.io/node2=
node/kube-node002 labeled
$ kubectl label nodes kube-node003 node-role.kubernetes.io/node3=
node/kube-node003 labeled

查看效果

$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
kube-node001   Ready    master   58m   v1.16.1
kube-node002   Ready    node2    45m   v1.16.1
kube-node003   Ready    node3    44m   v1.16.1

4.删除标签

在标签后面加上 - 号就可以了

$ kubectl label nodes kube-node002 node-role.kubernetes.io/node2-
node/kube-node002 labeled
$ kubectl get nodes
NAME           STATUS   ROLES    AGE   VERSION
kube-node001   Ready    master   59m   v1.16.1
kube-node002   Ready    <none>   46m   v1.16.1
kube-node003   Ready    node3    45m   v1.16.1

四、安装和部署 Helm 3.0

Helm Charts是 Kubernetes 项目中的一个子项目目的是提供 Kubernetes 的包管理平台。Helm 能够帮你管理 Kubernetes 的应用集合。Helm Charts 能够帮你定义，安装，升级最复杂的 Kubernetes 应用集合。

安装

$ wget https://get.helm.sh/helm-v3.0.0-linux-amd64.tar.gz
$ tar xf helm-v3.0.0-linux-amd64.tar.gz
$ mv linux-amd64/helm /usr/local/bin/

五、安装`ingress-nginx`

wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
sed -i 's/quay.io/quay.azk8s.cn/g' mandatory.yaml
kubectl apply -f mandatory.yaml

部署ingress-nginx的service

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/baremetal/service-nodeport.yaml

参考文档: https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md