kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具，伴随Kubernetes每个版本的发布都会同步更新，kubeadm会对集群配置方面的一些实践做调整，通过实验kubeadm可以学习到Kubernetes官方在集群配置上一些新的最佳实践。

安装前准备.

系统配置

同步各节点之间的时间

安装 时间同步工具ntpdate

yum install ntpdate -y

同步各服务器之间时间

ntpdate 0.asia.pool.ntp.org

准备主机修改主机名及DNS

在安装之前，需要先做如下准备。三台CentOS 7主机如下：

# 三台主机分别修改hostname
hostnamectl set-hostname master
hostnamectl set-hostname node1
hostnamectl set-hostname node2

# 重启三台节点
reboot

# 修改主机host 编辑下hosts文件，
# 给127.0.0.1添加hostname
vi /etc/hosts 
192.168.0.121 master
192.168.0.122 node1
192.168.0.123 node2

# 编辑后结果
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 master
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.121 master
192.168.0.122 node1
192.168.0.123 node2

检查主机防火墙

如果各个主机启用了防火墙，需要开放Kubernetes各个组件所需要的端口，可以查看Installing kubeadm中的"Check required ports"一节。 这里简单起见在各节点禁用防火墙：

systemctl stop firewalld
systemctl disable firewalld

禁用SELINUX：

setenforce 0

vi /etc/selinux/config
SELINUX=disabled

创建vi /etc/sysctl.d/k8s.conf文件，添加如下内容：

net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1

执行命令使修改生效。

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

kube-proxy开启ipvs的前置条件

由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：

ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack_ipv4

在所有的Kubernetes节点上执行以下脚本:

cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack_ipv4
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4

上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。

接下来还需要确保各个节点上已经安装了ipset软件包yum install ipset。 为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm yum install ipvsadm。

如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。

安装docker

卸载旧版本

    yum remove  docker \
                docker-client \
                docker-client-latest \
                docker-common \
                docker-latest \
                docker-latest-logrotate \
                docker-logrotate \
                docker-selinux \
                docker-engine-selinux \
                docker-engine

安装一些必要的系统工具：

yum install -y yum-utils device-mapper-persistent-data lvm2

添加软件源信息：

yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

更新 yum 缓存：

yum makecache fast

安装 Docker-ce：

yum -y install docker-ce

启动 Docker 后台服务

systemctl start docker
systemctl enable docker

测试运行 hello-world

docker run hello-world

确认一下iptables filter表中FOWARD链的默认策略(pllicy)为ACCEPT。

iptables -nvL

Docker从1.13版本开始调整了默认的防火墙规则，禁用了iptables filter表中FOWARD链，这样会引起Kubernetes集群中跨Node的Pod无法通信。但这里通过安装docker 1806，发现默认策略又改回了ACCEPT，这个不知道是从哪个版本改回的，因为我们线上版本使用的1706还是需要手动调整这个策略的。

使用kubeadm部署Kubernetes

安装kubeadm和kubelet

下面在各节点安装kubeadm和kubelet：

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg
        https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

测试地址https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64是否可用

curl https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

如果不可用需要将地址替换为阿里镜像

https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg 
        https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

安装kubelet kubeadm kubectl

yum makecache fast
yum install -y kubelet kubeadm kubectl

从安装结果可以看出还安装了cri-tools, kubernetes-cni, socat三个依赖：
官方从Kubernetes 1.9开始就将cni依赖升级到了0.6.0版本，在当前1.12中仍然是这个版本
socat是kubelet的依赖
cri-tools是CRI(Container Runtime Interface)容器运行时接口的命令行工具

运行kubelet –help可以看到原来kubelet的绝大多数命令行flag参数都被DEPRECATED了，如：

......
--address 0.0.0.0   The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.)
......

而官方推荐我们使用–config指定配置文件，并在配置文件中指定原来这些flag所配置的内容。具体内容可以查看这里Set Kubelet parameters via a config file。这也是Kubernetes为了支持动态Kubelet配置（Dynamic Kubelet Configuration）才这么做的，参考Reconfigure a Node’s Kubelet in a Live Cluster。

kubelet的配置文件必须是json或yaml格式，具体可查看这里。

Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。

关闭系统的Swap方法如下:

swapoff -a

修改 vi /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。 swappiness参数调整，修改vi /etc/sysctl.d/k8s.conf添加下面一行：

vm.swappiness=0

执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。

因为这里本次用于测试两台主机上还运行其他服务，关闭swap可能会对其他服务产生影响，所以这里修改kubelet的配置去掉这个限制。 之前的Kubernetes版本我们都是通过kubelet的启动参数–fail-swap-on=false去掉这个限制的。前面已经分析了Kubernetes不再推荐使用启动参数，而推荐使用配置文件。 所以这里我们改成配置文件配置的形式。

查看cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf，看到了下面的内容：

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

上面显示kubeadm部署的kubelet的配置文件–config=/var/lib/kubelet/config.yaml，实际去查看/var/lib/kubelet和这个config.yaml的配置文件都没有被创建。 可以猜想肯定是运行kubeadm初始化集群时会自动生成这个配置文件，而如果我们不关闭Swap的话，第一次初始化集群肯定会失败的。

所以还是老老实实的回到使用kubelet的启动参数–fail-swap-on=false去掉必须关闭Swap的限制。 修改vi /etc/sysconfig/kubelet，加入：

KUBELET_EXTRA_ARGS=--fail-swap-on=false

使用kubeadm init初始化集群

在各节点开机启动kubelet服务：

systemctl enable kubelet.service

接下来使用kubeadm初始化集群，选择master作为Master Node，在master上执行下面的命令：

kubeadm init \
  --kubernetes-version=v1.13.0 \
  --pod-network-cidr=10.244.0.0/16 \
  --apiserver-advertise-address=192.168.0.121

因为我们选择flannel作为Pod网络插件，所以上面的命令指定–pod-network-cidr=10.244.0.0/16。

执行时报了下面的错误：

[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
   [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
error execution phase preflight: [preflight] Some fatal errors occurred:
   [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

有一个错误信息是running with swap on is not supported. Please disable swap。因为我们决定配置failSwapOn: false，所以重新添加–ignore-preflight-errors=Swap参数忽略这个错误，重新运行。

kubeadm init \
   --kubernetes-version=v1.13.0 \
   --pod-network-cidr=10.244.0.0/16 \
   --apiserver-advertise-address=192.168.0.121 \
   --ignore-preflight-errors=Swap

如果出现以下错误提示

[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-apiserver:v1.13.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-controller-manager:v1.13.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-scheduler:v1.13.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/kube-proxy:v1.13.0: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/pause:3.1: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/etcd:3.2.24: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[ERROR ImagePull]: failed to pull image k8s.gcr.io/coredns:1.2.6: output: Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
, error: exit status 1
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

是因为 导入docker k8s 镜像 这些镜像默认在google是无法访问到的 需要从dockerhub官方网站上把需要的镜像先下载到本地 然后修改镜像的tag 再执行kubeadm init kubeadm首先会查看本机是否存在相应的docker镜像 如果有就直接启动本机的镜像如果没有就会从https://k8s.gcr.io仓库中下载相关镜像
上面的提示信息中包含所需要的镜像的名称和tag 根据提示所需的名称和tag来从dockerhub中下载镜像到本机

# 从dockerhub下载需要的镜像
docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.13.0
docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.13.0
docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.13.0
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.13.0
docker pull mirrorgooglecontainers/pause:3.1
docker pull mirrorgooglecontainers/etcd-amd64:3.2.24
docker pull coredns/coredns:1.2.6
# 修改dockerhub镜像tag为k8s.gcr.io
docker tag docker.io/mirrorgooglecontainers/kube-apiserver-amd64:v1.13.0 k8s.gcr.io/kube-apiserver:v1.13.0
docker tag docker.io/mirrorgooglecontainers/kube-controller-manager-amd64:v1.13.0 k8s.gcr.io/kube-controller-manager:v1.13.0
docker tag docker.io/mirrorgooglecontainers/kube-scheduler-amd64:v1.13.0 k8s.gcr.io/kube-scheduler:v1.13.0
docker tag docker.io/mirrorgooglecontainers/kube-proxy-amd64:v1.13.0 k8s.gcr.io/kube-proxy:v1.13.0
docker tag docker.io/mirrorgooglecontainers/pause:3.1  k8s.gcr.io/pause:3.1
docker tag docker.io/mirrorgooglecontainers/etcd-amd64:3.2.24  k8s.gcr.io/etcd:3.2.24
docker tag docker.io/coredns/coredns:1.2.6  k8s.gcr.io/coredns:1.2.6
# 删除多余镜像
docker rmi mirrorgooglecontainers/kube-apiserver-amd64:v1.13.0
docker rmi mirrorgooglecontainers/kube-controller-manager-amd64:v1.13.0
docker rmi mirrorgooglecontainers/kube-scheduler-amd64:v1.13.0
docker rmi mirrorgooglecontainers/kube-proxy-amd64:v1.13.0
docker rmi mirrorgooglecontainers/pause:3.1
docker rmi mirrorgooglecontainers/etcd-amd64:3.2.24
docker rmi coredns/coredns:1.2.6

再次运行init

[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
   [WARNING Swap]: running with swap on is not supported. Please disable swap
   [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.0.121]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master localhost] and IPs [192.168.0.121 127.0.0.1 ::1]
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 28.005180 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "master" as an annotation
[mark-control-plane] Marking the node master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: bag6qi.1qc7mfs75d5r1znq
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.0.121:6443 --token bag6qi.1qc7mfs75d5r1znq --discovery-token-ca-cert-hash sha256:5c3533240bf14b1aacc33e92ebde929f9e573752b629262fa43db02c4e656b8f

上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。
其中有以下关键内容：

[kubelet-start] 生成kubelet的配置文件/var/lib/kubelet/config.yaml
[certificates]生成相关的各种证书
[kubeconfig]生成相关的kubeconfig文件
[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到

下面的命令是配置常规用户如何使用kubectl访问集群：    
```bash
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

最后给出了将节点加入集群的命令kubeadm join 192.168.0.121:6443 --token bag6qi.1qc7mfs75d5r1znq --discovery-token-ca-cert-hash sha256:5c3533240bf14b1aacc33e92ebde929f9e573752b629262fa43db02c4e656b8f

查看一下集群状态：

kubectl get cs

确认个组件都处于healthy状态。

集群初始化如果遇到问题，可以使用下面的命令进行清理：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

安装Pod Network

接下来安装flannel network add-on：

kubectl create -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f  https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

执行结果

clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
daemonset.extensions/kube-flannel-ds-arm64 created
daemonset.extensions/kube-flannel-ds-arm created
daemonset.extensions/kube-flannel-ds-ppc64le created
daemonset.extensions/kube-flannel-ds-s390x created

这里注意kube-flannel.yml这个文件里的flannel的镜像是0.10.0，quay.io/coreos/flannel:v0.10.0-amd64

如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用–iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上–iface=<iface-name>

......
containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=eth1
......

使用kubectl get pod --all-namespaces -o wide确保所有的Pod都处于Running状态。

NAMESPACE     NAME                             READY   STATUS    RESTARTS   AGE   IP              NODE     NOMINATED NODE   READINESS GATES
default       curl-66959f6557-47582            1/1     Running   2          36h   10.244.0.6      master   <none>           <none>
kube-system   coredns-86c58d9df4-wdh4x         1/1     Running   1          37h   10.244.0.5      master   <none>           <none>
kube-system   coredns-86c58d9df4-zfrm5         1/1     Running   1          37h   10.244.0.7      master   <none>           <none>
kube-system   etcd-master                      1/1     Running   2          37h   192.168.0.121   master   <none>           <none>
kube-system   kube-apiserver-master            1/1     Running   2          37h   192.168.0.121   master   <none>           <none>
kube-system   kube-controller-manager-master   1/1     Running   3          37h   192.168.0.121   master   <none>           <none>
kube-system   kube-flannel-ds-amd64-66kns      1/1     Running   1          36h   192.168.0.123   node2    <none>           <none>
kube-system   kube-flannel-ds-amd64-6s7v5      1/1     Running   1          37h   192.168.0.121   master   <none>           <none>
kube-system   kube-flannel-ds-amd64-9j5sj      1/1     Running   1          36h   192.168.0.122   node1    <none>           <none>
kube-system   kube-proxy-4vn59                 1/1     Running   1          36h   192.168.0.123   node2    <none>           <none>
kube-system   kube-proxy-fsp2d                 1/1     Running   1          36h   192.168.0.121   master   <none>           <none>
kube-system   kube-proxy-qkg7t                 1/1     Running   1          36h   192.168.0.122   node1    <none>           <none>
kube-system   kube-scheduler-master            1/1     Running   3          37h   192.168.0.121   master   <none>           <none>

master node参与工作负载

使用kubeadm初始化的集群，出于安全考虑Pod不会被调度到Master Node上，也就是说Master Node不参与工作负载。这是因为当前的master节点master被打上了node-role.kubernetes.io/master:NoSchedule的污点：

kubectl describe node master | grep Taint
Taints:             node-role.kubernetes.io/master:NoSchedule

因为这里搭建的是测试环境，去掉这个污点使master参与工作负载：

kubectl taint nodes master node-role.kubernetes.io/master-
node "master" untainted

测试DNS

kubectl run curl --image=radial/busyboxplus:curl -it

该步骤时间较长，请耐心等待
当出现

[ root@curl-66959f6557-47582:/ ]$

时，执行

nslookup kubernetes.default

结果为

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

向Kubernetes集群中添加Node节点

下面我们将node2这个主机添加到Kubernetes集群中，因为我们同样在node2上的kubelet的启动参数中去掉了必须关闭swap的限制，所以同样需要–ignore-preflight-errors=Swap这个参数。 在node2上执行:

kubeadm \
    join 192.168.0.121:6443 \
    --token bag6qi.1qc7mfs75d5r1znq \
    --discovery-token-ca-cert-hash sha256:5c3533240bf14b1aacc33e92ebde929f9e573752b629262fa43db02c4e656b8f \
    --ignore-preflight-errors=Swap

结果

[preflight] Running pre-flight checks
   [WARNING Swap]: running with swap on is not supported. Please disable swap
   [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[discovery] Trying to connect to API Server "192.168.0.121:6443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.0.121:6443"
[discovery] Requesting info from "https://192.168.0.121:6443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.0.121:6443"
[discovery] Successfully established connection with API Server "192.168.0.121:6443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "node1" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the master to see this node join the cluster.

加入集群成功，在master节点执行kubectl get nodes结果

NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   36h   v1.13.2
node1    Ready    <none>   35h   v1.13.2
node2    Ready    <none>   35h   v1.13.2

如何从集群中移除Node
如果需要从集群中移除node2这个Node执行下面的命令：
在master节点上执行：

kubectl drain node1 --delete-local-data --force --ignore-daemonsets
kubectl delete node node1

在node1上执行：

kubeadm reset
ifconfig cni0 down
ip link delete cni0
ifconfig flannel.1 down
ip link delete flannel.1
rm -rf /var/lib/cni/

在master上执行：

kubectl delete node node1

kube-proxy开启ipvs

修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: "ipvs"：

kubectl edit cm kube-proxy -n kube-system

之后重启各个节点上的kube-proxy pod，在master节点运行：

kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'

kubectl get pod -n kube-system | grep kube-proxy

# 结果
kube-proxy-4vn59                 1/1     Running   0          100s
kube-proxy-fsp2d                 1/1     Running   0          86s
kube-proxy-qkg7t                 1/1     Running   0          97s

kubectl logs kube-proxy-4vn59 -n kube-system

# 结果
I0115 08:53:45.580868       1 server_others.go:189] Using ipvs Proxier.
W0115 08:53:45.581086       1 proxier.go:365] IPVS scheduler not specified, use rr by default
I0115 08:53:45.583075       1 server_others.go:216] Tearing down inactive rules.
I0115 08:53:45.614150       1 server.go:464] Version: v1.13.0
I0115 08:53:45.618996       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0115 08:53:45.619467       1 config.go:202] Starting service config controller
I0115 08:53:45.619474       1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0115 08:53:45.619482       1 config.go:102] Starting endpoints config controller
I0115 08:53:45.619485       1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0115 08:53:45.719664       1 controller_utils.go:1034] Caches are synced for service config controller
I0115 08:53:45.719739       1 controller_utils.go:1034] Caches are synced for endpoints config controller
# 结果

日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。