使用虚拟机搭建k8s集群流程以及相关问题解决

本文将介绍使用三台操作系统为centos7的虚拟机搭建k8s集群过程以及搭建过程中存在的一些问题

1. 虚拟机配置静态ip

虚拟机修改网络配置为桥接，选择当前主机使用的网卡

网络配置.png

ip route命令查看当前虚拟机使用的网卡:

default via 192.168.0.1 dev enp0s3 proto static metric 100

修改网卡配置,设置静态ip

vim /etc/sysconfig/network-scripts/ifcfg-enp0s3

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
#设置为静态
BOOTPROTO=static 
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=enp0s3
UUID=6f70c8cf-cc74-434e-86a4-b7b290936252
DEVICE=enp0s3
ONBOOT=yes
#192.168.0后随便使用一个未被占用的ip
IPADDR=192.168.0.212  
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
DNS1=119.29.29.29

重启服务

service network restart

2. 预环境搭建

#修改hostname
hostnamectl set-hostname k8s-node1

#修改host
cat >> /etc/hosts <<EOF
192.168.0.210 k8s-node1
192.168.0.211 k9s-node2
192.168.0.212 k8s-node3 
EOF


#关闭防火墙
systemctl disable firewalld
systemctl status firewalld


关闭linux安全策略
sed -i 's/enforcing/disabled/' /etc/selinux/config
setenforce 0


#关闭swap
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
free -g

#将桥接的IPv4流量传递到iptables的链:
cat > /etc/sysctl.d/k8s.conf <<EOF
net.bridge.bridge-nf-call-ip6tables=1
net.bridge.bridge-nf-call-iptables=1
EOF
sysctl --system

3.安装docekr（见官方文档）

4.安装k8s相关

为所有节点安装k8s

yum install -y kubelet-1.17.3 kubeadm-1.17.3 kubectl-1.17.3
systemctl enable kubelet
systemctl start kubelet

由于yum源中无对应版本软件，所以需要先添加yum源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

5.部署k8s

1. 提前下载相关镜像

由于在初始化过程中创建镜像时需要从外网获取，所以提前下载相关镜像，这里使用sh脚本安装，也可以手动安装

master_images.sh

#!/bin/bash

images=(
    kube-apiserver:v1.17.3
        kube-proxy:v1.17.3
    kube-controller-manager:v1.17.3
    kube-scheduler:v1.17.3
    coredns:1.6.5
    etcd:3.4.3-0
    pause:3.1
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
#   docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName  k8s.gcr.io/$imageName
done

node_images.sh

#!/bin/bash

images=(
    kube-proxy:v1.17.3
    pause:3.1
)

for imageName in ${images[@]} ; do
    docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName
#    docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/$imageName  k8s.gcr.io/$imageName
done

将两个文件分别上传到主从节点服务器，添加执行权限

chmod -R 777 对应文件.sh

执行./xxx.sh拉取相关镜像

2. 创建master节点

（1）初始化

kubeadm init \
--apiserver-advertise-address=192.168.0.210 \
--image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
--kubernetes-version v1.17.3 \
--service-cidr=10.96.0.0/16 \
--pod-network-cidr=10.244.0.0/16

apiserver-advertise-address为集群主节点的ip地址

遇到错误：

[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1

解决方案：

echo 1 > /proc/sys/net/ipv4/ip_forward

再次运行初始化命令，一段时间后控制台打印

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.0.210:6443 --token 5pm3nt.9emk7awc1j3irr4l \
    --discovery-token-ca-cert-hash sha256:ae6e24fa62360f3d84de0092e27ecf2c5e4c169d7f0ab081f969563918ae3e84

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

（2）配置使用cluster的相关配置

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

（3）安装pod网络

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

查看fiannel是否启动成功

kubectl get pods --all-namespaces

若等待初始化一段时间后报 Init:ImagePullBackOff 错误，说明从docker中拉取kube-flannel-ds-amd64镜像失败

可以先将kube-flannel.yml下载到本地，修改文件中kube-flannel-ds-amd64的镜像地址为可以拉取到的镜像

我这里修改为 jmgao1983/flannel:latest

修改完成后上传到服务器,重新配置

kubectl apply -f ./kube-flannel.yml 
kubectl get pods --all-namespaces

最终flannel启动成功

（4）查看节点状态

kubectl get nodes

输出如下：

NAME        STATUS     ROLES    AGE   VERSION
k8s-node1   Ready      master   77m   v1.17.3

3. 其他节点加入到主节点

安装主节点安装成功的提示执行如下命令

kubeadm join 192.168.0.210:6443 --token 5pm3nt.9emk7awc1j3irr4l \
    --discovery-token-ca-cert-hash sha256:ae6e24fa62360f3d84de0092e27ecf2c5e4c169d7f0ab081f969563918ae3e84

若丢失token与discovery-token-ca-cert-hash值,使用如下命令查看

kubeadm token list

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | \
   openssl dgst -sha256 -hex | sed 's/^.* //'

token在24小时后失效。失效后使用主节点重新创建token

kubeadm token create

参考博客

https://blog.51cto.com/3241766/2405624

http://www.amd5.cn/atang_3742.html