最近Kubernetes v1.16.3 刚刚发布，由于公司要上线业务，所以就先在线下自己搭环境测试。由于之前的数据中心在国外，所以安装Kubernetes去gcr.io，k8s.gcr.io拉取镜像的时候没有问题。 但是现在的公司VM在阿里云上，众所周知，由于一些不可抗力国内无法直接拉取google上的任何镜像，这就包括gcr.io。
之前用了一些做法，就是先找一台可以科学上网的机器，把镜像先拉下来，然后重新打tag。或者利用shell脚本自动pull，tag，然后push到private registry，然后安装的时候指定自己的私库。
但是由于版本更迭，有时候tag就变了，需要改脚本重新测试，所以感觉一直不完美。
后来发现国内可以直接用azure.cn的镜像加速，直接拉取k8s需要的所有镜像。之前用过阿里云的镜像加速站点，也是可以的。Azure.cn基本和google同步，速度可能更快一些，个人喜欢用azure.cn的镜像。下面就是Azure镜像的常用站点：
docker hub 镜像：
docker pull dockerhub.azk8s.cn/xxx/yyy:tag
gcr.io 镜像：
docker pull gcr.azk8s.cn/xxx/yyy:tag
k8s.gcr.io 镜像：
对于kubernetes相关的镜像，我们会使用到k8s.gcr.io开头的镜像。
k8s.gcr.io等价于gcr.io/google-containers，因此同上也可以使用中科大镜像或者Azure中国镜像。
docker pull gcr.azk8s.cn/google-containers/xxx:yyy
quay.io镜像：
docker pull quay.azk8s.cn/xxx/yyy:zzz
下面开始Kubernetes v1.16.3的正式安装：
系统要求：
One or more machines running one of:
Ubuntu 16.04+
Debian 9+
CentOS 7
Red Hat Enterprise Linux (RHEL) 7
Fedora 25+
HypriotOS v1.0.1+
Container Linux (tested with 1800.6.0)
2 GB or more of RAM per machine (any less will leave little room for your apps)
2 CPUs or more
Full network connectivity between all machines in the cluster (public or private network is fine)
Unique hostname, MAC address, and product_uuid for every node. See here for more details.
Certain ports are open on your machines. See here for more details.
Swap disabled. You MUST disable swap in order for the kubelet to work properly.
安装环境：
VMwareWorkstation or VirtualBOX：
1台master：4 core 8GB
2台worker：4 core 8GB
自动化工具：
ansible
Kubernetes安装工具：
kubeadm
利用kubeadm安装集群
首先，准备好3台VM
master IP：192.168.199.200
worker1：192.168.199.201
worker2：192.168.199.202
然后做一些初始化工作：
CentOS/RHEL:
关闭swap分区
关闭selinux
关闭NetworkManager
关闭firewalld
添加从master到worker节点的免密钥登录
由于没有DNS服务器，需要配置主机上的/etc/hosts，把master和worker节点对应添加进去
下面正式开始：
1 登录到master节点，安装ansible
#CentOS/RHEL:
yum install epel-release
yum install -y ansible python2-pip.noarch
pip install --upgrade ansible
2 编写ansible inventory文件
vim ~/inventory
[all]
k8s-master node_ip=192.168.199.200
node1 node_ip=192.168.199.201
node2 node_ip=192.168.199.202
3 编写playbook (下面的playbook是以Centos7为例，如果是Ubuntu需要把包管理器从yum改为apt,rpm_key改为apt_key)
vim ~/host-prepare.yml
---
- hosts: all
become: true
tasks:
- name: Disable NetworkManager
service:
name: NetworkManager
state: stopped
enabled: false
- name: Disable firewalld
service:
name: firewalld
state: stopped
enabled: false
- name: Install docker and its dependecies
yum:
name: docker
state: latest
update_cache: yes
- name: Start docker
service:
name: docker
state: started
enabled: yes
- name: Remove swapfile from /etc/fstab
mount:
name: "{{ item }}"
fstype: swap
state: absent
with_items:
- swap
- none
- name: Disable swap
command: swapoff -a
when: ansible_swaptotal_mb > 0
- name: Disable Selinux
selinux:
state: disabled
- name: Add kubernetes repository for stable version
yum_repository:
description: kubernetes
name: kubernetes
baseurl: https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled: yes
gpgcheck: yes
repo_gpgcheck: yes
gpgkey: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
state: present
- name: Add an yum signing key for Kubernetes
rpm_key:
key: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
state: present
- name: Add an rpm signing key for Kubernetes
rpm_key:
key: https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
state: present
- name: Install Kubernetes binaries
package:
name: "{{ packages }}"
state: present
use: yum
vars:
packages:
- kubelet
- kubeadm
- kubectl
- name: Configure node ip
lineinfile:
path: /etc/sysconfig/kubelet
line: KUBELET_EXTRA_ARGS=--node-ip={{ node_ip }}
- name: Restart kubelet
service:
name: kubelet
daemon_reload: yes
state: restarted
enabled: true
4 运行playbook
首先在master上生成ssh密钥：
ssh-keygen
ssh-copy-id root@k8s-master
ssh-copy-id root@node1
ssh-copy-id root@node2
sed -i s/#host_key_checking/host_key_checking/ /etc/ansible/ansible.cfg
ansible-playbook -i ~/inventory ~/host-prepare.yml
5 在master节点运行kubeadm安装kubernetes：
这里标红的参数，就是我指定image仓库的地址，默认为gcr.io，这里我改为Azure.cn的地址
kubeadm init --node-name k8s-master --apiserver-advertise-address <your master node IP> --pod-network-cidr 192.168.0.0/16 --image-repository gcr.azk8s.cn/google-containers
根据机器配置和网速，等待时间不等，大概5分钟，可以看到master节点就成功安装了。看到如下提示，就可以根据最后一条命令加入worker node了。
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 192.168.199.200:6443 --token 95u7da.08yb7leugjunvg3s --discovery-token-ca-cert-hash sha256:cc40687e79b5ea1486d3f1ea066789578758822a46d1d54516cc8d9ff28cf774
加入worker node之前，我们按照提示运行：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
我们还需要安装网络组件，这里我选择的是Calico，所以在master上运行以下命令即可安装Calico：
kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
然后利用kubectl查看集群健康状态（如果没运行上一步，安装CNI的过程，这里master的STATUS将会是NotReady）：
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 3m50s v1.16.3
这里我们看到集群master部署完成了，而且可以正常连接。如果master状态还是NotReady，说明没有部署Calico,重新部署Calico之后，状态会变成Ready
[root@k8s-master ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-55754f75c-zgxgv 1/1 Running 0 38s
kube-system calico-node-vsk9v 1/1 Running 0 38s
kube-system coredns-58ffd68966-4lf5s 1/1 Running 0 4m56s
kube-system coredns-58ffd68966-5xpvf 1/1 Running 0 4m56s
kube-system etcd-k8s-master 1/1 Running 0 4m5s
kube-system kube-apiserver-k8s-master 1/1 Running 0 4m12s
kube-system kube-controller-manager-k8s-master 1/1 Running 0 3m55s
kube-system kube-proxy-hmkv2 1/1 Running 0 4m56s
kube-system kube-scheduler-k8s-master 1/1 Running 0 3m46s
我们可以看到master节点已经正常了，各组件也正常启动，没问题。
6 最后一步，加入worker node
我们在两台worker节点上按照提示运行：
kubeadm join 192.168.199.200:6443 --token 95u7da.08yb7leugjunvg3s --discovery-token-ca-cert-hash sha256:cc40687e79b5ea1486d3f1ea066789578758822a46d1d54516cc8d9ff28cf774
这里注意，上面命令的token和hash是你自己机器生成的，一定要用你自己的token
最终，我们可以查看集群已经装好了：
[root@k8s-master ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-master Ready master 24m v1.16.3
node1 Ready <none> 2m51s v1.16.3
node2 Ready <none> 84s v1.16.3
[root@k8s-master ~]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system calico-kube-controllers-55754f75c-zgxgv 1/1 Running 0 19m
kube-system calico-node-rn95j 1/1 Running 0 2m47s
kube-system calico-node-rsgdb 1/1 Running 0 80s
kube-system calico-node-vsk9v 1/1 Running 0 19m
kube-system coredns-58ffd68966-4lf5s 1/1 Running 0 23m
kube-system coredns-58ffd68966-5xpvf 1/1 Running 0 23m
kube-system etcd-k8s-master 1/1 Running 0 22m
kube-system kube-apiserver-k8s-master 1/1 Running 0 22m
kube-system kube-controller-manager-k8s-master 1/1 Running 0 22m
kube-system kube-proxy-44c7r 1/1 Running 0 80s
kube-system kube-proxy-bsjn2 1/1 Running 0 2m47s
kube-system kube-proxy-hmkv2 1/1 Running 0 23m
kube-system kube-scheduler-k8s-master 1/1 Running 0 22m
