前置知识：docker入门，如何部署Django uwsgi nginx应用
参考文档：kubernetes setup doc
环境准备
练习环境采用VirtualBox运行ubuntu18.04，有现成的机器此环节可略过。
1、VirtualBox设置
1、新建虚拟电脑
2、设置》系统》cpu 2，内存2G（k8s最低要求）
3、下载Ubuntu Server 18.04 LTS，设置》存储》没有盘片》分配光驱，
选择刚刚下载的ubuntu镜像文件
4、启动，初次注意勾选安装sshd
5、配置网络：在VertualBox中，将虚拟机网络设置为桥接模式；参考宿主机网络信息，
修改虚拟机文件cat /etc/netplan/50-cloud-init.yaml，然后 netplan apply
network:
ethernets:
enp0s3:
addresses: [192.168.0.191/24] # [ip/子网掩码]
gateway4: 192.168.0.1 # 网关
dhcp4: no # 是否动态获取ip
nameservers:
addresses: [114.114.114.114] # dns
version: 2
备注：子网掩码设置请搜索CIDR（无类别域间路由，Classless Inter-Domain Routing），点击查看可用dns
6、安装docker apt-get install -y docker.io
2、安装k8s（阿里云源）
- 添加软件包密钥(apt-key)
sudo curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg --output /etc/apt/trusted.gpg.d/apt-key.gpg - 添加源
# /etc/apt/sources.list.d/kubernetes.list
deb http://mirrors.ustc.edu.cn/kubernetes/apt kubernetes-xenial main
- 安装
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
3、创建集群
1、确认机器开放端口， 如果使用aws或者阿里云机器务必配置开启相应端口的安全组
master节点开放端口
| 规则 | 方向 | 端口范围 | 作用 | 使用者 |
|---|---|---|---|---|
| TCP | Inbound | 6443* | Kubernetes API server | All |
| TCP | Inbound | 2379-2380 | etcd server client API | kube-apiserver, etcd |
| TCP | Inbound | 10250 | Kubelet API | Self, Control plane |
| TCP | Inbound | 10251 | kube-scheduler | Self |
| TCP | Inbound | 10252 | kube-controller-manager | Self |
node 节点开放端口
| 规则 | 方向 | 端口范围 | 作用 | 使用者 |
|---|---|---|---|---|
| TCP | Inbound | 10250 | Kubelet API | Self, Control plane |
| TCP | Inbound | 30000-32767 | NodePort Services** | All |
2、所有节点启用 docker service，关闭 swap
systemctl enable docker.service
swapoff -a
3、在master启动kubeadm
sudo kubeadm init --pod-network-cidr=182.168.0.0/16 --image-repository registry.aliyuncs.com/google_containers
输出：
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
Then you can join any number of worker nodes by running the following on each as root:
kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd
4、 设置 pod network calico
下载网络插件配置文件 calico.yaml
curl https://docs.projectcalico.org/v3.6/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml > calico.yaml
注意修改参数 CALICO_IPV4POOL_CIDR，与 kubeadm init 的参数 pod-network-cidr保持一致！
# The default IPv4 pool to create on startup if none exists. Pod IPs will be
# chosen from this range. Changing this value after installation will have
# no effect. This should fall within `--cluster-cidr`.
- name: CALICO_IPV4POOL_CIDR
value: "182.168.0.0/16"
执行 sudo kubectl apply -f calico.yaml
5、加入节点，在节点机器执行命令
yetongxue@vm2:~$ kubeadm join 192.168.0.191:6443 --token lgm81w.xjktiv2w32jozvgt --discovery-token-ca-cert-hash sha256:a3a535428d2302acfc4cdff1dc3a6db29bd01bbbbccae824f902842523035edd
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
在 master 检查 vm2 加入状态
yetongxue@vm1:~$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
vm1 Ready master 21m v1.15.0
vm2 Ready <none> 19m v1.15.0
如果出现问题，执行kubectl describe nodes <node_name>查看原因
k8s环境安装完成 ✌️
部署应用集群
前置知识
- pod 一个或一组容器组成，共同提供一种服务；定义文件pod.yaml中包含使用的镜像、端口、数据卷挂载、启动命令，为pod打标签label这些，类似编排多个容器的 docker-compose.yaml 的作用
- service 多个相同服务的pod组成一个服务，通过service.yaml中spec.selecter=[pod label]选择pod，负载均衡的感觉
- volume 数据卷，有很多类型：emptyDir适合pod内部各个容器共享内容，hostPath则会将目录挂载到宿主机，还有网络数据卷、分布式文件系统这些；还有一种专门当做配置文件来用的数据挂载方式——configMap，可以用一份文件来生成configMap，用来替换某些软件原来的配置文件；也可以设置很多的key value，在各种yaml文件中引用，以设置容器所需要的环境变量参数
参考资料
- kubernetes yaml格式的pod定义文件完整内容
- Creating the pod using the YAML file
- config map
- 通过configmap更新k8s里的mysql配置文件
- kubernetes 特殊存储卷ConfigMap
下面就将项目 django_demo 部署到 k8s 集群，参考 docker-compose.yaml、mysqld.cnf内容，分别创建如下文件（k8s命名一般是小驼峰/中划线）
1、通过文件 mysqld.cnf 创建 configmap
yetongxue@vm1:~$ sudo kubectl create configmap mysqld-config --from-file=mysqld.cnf
configmap/mysqld-config created
yetongxue@vm1:~$ sudo kubectl describe configmaps mysqld-config
[sudo] password for master:
Name: mysqld-config
Namespace: default
Labels: <none>
Annotations: <none>
Data
====
mysqld.cnf:
----
[mysqld]
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = /var/lib/mysql
#log-error = /var/log/mysql/error.log
# By default we only accept connections from localhost
bind-address = 0.0.0.0
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# utf8mb4
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_ci
init_connect='SET NAMES utf8mb4'
skip-character-set-client-handshake = true
Events: <none>
2、将 configmap mysql-config 挂载到 pod，替换 mysql 镜像原有的myqld.cnf；将mysql数据目录（见mysqd.cnf 中 datadir 字段）/var/lib/mysql 挂载到宿主机/var/lib/mysql-datadir
# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mysql
labels:
app: database
version: v1
release: dev
spec:
restartPolicy: Always
containers:
- name: mysql
image: mysql:5.7
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: MYSQL_ROOT_PASSWORD
value: root
- name: MYSQL_USER
value: yetongxue
- name: MYSQL_PASSWORD
value: qwerasdf
- name: MYSQL_DATABASE
value: django_demo
volumeMounts:
- name: config-volume-mysql
mountPath: /etc/mysql/mysql.conf.d/
- name: mysql-datadir
mountPath: /var/lib/mysql
volumes:
- name: config-volume-mysql
configMap:
name: mysqld-config
- name: mysql-datadir
hostPath:
path: /var/lib/mysql-datadir
启动并验证 mysql pod 是否正常
yetongxue@vm1:~$ sudo kubectl apply -f mysql-pod.yaml
pod/mysql created
yetongxue@vm1:~$ sudo kubectl get pod mysql
NAME READY STATUS RESTARTS AGE
mysql 1/1 Running 0 9d
yetongxue@vm1:~$ sudo kubectl exec -it mysql bash
root@mysql:/# mysql -u root -p
Enter password:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| django_demo |
| mysql |
| performance_schema |
| sys |
+--------------------+
5 rows in set (0.00 sec)
检查数据挂载是否成功
yetongxue@vm2:~$ ls /var/lib/mysql-datadir/
auto.cnf ca.pem client-key.pem ib_buffer_pool ib_logfile0 ibtmp1 performance_schema public_key.pem server-key.pem
ca-key.pem client-cert.pem django_demo ibdata1 ib_logfile1 mysql private_key.pem server-cert.pem sys
这里有一个优化的地方，因为 mysql-pod.yaml 中使用了许多的环境变量，这些变量在下面的 django-demo 部署中依然会使用到，所以我们将这些变量剥离出来，通过 configMap 引用。
创建 mysql 环境变量 configMap，参考 Configure all key-value pairs in a ConfigMap as container environment variables
# env-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: env-config
data:
MYSQL_ROOT_PASSWORD: root
MYSQL_USER: yetongxue
MYSQL_PASSWORD: qwerasdf
MYSQL_DATABASE: django_demo
使用前先创建 configMap
yetongxue@vm1:~$ sudo kubectl apply -f env-config.yaml
configmap/env-config created
下面将 mysql-pod.yaml 改造一下，环境变量通过 configMap 的方式引用，如下：
# mysql-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: mysql
labels:
app: database
version: v1
release: dev
spec:
restartPolicy: Always
containers:
- name: mysql
image: mysql:5.7
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: MYSQL_ROOT_PASSWORD
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_ROOT_PASSWORD
- name: MYSQL_USER
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_DATABASE
volumeMounts:
- name: config-volume-mysql
mountPath: /etc/mysql/mysql.conf.d/
- name: mysql-datadir
mountPath: /var/lib/mysql
volumes:
- name: config-volume-mysql
configMap:
name: mysqld-config
- name: mysql-datadir
hostPath:
path: /var/lib/mysql-datadir
这里还有一个优化的地方 😂，一般我们都不通过 pod.yaml 直接创建pod，而是通过 Deployment Controller 来自动创建 pod，Deployment 定义文件 deployment.yaml 中参数replicas设置期望运行的 pod 数量，Deployment 会自动增加或者删除 pod 来维持；另外参数template的内容便是刚刚的 pod.yaml 内容，所以 RC 便知道如何去创建 pod；弹性伸缩、滚动升级也是通过它来实现的。
# mysql-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql-deployment
spec:
selector:
matchLabels:
app: database
replicas: 1
template:
metadata:
name: mysql
labels:
app: database
spec:
restartPolicy: Always
containers:
- name: mysql
image: mysql:5.7
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: MYSQL_ROOT_PASSWORD
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_ROOT_PASSWORD
- name: MYSQL_USER
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_DATABASE
volumeMounts:
- name: config-volume-mysql
mountPath: /etc/mysql/mysql.conf.d/
- name: mysql-datadir
mountPath: /var/lib/mysql
volumes:
- name: config-volume-mysql
configMap:
name: mysqld-config
- name: mysql-datadir
hostPath:
path: /var/lib/mysql-datadir
删除刚刚创建的 pod: sudo kubectl delete pod mysql
通过 RC 创建 pod: sudo kubectl apply -f mysql-deployment.yaml
这里有两个地方需要注意：
- spec.selector.matchLabels务必和 spec.template.matadata.labels保持一致，否则RC通过template创建出pod后，通过matchLabels匹配不到相应的pod就会不停的创建。
- 对于mysql这种有数据状态的应用，不能简单的设置replicas为大于1的数值来扩容，这又是个优化的地方，有兴趣的同学可参考 Run a Replicated Stateful Application
3、创建 msyql service
Kubernetes 中 service 有三种类型:
- ClusterIP:提供一个集群内部的虚拟IP以供Pod访问。
- NodePort:在每个Node上打开一个端口以供外部访问。
- LoadBalancer:通过外部的负载均衡器来访问。
显然，这里的mysql用ClusterIP就好；待会下面创建django-demo service就用LoadBalancer。
# mysql-service.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql-service
spec:
selector:
app: database #通过label选择加入service的pod
ports:
- port: 3306
targetPort: 3306
protocol: TCP
启动mysql service
yetongxue@vm1:~$ sudo kubectl apply -f mysql-service.yaml
service/mysql-service created
yetongxue@vm1:~$ sudo kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 10h
mysql-service ClusterIP 10.108.192.143 <none> 3306/TCP 46s
yetongxue@vm1:~$ sudo kubectl get endpoints mysql-service
NAME ENDPOINTS AGE
mysql-service 182.168.185.194:3306 24m
ENDPOINTS表明已经将刚刚创建的mysql pod纳入了mysql service.
4、创建 django-demo deployment
先把通过 Dockerfile 构建的镜像上传到阿里云镜像仓库 registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev。
由于RC的好处是明显的，所以现在我们就直接通过RC来创建pod
注意将MYSQL_HOST修改成刚刚创建的mysql-service的ClusterIP，或者直接就用service name: mysql-service
# django-demo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: django-demo-deployment
spec:
selector:
matchLabels:
app: web
replicas: 2
template:
metadata:
name: django-demo
labels:
app: web
spec:
restartPolicy: Always
containers:
- name: demo
image: registry.cn-hangzhou.aliyuncs.com/yetongxue/django_demo:dev
imagePullPolicy: IfNotPresent
env:
- name: TZ
value: Asia/Shanghai
- name: MYSQL_HOST
value: mysql-service
- name: MYSQL_USER
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_USER
- name: MYSQL_PASSWORD
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_PASSWORD
- name: MYSQL_DATABASE
valueFrom:
configMapKeyRef:
name: env-config
key: MYSQL_DATABASE
ports:
- containerPort: 80
启动 django-demo deployment，并验证连接数据库是否正常。
yetongxue@vm1:~$ sudo kubectl apply -f django-demo-deployment.yaml
deployment.apps/django-demo-deployment created
yetongxue@vm1:~$ sudo kubectl get pods
NAME READY STATUS RESTARTS AGE
django-demo-deployment-8477cd6dbb-4dz9b 0/1 ContainerCreating 0 57s
django-demo-deployment-8477cd6dbb-7jssg 0/1 ContainerCreating 0 57s
mysql-deployment-588c96799f-7md5n 1/1 Running 0 24m
yetongxue@vm1:~$ sudo kubectl exec -it mysql-deployment-588c96799f-7md5n bash
root@mysql-deployment-588c96799f-7md5n:/# mysql -u root -p
Enter password:
mysql> use django_demo
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------------------------+
| Tables_in_django_demo |
+--------------------------------+
| auth_group |
| auth_group_permissions |
| auth_permission |
| authtoken_token |
| django_admin_log |
| django_content_type |
| django_migrations |
| django_session |
| test_app_book |
| test_app_user |
| test_app_user_groups |
| test_app_user_user_permissions |
+--------------------------------+
12 rows in set (0.01 sec)
5、创建django-demo service
# django-demo-service.yaml
apiVersion: v1
kind: Service
metadata:
name: django-demo-service
spec:
selector:
app: web
externalIPs: ["192.168.0.191"] # 指定一个外部IP，这里就用master节点IP
ports:
- name: http
port: 8000
targetPort: 80
protocol: TCP
type: LoadBalancer # 类型负载均衡
启动并验证是否成功
yetongxue@vm1:~$ sudo kubectl apply -f django-demo-service.yaml
service/django-demo-service created
访问地址：http://192.168.0.191:8000/admin
账号：admin
密码：qwerasdf
截图纪念如下：
image.png
问题收集
在参照文档进行练习时，遇到的一些问题记录如下
1、在master执行kubeadm reset之后，如果你之前执行过命令sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config，则需要再次执行，否则权限无法验证，因为$HOME/.kube/config里面存的还是上次kubeadm init生成的文件；node被移除或者master执行了kubeadm reset， 在执行kubeadm join之前还需要执行 kubeadm reset，否则在进行 kubeadm join时产生如下错误
[kubelet-start] Activating the kubelet service
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
[kubelet-check] Initial timeout of 40s passed.
packet_write_wait: Connection to 192.168.0.192 port 22: Broken pipe
2、在使用VirtualBox时，安装完ubuntu我reboot了（以后一定要通过VirtualBox操作），结果当前虚拟机在VirtualBox的状态就变成了Guru Meditation，无法关闭启动删除，只可查看日志，又没看懂日志。不知道为啥，最后直接kill相应进程解决
常用命令
- 查看 pod:
kubectl describe pod --namespace=<name_space> <pod_name> - 进入pod:
kubectl exec -it <pod_name> bash - 查看pod日志：
kubectl logs <pod_name>
