高阶k8s HA 集群搭建（一）

前言

尝到k8s甜头以后，我们就想着应用到生产环境里去，以提高业务迭代效率，可是部署在生产环境里有一个要求，就是k8s集群不可以存在单点故障。。。诶唷我的乖乖，这不就要求k8s集群高可用吗，好，下面就是介绍两种目前比较火的k8s集群master高可用方式。

介绍

首先介绍的第一种k8sHA集群我觉得更应该叫做主从结构k8s集群，它由三台master组成，有三个keepalived提供一个vip 来作为apiserver的ip入口，keepalived设置权重，使得vip落在权重大的master节点上，node节点通过访问这个vip从而访问到这一台master，另外两台master则通过etcd集群，来完成数据同步。

缺点：这样的集群是通过keepalived来实现高可用的，也就是说在权重较大的节点没有故障之前，keepalived所指向的流量永远都是经过主master，只有当主master出现故障或者宕机的情况下，才有可能转移到另外两台从master节点上。这样会导致主master节点压力过大，而另外两台从master可能永远不会被调用，导致资源浪费等等情况。

不过，这也是排除单点故障的一种方式。

下面是理想的高可用架构图。

k8s 理想HA高可用

本文中要部署高可用的架构图：

本文高可用架构

上图摘抄至https://www.kubernetes.org.cn/3536.html

好了，到此我们整理一下本文中需要使用的技术栈

keepalived+etcd+k8s master

其中keepalived提供vip供node做apiserver入口，etcd必须是高可用集群，实现数据同步；以及基本的k8s master节点部署。

安装准备

节点部署相关情况

软件版本：

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

以上软件在上一篇初阶k8s集群搭建里已经介绍并附有下载地址。

环境配置

systemctl stop firewalldsystemctl disable firewalld

修改每个节点hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

keepalived安装

libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

wget http://www.keepalived.org/software/keepalived-1.4.3tar.gz

yum install -y libnfnetlink-devel-1.0.1-4.el7.x86_64.rpm

yum -y install libnl libnl-devel

tar -xzvf keepalived-1.4.3.tar.gz

cd keepalived-1.4.3

./configure --prefix=/usr/local/keepalived #检查环境配置

出现上图即为正确环境，如果出现错误

checking openssl/ssl.h usability... no

checking openssl/ssl.h presence... no

checking foropenssl/ssl.h... no

configure: error:

!!! OpenSSL is not properly installed on your system. !!!

!!! Can not include OpenSSL headers files. !!!

则：安装openssl和openssl-devel包，然后从新编译配置文件。

yum install openssl openssl-devel

./configure --prefix=/usr/local/keepalived

make && make install

cp keepalived/etc/init.d/keepalived /etc/init.d/

mkdir /etc/keepalived

cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

cp keepalived/etc/sysconfig/keepalived /etc/sysconfig/

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/

ps -aux |grep keepalived

chkconfig keepalived on

通过systemctl status keepalived查看keepalived状态

三台master重复以上步骤，直到完成keepalived的安装。

安装完成后编写配置文件：

master1的keepalived.conf

cat >/etc/keepalived/keepalived.conf <<EOF

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state MASTER

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 120 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.1 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

#192.168.100.1

192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

master2的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 110 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.2 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

192.168.100.1

#192.168.100.2

192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

master3的keepalived.conf

cat >/etc/keepalived/keepalived.conf <

global_defs {

router_id LVS_k8s

}

vrrp_script CheckK8sMaster{

script "curl -k https://192.168.100.4:6443"

interval 3

timeout 9

fall 2

rise 2

}

vrrp_instance VI_1 {

state BACKUP

interface ens33 #本机物理网卡名字,可通过ip a来查看

virtual_router_id 61

priority 100 # 主节点权重最高依次减少

advert_int 1

mcast_src_ip 192.168.100.3 #修改为本地IP

nopreempt

authentication {

auth_type PASS

auth_pass awzhXylxy.T

}

unicast_peer{

#注释掉本地IP

192.168.100.1

192.168.100.2

#192.168.100.3

}

virtual_ipaddress {

192.168.100.4/22 #VIP

}

track_script {

#CheckK8sMaster#这个方法在没部署k8s之前最好注释掉，因为很可能因为这个报错

}

EOF

启动keepalived

systemctl restart keepalived

通过ip a可以查看

除了本机ip还多了一个虚拟ip

也可以通过ping ip去验证vip是否生效。

安装ETCD

1：设置cfssl环境

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

chmod +x cfssljson_linux-amd64

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

chmod +x cfssl-certinfo_linux-amd64

mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfoexport PATH=/usr/local/bin:$PATH

2：创建 CA 配置文件（下面配置的IP为etc节点的IP

mkdir /root/ssl

cd /root/ssl

cat > ca-config.json <<EOF

{"signing": {"default": { "expiry": "8760h"},"profiles": { "kubernetes-Soulmate": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "8760h" }}}}EOF

cat > ca-csr.json <<EOF

{"CN": "kubernetes-Soulmate","key": {"algo": "rsa","size": 2048},"names": [{ "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System"}]}EOF

cfssl gencert -initca ca-csr.json | cfssljson -bare ca

cat > etcd-csr.json <<EOF

{ "CN": "etcd", "hosts": [ "127.0.0.1", "192.168.100.1", "192.168.100.2", "192.168.100.3" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "shanghai", "L": "shanghai", "O": "k8s", "OU": "System" } ]}EOF

cfssl gencert -ca=ca.pem \

-ca-key=ca-key.pem \

-config=ca-config.json \

-profile=kubernetes-Soulmate etcd-csr.json | cfssljson -bare etcd

3:master1分发etcd证书到master2、master3上面

mkdir -p /etc/etcd/ssl

cp etcd.pem etcd-key.pem ca.pem /etc/etcd/ssl/

ssh -n master2 "mkdir -p /etc/etcd/ssl && exit"

ssh -n master3 "mkdir -p /etc/etcd/ssl && exit"

scp -r /etc/etcd/ssl/*.pem master2:/etc/etcd/ssl/

scp -r /etc/etcd/ssl/*.pem master3:/etc/etcd/ssl/

解压etcd-v3.3.2-linux-amd64.tar.gz并安装

wget https://github.com/coreos/etcd/releases/download/v3.3.2/etcd-v3.3.2-linux-amd64.tar.gz

tar -xzvf etcd-v3.3.2-linux-amd64.tar.gz

cd etcd-v3.3.2-linux-amd64

cp etcd* /bin/

#查看是否安装好

etcd -version

etcd Version: 3.3.2Git SHA: c9d46ab37Go Version: go1.9.4Go OS/Arch: linux/amd64

etcdctl -version

etcdctl version: 3.3.2API version: 2

在每一个master上都创建一个etcd存储目录mkdir -p /u03/etcd/

这里可以自行选择储存数据地址，但是要记得在下面配置文件中做修改

master1

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

--name master1 \

--cert-file=/etc/etcd/ssl/etcd.pem \

--key-file=/etc/etcd/ssl/etcd-key.pem \

--peer-cert-file=/etc/etcd/ssl/etcd.pem \

--peer-key-file=/etc/etcd/ssl/etcd-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--initial-advertise-peer-urls https://192.168.100.1:2380 \

--listen-peer-urls https://192.168.100.1:2380 \

--listen-client-urls https://192.168.100.1:2379,http://127.0.0.1:2379 \

--advertise-client-urls https://192.168.100.1:2379 \

--initial-cluster-token etcd-cluster-0 \

--initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

--initial-cluster-state new \

--data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

master2

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

--name master2 \

--cert-file=/etc/etcd/ssl/etcd.pem \

--key-file=/etc/etcd/ssl/etcd-key.pem \

--peer-cert-file=/etc/etcd/ssl/etcd.pem \

--peer-key-file=/etc/etcd/ssl/etcd-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--initial-advertise-peer-urls https://192.168.100.2:2380 \

--listen-peer-urls https://192.168.100.2:2380 \

--listen-client-urls https://192.168.100.2:2379,http://127.0.0.1:2379 \

--advertise-client-urls https://192.168.220.146:2379 \

--initial-cluster-token etcd-cluster-0 \

--initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

--initial-cluster-state new \

--data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

master3

cat <<EOF >/etc/systemd/system/etcd.service

[Unit]

Description=Etcd Server

After=network.target

After=network-online.target

Wants=network-online.target

Documentation=https://github.com/coreos

[Service]

Type=notify

WorkingDirectory=/u03/etcd/

ExecStart=/usr/bin/etcd \

--name master3 \

--cert-file=/etc/etcd/ssl/etcd.pem \

--key-file=/etc/etcd/ssl/etcd-key.pem \

--peer-cert-file=/etc/etcd/ssl/etcd.pem \

--peer-key-file=/etc/etcd/ssl/etcd-key.pem \

--trusted-ca-file=/etc/etcd/ssl/ca.pem \

--peer-trusted-ca-file=/etc/etcd/ssl/ca.pem \

--initial-advertise-peer-urls https://192.168.100.3:2380 \

--listen-peer-urls https://192.168.100.3:2380 \

--listen-client-urls https://192.168.100.3:2379,http://127.0.0.1:2379 \

--advertise-client-urls https://192.168.100.3:2379 \

--initial-cluster-token etcd-cluster-0 \

--initial-cluster master1=https://192.168.100.1:2380,master2=https://192.168.100.2:2380,master3=https://192.168.100.3:2380 \

--initial-cluster-state new \

--data-dir=/u03/etcd/

Restart=on-failure

RestartSec=5

LimitNOFILE=65536

[Install]

WantedBy=multi-user.target

EOF

每个master都执行以下命令以启动etcd集群

cd /etc/systemd/system/

mv etcd.service /usr/lib/systemd/system/

systemctl daemon-reload

systemctl enable etcd

systemctl restart etcd

systemctl status etcd

通过以下命令检测集群是否正常

etcdctl --endpoints=https://192.168.100.1:2379,https://192.168.100.2:2379,https://192.168.100.3:2379 \

--ca-file=/etc/etcd/ssl/ca.pem \

--cert-file=/etc/etcd/ssl/etcd.pem \

--key-file=/etc/etcd/ssl/etcd-key.pem cluster-health

keepalived+etcd安装完成后，开始部署k8s

安装docker、k8s相关rpm包，以及上传k8s相关镜像。请看我上一篇初阶k8s集群搭建。

所有节点修改kubelet配置文件

sed -i -e 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/g' /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

重启kubelet

systemctl daemon-reload && systemctl restart kubelet

初始化集群，创建集群配置文件

# 生成token

# 保留token后面还要使用

我们使用coreDNS作为k8s集群内部DNS解析，使用canal作为网络服务

kubeadm token generate

cat <<EOF > config.yaml

apiVersion: kubeadm.k8s.io/v1alpha1

kind: MasterConfiguration

etcd:

endpoints:

- https://192.168.100.1:2379

- https://192.168.100.2:2379

- https://192.168.100.3:2379

caFile: /etc/etcd/ssl/ca.pem

certFile: /etc/etcd/ssl/etcd.pem

keyFile: /etc/etcd/ssl/etcd-key.pem

dataDir: /var/lib/etcd

networking:

podSubnet: 10.244.0.0/16

kubernetesVersion: 1.10.0

api:

advertiseAddress: "192.168.150.186"

token: "hpobow.vw1g1ya5dre7sq06" #刚刚保存的token

tokenTTL: "0s"#表示永不过期

apiServerCertSANs:

- master1

- master2

- master3

- 192.168.100.1

- 192.168.100.2

- 192.168.100.3

- 192.168.100.4

featureGates:

CoreDNS: true

EOF

编辑完成后执行kubeadm init --config config.yaml

如果失败则查看错误journalctl -xeu kubelet 查看服务启动日志或根据相关日志查看问题

通过kubeadm reset重置

注意，如果etcd已经写入数据，请先到etcd存储数据路径下清空数据记录。

若成功，你会看到以下内容

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node

as root:

kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315

按照上面提示，此时root用户还不能使用kubelet控制集群需要，配置下环境变量

对于非root用户

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

对于root用户

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source一下环境变量

source ~/.bash_profile

kubeadm生成证书密码文件分发到master2和master3上面去

scp -r /etc/kubernetes/pki master2:/etc/kubernetes/

scp -r /etc/kubernetes/pki master3:/etc/kubernetes/

部署canal网络，在master1执行

kubectl apply -f \

https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml

kubectl apply-f \https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml

可能镜像下载需要一点时间，也可以先将yaml文件下载到本地，自行修改镜像路径，使用自己下载好的镜像

wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/rbac.yaml

wget https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/canal/canal.yaml

等待部署好了之后查看当前节点是否准备好

[root@master1 ~]# kubectl get node

NAME STATUS ROLES AGE VERSION

master1 Ready master 31m v1.10.0

通过kubectl get pods --all-namespaces查看是否所有的容器都已经运行，如果出现error或crash，就使用kubectl describe pod -n kube-system来查看出现的问题。

在master2和master3上面分别执行初始化

使用之前在master1执行的配置config.yaml在另外两个节点上执行kubeadm init --config config.yaml，将获得与master1一样的结果

同样的配置下环境变量

对于非root用户

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

对于root用户

echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile

source一下环境变量

source ~/.bash_profile

[root@master1 ~]# kubectl get nodes

NAME STATUS ROLES AGE VERSION

master1 Ready master 1h v1.10.0

master2 Ready master 1h v1.10.0

master3 Ready master 1h v1.10.0

查看所有节点运行的容器kubectl get pods --all-namespaces -o wide

这样，基本的主备模式的高可用就搭建完成了，若要部署dashboard请看我上一篇文章初阶k8s集群搭建，值得注意的是，设置dashboard的basicauth的方式进行apiserver的验证，这个设置需要在每一台master上执行以保证高可用。

另外，在k8s 1.10中想使用HPA需要在每个master节点 /etc/kubernetes/manifests/kube-controller-manager.yaml中增加 - --horizontal-pod-autoscaler-use-rest-clients=false 才可以监控到cpu使用率来完成自动扩容。

监控插件heapster

需要有heapster.yaml、influxdb.yaml、grafana.yaml

vim heapster.yaml

---

apiVersion: v1

kind: ServiceAccount

metadata:

namespace: kube-system

---

kind: ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

subjects:

- kind: ServiceAccount

namespace: kube-system

roleRef:

kind: ClusterRole

apiGroup: rbac.authorization.k8s.io

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

task: monitoring

k8s-app: heapster

spec:

serviceAccountName: heapster

containers:

- name: heapster

image: 192.168.220.84/third_party/heapster-amd64:v1.3.0 #这里我用的是自己的私服镜像地址

imagePullPolicy: IfNotPresent

command:

- /heapster

- --source=kubernetes:https://kubernetes.default

- --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086

---

apiVersion: v1

kind: Service

metadata:

labels:

task: monitoring

# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

# If you are NOT using this as an addon, you should comment out this line.

kubernetes.io/cluster-service: 'true'

kubernetes.io/name: Heapster

namespace: kube-system

spec:

ports:

- port: 80

targetPort: 8082

selector:

k8s-app: heapster

vim influxdb.yaml

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

task: monitoring

k8s-app: influxdb

spec:

containers:

- name: influxdb

image: 192.168.220.84/third_party/heapster-influxdb-amd64:v1.1.1 #私服地址，需要自行更换

volumeMounts:

- mountPath: /data

volumes:

- name: influxdb-storage

emptyDir: {}

---

apiVersion: v1

kind: Service

metadata:

labels:

task: monitoring

# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

# If you are NOT using this as an addon, you should comment out this line.

kubernetes.io/cluster-service: 'true'

kubernetes.io/name: monitoring-influxdb

namespace: kube-system

spec:

ports:

- port: 8086

targetPort: 8086

selector:

k8s-app: influxdb

vim grafana.yaml

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

namespace: kube-system

spec:

replicas: 1

template:

metadata:

labels:

task: monitoring

k8s-app: grafana

spec:

containers:

- name: grafana

image: 192.168.220.84/third_party/heapster-grafana-amd64:v4.4.1 #私服地址，需自行更换

ports:

- containerPort: 3000

protocol: TCP

volumeMounts:

- mountPath: /var

env:

- name: INFLUXDB_HOST

value: monitoring-influxdb

- name: GF_SERVER_HTTP_PORT

value: "3000"

# The following env variables are required to make Grafana accessible via

# the kubernetes api-server proxy. On production clusters, we recommend

# removing these env variables, setup auth for grafana, and expose the grafana

# service using a LoadBalancer or a public IP.

- name: GF_AUTH_BASIC_ENABLED

value: "false"

- name: GF_AUTH_ANONYMOUS_ENABLED

value: "true"

- name: GF_AUTH_ANONYMOUS_ORG_ROLE

value: Admin

- name: GF_SERVER_ROOT_URL

# If you're only using the API Server proxy, set this value instead:

# value: /api/v1/proxy/namespaces/kube-system/services/monitoring-grafana/

value: /

volumes:

- name: grafana-storage

emptyDir: {}

---

apiVersion: v1

kind: Service

metadata:

labels:

# For use as a Cluster add-on (https://github.com/kubernetes/kubernetes/tree/master/cluster/addons)

# If you are NOT using this as an addon, you should comment out this line.

kubernetes.io/cluster-service: 'true'

kubernetes.io/name: monitoring-grafana

namespace: kube-system

spec:

# In a production setup, we recommend accessing Grafana through an external Loadbalancer

# or through a public IP.

# type: LoadBalancer

# You could also use NodePort to expose the service at a randomly-generated port

type: NodePort

ports:

- port: 80

targetPort: 3000

nodePort: 31236

selector:

k8s-app: grafana

执行kubectl apply -f heapster.yaml -f influxdb.yaml -f grafana.yaml

在dashboard上的展示效果

heapster展示

grafana展示

加入的node工作节点

安装以下软件版本，文章开头有说道

docker17.03.2-ce

socat-1.7.3.2-2.el7.x86_64

kubelet-1.10.0-0.x86_64

kubernetes-cni-0.6.0-0.x86_64

kubectl-1.10.0-0.x86_64

kubeadm-1.10.0-0.x86_64

环境配置

systemctl stop firewalldsystemctl disable firewalld

修改每个节点hostname

cat < /etc/hosts > /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4

::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.100.1 master1

192.168.100.2 master2

192.168.100.3 master3

EOF

swapoff -a

sed -i 's/.*swap.*/#&/' /etc/fstab

setenforce 0

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf

echo "* soft nproc 65536" >> /etc/security/limits.conf

echo "* hard nproc 65536" >> /etc/security/limits.conf

echo "* soft memlock unlimited" >> /etc/security/limits.conf

echo "* hard memlock unlimited" >> /etc/security/limits.conf

echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables

echo 1 > /proc/sys/net/bridge/bridge-nf-call-ip6tables

echo 1 > /proc/sys/net/ipv4/ip_forward

sysctl -w net.bridge.bridge-nf-call-iptables=1

vim /etc/sysctl.conf

net.ipv4.ip_forward=1

net.bridge.bridge-nf-call-iptables=1

net.bridge.bridge-nf-call-ip6tables=1

sysctl -p

然后执行在master上留下的kubeadm join 192.168.100.1:6443 --token hpobow.vw1g1ya5dre7sq06 --discovery-token-ca-cert-hash sha256:f79b68fb698c92b9336474eb3bf184e847fgerbc58a6296911892662b98b1315即可。

主要参考文章：

kubeadm安装Kubernetes V1.10集群详细文档

kubernetes1.9离线部署

最后编辑于：2018.07.23 15:25:12

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 194,242评论 5赞 459
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 81,769评论 2赞 371
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 141,484评论 0赞 319
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 52,133评论 1赞 263
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 61,007评论 4赞 355
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 46,080评论 1赞 272
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 36,496评论 3赞 381
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 35,190评论 0赞 253
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 39,464评论 1赞 290
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 34,549评论 2赞 309
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 36,330评论 1赞 326
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 32,205评论 3赞 312
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 37,567评论 3赞 298
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 28,889评论 0赞 17
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 30,160评论 1赞 250
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 41,475评论 2赞 341
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 40,650评论 2赞 335

高阶k8s HA 集群搭建（一）

前言

介绍

安装准备

keepalived安装

master1的keepalived.conf

master2的keepalived.conf

master3的keepalived.conf

安装ETCD

master1

master2

master3

推荐阅读更多精彩内容