kubeadm搭建过程错误记录

1.Kubelet的任何命令卡住

是因为 $HOME/.kube/config有问题

将/etc/kubernetes/admin.conf 粘贴复制过去

mkdir -p $HOME/.kube

cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

chown $(id -u):$(id -g) $HOME/.kube/config

要是还不行，查看集群信息

kubectl cluster-info

是$HOME/.kube/config的IP错了

2.在this might take a minute or longer if the control plane images have to be pulled

这里卡住肯定是有几个镜像拉取不下来，

一些是在/etc/kubernetes/manifests/下，查看每一个yaml文件的镜像地址

如果修改了私有仓库的地址，那这里的镜像地址就会修改

直接docker pull 拉取一下试一试，

A.如果不行说明是docker的配置问题，查看修改/etc/docker/daemon.json,配置成私有仓库的地址，或者阿里云，网易云之类的国内开源网站，文件具体内容在最下面。

B.如果生成的地址不是你想要的私有仓库的地址，或者国内源的地址。说明是kubeadm的配置问题，他其中有一个。

imageRepository: 172.25.0.112/library

这样在kubeadm生成的yaml文件的镜像地址，会修改成相应的仓库地址

C.如果还是失败，那就是有一个隐藏的镜像pause的问题，因为他的地址是另外生成的，上面的修改不能影响到他。他的修改是在

vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

最后加上，文件完整在最后

--pod-infra-container-image=172.25.0.112/library/pause-amd64:3.0

3.这个问题是上一个的延深

Unfortunately, an error has occurred:timed out waiting for the condition

This error is likely caused by:

- The kubelet is not running

- The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

- Either there is no internet connection, or imagePullPolicy is set to "Never",

so the kubelet cannot pull or find the following control plane images:

- k8s.gcr.io/kube-apiserver-amd64:v1.10.2

- k8s.gcr.io/kube-controller-manager-amd64:v1.10.2

- k8s.gcr.io/kube-scheduler-amd64:v1.10.2

- k8s.gcr.io/etcd-amd64:3.1.12 (only if no external etcd endpoints are configured)

If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:

- 'systemctl status kubelet'

- 'journalctl -xeu kubelet'

couldn't initialize a Kubernetes cluster

Apr 20 20:14:49 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: W0420 20:14:49.167886 67016 status_manager.go:461] Failed to get status for pod "kube-apiserver-sz-pg-oam-k8stest-007.tendcloud.com_kube-system(4ab323b863d89dbe339bace4f59a9674)": Get https://172.23.5.255:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-sz-pg-oam-k8stest-007.tendcloud.com: dial tcp 172.23.5.255:6443: getsockopt: connection refused

Apr 20 20:14:49 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: I0420 20:14:49.175971 67016 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

Apr 20 20:14:49 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: I0420 20:14:49.176384 67016 kubelet_node_status.go:271] Setting node annotation to enable volume controller attach/detach

Apr 20 20:14:49 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: W0420 20:14:49.179894 67016 status_manager.go:461] Failed to get status for pod "kube-controller-manager-sz-pg-oam-k8stest-007.tendcloud.com_kube-system(015b3a6b417294ccf28ff1f53295ec79)": Get https://172.23.5.255:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-sz-pg-oam-k8stest-007.tendcloud.com: dial tcp 172.23.5.255:6443: getsockopt: connection refused

表面上看着是kubelet除了问题

但是从端口上，可以看到是请求6443，出了问题。

6443是kube-api的端口，所以是kube-api除了问题

kubeadm的kube-api是镜像部署，dokcer images查看镜像，发现kube-apiserver-amd64这个镜像不存在

cd /etc/kubernetes/manifests

kubectl create -f kube-apiserver.yaml

输出：

sUnable to connect to the server: dial tcp 192.168.61.11:6443: i/o timeout

vi kube-apiserver.yaml

查看这一行

image: k8s.gcr.io/kube-apiserver-amd64:v1.10.2

所以确定是镜像拉不下来导致一直报错

创建配置文件 kubeadm.yaml

imageRepository: "172.20.0.112/library"

这个地方写入私有仓库的路径，

则从新启动后

kube-apiserver.yaml文件的image就变成了

image: 172.20.0.112/library/kube-apiserver-amd64:v1.10.2

总之这个路径，docker pull 172.20.0.112/library/kube-apiserver-amd64:v1.10.2

可以拉去下来，则就没有问题

对了如果还是失败，查看镜像发现没有pause的镜像

就根据下面的进行修改

布置私有仓库

https://www.itnotebooks.com/?p=302

4.etcd只能访问localhost和127.0.0.1 不能请求ip地址

是因为配置文件的问题

etcd安装，找到合适的版本下载

https://github.com/coreos/etcd/releases

解压，加上配置文件

（单是etcd启动只能访问127.0.0.1和localhost）

配置文件启动

etcd --config-file etcd.yaml &

测试

curl -v -X PUT http://172.27.64.3:2379/v2/keys/test?value="test"

5.启动kubeadm后，查看/etc/log/message报错

Apr 20 20:15:43 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: W0420 20:15:43.244788 67016 cni.go:171] Unable to update cni config: No networks found in /etc/cni/net.d

Apr 20 20:15:43 sz-pg-oam-k8stest-007.tendcloud.com kubelet[67016]: E0420 20:15:43.244889 67016 kubelet.go:2125] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized

这里是cni网络插件的还没有成功启动，所以报错

可以选择macvlan，fannel等方案

！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！！

文件内容

Daemon.json(172.0.0.1是私有仓库的地址)

-------------------------------------------------------------------------

{

"bridge": "none",

"graph": "/data1/docker",

"log-level": "warn",

"insecure-registries": ["172.0.0.1","172.0.0.2","bj-yh-oam-docker-hub-001.tendcloud.com"],

"registry-mirror": "172.0.0.1",

"exec-opts": ["native.cgroupdriver=systemd"],

"storage-driver": "overlay2",

"storage-opts": [

"overlay2.override_kernel_check=true"

]

}

-------------------------------------------------------------------------

Kubeadm-config.yaml

-------------------------------------------------------------------------

api:

advertiseAddress: 172.66.66.66

bindPort: 6443

controlPlaneEndpoint: ""

apiServerCertSANs:

- 172.66.66.16

- 172.66.66.15

- 172.66.66.14

auditPolicy:

logDir: /var/log/kubernetes/audit

logMaxAge: 2

path: ""

authorizationModes:

- Node

- RBAC

certificatesDir: /etc/kubernetes/pki

cloudProvider: ""

criSocket: /var/run/dockershim.sock

etcd:

caFile: ""

certFile: ""

dataDir: /var/lib/etcd

endpoints: null

image: ""

keyFile: ""

featureGates:

CoreDNS: true

imageRepository: 172.66.66.112/library

kubeProxy:

config:

bindAddress: 0.0.0.0

clientConnection:

acceptContentTypes: ""

burst: 10

contentType: application/vnd.kubernetes.protobuf

kubeconfig: /var/lib/kube-proxy/kubeconfig.conf

qps: 5

clusterCIDR: ""

configSyncPeriod: 15m0s

conntrack:

max: null

maxPerCore: 32768

min: 131072

tcpCloseWaitTimeout: 1h0m0s

tcpEstablishedTimeout: 24h0m0s

enableProfiling: false

featureGates:

SupportIPVSProxyMode: true

healthzBindAddress: 0.0.0.0:10256

hostnameOverride: ""

iptables:

masqueradeAll: true

masqueradeBit: 14

minSyncPeriod: 0s

syncPeriod: 30s

ipvs:

minSyncPeriod: 0s

scheduler: rr

syncPeriod: 30s

metricsBindAddress: 127.0.0.1:10249

mode: ipvs

nodePortAddresses: null

oomScoreAdj: -999

portRange: ""

resourceContainer: /kube-proxy

udpIdleTimeout: 250ms

kubeletConfiguration: {}

kubernetesVersion: v1.10.2

networking:

dnsDomain: jdtest.tendcloud.com

podSubnet: ""

serviceSubnet: 172.66.66.1/24

nodeName: bj-jd-dc-datanode-1

privilegedPods: false

token: ""

tokenGroups:

- system:bootstrappers:kubeadm:default-node-token

tokenTTL: 24h0m0s

tokenUsages:

- signing

- authentication

unifiedControlPlaneImage: ""

-------------------------------------------------------------------------

10-kubeadm.conf

-------------------------------------------------------------------------

[Service]

Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"

Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"

Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"

Environment="KUBELET_DNS_ARGS=--cluster-dns=172.88.28.19 --cluster-domain=yhk8s.tendcloud.com"

Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"

Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"

Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"

Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"

Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/start-kubelet.conf"

Environment="KUBELET_EXTRA_ARGS=--pod-infra-container-image=172.66.66.61/library/pause-amd64:3.1"

ExecStart=

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_CONFIG_ARGS

-------------------------------------------------------------------------

Etcd-config.yaml

-------------------------------------------------------------------------

name: etcd-1

data-dir: /opt/etcd-v3.2.6/data

listen-client-urls: http://172.66.66.3:2379,http://127.0.0.1:2379

advertise-client-urls: http://172.66.66.3:2379,http://127.0.0.1:2379

listen-peer-urls: http://172.66.66.3:2380

initial-advertise-peer-urls: http://172.66.66.3:2380

initial-cluster: etcd-1=http://172.66.66.3:2380

initial-cluster-token: etcd-cluster-token

initial-cluster-state: new

-------------------------------------------------------------------------