要求:
| 要求 |
|---|
| Kubernetes cluster 1.16+ |
| Helm 2.10.0+ |
部署k8s
- 安装docker
$ cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://2a2urxbq.mirror.aliyuncs.com"],
"exec-opts": ["native.cgroupdriver=systemd"],
"insecure-registries": ["10.20.8.133"], #这里要为本地得地址,否则harbor部署完成后,docker login 会报错
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
- 安装k8s
部署horbor
安装helm
- 下载
官方链接: https://github.com/helm/helm/releases
$ wget https://get.helm.sh/helm-v3.4.0-linux-amd64.tar.gz
$ tar fx helm-v3.4.0-linux-amd64.tar.gz
$ cp linux-amd64/helm /usr/local/bin
$ helm version
version.BuildInfo{Version:"v3.4.0", GitCommit:"7090a89efc8a18f3d8178bf47d2462450349a004", GitTreeState:"clean", GoVersion:"go1.14.10"}
安装harbor
- 创建namespace
$ kubectl create namespace harbor
- 安装ingress controller
注意版本
$ kubectl apply -f https://kuboard.cn/install-script/v1.16.2/nginx-ingress.yaml
- 创建pv
生产环境使用ceph 的sc
# 试验使用本地目录文件
# 编辑yml文件,创建pv
$ kubectl apply -f . -n harbor
- 下载源码
$ wget https://github.com/goharbor/harbor-helm/archive/v1.5.0.tar.gz
$ tar fx v1.5.0.tar.gz
$ cd harbor-helm-1.5.0/
- 修改ingress
默认的harbor ingress 证书为1年时间,这里可以使用secret的方式进行更改,参考附录1 创建证书和创建secret部分，使用方法1即可
$ vim values.yaml
expose:
type: ingress
tls:
enabled: true
certSource: secret #修改为secret
auto:
commonName: ""
secret:
secretName: "harbor-self-tls" #指定secret的名字
notarySecretName: ""
ingress:
hosts:
core: core.harbor.domain
notary: notary.harbor.domain
controller: default
annotations:
ingress.kubernetes.io/ssl-redirect: "true"
ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/proxy-body-size: "0"
- 修改 StorageClass为ceph的sc
前提ceph已部署,ceph的sc已经创建，修改persistence配置块的storageClass为创建好的ceph sc
$ vim values.yaml
persistence:
enabled: true
resourcePolicy: "keep"
persistentVolumeClaim:
registry:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
chartmuseum:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
jobservice:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
database:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
redis:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 1Gi
trivy:
existingClaim: ""
storageClass: "rook-ceph-block"
subPath: ""
accessMode: ReadWriteOnce
size: 5Gi
- 安装
$ helm install harbor --debug --namespace harbor .
安装过程中harbor-harbor-database-0 一直无法Running的话，查看容器对应的volume,将dump.rdb删掉即可
- 卸载
$ helm uninstall harbor --debug --namespace harbor
- 查看pod
$ kubectl get pods -n harbor
NAME READY STATUS RESTARTS AGE
harbor-harbor-chartmuseum-75fcf4bccc-4dhlh 1/1 Running 0 88m
harbor-harbor-clair-d8f59f74-qzkrw 2/2 Running 9 88m
harbor-harbor-core-7fcbd6d86c-mqqwr 1/1 Running 8 88m
harbor-harbor-database-0 1/1 Running 0 68m
harbor-harbor-jobservice-55f8b5f8bd-k8s9k 1/1 Running 0 62m
harbor-harbor-notary-server-597c779966-zhvz2 1/1 Running 8 88m
harbor-harbor-notary-signer-6c9fdc8655-qj9ss 1/1 Running 8 88m
harbor-harbor-portal-8456c5d77-j6ftj 1/1 Running 0 88m
harbor-harbor-redis-0 1/1 Running 0 88m
harbor-harbor-registry-8456589f68-f5r2p 2/2 Running 0 88m
harbor-harbor-trivy-0 1/1 Running 0 88m
- 查看service
$ kubectl get svc -n harbor
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
harbor-harbor-chartmuseum ClusterIP 10.110.135.58 <none> 80/TCP 92m
harbor-harbor-clair ClusterIP 10.103.49.219 <none> 8080/TCP 92m
harbor-harbor-core ClusterIP 10.103.229.60 <none> 80/TCP 92m
harbor-harbor-database ClusterIP 10.110.126.185 <none> 5432/TCP 92m
harbor-harbor-jobservice ClusterIP 10.105.41.118 <none> 80/TCP 92m
harbor-harbor-notary-server ClusterIP 10.101.181.231 <none> 4443/TCP 92m
harbor-harbor-notary-signer ClusterIP 10.100.122.224 <none> 7899/TCP 92m
harbor-harbor-portal ClusterIP 10.108.65.32 <none> 80/TCP 92m
harbor-harbor-redis ClusterIP 10.109.188.51 <none> 6379/TCP 92m
harbor-harbor-registry ClusterIP 10.107.14.62 <none> 5000/TCP,8080/TCP 92m
harbor-harbor-trivy ClusterIP 10.107.82.132 <none> 8080/TCP 92m
- 查看pv和pvc
$ kubectl get pv,pvc -n harbor | grep harbor
persistentvolume/disk1.yml 5Gi RWO Recycle Bound harbor/harbor-harbor-chartmuseum 94m
persistentvolume/disk2.yml 5Gi RWO Recycle Bound harbor/data-harbor-harbor-trivy-0 94m
persistentvolume/disk3.yml 1Gi RWO Recycle Bound harbor/harbor-harbor-jobservice 94m
persistentvolume/disk4.yml 1Gi RWO Recycle Bound harbor/data-harbor-harbor-redis-0 94m
persistentvolume/disk5.yml 1Gi RWO Recycle Bound harbor/database-data-harbor-harbor-database-0 94m
persistentvolume/disk6.yml 5Gi RWO Recycle Bound harbor/harbor-harbor-registry 94m
persistentvolumeclaim/data-harbor-harbor-redis-0 Bound disk4.yml 1Gi RWO 93m
persistentvolumeclaim/data-harbor-harbor-trivy-0 Bound disk2.yml 5Gi RWO 93m
persistentvolumeclaim/database-data-harbor-harbor-database-0 Bound disk5.yml 1Gi RWO 93m
persistentvolumeclaim/harbor-harbor-chartmuseum Bound disk1.yml 5Gi RWO 93m
persistentvolumeclaim/harbor-harbor-jobservice Bound disk3.yml 1Gi RWO 93m
persistentvolumeclaim/harbor-harbor-registry Bound disk6.yml 5Gi RWO 93m
- 查看harbor ingress
$ kubectl get ingress -n harbor
- 查看ingress 配置
$ kubectl edit ingress harbor-harbor-ingress -n harbor
访问harbor
- 配置/etc/hosts
$ vim /etc/hosts
10.20.8.150 core.harbor.domain
- 访问
https://core.harbor.domain
默认用户名/密码 admin/Harbor12345
- 在k8s中使用harbor
$ kubectl get secrets/harbor-harbor-ingress -n harbor -o jsonpath="{.data.ca\.crt}" | base64 --decode
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgIQS7zjdWHqFMC7RsAd3jYT/DANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjAxMDI4MDc1MDAwWhcNMjExMDI4MDc1
MDAwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCuu51lc9wZ2n1cXjn36cyNsUgZmDBJbJ9GMBjNHXa7ctEZQJdo
cMhFr315qxteeeZicNgnmlf00ETz6q9mRSXvfezY82yBdsYz2ZeLLcHfLdl5yu7J
RCSwTOAvOGPUvZIkkCU4L/2WCsfZrHLLMzQTnrT2LVYyzI9QkZMW2biafjfPdAC3
XVY+FPd6jUYpLXb5pEKuRusFTJWSP4Lu6Jw1f+ZwgMhyRWVmDVZlBPtcX8s108Fb
R097n3bvZ4YujM4HYTZ8dEKdj5jwxbNWntMLNYPkK7lgxWuwZJ0tzuWGZQI0qQzg
s9DFnetUeUnbMZ0F65iKaQW1CEBO+UajPyQDAgMBAAGjQjBAMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAK6ii5qshCaUednjcvOieHoCu1CsCIURK
D6Zq5RoCdvU0v70s9fK5XohYMjw5InwYEYSTB05H2kbePjk+D4BgRmmg7Svc6Dr0
rsRfFCkG+cQFQ6/lf5flerAGCHGIsxJ2MLSG5P5s81wOPMRtwSF2bhZIcvjchPZE
1ExCQFeRkqgEP1zy3++XW5BL77k8pR0uK2NI16VM6GzT4IR7lEfrlYgjGEPjWma+
ZxL5UprZofuxkfiFzTcEUHqvmwMZo2Iwros5ARPtJikbeHn7BbSMBX2U4HOqHih9
6661XWyZAhdCC2NlPYeF+1Bd7rRLWyH7MU5BDVodvX7WesXmmbkFsg==
-----END CERTIFICATE-----
- 创建证书
$ mkdir -pv /etc/docker/certs.d/core.harbor.domain/
$ cat <<EOF > /etc/docker/certs.d/core.harbor.domain/ca.crt
-----BEGIN CERTIFICATE-----
MIIC9DCCAdygAwIBAgIQS7zjdWHqFMC7RsAd3jYT/DANBgkqhkiG9w0BAQsFADAU
MRIwEAYDVQQDEwloYXJib3ItY2EwHhcNMjAxMDI4MDc1MDAwWhcNMjExMDI4MDc1
MDAwWjAUMRIwEAYDVQQDEwloYXJib3ItY2EwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCuu51lc9wZ2n1cXjn36cyNsUgZmDBJbJ9GMBjNHXa7ctEZQJdo
cMhFr315qxteeeZicNgnmlf00ETz6q9mRSXvfezY82yBdsYz2ZeLLcHfLdl5yu7J
RCSwTOAvOGPUvZIkkCU4L/2WCsfZrHLLMzQTnrT2LVYyzI9QkZMW2biafjfPdAC3
XVY+FPd6jUYpLXb5pEKuRusFTJWSP4Lu6Jw1f+ZwgMhyRWVmDVZlBPtcX8s108Fb
R097n3bvZ4YujM4HYTZ8dEKdj5jwxbNWntMLNYPkK7lgxWuwZJ0tzuWGZQI0qQzg
s9DFnetUeUnbMZ0F65iKaQW1CEBO+UajPyQDAgMBAAGjQjBAMA4GA1UdDwEB/wQE
AwICpDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDwYDVR0TAQH/BAUw
AwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAK6ii5qshCaUednjcvOieHoCu1CsCIURK
D6Zq5RoCdvU0v70s9fK5XohYMjw5InwYEYSTB05H2kbePjk+D4BgRmmg7Svc6Dr0
rsRfFCkG+cQFQ6/lf5flerAGCHGIsxJ2MLSG5P5s81wOPMRtwSF2bhZIcvjchPZE
1ExCQFeRkqgEP1zy3++XW5BL77k8pR0uK2NI16VM6GzT4IR7lEfrlYgjGEPjWma+
ZxL5UprZofuxkfiFzTcEUHqvmwMZo2Iwros5ARPtJikbeHn7BbSMBX2U4HOqHih9
6661XWyZAhdCC2NlPYeF+1Bd7rRLWyH7MU5BDVodvX7WesXmmbkFsg==
-----END CERTIFICATE-----
EOF
- 重启docker然后使用docker login 登录
用户名/密码: admin/Harbor12345
$ docker login core.harbor.domian
Username: admin
Password:
Login Succeeded
- 上传镜像
$ docker tag nginx core.harbor.domain/library/nginx
$ docker push core.harbor.domain/library/nginx
报错1：push镜像提示文件大
rror parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.17.3</center>\r\n</body>\r\n</html>\r\n"
参考链接(https://blog.pragtechnologies.com/file-upload-limit-in-kubernetes/)
解决办法: 修改ingress
$ kubectl edit ingress harbor-harbor-ingress -n harbor
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
ingress.kubernetes.io/proxy-body-size: "0"
ingress.kubernetes.io/ssl-redirect: "true"
meta.helm.sh/release-name: harbor
meta.helm.sh/release-namespace: harbor
nginx.ingress.kubernetes.io/client-max-body-size: 10240m #增加该字段
nginx.ingress.kubernetes.io/proxy-body-size: 50m # 修改该值
nginx.ingress.kubernetes.io/ssl-redirect: "true"
修改configmap
$ kubectl edit configmap nginx-config -n nginx-ingress
apiVersion: v1
data:
body-size: 50m #增加该字段
client-max-body-size: 10240m #增加该字段
server-names-hash-bucket-size: "1024"
- 查看log
$ cd /var/log/pods
$ ls
harbor_harbor-harbor-chartmuseum-75fcf4bccc-4dhlh_a10728ea-ab87-404b-a6eb-7191c2aa01ce
harbor_harbor-harbor-clair-d8f59f74-qzkrw_516a1ece-0625-45d4-ba86-167c5d27d34f
harbor_harbor-harbor-core-7fcbd6d86c-mqqwr_130f54e3-3f99-4f3d-a468-079033903915
harbor_harbor-harbor-database-0_b90fda78-0fd3-4230-8c0d-a680f1d00010
harbor_harbor-harbor-jobservice-55f8b5f8bd-k8s9k_eae58e10-6118-4827-b75a-23a1af5bf0b5
harbor_harbor-harbor-notary-server-597c779966-zhvz2_5fcb564b-242b-4429-9560-85c6c3a2d333
harbor_harbor-harbor-notary-signer-6c9fdc8655-qj9ss_0eeb3aab-227d-46fa-98fa-a443bbb22bb5
harbor_harbor-harbor-portal-8456c5d77-j6ftj_e582f16e-3269-42aa-b16a-ad4c2c9ec02f
harbor_harbor-harbor-redis-0_eee073c7-1a0e-43c5-b86a-ef64827d5d80
harbor_harbor-harbor-registry-8456589f68-f5r2p_5a203290-6a60-4119-afbc-dd8a4263173b
harbor_harbor-harbor-trivy-0_d2f15569-a856-4b10-92a4-f7f7dfe9e15d
附录1
创建证书
FQDN要为harbor使用的域名，vaules.yaml里面externalURL配置块的值
- 方法
## 获得证书
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout ca.key -x509 -days 3650 -out ca.crt
## 生成证书签名请求
$ openssl req -newkey rsa:4096 -nodes -sha256 -keyout tls.key -out tls.csr
## 生成证书
$ openssl x509 -req -days 3650 -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out tls.crt
- 创建secrect
$ cp core.harbor.domain.crt tls.crt
$ cp core.harbor.domain.key tls.key
$ kubectl create secret generic harbor-harbor-ingress --from-file=tls.crt --from-file=tls.key --from-file=ca.crt -n harbor
- 查看证书过期时间
$ openssl x509 -in ca.crt -noout -dates
