kubernetes1.13.1集群集成harbor-helm.md

参考文章

https://github.com/goharbor/harbor-helm

https://www.hi-linux.com/posts/14136.html

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd

https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/rbd/deploy/rbac

https://github.com/helm/helm/issues/3130

https://www.kancloud.cn/huyipow/kubernetes/531999

https://www.hi-linux.com/posts/14136.html

https://li-sen.github.io/2018/10/08/k8s%E9%83%A8%E7%BD%B2%E9%AB%98%E5%8F%AF%E7%94%A8harbor/

https://github.com/minminmsn/k8s1.13/blob/master/harbor-helm/kubernetes1.13.1%E9%9B%86%E7%BE%A4%E9%9B%86%E6%88%90harbor-helm.md#%E5%8F%82%E8%80%83%E6%96%87%E7%AB%A0

依赖关系

Kubernetes cluster 1.10+

kubernetes集群版本1.13.1

Helm 2.8.0+

ingress

用于外部访问集群内部环境

rbd-provisioner

ceph rbd 客户端，可以创建、删除ceph rbd pool、image等

storageclass

用于自动创建pv与pvc

ceph rbd

ceph集群luminous版本

操作步骤

一、部署rbd-provisioner

1、下载external-storage

[root@k8s-master yaml]# git clone https://github.com/kubernetes-incubator/external-storage

[root@k8s-master yaml]# cd external-storage/ceph/rbd/deploy/rbac/

[root@k8s-masterac]# ls

clusterrolebinding.yaml deployment.yaml role.yaml

clusterrole.yaml rolebinding.yaml serviceaccount.yaml

[root@elasticsearch01 rbac]# mkdir /k8s/yaml/volumes/rbd-provisioner

[root@elasticsearch01 rbac]# cp * /k8s/yaml/volumes/rbd-provisioner/

[root@elasticsearch01 rbac]# cd /k8s/yaml/volumes/rbd-provisioner/

2、创建rbd-provisioner角色、pod

[root@k8s-master rbd-provisioner]# ls

clusterrolebinding.yaml deployment.yaml role.yaml

clusterrole.yaml rolebinding.yaml serviceaccount.yaml

[root@k8s-master rbd-provisioner]# kubectl create -f ./

clusterrole.rbac.authorization.k8s.io/rbd-provisioner created

clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created

deployment.extensions/rbd-provisioner created

role.rbac.authorization.k8s.io/rbd-provisioner created

rolebinding.rbac.authorization.k8s.io/rbd-provisioner created

serviceaccount/rbd-provisioner created

3、验证rbd-provisioner

[root@k8s-master rbd-provisioner]# kubectl get pods

NAME READY STATUS RESTARTS AGE

busybox 1/1 Running 600 25d

ceph-rbd-pv-pod1 1/1 Running 10 6d23h

jenkins-0 1/1 Running 0 6d1h

rbd-provisioner-67b4857bcd-xxwx5 1/1 Running 0 9s

二、部署storageclass

1、修改storageclass配置 参考external-storage/gluster/glusterfs/deploy/storageclass.yaml样例根据自己情况修改，其中secretName在kubernetes集群使用ceph rbd块存储时已经创建过

[root@elasticsearch01 rbd-provisioner]# cat storageclass.yaml

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

provisioner: ceph.com/rbd

parameters:

monitors: 10.0.4.10:6789

pool: rbd-k8s

adminId: admin

adminSecretNamespace: default

adminSecretName: ceph-secret

userSecretName: ceph-secret

userId: admin

userSecretNamespace: default

userSecretName: ceph-secret

imageFormat: "2"

imageFeatures: layering

2、创建storageclass rbd

[root@elasticsearch01 harbor-helm]# kubectl create -f storageclass.yaml

storageclass.storage.k8s.io/rbd created

[root@elasticsearch01 harbor-helm]# kubectl get storageclasses

NAME PROVISIONER AGE

rbd ceph.com/rbd 2m

三、部署harbor-helm

1、下载harbor-helm 1.0.0版本的源码

[root@elasticsearch01 yaml]# git clone https://github.com/goharbor/harbor-helm.git

[root@elasticsearch01 yaml]# cd harbor-helm/

[root@elasticsearch01 harbor-helm]# git checkout 1.0.0

[root@elasticsearch01 harbor-helm]# ls

Chart.yaml CONTRIBUTING.md docs LICENSE README.md templates values.yaml

2、修改values.yaml配置 需要根据实际情况修改values.yaml配置文件，主要修改如下几个地方

admin

登陆密码

storageclass

这里是rbd

ingress

修改自己的域名

修改后具体如下

[root@elasticsearch01 harbor-helm]# cat values.yaml

expose:

# Set the way how to expose the service. Set the type as "ingress",

# "clusterIP" or "nodePort" and fill the information in the corresponding

# section

type: ingress

tls:

# Enable the tls or not. Note: if the type is "ingress" and the tls

# is disabled, the port must be included in the command when pull/push

# images. Refer to https://github.com/goharbor/harbor/issues/5291

# for the detail.

enabled: true

# Fill the name of secret if you want to use your own TLS certificate

# and private key. The secret must contain keys named tls.crt and

# tls.key that contain the certificate and private key to use for TLS

# The certificate and private key will be generated automatically if

# it is not set

secretName: ""

# By default, the Notary service will use the same cert and key as

# described above. Fill the name of secret if you want to use a

# separated one. Only needed when the type is "ingress".

notarySecretName: ""

# The commmon name used to generate the certificate, it's necessary

# when the type is "clusterIP" or "nodePort" and "secretName" is null

commonName: ""

ingress:

hosts:

core: core-harbor.dashboard.com

notary: notary-harbor.dashboard.com

annotations:

ingress.kubernetes.io/ssl-redirect: "true"

nginx.ingress.kubernetes.io/ssl-redirect: "true"

ingress.kubernetes.io/proxy-body-size: "0"

nginx.ingress.kubernetes.io/proxy-body-size: "0"

clusterIP:

# The name of ClusterIP service

ports:

# The service port Harbor listens on when serving with HTTP

httpPort: 80

# The service port Harbor listens on when serving with HTTPS

httpsPort: 443

# The service port Notary listens on. Only needed when notary.enabled

# is set to true

notaryPort: 4443

nodePort:

# The name of NodePort service

ports:

http:

# The service port Harbor listens on when serving with HTTP

port: 80

# The node port Harbor listens on when serving with HTTP

nodePort: 30002

https:

# The service port Harbor listens on when serving with HTTPS

port: 443

# The node port Harbor listens on when serving with HTTPS

nodePort: 30003

# Only needed when notary.enabled is set to true

notary:

# The service port Notary listens on

port: 4443

# The node port Notary listens on

nodePort: 30004

# The external URL for Harbor core service. It is used to

# 1) populate the docker/helm commands showed on portal

# 2) populate the token service URL returned to docker/notary client

# Format: protocol://domain[:port]. Usually:

# 1) if "expose.type" is "ingress", the "domain" should be

# the value of "expose.ingress.hosts.core"

# 2) if "expose.type" is "clusterIP", the "domain" should be

# the value of "expose.clusterIP.name"

# 3) if "expose.type" is "nodePort", the "domain" should be

# the IP address of k8s node

# If Harbor is deployed behind the proxy, set it as the URL of proxy

externalURL: https://core-harbor.dashboard.com

# The persistence is enabled by default and a default StorageClass

# is needed in the k8s cluster to provision volumes dynamicly.

# Specify another StorageClass in the "storageClass" or set "existingClaim"

# if you have already existing persistent volumes to use

# For storing images and charts, you can also use "azure", "gcs", "s3",

# "swift" or "oss". Set it in the "imageChartStorage" section

persistence:

enabled: true

# Setting it to "keep" to avoid removing PVCs during a helm delete

# operation. Leaving it empty will delete PVCs after the chart deleted

resourcePolicy: "keep"

persistentVolumeClaim:

registry:

# Use the existing PVC which must be created manually before bound

existingClaim: ""

# Specify the "storageClass" used to provision the volume. Or the default

# StorageClass will be used(the default).

# Set it to "-" to disable dynamic provisioning

storageClass: "rbd"

subPath: ""

accessMode: ReadWriteOnce

size: 50Gi

chartmuseum:

existingClaim: ""

storageClass: "rbd"

subPath: ""

accessMode: ReadWriteOnce

size: 5Gi

jobservice:

existingClaim: ""

storageClass: "rbd"

subPath: ""

accessMode: ReadWriteOnce

size: 2Gi

# If external database is used, the following settings for database will

# be ignored

database:

existingClaim: ""

storageClass: "rbd"

subPath: ""

accessMode: ReadWriteOnce

size: 2Gi

# If external Redis is used, the following settings for Redis will

# be ignored

redis:

existingClaim: ""

storageClass: "rbd"

subPath: ""

accessMode: ReadWriteOnce

size: 2Gi

# Define which storage backend is used for registry and chartmuseum to store

# images and charts. Refer to

# https://github.com/docker/distribution/blob/master/docs/configuration.md#storage

# for the detail.

imageChartStorage:

# Specify the type of storage: "filesystem", "azure", "gcs", "s3", "swift",

# "oss" and fill the information needed in the corresponding section. The type

# must be "filesystem" if you want to use persistent volumes for registry

# and chartmuseum

type: filesystem

filesystem:

rootdirectory: /storage

#maxthreads: 100

azure:

accountname: accountname

accountkey: base64encodedaccountkey

container: containername

#realm: core.windows.net

gcs:

bucket: bucketname

# TODO: support the keyfile of gcs

#keyfile: /path/to/keyfile

#rootdirectory: /gcs/object/name/prefix

#chunksize: "5242880"

s3:

region: us-west-1

bucket: bucketname

#accesskey: awsaccesskey

#secretkey: awssecretkey

#regionendpoint: http://myobjects.local

#encrypt: false

#keyid: mykeyid

#secure: true

#v4auth: true

#chunksize: "5242880"

#rootdirectory: /s3/object/name/prefix

#storageclass: STANDARD

swift:

authurl: https://storage.myprovider.com/v3/auth

username: username

password: password

container: containername

#region: fr

#tenant: tenantname

#tenantid: tenantid

#domain: domainname

#domainid: domainid

#trustid: trustid

#insecureskipverify: false

#chunksize: 5M

#prefix:

#secretkey: secretkey

#accesskey: accesskey

#authversion: 3

#endpointtype: public

#tempurlcontainerkey: false

#tempurlmethods:

oss:

accesskeyid: accesskeyid

accesskeysecret: accesskeysecret

region: regionname

bucket: bucketname

#endpoint: endpoint

#internal: false

#encrypt: false

#secure: true

#chunksize: 10M

#rootdirectory: rootdirectory

imagePullPolicy: IfNotPresent

logLevel: debug

# The initial password of Harbor admin. Change it from portal after launching Harbor

harborAdminPassword: "newpassword"

# The secret key used for encryption. Must be a string of 16 chars.

secretKey: "not-a-secure-key"

# If expose the service via "ingress", the Nginx will not be used

nginx:

image:

repository: goharbor/nginx-photon

tag: v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

portal:

image:

repository: goharbor/harbor-portal

tag: v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

core:

image:

repository: goharbor/harbor-core

tag: v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

adminserver:

image:

repository: goharbor/harbor-adminserver

tag: v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

jobservice:

image:

repository: goharbor/harbor-jobservice

tag: v1.7.0

replicas: 1

maxJobWorkers: 10

# The logger for jobs: "file", "database" or "stdout"

jobLogger: file

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

registry:

image:

repository: goharbor/registry-photon

tag: v2.6.2-v1.7.0

controller:

image:

repository: goharbor/harbor-registryctl

tag: v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

chartmuseum:

enabled: true

image:

repository: goharbor/chartmuseum-photon

tag: v0.7.1-v1.7.0

replicas: 1

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

clair:

enabled: true

image:

repository: goharbor/clair-photon

tag: v2.0.7-v1.7.0

replicas: 1

# The http(s) proxy used to update vulnerabilities database from internet

httpProxy:

httpsProxy:

# The interval of clair updaters, the unit is hour, set to 0 to

# disable the updaters

updatersInterval: 12

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

notary:

enabled: true

server:

image:

repository: goharbor/notary-server-photon

tag: v0.6.1-v1.7.0

replicas: 1

signer:

image:

repository: goharbor/notary-signer-photon

tag: v0.6.1-v1.7.0

replicas: 1

nodeSelector: {}

tolerations: []

affinity: {}

## Additional deployment annotations

podAnnotations: {}

database:

# if external database is used, set "type" to "external"

# and fill the connection informations in "external" section

type: internal

internal:

image:

repository: goharbor/harbor-db

tag: v1.7.0

# The initial superuser password for internal database

password: "changeit"

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

external:

host: "192.168.0.1"

port: "5432"

username: "user"

password: "password"

coreDatabase: "registry"

clairDatabase: "clair"

notaryServerDatabase: "notary_server"

notarySignerDatabase: "notary_signer"

sslmode: "disable"

## Additional deployment annotations

podAnnotations: {}

redis:

# if external Redis is used, set "type" to "external"

# and fill the connection informations in "external" section

type: internal

internal:

image:

repository: goharbor/redis-photon

tag: v1.7.0

# resources:

# requests:

# memory: 256Mi

# cpu: 100m

nodeSelector: {}

tolerations: []

affinity: {}

external:

host: "10.2.8.44"

port: "6379"

# The "coreDatabaseIndex" must be "0" as the library Harbor

# used doesn't support configuring it

coreDatabaseIndex: "0"

jobserviceDatabaseIndex: "1"

registryDatabaseIndex: "2"

chartmuseumDatabaseIndex: "3"

password: ""

## Additional deployment annotations

podAnnotations: {}

4、helm 初始化安装harbor

[root@k8s-master harbor-helm]# helm install . --name mymin

NAME: min

LAST DEPLOYED: Mon Jan 28 17:01:09 2019

NAMESPACE: default

STATUS: DEPLOYED

RESOURCES:

==> v1/PersistentVolumeClaim

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

mymin-harbor-chartmuseum Pending 1s

mymin-harbor-jobservice Pending 1s

mymin-harbor-registry Pending 1s

==> v1/Service

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

mymin-harbor-adminserver ClusterIP 10.254.7.52 <none> 80/TCP 1s

mymin-harbor-chartmuseum ClusterIP 10.254.80.86 <none> 80/TCP 1s

mymin-harbor-clair ClusterIP 10.254.221.71 <none> 6060/TCP 0s

mymin-harbor-core ClusterIP 10.254.114.190 <none> 80/TCP 0s

mymin-harbor-database ClusterIP 10.254.146.141 <none> 5432/TCP 0s

mymin-harbor-jobservice ClusterIP 10.254.21.20 <none> 80/TCP 0s

mymin-harbor-notary-server ClusterIP 10.254.255.218 <none> 4443/TCP 0s

mymin-harbor-notary-signer ClusterIP 10.254.203.88 <none> 7899/TCP 0s

mymin-harbor-portal ClusterIP 10.254.73.42 <none> 80/TCP 0s

mymin-harbor-redis ClusterIP 10.254.134.216 <none> 6379/TCP 0s

mymin-harbor-registry ClusterIP 10.254.69.96 <none> 5000/TCP,8080/TCP 0s

==> v1/Deployment

NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE

mymin-harbor-adminserver 1 1 1 0 0s

mymin-harbor-chartmuseum 1 1 1 0 0s

mymin-harbor-clair 1 1 1 0 0s

mymin-harbor-core 1 0 0 0 0s

mymin-harbor-jobservice 1 0 0 0 0s

mymin-harbor-notary-server 1 0 0 0 0s

mymin-harbor-notary-signer 1 0 0 0 0s

mymin-harbor-portal 1 0 0 0 0s

mymin-harbor-registry 1 0 0 0 0s

==> v1/StatefulSet

NAME DESIRED CURRENT AGE

mymin-harbor-database 1 1 0s

mymin-harbor-redis 1 1 0s

==> v1beta1/Ingress

NAME HOSTS ADDRESS PORTS AGE

mymin-harbor-ingress core-harbor.dashboard.com,notary-harbor.dashboard.com 80, 443 0s

==> v1/Pod(related)

NAME READY STATUS RESTARTS AGE

mymin-harbor-adminserver-54877f95bd-45vq2 0/1 ContainerCreating 0 0s

mymin-harbor-chartmuseum-7d59b659df-jkt9f 0/1 Pending 0 0s

mymin-harbor-clair-69f89c644-hg6qp 0/1 ContainerCreating 0 0s

mymin-harbor-core-5cdff64cc8-9vw2w 0/1 ContainerCreating 0 0s

mymin-harbor-jobservice-bbdf5bbcd-qsz9h 0/1 Pending 0 0s

mymin-harbor-notary-server-dcbccf89b-9gpsp 0/1 Pending 0 0s

mymin-harbor-notary-signer-5d45d46d64-d4sjg 0/1 ContainerCreating 0 0s

mymin-harbor-database-0 0/1 Pending 0 0s

mymin-harbor-redis-0 0/1 Pending 0 0s

==> v1/Secret

NAME TYPE DATA AGE

min-harbor-adminserver Opaque 4 1s

mymin-harbor-chartmuseum Opaque 1 1s

mymin-harbor-core Opaque 4 1s

mymin-harbor-database Opaque 1 1s

mymin-harbor-ingress kubernetes.io/tls 3 1s

mymin-harbor-jobservice Opaque 1 1s

mymin-harbor-registry Opaque 1 1s

==> v1/ConfigMap

NAME DATA AGE

mymin-harbor-adminserver 39 1s

mymin-harbor-chartmuseum 24 1s

mymin-harbor-clair 1 1s

mymin-harbor-core 1 1s

mymin-harbor-jobservice 1 1s

mymin-harbor-notary-server 5 1s

mymin-harbor-registry 2 1s

NOTES:

Please wait for several minutes for Harbor deployment to complete.

Then you should be able to visit the Harbor portal at https://core-harbor.dashboard.com.

For more details, please visit https://github.com/goharbor/harbor.

5、验证pv与pvc 主要是pv与pvc如果没有自动创建存储的条件需要提前手动创建好pv几pvc，然后value.yaml文件里选择existingClaim，填写各自pvc的名字即可

[root@elasticsearch01 harbor-helm]# kubectl get pv

NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE

ceph-rbd-pv 20Gi RWO Recycle Bound default/ceph-rbd-pv-claim 7d1h

jenkins-home-pv 40Gi RWO Recycle Bound default/jenkins-home-pvc 6d2h

pvc-84079273-22de-11e9-a09d-52540089b2b6 5Gi RWO Delete Bound default/mymin-harbor-chartmuseum rbd 43s

pvc-84085284-22de-11e9-a09d-52540089b2b6 2Gi RWO Delete Bound default/mymin-harbor-jobservice rbd 56s

pvc-840a9404-22de-11e9-a09d-52540089b2b6 50Gi RWO Delete Bound default/mymin-harbor-registry rbd 56s

pvc-844d2f2d-22de-11e9-a09d-52540089b2b6 2Gi RWO Delete Bound default/database-data-mymin-harbor-database-0 rbd 43s

pvc-8455d703-22de-11e9-a09d-52540089b2b6 2Gi RWO Delete Bound default/data-mymin-harbor-redis-0 rbd 43s

[root@k8s-master harbor-helm]# kubectl get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

ceph-rbd-pv-claim Bound ceph-rbd-pv 20Gi RWO 7d1h

data-mymin-harbor-redis-0 Bound pvc-8455d703-22de-11e9-a09d-52540089b2b6 2Gi RWO rbd 46s

database-data-mymin-harbor-database-0 Bound pvc-844d2f2d-22de-11e9-a09d-52540089b2b6 2Gi RWO rbd 46s

jenkins-home-pvc Bound jenkins-home-pv 40Gi RWO 6d2h

mymin-harbor-chartmuseum Bound pvc-84079273-22de-11e9-a09d-52540089b2b6 5Gi RWO rbd 46s

min-harbor-jobservice Bound pvc-84085284-22de-11e9-a09d-52540089b2b6 2Gi RWO rbd 46s

mymin-harbor-registry Bound pvc-840a9404-22de-11e9-a09d-52540089b2b6 50Gi RWO rbd 46s

6、验证ceph rbd

[root@ceph01 ~]# rbd list rbd-k8s

cephimage1

cephimage2

cephimage3

kubernetes-dynamic-pvc-8420311c-22de-11e9-b7ec-02420afe4907

kubernetes-dynamic-pvc-84203268-22de-11e9-b7ec-02420afe4907

kubernetes-dynamic-pvc-8bfd862e-22de-11e9-b7ec-02420afe4907

kubernetes-dynamic-pvc-8bfe7a4f-22de-11e9-b7ec-02420afe4907

kubernetes-dynamic-pvc-8bfe9445-22de-11e9-b7ec-02420afe4907

7、验证pods

[root@elasticsearch01 harbor-helm]# kubectl get pods

NAME READY STATUS RESTARTS AGE

busybox 1/1 Running 600 25d

ceph-rbd-pv-pod1 1/1 Running 10 6d23h

jenkins-0 1/1 Running 0 6d2h

mymin-harbor-adminserver-685ccf67d7-k6z4p 1/1 Running 1 5m10s

mymin-harbor-chartmuseum-7d59b659df-nglbx 1/1 Running 0 5m10s

mymin-harbor-clair-69f89c644-62428 1/1 Running 1 5m10s

mymin-harbor-core-5cdd9c7bc9-z2lnd 1/1 Running 1 5m10s

mymin-harbor-database-0 1/1 Running 0 5m10s

mymin-harbor-jobservice-9889c95b9-s656x 1/1 Running 0 5m10s

mymin-harbor-notary-server-588bc8bf45-t7mkz 1/1 Running 0 5m10s

mymin-harbor-notary-signer-6d967d4c-jhvfs 1/1 Running 0 5m10s

mymin-harbor-portal-798ff99d56-vxnnx 1/1 Running 0 5m9s

mymin-harbor-redis-0 1/1 Running 0 5m10s

mymin-harbor-registry-54b5cd848d-4nr95 2/2 Running 0 5m9s

rbd-provisioner-67b4857bcd-xxwx5 1/1 Running 0 42m

期间遇到各种报错可以重置helm环境

[root@k8s-master harbor-helm]# helm install . --name mymin

helm delete --purge mymin

These resources were kept due to the resource policy:

[PersistentVolumeClaim] mymin-harbor-chartmuseum

[PersistentVolumeClaim] mymin-harbor-jobservice

[PersistentVolumeClaim] mymin-harbor-registry

release "mymin" deleted

把所有创建得pvc全部删除

四、访问harobr

1、获取harbor ingress 服务

[root@elasticsearch01 harbor-helm]# kubectl get ingress

NAME HOSTS ADDRESS PORTS AGE

jenkins jenkins.dashboard.com 80, 443 6d2h

mymin-harbor-ingress core-harbor.dashboard.com,notary-harbor.dashboard.com 80, 443 6m43s

2、docker login登陆验证 注意这里docker login默认是走https协议，需要ingress的node节点443对外开放，之前部署的ingress没有启动hostNetwork为true，这里需要启动，可以通过kubectl edit deployment/nginx-ingress-controller -n ingress-nginx修改，然后docker login就没问题了登陆测试

添加在dnsPolicy: ClusterFirst下面

hostNetwork: true

网页登录：https://core-harbor.dashboard.com/

机器登录上传镜像操作：

注意：登陆需要再nginx-ingress-controller 这个pods 所在得节点机器上

1、mkdir -p /etc/docker/certs.d/ore-harbor.dashboard.com/

2、登录网页把证书导入（登录网页下载证书）

重启docker 服务

[root@elasticsearch02 ~]# docker login core-harbor.dashboard.com

Username: admin

Password:

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.

Configure a credential helper to remove this warning. See

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

cat /etc/docker/certs.d/core-harbor.dashboard.com/ca.crt >> /etc/pki/tls/certs/ca-bundle.crt

上传下载测试

[root@k8s-node2~]# docker tag registry.cn-beijing.aliyuncs.com/kubernetes-dashboard:v1.10.1 core-harbor.dashboard.com/public/kubernetes-dashboard:v1.10.1

[root@elasticsearch02 ~]# docker push core-harbor.dashboard.com/public/kubernetes-dashboard:v1.10.1

The push refers to repository [core-harbor.doshboard.com/public/kubernetes-dashboard]

fbdfe08b001c: Pushed

v1.10.1: digest: sha256:54cc02a35d33a5ff9f8aa1a1b43f375728bcd85034cb311bdaf5c14f48340733 size: 529

人面猴
序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
沈念sama阅读 203,456评论 5赞 477
死咒
序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
沈念sama阅读 85,370评论 2赞 381
救了他两次的神仙让他今天三更去死
文/潘晓璐我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
开封第一讲书人阅读 150,337评论 0赞 337
道士缉凶录：失踪的卖姜人
文/不坏的土叔我叫张陵，是天一观的道长。经常有香客问我，道长，这世上最难降的妖魔是什么？我笑而不...
开封第一讲书人阅读 54,583评论 1赞 273
港岛之恋（遗憾婚礼）
正文为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
茶点故事阅读 63,596评论 5赞 365
恶毒庶女顶嫁案：这布局不是一般人想出来的
文/花漫我一把揭开白布。她就那样静静地躺着，像睡着了一般。火红的嫁衣衬着肌肤如雪。梳的纹丝不乱的头发上，一...
开封第一讲书人阅读 48,572评论 1赞 281
城市分裂传说
那天，我揣着相机与录音，去河边找鬼。笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。我是一名探鬼主播，决...
沈念sama阅读 37,936评论 3赞 395
双鸳鸯连环套：你想象不到人心有多黑
文/苍兰香墨我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
开封第一讲书人阅读 36,595评论 0赞 258
万荣杀人案实录
序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
沈念sama阅读 40,850评论 1赞 297
护林员之死
正文独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋以下内容为张勋视角年9月15日...
茶点故事阅读 35,601评论 2赞 321
白月光启示录
正文我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
茶点故事阅读 37,685评论 1赞 329
活死人
序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
沈念sama阅读 33,371评论 4赞 318
日本核电站爆炸内幕
正文年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
茶点故事阅读 38,951评论 3赞 307
男人毒药：我在死后第九天来索命
文/蒙蒙一、第九天我趴在偏房一处隐蔽的房顶上张望。院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
开封第一讲书人阅读 29,934评论 0赞 19
一桩弑父案，背后竟有这般阴谋
文/苍兰香墨我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。一阵脚步声响...
开封第一讲书人阅读 31,167评论 1赞 259
情欲美人皮
我被黑心中介骗来泰国打工，没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。一个月前我还...
沈念sama阅读 43,636评论 2赞 349
代替公主和亲
正文我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
茶点故事阅读 42,411评论 2赞 342

kubernetes1.13.1集群集成harbor-helm.md

推荐阅读更多精彩内容