mongodb.png
Kubernetes部署Mongodb高可用集群
Mongo介绍
Mongodb是时下流行的NoSql数据库，它的存储方式是文档式存储，并不是Key-Value形式。关于Mongodb的特点，这里就不多介绍了，大家查看官方说明。
Mongodb高可用集群方案介绍
Replica set 模式
mongo_replica_set.png
如上图所示是mongodb 的 Replica Set集群模式，主备节点存储数据，仲裁节点不存储数据。客户端同时连接主节点与备节点，不连接仲裁节点。
默认设置下，主节点提供所有增删查改服务，备节点不提供任何服务。但是可以通过设置使备节点提供查询服务，这样就可以减少主节点的压力，当客户端进行数据查询时，请求自动转到备节点上。这个设置叫做Read Preference Modes，同时客户端提供了简单的配置方式，可以不必直接对数据库进行操作。
仲裁节点是一种特殊的节点，它本身并不存储数据，主要的作用是决定哪一个备节点在主节点挂掉之后提升为主节点，所以客户端不需要连接此节点。这里虽然只有一个备节点，但是仍然需要一个仲裁节点来提升备节点级别。
Mongodb的Replica Set即副本集方式主要有两个目的，一个是数据冗余做故障恢复使用，当发生硬件故障或者其它原因造成的宕机时，可以使用副本进行恢复。另一个是做读写分离，读的请求分流到副本上，减轻主（Primary）的读压力。
Master-Slaver 模式
这个是最简答的集群搭建，不过准确说也不能算是集群，只能说是主备。并且官方已经不推荐这种方式。
Sharding 模式
sharding模式最为完备，和Replica Set类似，都需要一个仲裁节点，但是Sharding还需要配置节点和路由节点。就三种集群搭建方式来说，这种是最复杂的。
Mongodb Kubernetes 部署(Replica set 模式)
注： 部署MongoDB主要用到的是mongo-db-sidecar。来设置replica set集群模式。
创建headless service
apiVersion: v1
kind: Service
metadata:
name: mongo
labels:
app: mongo
spec:
ports:
- name: mongo
port: 27017
targetPort: 27017
clusterIP: None
selector:
app: mongo
创建StorageClass
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: mongodb-data
provisioner: fuseim.pri/ifs
创建Statefulset
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: mongo
spec:
selector:
matchLabels:
app: mongo
serviceName: "mongo"
replicas: 3
template:
metadata:
labels:
app: mongo
role: mongo
environment: prod
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: 10.19.85.58:6666/mongo:latest
command:
- mongod
- "--replSet"
- rs0
- "--smallfiles"
- "--noprealloc"
ports:
- containerPort: 27017
volumeMounts:
- name: mongo-persistent-storage
mountPath: /data/db
- name: mongo-sidecar
image: 10.19.85.58:6666/mongo-k8s-sidecar:latest
env:
- name: MONGO_SIDECAR_POD_LABELS
value: "role=mongo,environment=prod"
resources:
limits:
memory: "{{ .ContainerMemory }}Gi"
cpu: "{{ .ContainerCPU }}"
volumeClaimTemplates:
- metadata:
name: mongo-persistent-storage
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: mongodb-data
resources:
requests:
storage: 10Gi
创建并查看以上服务
查看service服务
# kubectl get service -n mongo
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mongo None <none> 27017/TCP 2m
查看mongo服务
# kubectl get pod -n mongo
NAME READY STATUS RESTARTS AGE
mongo-0 2/2 Running 0 2m
mongo-1 2/2 Running 2 2m
mongo-2 2/2 Running 0 2m
查看集群状态
#kubectl exec -it mongo-0 -- mongo -n mongo
2020-04-24T11:17:33.030+0000 I CONTROL [initandlisten]
2020-04-24T11:17:33.030+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2020-04-24T11:17:33.030+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2020-04-24T11:17:33.030+0000 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended.
2020-04-24T11:17:33.030+0000 I CONTROL [initandlisten]
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten]
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten]
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten] ** We suggest setting it to 'never'
2020-04-24T11:17:34.040+0000 I CONTROL [initandlisten]
> rs.status()
{
"info" : "run rs.initiate(...) if not yet done for the set",
"ok" : 0,
"errmsg" : "no replset config has been received",
"code" : 94,
"codeName" : "NotYetInitialized"
}
结果显示集群状态不可以，没有组建成集群。
查看日志结果如下：
#kubectl logs mongo-0 mongo-sidecar -n mongo
...
Error in workloop { [Error: [object Object]]
message:
{ kind: 'Status',
apiVersion: 'v1',
metadata: {},
status: 'Failure',
message:
'pods is forbidden: User "system:serviceaccount:mongo:default" cannot list resource "pods" in API group "" at the cluster scope',
reason: 'Forbidden',
details: { kind: 'pods' },
code: 403 },
statusCode: 403 }
日志信息显示默认分配的sa账号没有list此namespace下pods的权限。
namespace 权限问题解决方案
对默认的clusterrole view权限进行赋权，我们可以使用clusterrole对sa特定namespace进行赋权，如下ClusterRoleBinding yaml方案：
apiVersion: v1
kind: ClusterRoleBinding
metadata:
name: view
roleRef:
name: view
subjects:
- kind: ServiceAccount
name: default
namespace: mongo
userNames:
- system:serviceaccount:mongo:default
添加完该namespace权限之后，再次进行查看。
kubectl exec -it mongo-0 -- mongo -n mongo
...
rs0:PRIMARY> rs.status()
{
"set" : "rs0",
"date" : ISODate("2020-04-23T03:30:31.502Z"),
"myState" : 1,
"term" : NumberLong(3),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"heartbeatIntervalMillis" : NumberLong(2000),
"majorityVoteCount" : 2,
"writeMajorityCount" : 2,
"optimes" : {
"lastCommittedOpTime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"lastCommittedWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
"readConcernMajorityOpTime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"readConcernMajorityWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
"appliedOpTime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"durableOpTime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"lastAppliedWallTime" : ISODate("2020-04-23T03:30:21.699Z"),
"lastDurableWallTime" : ISODate("2020-04-23T03:30:21.699Z")
},
"lastStableRecoveryTimestamp" : Timestamp(1587612591, 1),
"lastStableCheckpointTimestamp" : Timestamp(1587612591, 1),
"electionCandidateMetrics" : {
"lastElectionReason" : "electionTimeout",
"lastElectionDate" : ISODate("2020-04-22T10:42:49.199Z"),
"electionTerm" : NumberLong(3),
"lastCommittedOpTimeAtElection" : {
"ts" : Timestamp(1587552113, 1),
"t" : NumberLong(1)
},
"lastSeenOpTimeAtElection" : {
"ts" : Timestamp(1587552113, 1),
"t" : NumberLong(1)
},
"numVotesNeeded" : 2,
"priorityAtElection" : 1,
"electionTimeoutMillis" : NumberLong(10000),
"numCatchUpOps" : NumberLong(0),
"newTermStartDate" : ISODate("2020-04-22T10:42:49.910Z"),
"wMajorityWriteAvailabilityDate" : ISODate("2020-04-22T10:42:50.777Z")
},
"members" : [
{
"_id" : 0,
"name" : "171.248.3.105:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 2012,
"optime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"optimeDurable" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"optimeDate" : ISODate("2020-04-23T03:30:21Z"),
"optimeDurableDate" : ISODate("2020-04-23T03:30:21Z"),
"lastHeartbeat" : ISODate("2020-04-23T03:30:29.622Z"),
"lastHeartbeatRecv" : ISODate("2020-04-23T03:30:29.622Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "",
"syncingTo" : "171.248.1.86:27017",
"syncSourceHost" : "171.248.1.86:27017",
"syncSourceId" : 2,
"infoMessage" : "",
"configVersion" : 143974
},
{
"_id" : 1,
"name" : "171.248.2.3:27017",
"health" : 1,
"state" : 2,
"stateStr" : "SECONDARY",
"uptime" : 60495,
"optime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"optimeDurable" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"optimeDate" : ISODate("2020-04-23T03:30:21Z"),
"optimeDurableDate" : ISODate("2020-04-23T03:30:21Z"),
"lastHeartbeat" : ISODate("2020-04-23T03:30:31.410Z"),
"lastHeartbeatRecv" : ISODate("2020-04-23T03:30:31.408Z"),
"pingMs" : NumberLong(0),
"lastHeartbeatMessage" : "",
"syncingTo" : "171.248.1.86:27017",
"syncSourceHost" : "171.248.1.86:27017",
"syncSourceId" : 2,
"infoMessage" : "",
"configVersion" : 143974
},
{
"_id" : 2,
"name" : "171.248.1.86:27017",
"health" : 1,
"state" : 1,
"stateStr" : "PRIMARY",
"uptime" : 60503,
"optime" : {
"ts" : Timestamp(1587612621, 1),
"t" : NumberLong(3)
},
"optimeDate" : ISODate("2020-04-23T03:30:21Z"),
"syncingTo" : "",
"syncSourceHost" : "",
"syncSourceId" : -1,
"infoMessage" : "",
"electionTime" : Timestamp(1587552169, 1),
"electionDate" : ISODate("2020-04-22T10:42:49Z"),
"configVersion" : 143974,
"self" : true,
"lastHeartbeatMessage" : ""
}
],
"ok" : 1,
"$clusterTime" : {
"clusterTime" : Timestamp(1587612621, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
},
"operationTime" : Timestamp(1587612621, 1)
}
此时集群状态显示正常节点同步信息也正常。
