高可用 ks集群，停一个master后报错

ryuhai083

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。如果未按模板创建问题，管理员有权关闭问题。
确保帖子格式清晰易读，用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。

操作系统信息
例如：虚拟机/物理机，Centos7.5/Ubuntu18.04，4C/8G

虚拟机 centos 7 2c/8G

Kubernetes版本信息
将 kubectl version 命令执行结果贴在下方

v1.20.4

容器运行时
将 docker version / crictl version / nerdctl version 结果贴在下方

19.03.8

KubeSphere版本信息
例如：v2.1.1/v3.0.0。离线安装还是在线安装。在已有K8s上安装还是使用kk安装。

v3.11 在线安装，使用kk安装

问题是什么

原来的架构，k8s-master1 和k8s-master2可以时间高可用。停k8s-master1或者k8s-master2.不影响服务。
k8s-master1 Ready control-plane,master 9d v1.20.4

k8s-master2 Ready control-plane,master 9d v1.20.4

k8s-node1 Ready worker 9d v1.20.4

k8s-node2 Ready worker 9d v1.20.4报错日志是什么，最好有截图。

先有的架构。 etcd的集群是 k8s-master1 ，k8s-master2和 k8s-node1 3个节点

[root@k8s-master2 ~]# kubectl get node

NAME STATUS ROLES AGE VERSION

k8s-master
k8s-master1 Ready control-plane,master 9d v1.20.4

k8s-master2 Ready control-plane,master 9d v1.20.4

k8s-node1 Ready worker 9d v1.20.4

k8s-node2 Ready worker 9d v1.20.4

k8s-node3 Ready worker 16h v1.20.4

现在要新加个 k8s-master3 的节点

sample.yaml 文件如下

[root@k8s-master2 ~]# cat sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha1

kind: Cluster

metadata:

name: sample

spec:

hosts:

You should complete the ssh information of the hosts

{name: k8s-master1, address: 192.168.1.237, internalAddress: 192.168.1.237}
{name: k8s-master2, address: 192.168.1.231, internalAddress: 192.168.1.231}
{name: k8s-master3, address: 192.168.1.236, internalAddress: 192.168.1.236}
{name: k8s-node1, address: 192.168.1.235, internalAddress: 192.168.1.235}
{name: k8s-node2, address: 192.168.1.251, internalAddress: 192.168.1.251}
{name: k8s-node3, address: 192.168.1.232, internalAddress: 192.168.1.232}

roleGroups:

etcd:
- k8s-master1
- k8s-master2
- k8s-node1
  
  master:
- k8s-master1
- k8s-master2
- k8s-master3
  
  worker:
- k8s-node1
- k8s-node2
- k8s-node3
controlPlaneEndpoint:

If loadbalancer was used, ‘address’ should be set to loadbalancer’s ip.

domain: lb.kubesphere.local

address: “192.168.1.239”

port: 16443

kubernetes:

version: v1.20.4

clusterName: cluster.local

proxyMode: ipvs

masqueradeAll: false

maxPods: 110

nodeCidrMaskSize: 24

network:

plugin: calico

kubePodsCIDR: 10.233.64.0/18

kubeServiceCIDR: 10.233.0.0/18

registry:

privateRegistry: ""

报错如下

[root@k8s-master2 ~]# ./kk add nodes -f sample.yaml

Failed to start etcd cluster: Failed to start etcd cluster: Failed to exec command: sudo -E /bin/sh -c “export ETCDCTL_API=2;export ETCDCTL_CERT_FILE=‘/etc/ssl/etcd/ssl/admin-k8s-master1.pem’;export ETCDCTL_KEY_FILE=‘/etc/ssl/etcd/ssl/admin-k8s-master1-key.pem’;export ETCDCTL_CA_FILE=‘/etc/ssl/etcd/ssl/ca.pem’;/usr/local/bin/etcdctl –endpoints=https://192.168.1.237:2379,https://192.168.1.231:2379,https://192.168.1.235:2379 cluster-health | grep -q ‘cluster is healthy’”

: Process exited with status 1

laminar

ryuhai083
etcd:

k8s-master1

k8s-master2

k8s-node1

如果要将 etcd 部署到 master3 的话需要在这个部分添加 master3

ryuhai083

标题忘记改了。是新加master是报错

ryuhai083

新加下报错信息

laminar

可以查看下 etcd 服务的日志吗，类似 journalctl -u etcd -f

ryuhai083

journalctl -u etcd -f

ryuhai083

journalctl -u etcd -f

1.236 是要新加的k8s-master3 节点

ryuhai083

laminar
m1 和 m2 和 n1 3个本来就是3个etcd的集群。

ryuhai083

然后我也试过在这个部分添加 master3 ，还是报同样的错误

laminar

也就是说现在其实不需要在 master3 上搭建 etcd，但在集群配置中已经配置上了 master3 的 etcd？可以检查下集群中关于 etcd 部分的配置，如果不需要在 master3 上搭建 etcd 的话可以 192.168.1.236:2380 类似的配置清除掉

ryuhai083

laminar
现在看下来是原来的etcd集群有问题，这个怎么解决呢
231 237 和 235是原来的3个etcd

不过这样查看到是正常的

原来的是这样kk部署的
`[root@k8s-master1 ~]# cat config-sample.yaml
apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
name: sample
spec:
hosts:

{name: k8s-master1, address: 192.168.1.237, internalAddress: 192.168.1.237, user: root, password: ‘1qaz2wsx’}
{name: k8s-master2, address: 192.168.1.231, internalAddress: 192.168.1.231, user: root, password: ‘1qaz2wsx’}
{name: k8s-node1, address: 192.168.1.235, internalAddress: 192.168.1.235, user: root, password: ‘1qaz2wsx’}
{name: k8s-node2, address: 192.168.1.251, internalAddress: 192.168.1.251, user: root, password: ‘1qaz2wsx’}
roleGroups:
etcd:
- k8s-master1
- k8s-master2
- k8s-node1
  master:
- k8s-master1
- k8s-master2
  worker:
- k8s-node1
- k8s-node2
  controlPlaneEndpoint:
  domain: lb.kubesphere.local
  address: “192.168.1.239”
  port: 16443
  kubernetes:
  version: v1.20.4
  imageRepo: kubesphere
  clusterName: cluster.local
  network:
  plugin: calico
  kubePodsCIDR: 10.233.64.0/18
  kubeServiceCIDR: 10.233.0.0/18
  registry:
  registryMirrors: []
  insecureRegistries: []
  addons:
name: nfs-client
namespace: kube-system
sources:
chart:
name: nfs-client-provisioner
repo: https://charts.kubesphere.io/main
valuesFile: /tmp/nfs-client.yaml

—
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.1.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
zone: ""
local_registry: ""
etcd:
monitoring: false
endpointIps: localhost
port: 2379
tlsEnable: true
common:
redis:
enabled: false
redisVolumSize: 2Gi
openldap:
enabled: false
openldapVolumeSize: 2Gi
minioVolumeSize: 20Gi
monitoring:
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
es:
elasticsearchMasterVolumeSize: 4Gi
elasticsearchDataVolumeSize: 20Gi
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchUrl: ""
externalElasticsearchPort: ""
console:
enableMultiLogin: true
port: 30880
alerting:
enabled: false

thanosruler:

replicas: 1

resources: {}

auditing:
enabled: false
devops:
enabled: false
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: false
ruler:
enabled: true
replicas: 2
logging:
enabled: false
logsidecar:
enabled: true
replicas: 2
metrics_server:
enabled: false
monitoring:
storageClass: ""
prometheusMemoryRequest: 400Mi
prometheusVolumeSize: 20Gi
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: false
servicemesh:
enabled: false
kubeedge:
enabled: false
cloudCore:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
cloudhubPort: “10000”
cloudhubQuicPort: “10001”
cloudhubHttpsPort: “10002”
cloudstreamPort: “10003”
tunnelPort: “10004”
cloudHub:
advertiseAddress:
- ""
nodeLimit: “100”
service:
cloudhubNodePort: “30000”
cloudhubQuicNodePort: “30001”
cloudhubHttpsNodePort: “30002”
cloudstreamNodePort: “30003”
tunnelNodePort: “30004”
edgeWatcher:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {“node-role.kubernetes.io/worker”: ""}
tolerations: []
`

ryuhai083

laminar
现在就是要新加一个节点，当做master3，集群里面本来已经有3个etcd了。master3不用跑etcd，并没有配置master3为etcd。
现在测试下来本来的 master1， master2 和node1 这个3个etcd都是正常的。现在新加 master3这个节点为master节点时候有报错。

Cauchy

ryuhai083

把证书加上再试试看

export ETCDCTL_API=2;export ETCDCTL_CERT_FILE=‘/etc/ssl/etcd/ssl/admin-k8s-master1.pem’;export ETCDCTL_KEY_FILE=‘/etc/ssl/etcd/ssl/admin-k8s-master1-key.pem’;export ETCDCTL_CA_FILE=‘/etc/ssl/etcd/ssl/ca.pem’;/usr/local/bin/etcdctl –endpoints=https://192.168.1.237:2379,https://192.168.1.231:2379,https://192.168.1.235:2379 cluster-health

ryuhai083

Cauchy

ryuhai083

ryuhai083

现在发现是新加节点 k8s-master3 192.168.1.236 这里的etcd服务没有正常启动