ks正常运行，node节点突然notready

1735802356

生产使用得ks2.1，然后node节点突然notready。

kubectl describe pod ks-apigateway-65fddf9488-w5vr8 -n kubesphere-system
Name: ks-apigateway-65fddf9488-w5vr8
Namespace: kubesphere-system
Priority: 0
Node: node28/172.19.186.57
Start Time: Sun, 27 Sep 2020 17:59:48 +0800
Labels: app=ks-apigateway
pod-template-hash=65fddf9488
tier=backend
version=v2.1.1
Annotations: kubectl.kubernetes.io/restartedAt: 2020-09-27T09:59:25Z
Status: Terminating (lasts 3h59m)
Termination Grace Period: 30s
IP: 10.233.67.3
IPs:
IP: 10.233.67.3
Controlled By: ReplicaSet/ks-apigateway-65fddf9488
Containers:
ks-apigateway:
Container ID: docker://0d2dd0bf04ff83fb553e785c51aeb2b6bb8a101684ec7a5e6a8894fa6f20f017
Image: kubesphere/ks-apigateway:v2.1.1
Image ID: docker-pullable://kubesphere/ks-apigateway@sha256:805d1e89aebd391d04bcc0d10fcef2529bad88e0919b5751c043a83fd1781592
Port: 2018/TCP
Host Port: 0/TCP
Command:
/bin/sh
-c
export KUBESPHERE_TOKEN=cat /var/run/secrets/kubernetes.io/serviceaccount/token && ks-apigateway –conf=/etc/caddy/Caddyfile –log=stderr
State: Running
Started: Sun, 27 Sep 2020 18:00:07 +0800
Ready: True
Restart Count: 0
Limits:
cpu: 1
memory: 500Mi
Requests:
cpu: 20m
memory: 100Mi
Environment:
JWT_SECRET: <set to the key ‘jwt-secret’ in secret ‘ks-account-secret’> Optional: false
Mounts:
/etc/caddy from caddyfile (rw)
/etc/kubesphere from kubesphere-config (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kubesphere-token-kq4wk (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
caddyfile:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: caddyfile
Optional: false
kubesphere-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kubesphere-config
Optional: false
kubesphere-token-kq4wk:
Type: Secret (a volume populated by a Secret)
SecretName: kubesphere-token-kq4wk
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: CriticalAddonsOnly
node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute for 60s
node.kubernetes.io/unreachable:NoExecute for 60s
Events: <none>

有没有大佬帮忙看下

1735802356

[root@master1 ~]# kubectl describe pod s2ioperator-0 -n kubesphere-devops-system
Name: s2ioperator-0
Namespace: kubesphere-devops-system
Priority: 0
Node: node28/172.19.186.57
Start Time: Sun, 27 Sep 2020 18:00:14 +0800
Labels: control-plane=s2i-controller-manager
controller-revision-hash=s2ioperator-6cdf88fb5f
controller-tools.k8s.io=1.0
statefulset.kubernetes.io/pod-name=s2ioperator-0
Annotations: prometheus.io/scrape: true
Status: Terminating (lasts 4h1m)
Termination Grace Period: 10s
IP: 10.233.67.4
IPs:
IP: 10.233.67.4
Controlled By: StatefulSet/s2ioperator
Containers:
manager:
Container ID: docker://c20bce5cfddf7edc61e30dc08689d7f20e68c238d8f03c79d693fe31cb8300be
Image: kubesphere/s2ioperator:v2.1.1
Image ID: docker-pullable://kubesphere/s2ioperator@sha256:48c67c38344f2be958f6b65e425f35a9ec19cb2d0ac9620d3bfa9a066103697c
Ports: 8080/TCP, 9876/TCP
Host Ports: 0/TCP, 0/TCP
Command:
/manager
State: Running
Started: Sun, 27 Sep 2020 18:00:39 +0800
Ready: True
Restart Count: 1
Limits:
cpu: 100m
memory: 500Mi
Requests:
cpu: 100m
memory: 20Mi
Environment:
POD_NAMESPACE: kubesphere-devops-system (v1:metadata.namespace)
SECRET_NAME: webhook-server-secret
S2IIMAGENAME: kubesphere/s2irun:v2.1.1
Mounts:
/tmp/cert from cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-hfzld (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady True
PodScheduled True
Volumes:
cert:
Type: Secret (a volume populated by a Secret)
SecretName: webhook-server-secret
Optional: false
default-token-hfzld:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-hfzld
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>

1735802356

节点日志

yunkunrao

执行kubectl decribe node看一下事件有相关提示吗，另外环境本身是ready的吗，比如防火墙，节点资源是否充足。