一个master，两个node部署成功，几小时后apiserver无法访问，
cluster-info
To further debug and diagnose cluster problems, use ‘kubectl cluster-info dump’.
The connection to the server 10.119.31.220:6443 was refused - did you specify the right host or port?
现在看apiserver docker进程状态如下
以下是master节点/var/log/messages相关日志，求助
> Feb 25 22:52:35 master kubelet: I0225 22:46:50.224415 29197 prober.go:116] Liveness probe for "calico-node-2v6rm_kube-system(4130c112-9c93-4553-a115-5cd1b0f217cf):calico-node" failed (failure): Get http://127.0.0.1:9099/liveness: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Feb 25 22:56:41 master kubelet: I0225 22:56:04.886774 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 5m5.110325701s: [ /var/lib/docker/containers/d9c5b4ba844629e497fe5f6547b63e42ec67f18cffb4bf6dd4fa75eac04e8c27]; will not log again for this container unless duration exceeds 2s
Feb 25 22:57:29 master kubelet: E0225 22:57:25.962157 29197 remote_runtime.go:453] Status from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:30 master kubelet: I0225 22:57:26.990329 29197 prober.go:116] Liveness probe for "kube-controller-manager-master_kube-system(151a1c6c0f2a3a780b7963a5020276ea):kube-controller-manager" failed (failure): Get http://127.0.0.1:10252/healthz: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Feb 25 22:57:31 master kubelet: W0225 22:57:27.278183 29197 prober.go:108] No ref for container "docker://28bcc85afaa5bc6d264158541d792e496fc455020a5d769ee61f2570e18c9d0f" (coredns-7f9d8dc6c8-stghs_kube-system(a404ad94-2eb6-4324-adb9-88afdf5715da):coredns)
Feb 25 22:57:31 master kubelet: I0225 22:57:27.278298 29197 prober.go:116] Readiness probe for "coredns-7f9d8dc6c8-stghs_kube-system(a404ad94-2eb6-4324-adb9-88afdf5715da):coredns" failed (failure): Get http://10.233.70.3:8181/ready: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Feb 25 22:57:31 master kubelet: I0225 22:57:27.353458 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 8m41.537054271s: [ /var/lib/docker/containers/0c27eb1fbcfa8f3beff9382c63d82b8b171f6e027ebdcd6bb3f666f569c4e609]; will not log again for this container unless duration exceeds 2s
> Feb 25 22:57:31 master kubelet: I0225 22:57:28.530654 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 8m42.164205869s: [ /var/lib/docker/containers/4d99e4ed85a2e137b86f7c
2b4c43ad040135d5cdf3134d2f00a4f115e6cfabba]; will not log again for this container unless duration exceeds 2s
Feb 25 22:57:31 master kubelet: E0225 22:57:28.525799 29197 remote_image.go:71] ListImages with filter nil from image service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:31 master kubelet: E0225 22:57:28.569799 29197 container_manager_linux.go:89] Unable to get docker version: operation timeout: context deadline exceeded
Feb 25 22:57:31 master kubelet: W0225 22:57:28.580933 29197 prober.go:108] No ref for container "docker://4ea1f4bd538fdd5c7c1ff75e9ff2cbcfa7bf4f110473a10dce424a2d2c7cebb3" (openldap-0_kubesphere-system(9d9e0c6e-11d9-48b0-bb90-ca3819840121):openldap-ha)
Feb 25 22:57:31 master kubelet: I0225 22:57:28.580990 29197 prober.go:116] Liveness probe for "openldap-0_kubesphere-system(9d9e0c6e-11d9-48b0-bb90-ca3819840121):openldap-ha" failed (failure): dial tcp 10.233.70.5:389: i/o timeout
> Feb 25 22:57:32 master kubelet: I0225 22:57:32.577912 29197 prober.go:116] Liveness probe for "calico-node-2v6rm_kube-system(4130c112-9c93-4553-a115-5cd1b0f217cf):calico-node" failed (failure): Get http://127.0.0.1:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Feb 25 22:57:32 master kubelet: E0225 22:57:32.578906 29197 remote_runtime.go:182] ListPodSandbox with filter nil from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: E0225 22:57:32.579187 29197 kuberuntime_sandbox.go:210] ListPodSandbox failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: E0225 22:57:32.579234 29197 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: E0225 22:57:32.575645 29197 remote_runtime.go:351] ExecSync c1aac479daaf071e7b6fdf957874c7404c9905d8e407c79945097e1a569da3e1 '/bin/calico-node -bird-ready -felix-ready' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
> Feb 25 22:57:32 master kubelet: I0225 22:57:32.589042 29197 kubelet_getters.go:172] status for pod kube-scheduler-master updated to {Running [{Initialized True 0001-01-01 00:00:00 +0000 UTC 2020-02-25 22:25:41 +0800 CST } {Ready True 0001-01-01 00:00:00 +0000 UTC 2020-02-25 22:38:58 +0800 CST } {ContainersReady True 0001-01-01 00:00:00 +0000 UTC 2020-02-25 22:38:58 +0800 CST } {PodScheduled True 0001-01-01 00:00:00 +0000 UTC 2020-02-25 22:25:41 +0800 CST }] 10.119.31.220 10.119.31.220 [{10.119.31.220}] 2020-02-25 22:25:41 +0800 CST [] [{kube-scheduler {nil &ContainerStateRunning{StartedAt:2020-02-25 22:38:56 +0800 CST,} nil} {nil nil &ContainerStateTerminated{ExitCode:255,Signal:0,Reason:Error,Message:,StartedAt:2020-02-25 22:28:40 +0800 CST,FinishedAt:2020-02-25 22:38:48 +0800 CST,ContainerID:docker://158fae7b4a72c74ca4f208145628c1f20d088aa30057efa577f38bbdacf9d67f,}} true 8 gcr.azk8s.cn/google-containers/hyperkube:v1.16.7 docker-pullable://gcr.azk8s.cn/google-containers/hyperkube@sha256:7e476647f86543072a0aad7acf44f6460ac1400e690094ccbb39b3c5c3c709fa docker://6c2127ae0363358f611026b2b2ce31526bc782b93018a1b2affa51367ec53b56 0xc0018fa999}] Burstable []}
Feb 25 22:57:32 master kubelet: I0225 22:57:00.890062 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 4m55.849754507s: [ /var/lib/docker/containers/4ea1f4bd538fdd5c7c1ff75e9ff2cbcfa7bf4f110473a10dce424a2d2c7cebb3]; will not log again for this container unless duration exceeds 2s
Feb 25 22:57:32 master kubelet: E0225 22:57:32.508730 29197 kubelet.go:2174] Container runtime sanity check failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: I0225 22:57:32.648545 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 7m18.191045792s: [ /var/lib/docker/containers/9c0801d1d0be0a70a0a79279e572fef95c2af02d1ea66b796721bd0a34720814]; will not log again for this container unless duration exceeds 2s
Feb 25 22:57:32 master kubelet: E0225 22:57:32.656223 29197 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "master": Get https://10.119.31.220:6443/api/v1/nodes/master?resourceVersion=0&timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Feb 25 22:57:32 master kubelet: E0225 22:57:32.680972 29197 remote_image.go:87] ImageStatus "mirrorgooglecontainers/pause-amd64:3.1" from image service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: E0225 22:57:32.681029 29197 kuberuntime_image.go:85] ImageStatus for image {"mirrorgooglecontainers/pause-amd64:3.1"} failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: I0225 22:57:32.702992 29197 log.go:172] http: TLS handshake error from 10.119.31.208:45132: write tcp 10.119.31.220:10250->10.119.31.208:45132: write: broken pipe
Feb 25 22:57:33 master kubelet: W0225 22:57:33.055199 29197 prober.go:108] No ref for container "docker://6a5b48a2783ac7ccadc6308d14d9116ac598dca23aeab2eb83c5a91177496ef4" (nodelocaldns-lvf9h_kube-system(974f08dc-150b-4c45-a56f-d746f90a42b7):node-cache)
> Feb 25 22:57:32 master kubelet: E0225 22:57:32.508730 29197 kubelet.go:2174] Container runtime sanity check failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: I0225 22:57:32.648545 29197 fsHandler.go:132] fs: disk usage and inodes count on following dirs took 7m18.191045792s: [ /var/lib/docker/containers/9c0801d1d0be0a70a0a79279e572fef95c2af02d1ea66b796721bd0a34720814]; will not log again for this container unless duration exceeds 2s
Feb 25 22:57:32 master kubelet: E0225 22:57:32.656223 29197 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "master": Get https://10.119.31.220:6443/api/v1/nodes/master?resourceVersion=0&timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Feb 25 22:57:32 master kubelet: E0225 22:57:32.680972 29197 remote_image.go:87] ImageStatus "mirrorgooglecontainers/pause-amd64:3.1" from image service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: E0225 22:57:32.681029 29197 kuberuntime_image.go:85] ImageStatus for image {"mirrorgooglecontainers/pause-amd64:3.1"} failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 22:57:32 master kubelet: I0225 22:57:32.702992 29197 log.go:172] http: TLS handshake error from 10.119.31.208:45132: write tcp 10.119.31.220:10250->10.119.31.208:45132: write: broken pipe
Feb 25 22:57:33 master kubelet: W0225 22:57:33.055199 29197 prober.go:108] No ref for container "docker://6a5b48a2783ac7ccadc6308d14d9116ac598dca23aeab2eb83c5a91177496ef4" (nodelocaldns-lvf9h_kube-system(974f08dc-150b-4c45-a56f-d746f90a42b7):node-cache)
> Feb 25 23:02:27 master kubelet: E0225 22:57:44.638576 29197 controller.go:170] failed to update node lease, error: Put https://10.119.31.220:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/master?timeout=10s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Feb 25 23:02:28 master kubelet: E0225 22:58:28.400182 29197 kubelet_node_status.go:388] Error updating node status, will retry: error getting node "master": Get https://10.119.31.220:6443/api/v1/nodes/master?timeout=10s: context deadline exceeded
Feb 25 23:02:28 master dockerd: time="2020-02-25T23:02:28.575169581+08:00" level=error msg="Handler for GET /images/json returned error: write unix /var/run/docker.sock->@: write: broken pipe"
Feb 25 23:02:28 master dockerd: http: multiple response.WriteHeader calls
Feb 25 23:02:43 master kubelet: W0225 23:02:39.857197 29197 prober.go:108] No ref for container "docker://6a5b48a2783ac7ccadc6308d14d9116ac598dca23aeab2eb83c5a91177496ef4" (nodelocaldns-lvf9h_kube-system(974f08dc-150b-4c45-a56f-d746f90a42b7):node-cache)
Feb 25 23:02:46 master kubelet: W0225 23:02:39.892534 29197 prober.go:108] No ref for container "docker://6a5b48a2783ac7ccadc6308d14d9116ac598dca23aeab2eb83c5a91177496ef4" (nodelocaldns-lvf9h_kube-system(974f08dc-150b-4c45-a56f-d746f90a42b7):node-cache)
Feb 25 23:02:47 master kubelet: I0225 23:02:39.892593 29197 prober.go:116] Liveness probe for "nodelocaldns-lvf9h_kube-system(974f08dc-150b-4c45-a56f-d746f90a42b7):node-cache" failed (failure): Get http://169.254.25.10:9254/health: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Feb 25 23:02:47 master kubelet: W0225 23:02:24.469975 29197 prober.go:108] No ref for container "docker://c1aac479daaf071e7b6fdf957874c7404c9905d8e407c79945097e1a569da3e1" (calico-node-2v6rm_kube-system(4130c112-9c93-4553-a115-5cd1b0f217cf):calico-node)
Feb 25 23:02:47 master kubelet: I0225 23:02:39.996389 29197 prober.go:116] Liveness probe for "calico-node-2v6rm_kube-system(4130c112-9c93-4553-a115-5cd1b0f217cf):calico-node" failed (failure): Get http://127.0.0.1:9099/liveness: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Feb 25 23:02:47 master kubelet: E0225 23:02:40.036387 29197 remote_runtime.go:182] ListPodSandbox with filter nil from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.036563 29197 kuberuntime_sandbox.go:210] ListPodSandbox failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.036609 29197 generic.go:205] GenericPLEG: Unable to retrieve pods: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.039816 29197 remote_image.go:71] ListImages with filter nil from image service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.039889 29197 kuberuntime_image.go:134] ListImages failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.039936 29197 kubelet.go:1302] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get image stats: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.040196 29197 webhook.go:107] Failed to make webhook authenticator request: Post https://10.119.31.220:6443/apis/authentication.k8s.io/v1beta1/tokenreviews: read tcp 10.119.31.220:51918->10.119.31.220:6443: use of closed network connection
Feb 25 23:02:47 master kubelet: E0225 23:02:40.040277 29197 server.go:252] Unable to authenticate the request due to an error: Post https://10.119.31.220:6443/apis/authentication.k8s.io/v1beta1/tokenreviews: read tcp 10.119.31.220:51918->10.119.31.220:6443: use of closed network connection
Feb 25 23:02:47 master kubelet: E0225 23:02:40.040620 29197 remote_image.go:71] ListImages with filter nil from image service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: E0225 23:02:40.040671 29197 kuberuntime_image.go:100] ListImages failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: W0225 23:02:40.040691 29197 image_gc_manager.go:181] [imageGCManager] Failed to monitor images: rpc error: code = DeadlineExceeded desc = context deadline exceeded
Feb 25 23:02:47 master kubelet: I0225 23:02:40.040702 29197 kubelet.go:1839] skipping pod synchronization - [container runtime is down, PLEG is not healthy: pleg was last seen active 15m45.117330616s ago; threshold is 3m0s]
> Feb 25 23:09:56 master kubelet: I0225 23:09:56.672225 16698 server.go:384] Adding debug handlers to kubelet server.
Feb 25 23:09:56 master kubelet: E0225 23:09:56.673996 16698 kubelet.go:1302] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Feb 25 23:09:56 master kubelet: I0225 23:09:56.684466 16698 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Feb 25 23:09:56 master kubelet: I0225 23:09:56.684540 16698 status_manager.go:156] Starting to sync pod status with apiserver
Feb 25 23:09:56 master kubelet: I0225 23:09:56.684582 16698 kubelet.go:1822] Starting kubelet main sync loop.
Feb 25 23:09:56 master kubelet: I0225 23:09:56.684617 16698 kubelet.go:1839] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Feb 25 23:09:56 master kubelet: I0225 23:09:56.688373 16698 volume_manager.go:247] The desired_state_of_world populator starts
Feb 25 23:09:56 master kubelet: I0225 23:09:56.688394 16698 volume_manager.go:249] Starting Kubelet Volume Manager
Feb 25 23:09:56 master kubelet: I0225 23:09:56.689487 16698 desired_state_of_world_populator.go:131] Desired state populator starts to run
Feb 25 23:09:56 master kubelet: I0225 23:09:56.911336 16698 kubelet.go:1839] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Feb 25 23:09:56 master kubelet: E0225 23:09:56.923624 16698 kubelet.go:2267] node "master" not found
Feb 25 23:09:57 master kubelet: I0225 23:09:57.024886 16698 kubelet_node_status.go:286] Setting node annotation to enable volume controller attach/detach
Feb 25 23:09:57 master kubelet: I0225 23:09:57.026959 16698 setters.go:73] Using node IP: "10.119.31.220"
Feb 25 23:09:57 master kubelet: I0225 23:09:56.986948 16698 oom_watcher_linux.go:62] Got sys oom event: &{20091 java 2020-02-25 21:17:13.936559073 +0800 CST m=-6751.264839585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.080998 16698 oom_watcher_linux.go:62] Got sys oom event: &{30686 java 2020-02-25 21:24:35.212216073 +0800 CST m=-6309.989182585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081051 16698 oom_watcher_linux.go:62] Got sys oom event: &{7279 java 2020-02-25 21:27:18.110321073 +0800 CST m=-6147.091077585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081069 16698 oom_watcher_linux.go:62] Got sys oom event: &{12924 java 2020-02-25 21:30:35.683904073 +0800 CST m=-5949.517494585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081095 16698 oom_watcher_linux.go:62] Got sys oom event: &{19415 java 2020-02-25 21:36:43.490455073 +0800 CST m=-5581.710943585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081109 16698 oom_watcher_linux.go:62] Got sys oom event: &{27862 java 2020-02-25 21:48:31.357858073 +0800 CST m=-4873.843540585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081183 16698 oom_watcher_linux.go:62] Got sys oom event: &{6715 java 2020-02-25 22:09:34.213301073 +0800 CST m=-3610.988097585 / /}
Feb 25 23:09:57 master kubelet: I0225 23:09:57.081216 16698 oom_watcher_linux.go:62] Got sys oom event: &{16498 java 2020-02-25 22:19:10.691576073 +0800 CST m=-3034.509822585 / /}
