journalctl -u kubelet –no-pager

Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.754019 1729 kubelet_node_status.go:94] Unable to register node “master” with API server: Post https://192.168.110.128:6443/api/v1/nodes: dial tcp 192.168.110.128:6443: connect: connection refused
Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.756764 1729 kubelet.go:2267] node “master” not found
Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.780410 1729 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://192.168.110.128:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dmaster&limit=500&resourceVersion=0: dial tcp 192.168.110.128:6443: connect: connection refused
Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.857437 1729 kubelet.go:2267] node “master” not found
Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.906390 1729 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: Get https://192.168.110.128:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 192.168.110.128:6443: connect: connection refused
Dec 07 20:44:14 master kubelet[1729]: E1207 20:44:14.957606 1729 kubelet.go:2267] node “master” not found
Dec 07 20:44:15 master kubelet[1729]: E1207 20:44:15.058525 1729 kubelet.go:2267] node “master” not found
Dec 07 20:44:15 master kubelet[1729]: E1207 20:44:15.106037 1729 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: Get https://192.168.110.128:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 192.168.110.128:6443: connect: connection refused
Dec 07 20:44:15 master kubelet[1729]: E1207 20:44:15.159565 1729 kubelet.go:2267] node “master” not found
Dec 07 20:44:15 master kubelet[1729]: E1207 20:44:15.260503 1729 kubelet.go:2267] node “master” not found

  • Jeff 回复了此帖

  • Jeff 都检查过了 没有问题，不知道为什么起不起来，没找到原因

      yuwengoku 兄弟啊，你先检查下防火墙是不是开了，主机ip是否变了，搜下帖子。服务器软件还是要提供一个稳定的环境，关机可以，开机之后状态至少要能恢复到关机之前的状态。

          Jeff 都检查过了 没有问题，不知道为什么起不起来，没找到原因

            yuwengoku 这个日志看着是 kube-apiserver 没有起来，docker ps -a | grep kube-apiserver 看下相关的container是否运行，如果有问题，可以用 docker logs 看下对应的container日志，先解决 kube-apiserver 的问题

              jcm 没办法，ip不能变。变了ip，证书那些都得重新生成，和重装没啥区别了

              • jcm 回复了此帖

                  Jeff 我重装了一遍，现在重启后访问不了30880了。。

                    检查有没有异常pod

                    kubectl get po -A

                    观察日志

                    防火墙

                    • kumu 回复了此帖
                      6 个月 后

                      zackzhang 我也遇到了，IP 没变 就是起不来
                      开始有一台 etcd 没起来，把etcd 弄起来还是不想。
                      删除pod 就报这个异常
                      pod “kube-scheduler-hdss7-21.host.com” deleted
                      error: An error occurred while waiting for the object to be deleted: too old resource version: 3033891 (3108218)error: An error occurred while waiting for the object to be deleted: too old resource version: 3033891 (3108218)

                        1 年 后

                        -- Logs begin at Sun 2022-07-17 01:42:30 PDT, end at Sun 2022-07-17 02:25:47 PDT. –

                        Jul 17 01:42:48 k8s-master systemd[1]: Started kubelet: The Kubernetes Node Agent.

                        Jul 17 01:42:48 k8s-master systemd[1]: Starting kubelet: The Kubernetes Node Agent…

                        Jul 17 01:43:26 k8s-master kubelet[665]: Flag –cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet’s –config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

                        Jul 17 01:43:26 k8s-master kubelet[665]: Flag –network-plugin has been deprecated, will be removed along with dockershim.

                        Jul 17 01:43:26 k8s-master kubelet[665]: Flag –cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet’s –config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.

                        Jul 17 01:43:26 k8s-master kubelet[665]: Flag –network-plugin has been deprecated, will be removed along with dockershim.

                        Jul 17 01:43:26 k8s-master kubelet[665]: I0717 01:43:26.624563 665 server.go:440] “Kubelet version” kubeletVersion=“v1.21.5”

                        Jul 17 01:43:26 k8s-master kubelet[665]: I0717 01:43:26.747939 665 server.go:851] “Client rotation is on, will bootstrap in background”

                        Jul 17 01:43:27 k8s-master kubelet[665]: I0717 01:43:27.319832 665 certificate_store.go:130] Loading cert/key pair from “/var/lib/kubelet/pki/kubelet-client-current.pem”.

                        Jul 17 01:43:30 k8s-master kubelet[665]: I0717 01:43:30.100553 665 dynamic_cafile_content.go:167] Starting client-ca-bundle::/etc/kubernetes/pki/ca.crt

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.042907 665 server.go:660] “–cgroups-per-qos enabled, but –cgroup-root was not specified. defaulting to /”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.087186 665 container_manager_linux.go:278] “Container manager verified user specified cgroup-root exists” cgroupRoot=[]

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.087519 665 container_manager_linux.go:283] “Creating Container Manager object based on Node Config” nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.116862 665 topology_manager.go:120] “Creating topology manager with policy per scope” topologyPolicyName=“none” topologyScopeName=“container”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.116919 665 container_manager_linux.go:314] “Initializing Topology Manager” policy=“none” scope=“container”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.116929 665 container_manager_linux.go:319] “Creating device plugin manager” devicePluginEnabled=true

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.117273 665 kubelet.go:307] “Using dockershim is deprecated, please consider using a full-fledged CRI implementation”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.316577 665 client.go:78] “Connecting to docker on the dockerEndpoint” endpoint=“unix:///var/run/docker.sock”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.316641 665 client.go:97] “Start docker client with request timeout” timeout=“2m0s”

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.329099 665 docker_service.go:566] “Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth” hairpinMode=promiscuous-bridge

                        Jul 17 01:44:22 k8s-master kubelet[665]: I0717 01:44:22.373470 665 docker_service.go:242] “Hairpin mode is set” hairpinMode=hairpin-veth

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.836321 665 docker_service.go:257] “Docker cri networking managed by the network plugin” networkPluginName=“cni”

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.873224 665 docker_service.go:264] “Docker Info” dockerInfo=&{ID😛K2Y:QN57:3XX5:Q3AO:MP5I:45VB:S3ZK:WLBL:VFKX:IDFG😃NRI:ESEX Containers:159 ContainersRunning:77 ContainersPaused:0 ContainersStopped:82 Images:73 Driver:overlay2

                        Jul 17 01:44:24 k8s-master kubelet[665]: :[] Warnings:[]}

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.873355 665 docker_service.go:277] “Setting cgroupDriver” cgroupDriver=“systemd”

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900407 665 remote_runtime.go:62] parsed scheme: ""

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900431 665 remote_runtime.go:62] scheme "" not registered, fallback to default scheme

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900533 665 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock <nil> 0 <nil>}] <nil> <nil>}

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900608 665 clientconn.go:948] ClientConn switching balancer to “pick_first”

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900786 665 remote_image.go:50] parsed scheme: ""

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900794 665 remote_image.go:50] scheme "" not registered, fallback to default scheme

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900803 665 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock <nil> 0 <nil>}] <nil> <nil>}

                        Jul 17 01:44:24 k8s-master kubelet[665]: I0717 01:44:24.900808 665 clientconn.go:948] ClientConn switching balancer to “pick_first”

                        Jul 17 01:44:25 k8s-master kubelet[665]: I0717 01:44:25.345694 665 kubelet.go:404] “Attempting to sync node with API server”

                        Jul 17 01:44:25 k8s-master kubelet[665]: I0717 01:44:25.345937 665 kubelet.go:272] “Adding static pod path” path=“/etc/kubernetes/manifests”

                        Jul 17 01:44:25 k8s-master kubelet[665]: I0717 01:44:25.346078 665 kubelet.go:283] “Adding apiserver pod source”

                        Jul 17 01:44:25 k8s-master kubelet[665]: I0717 01:44:25.346223 665 apiserver.go:42] “Waiting for node sync before watching apiserver pods”

                        Jul 17 01:44:25 k8s-master kubelet[665]: E0717 01:44:25.451339 665 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://lb.kubesphere.local:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-master&limit=500&r

                        Jul 17 01:44:25 k8s-master kubelet[665]: E0717 01:44:25.451700 665 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get “https://lb.kubesphere.local:6443/api/v1/services?limit=500&resourceVersion=0”: dial tcp 192.

                        Jul 17 01:44:25 k8s-master kubelet[665]: I0717 01:44:25.784429 665 kuberuntime_manager.go:222] “Container runtime initialized” containerRuntime=“docker” version=“20.10.8” apiVersion=“1.41.0”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.257748 665 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.

                        Jul 17 01:44:26 k8s-master kubelet[665]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.398474 665 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get “https://lb.kubesphere.local:6443/api/v1/services?limit=500&resourceVersion=0”: dial tcp 192.

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.459645 665 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://lb.kubesphere.local:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-master&limit=500&r

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.510391 665 server.go:1190] “Started kubelet”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.512251 665 kubelet.go:1306] “Image garbage collection failed once. Stats initialization may not have completed yet” err=“failed to get imageFs info: unable to find data in memory cache”

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.514693 665 server.go:149] “Starting to listen” address=“0.0.0.0” port=10250

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.625973 665 kubelet.go:1263] “Unprivileged containerized plugins might not work, could not set selinux context on plugin registration dir” path=“/var/lib/kubelet/plugins_registry” err="failed to set file label on /var/lib/kubel

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.626178 665 kubelet.go:1267] “Unprivileged containerized plugins might not work, could not set selinux context on plugins dir” path=“/var/lib/kubelet/plugins” err="failed to set file label on /var/lib/kubelet/plugins: invalid a

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.641484 665 fs_resource_analyzer.go:67] “Starting FS ResourceAnalyzer”

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.642644 665 volume_manager.go:271] “Starting Kubelet Volume Manager”

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.643255 665 desired_state_of_world_populator.go:141] “Desired state populator starts to run”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.645024 665 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get “https://lb.kubesphere.local:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-master?timeout=10s”: dial tcp 192

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.645128 665 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://lb.kubesphere.local:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVer

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.684772 665 server.go:409] “Adding debug handlers to kubelet server”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.718240 665 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"“, APIVersion:”"}, ObjectMeta:v1.ObjectMeta{Name:“k8s-master.17029162d61365e5”, GenerateName:"", Namespace:“default”, SelfLink:"“, UID:”", R

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.723187 665 scope.go:111] “RemoveContainer” containerID=“706172edd5121d56f9cf79ccf327946ff701bd2df035bdece7abf453ee42721f”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.742935 665 kubelet.go:2291] “Error getting node” err="node \“k8s-master\” not found"

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.827882 665 kubelet_node_status.go:71] “Attempting to register node” node=“k8s-master”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.828673 665 kubelet_node_status.go:93] “Unable to register node with API server” err="Post \“https://lb.kubesphere.local:6443/api/v1/nodes\”: dial tcp 192.168.25.137:6443: connect: connection refused" node=“k8s-master”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.843096 665 kubelet.go:2291] “Error getting node” err="node \“k8s-master\” not found"

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.847093 665 client.go:86] parsed scheme: “unix”

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.847125 665 client.go:86] scheme “unix” not registered, fallback to default scheme

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.847711 665 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}

                        Jul 17 01:44:26 k8s-master kubelet[665]: I0717 01:44:26.847722 665 clientconn.go:948] ClientConn switching balancer to “pick_first”

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.847793 665 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get “https://lb.kubesphere.local:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-master?timeout=10s”: dial tcp 192

                        Jul 17 01:44:26 k8s-master kubelet[665]: E0717 01:44:26.943631 665 kubelet.go:2291] “Error getting node” err="node \“k8s-master\” not found"

                        Jul 17 01:44:27 k8s-master kubelet[665]: E0717 01:44:27.044682 665 kubelet.go:2291] “Error getting node” err="node \“k8s-master\” not found"

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.104682 665 kubelet_node_status.go:71] “Attempting to register node” node=“k8s-master”

                        Jul 17 01:44:27 k8s-master kubelet[665]: E0717 01:44:27.106446 665 kubelet_node_status.go:93] “Unable to register node with API server” err="Post \“https://lb.kubesphere.local:6443/api/v1/nodes\”: dial tcp 192.168.25.137:6443: connect: connection refused" node=“k8s-master”

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.117042 665 cpu_manager.go:199] “Starting CPU manager” policy=“none”

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.117082 665 cpu_manager.go:200] “Reconciling” reconcilePeriod=“10s”

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.117149 665 state_mem.go:36] “Initialized new in-memory state store”

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.119173 665 kubelet_network_linux.go:56] “Initialized protocol iptables rules.” protocol=IPv4

                        Jul 17 01:44:27 k8s-master kubelet[665]: E0717 01:44:27.146615 665 kubelet.go:2291] “Error getting node” err="node \“k8s-master\” not found"

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.229530 665 state_mem.go:88] “Updated default CPUSet” cpuSet=""

                        Jul 17 01:44:27 k8s-master kubelet[665]: I0717 01:44:27.229578 665 state_mem.go:96] “Updated CPUSet assignments” assignments=map[]

                        我也是服务意外断电后服务都起不来。的ip 和防火否都关闭了。但是就是不行。