云上部署kubesphere 3.3.1报错

SYB

部署环境国内AWS服务器ubuntu 22.04版本部署kubesphere v3.3.1版本，k8s v1.26.12版本，按照官网文档部署出现node没有就绪，发现其中网络组件有报错

提供信息如下：

root@qc-master-01:/opt/kubekey# kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE

kube-system calico-kube-controllers-57db949bd8-vxkg2 1/1 Running 0 101m

kube-system calico-node-c6zq7 0/1 Init:↉ 0 101m

kube-system calico-node-dsj7l 1/1 Running 0 101m

kube-system calico-node-kt6c7 0/1 Init:↉ 0 101m

kube-system calico-node-mm7mf 0/1 Init:↉ 0 101m

kube-system coredns-6788f9ff4-btfkc 1/1 Running 0 101m

kube-system coredns-6788f9ff4-vb44t 1/1 Running 0 101m

kube-system kube-apiserver-qc-master-01 1/1 Running 0 101m

kube-system kube-controller-manager-qc-master-01 1/1 Running 0 101m

kube-system kube-proxy-bgb4p 1/1 Running 0 101m

kube-system kube-proxy-lkbvp 0/1 ContainerCreating 0 101m

kube-system kube-proxy-m9c8d 0/1 ContainerCreating 0 101m

kube-system kube-proxy-tkj6d 0/1 ContainerCreating 0 101m

kube-system kube-scheduler-qc-master-01 1/1 Running 0 101m

kube-system nodelocaldns-b8mjt 0/1 ContainerCreating 0 101m

kube-system nodelocaldns-h5d4t 0/1 ContainerCreating 0 101m

kube-system nodelocaldns-wmffn 1/1 Running 0 101m

kube-system nodelocaldns-ws9c4 0/1 ContainerCreating 0 101m

kube-system openebs-localpv-provisioner-589cc46f59-xwq9n 0/1 Pending 0 101m

kubesphere-system ks-installer-9fd999d6f-lgnvx 0/1 Pending 0 101m

root@qc-master-01:/opt/kubekey# kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

qc-master-01 Ready control-plane 101m v1.26.12 10.0.12.29 <none> Ubuntu 22.04.5 LTS 6.5.0-1023-aws containerd://1.7.24

qc-node-01 NotReady worker 101m v1.26.12 10.0.15.86 <none> Ubuntu 22.04.4 LTS 6.5.0-1023-aws containerd://1.7.24

qc-node-02 NotReady worker 101m v1.26.12 10.0.15.91 <none> Ubuntu 22.04.5 LTS 6.5.0-1023-aws containerd://1.7.24

qc-node-03 NotReady worker 101m v1.26.12 10.0.15.211 <none> Ubuntu 22.04.4 LTS 6.5.0-1023-aws containerd://1.7.24

—

root@qc-master-01:/opt/kubekey# kubectl describe node qc-node-01

Name: qc-node-01

Roles: worker

Labels: beta.kubernetes.io/arch=amd64

                beta.kubernetes.io/os=linux

                kubernetes.io/arch=amd64

                kubernetes.io/hostname=qc-node-01

                kubernetes.io/os=linux

                node-role.kubernetes.io/worker=

Annotations: kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock

                node.alpha.kubernetes.io/ttl: 0

                volumes.kubernetes.io/controller-managed-attach-detach: true

CreationTimestamp: Tue, 11 Feb 2025 13:52:26 +0800

Taints: node.kubernetes.io/not-ready:NoSchedule

Unschedulable: false

Lease:

HolderIdentity: qc-node-01

AcquireTime: <unset>

RenewTime: Tue, 11 Feb 2025 15:41:08 +0800

Conditions:

Type Status LastHeartbeatTime LastTransitionTime Reason Message

MemoryPressure False Tue, 11 Feb 2025 15:39:48 +0800 Tue, 11 Feb 2025 13:52:26 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available

DiskPressure False Tue, 11 Feb 2025 15:39:48 +0800 Tue, 11 Feb 2025 13:52:26 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure

PIDPressure False Tue, 11 Feb 2025 15:39:48 +0800 Tue, 11 Feb 2025 13:52:26 +0800 KubeletHasSufficientPID kubelet has sufficient PID available

Ready False Tue, 11 Feb 2025 15:39:48 +0800 Tue, 11 Feb 2025 13:52:26 +0800 KubeletNotReady container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

Addresses:

root@qc-master-01:/opt/kubekey# kubectl describe pod kube-proxy-lkbvp -n kube-system

Name: kube-proxy-lkbvp

Namespace: kube-system

Priority: 2000001000

Priority Class Name: system-node-critical

Service Account: kube-proxy

Node: qc-node-01/10.0.15.86

Start Time: Tue, 11 Feb 2025 13:52:29 +0800

Labels: controller-revision-hash=56c7d6f877

                  k8s-app=kube-proxy

                  pod-template-generation=1

Annotations: <none>

Status: Pending

IP: 10.0.15.86

IPs:

IP: 10.0.15.86

Controlled By: DaemonSet/kube-proxy

Containers:

kube-proxy:

Container ID:  

Image:         registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.26.12

Image ID:      

Port:          <none>

Host Port:     <none>

Command:

  /usr/local/bin/kube-proxy

  --config=/var/lib/kube-proxy/config.conf

  --hostname-override=$(NODE_NAME)

State:          Waiting

  Reason:       ContainerCreating

Ready:          False

Restart Count:  0

Environment:

  NODE_NAME:   (v1:spec.nodeName)

Mounts:

  /lib/modules from lib-modules (ro)

  /run/xtables.lock from xtables-lock (rw)

  /var/lib/kube-proxy from kube-proxy (rw)

  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pw27r (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

kube-proxy:

Type:      ConfigMap (a volume populated by a ConfigMap)

Name:      kube-proxy

Optional:  false

xtables-lock:

Type:          HostPath (bare host directory volume)

Path:          /run/xtables.lock

HostPathType:  FileOrCreate

lib-modules:

Type:          HostPath (bare host directory volume)

Path:          /lib/modules

HostPathType:

kube-api-access-pw27r:

Type:                    Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds:  3607

ConfigMapName:           kube-root-ca.crt

ConfigMapOptional:       <nil>

DownwardAPI:             true

QoS Class: BestEffort

Node-Selectors: kubernetes.io/os=linux

Tolerations: op=Exists

                         node.kubernetes.io/disk-pressure:NoSchedule op=Exists

                         node.kubernetes.io/memory-pressure:NoSchedule op=Exists

                         node.kubernetes.io/network-unavailable:NoSchedule op=Exists

                         node.kubernetes.io/not-ready:NoExecute op=Exists

                         node.kubernetes.io/pid-pressure:NoSchedule op=Exists

                         node.kubernetes.io/unreachable:NoExecute op=Exists

                         node.kubernetes.io/unschedulable:NoSchedule op=Exists

Events:

Type Reason Age From Message

Warning FailedCreatePodSandBox 37m (x2 over 94m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

Warning FailedCreatePodSandBox 12m (x9 over 64m) kubelet Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

Warning FailedCreatePodSandBox 7m24s (x7 over 89m) kubelet Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

Warning FailedCreatePodSandBox 2m24s (x56 over 90m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

● kubelet.service - kubelet: The Kubernetes Node Agent

 Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)

Drop-In: /etc/systemd/system/kubelet.service.d

         └─10-kubeadm.conf

 Active: active (running) since Tue 2025-02-11 13:51:56 CST; 57min ago

   Docs: http://kubernetes.io/docs/

Main PID: 40131 (kubelet)

  Tasks: 12 (limit: 9830)

 Memory: 43.7M

    CPU: 1min 23.127s

 CGroup: /system.slice/kubelet.service

         └─40131 /usr/local/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --container-runtime-endpoint=unix:///run/containerd/containerd.sock --pod-infra-container-image=registry.cn-beijing.aliyuncs.com/kubesphereio/pause:3.9 --node-ip=10.0.12.29 --hostname-override=qc-master-01

Feb 11 13:52:35 qc-master-01 kubelet[40131]: E0211 13:52:35.868445 40131 kuberuntime_sandbox.go:72] “Failed to create sandbox for pod” err="rpc error: code = Unknown desc = failed to setup network for sandbox \“b0d42f564f776af10433ab1b2ea9a9a49734be5d42b3a11ae1087e9d6cc9b6cd\”: plugin type=\“calico\” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" pod=“kube-system/coredns-6788f9ff4-btfkc”

Feb 11 13:52:35 qc-master-01 kubelet[40131]: E0211 13:52:35.868478 40131 kuberuntime_manager.go:782] “CreatePodSandbox for pod failed” err="rpc error: code = Unknown desc = failed to setup network for sandbox \“b0d42f564f776af10433ab1b2ea9a9a49734be5d42b3a11ae1087e9d6cc9b6cd\”: plugin type=\“calico\” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/" pod=“kube-system/coredns-6788f9ff4-btfkc”

Feb 11 13:52:35 qc-master-01 kubelet[40131]: E0211 13:52:35.868549 40131 pod_workers.go:965] “Error syncing pod, skipping” err="failed to \“CreatePodSandbox\” for \“coredns-6788f9ff4-btfkc_kube-system(ed0d168f-24a0-41d0-9739-4fd707ad637a)\” with CreatePodSandboxError: \"Failed to create sandbox for pod \\\“coredns-6788f9ff4-btfkc_kube-system(ed0d168f-24a0-41d0-9739-4fd707ad637a)\\\”: rpc error: code = Unknown desc = failed to setup network for sandbox \\\“b0d42f564f776af10433ab1b2ea9a9a49734be5d42b3a11ae1087e9d6cc9b6cd\\\”: plugin type=\\\“calico\\\” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/\"" pod=“kube-system/coredns-6788f9ff4-btfkc” podUID=ed0d168f-24a0-41d0-9739-4fd707ad637a

Feb 11 13:52:36 qc-master-01 kubelet[40131]: I0211 13:52:36.357678 40131 pod_container_deletor.go:80] “Container not found in pod’s containers” containerID=“b0d42f564f776af10433ab1b2ea9a9a49734be5d42b3a11ae1087e9d6cc9b6cd”

Feb 11 13:52:36 qc-master-01 kubelet[40131]: I0211 13:52:36.365405 40131 pod_container_deletor.go:80] “Container not found in pod’s containers” containerID=“d2e6a5ede8d0567d44fe3c7a0902f374a5a5c7bec63ef20f9d710305926e8637”

Feb 11 13:52:36 qc-master-01 kubelet[40131]: I0211 13:52:36.370791 40131 pod_container_deletor.go:80] “Container not found in pod’s containers” containerID=“0f47fdfa141d4dc1765e07903842cab395b0a097aa2d1a7e31ff9f122e7c2e93”

Feb 11 13:52:36 qc-master-01 kubelet[40131]: I0211 13:52:36.442290 40131 pod_startup_latency_tracker.go:102] “Observed pod startup duration” pod=“kube-system/calico-node-dsj7l” podStartSLOduration=7.442251809 pod.CreationTimestamp=“2025-02-11 13:52:29 +0800 CST” firstStartedPulling=“0001-01-01 00:00:00 +0000 UTC” lastFinishedPulling=“0001-01-01 00:00:00 +0000 UTC” observedRunningTime=“2025-02-11 13:52:36.441951659 +0800 CST m=+39.594642162” watchObservedRunningTime=“2025-02-11 13:52:36.442251809 +0800 CST m=+39.594942311”

Feb 11 13:52:38 qc-master-01 kubelet[40131]: I0211 13:52:38.463073 40131 pod_startup_latency_tracker.go:102] “Observed pod startup duration” pod=“kube-system/calico-kube-controllers-57db949bd8-vxkg2” podStartSLOduration=9.463031675 pod.CreationTimestamp=“2025-02-11 13:52:29 +0800 CST” firstStartedPulling=“0001-01-01 00:00:00 +0000 UTC” lastFinishedPulling=“0001-01-01 00:00:00 +0000 UTC” observedRunningTime=“2025-02-11 13:52:38.43814195 +0800 CST m=+41.590832453” watchObservedRunningTime=“2025-02-11 13:52:38.463031675 +0800 CST m=+41.615722177”

Feb 11 13:52:38 qc-master-01 kubelet[40131]: I0211 13:52:38.483046 40131 pod_startup_latency_tracker.go:102] “Observed pod startup duration” pod=“kube-system/coredns-6788f9ff4-btfkc” podStartSLOduration=27.482992381 pod.CreationTimestamp=“2025-02-11 13:52:11 +0800 CST” firstStartedPulling=“0001-01-01 00:00:00 +0000 UTC” lastFinishedPulling=“0001-01-01 00:00:00 +0000 UTC” observedRunningTime=“2025-02-11 13:52:38.463957586 +0800 CST m=+41.616648107” watchObservedRunningTime=“2025-02-11 13:52:38.482992381 +0800 CST m=+41.635682884”

Feb 11 13:52:38 qc-master-01 kubelet[40131]: I0211 13:52:38.575120 40131 pod_startup_latency_tracker.go:102] “Observed pod startup duration” pod=“kube-system/coredns-6788f9ff4-vb44t” podStartSLOduration=27.57506763 pod.CreationTimestamp=“2025-02-11 13:52:11 +0800 CST” firstStartedPulling=“0001-01-01 00:00:00 +0000 UTC” lastFinishedPulling=“0001-01-01 00:00:00 +0000 UTC” observedRunningTime=“2025-02-11 13:52:38.544729963 +0800 CST m=+41.697420463” watchObservedRunningTime=“2025-02-11 13:52:38.57506763 +0800 CST m=+41.727758133”

NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE

default 18m Normal Starting node/qc-master-01 Starting kubelet.

default 18m Warning InvalidDiskCapacity node/qc-master-01 invalid capacity 0 on image filesystem

default 18m Normal NodeHasSufficientMemory node/qc-master-01 Node qc-master-01 status is now: NodeHasSufficientMemory

default 18m Normal NodeHasNoDiskPressure node/qc-master-01 Node qc-master-01 status is now: NodeHasNoDiskPressure

default 18m Normal NodeHasSufficientPID node/qc-master-01 Node qc-master-01 status is now: NodeHasSufficientPID

default 18m Normal NodeAllocatableEnforced node/qc-master-01 Updated Node Allocatable limit across pods

default 17m Normal Starting node/qc-master-01 Starting kubelet.

default 17m Warning InvalidDiskCapacity node/qc-master-01 invalid capacity 0 on image filesystem

default 17m Normal NodeAllocatableEnforced node/qc-master-01 Updated Node Allocatable limit across pods

default 17m Normal NodeHasSufficientMemory node/qc-master-01 Node qc-master-01 status is now: NodeHasSufficientMemory

default 17m Normal NodeHasNoDiskPressure node/qc-master-01 Node qc-master-01 status is now: NodeHasNoDiskPressure

default 17m Normal NodeHasSufficientPID node/qc-master-01 Node qc-master-01 status is now: NodeHasSufficientPID

default 17m Normal RegisteredNode node/qc-master-01 Node qc-master-01 event: Registered Node qc-master-01 in Controller

default 17m Normal Starting node/qc-master-01

default 16m Normal NodeReady node/qc-master-01 Node qc-master-01 status is now: NodeReady

default 17m Normal NodeHasSufficientMemory node/qc-node-01 Node qc-node-01 status is now: NodeHasSufficientMemory

default 17m Normal RegisteredNode node/qc-node-01 Node qc-node-01 event: Registered Node qc-node-01 in Controller

default 17m Normal NodeHasSufficientMemory node/qc-node-02 Node qc-node-02 status is now: NodeHasSufficientMemory

default 17m Normal RegisteredNode node/qc-node-02 Node qc-node-02 event: Registered Node qc-node-02 in Controller

default 17m Normal NodeHasSufficientMemory node/qc-node-03 Node qc-node-03 status is now: NodeHasSufficientMemory

default 17m Normal RegisteredNode node/qc-node-03 Node qc-node-03 event: Registered Node qc-node-03 in Controller

kube-system 17m Warning FailedScheduling pod/calico-kube-controllers-57db949bd8-9rbtz 0/4 nodes are available: 4 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..

kube-system 16m Normal Scheduled pod/calico-kube-controllers-57db949bd8-9rbtz Successfully assigned kube-system/calico-kube-controllers-57db949bd8-9rbtz to qc-master-01

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “829036fd5566b8590f8fe84747a5abebbdab42659d424194f58bb750290d982a”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Normal SandboxChanged pod/calico-kube-controllers-57db949bd8-9rbtz Pod sandbox changed, it will be killed and re-created.

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “898f21c8e0a9f181a3b44fe64f48a1860941c81d7e120998b8d624579a2b4cdb”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “83210b70d882e72bf0700f04611c4f31b05a9ef967366f34049ef1ef8ee01c0b”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “366475da2f874b2c46612fd0a1f36ebd007d73b68a7cc32214bf4fb18a2eea6a”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “c1fe0735cb199c65bdc73816d4f4d5e9877720dbfdf3343f69c4bf5b93262941”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “dff3afa7b558967bfee19108cba535f146466a169cddb8e358d0c394a23646da”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “b3c657092ff412640dd0c6ac9b49524b43baf830fd9c649d0f4072f91c2a4571”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “a5dcc57b2dd5c2ddb77d9e64129b89b500fa6b91920568afb76bf1b4e8ba928f”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “bb6c29a0ac11e6f7d38ec4f46135513371d344ee135db647a4f564154feb4556”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/calico-kube-controllers-57db949bd8-9rbtz (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “34455fb7bdfa3f1ee813ee10970f54bfae0b0b4f949c85c1b3df0d03472e59ac”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 17m Normal SuccessfulCreate replicaset/calico-kube-controllers-57db949bd8 Created pod: calico-kube-controllers-57db949bd8-9rbtz

kube-system 17m Normal NoPods poddisruptionbudget/calico-kube-controllers No matching pods found

kube-system 17m Normal ScalingReplicaSet deployment/calico-kube-controllers Scaled up replica set calico-kube-controllers-57db949bd8 to 1

kube-system 17m Normal Scheduled pod/calico-node-b7m2h Successfully assigned kube-system/calico-node-b7m2h to qc-node-01

kube-system 14m Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 13m Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 9m33s Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 11m Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 11m Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 91s Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 4m30s Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 2m59s Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 47s Warning FailedCreatePodSandBox pod/calico-node-b7m2h Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 4s Warning FailedCreatePodSandBox pod/calico-node-b7m2h (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/calico-node-fhrrg Successfully assigned kube-system/calico-node-fhrrg to qc-master-01

kube-system 17m Normal Pulling pod/calico-node-fhrrg Pulling image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4”

kube-system 16m Normal Pulled pod/calico-node-fhrrg Successfully pulled image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” in 47.390062076s (47.390081227s including waiting)

kube-system 16m Normal Created pod/calico-node-fhrrg Created container upgrade-ipam

kube-system 16m Normal Started pod/calico-node-fhrrg Started container upgrade-ipam

kube-system 16m Normal Pulled pod/calico-node-fhrrg Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

kube-system 16m Normal Created pod/calico-node-fhrrg Created container install-cni

kube-system 16m Normal Started pod/calico-node-fhrrg Started container install-cni

kube-system 16m Normal Pulling pod/calico-node-fhrrg Pulling image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4”

kube-system 15m Normal Pulled pod/calico-node-fhrrg Successfully pulled image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” in 1m5.057504978s (1m5.05751111s including waiting)

kube-system 15m Normal Created pod/calico-node-fhrrg Created container mount-bpffs

kube-system 15m Normal Started pod/calico-node-fhrrg Started container mount-bpffs

kube-system 15m Normal Pulled pod/calico-node-fhrrg Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

kube-system 15m Normal Created pod/calico-node-fhrrg Created container calico-node

kube-system 15m Normal Started pod/calico-node-fhrrg Started container calico-node

kube-system 15m Warning Unhealthy pod/calico-node-fhrrg Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

kube-system 17m Normal Scheduled pod/calico-node-jdkp5 Successfully assigned kube-system/calico-node-jdkp5 to qc-node-03

kube-system 8m Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.142.82:443: i/o timeout

kube-system 16m Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 12m Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 14m Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 9m27s Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 43s Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 6m34s Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 5m4s Warning FailedCreatePodSandBox pod/calico-node-jdkp5 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.142.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/calico-node-sz5sr Successfully assigned kube-system/calico-node-sz5sr to qc-node-02

kube-system 2m16s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 51s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 12m Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 3m1s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 7m58s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 7m15s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 6m32s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 94s Warning FailedCreatePodSandBox pod/calico-node-sz5sr Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 8s Warning FailedCreatePodSandBox pod/calico-node-sz5sr (combined from similar events): Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 17m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-jdkp5

kube-system 17m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-sz5sr

kube-system 17m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-b7m2h

kube-system 17m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-fhrrg

kube-system 17m Warning FailedScheduling pod/coredns-6fc8c459c9-2mnj8 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

kube-system 16m Normal Scheduled pod/coredns-6fc8c459c9-2mnj8 Successfully assigned kube-system/coredns-6fc8c459c9-2mnj8 to qc-master-01

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “5feb00fa9c9ae48f865af89543a9bda2f428968db3193246e0b096d6c9bbce77”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Normal SandboxChanged pod/coredns-6fc8c459c9-2mnj8 Pod sandbox changed, it will be killed and re-created.

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “c33e38ffc31d9de063e3f3c6d4dce66ffbde4319d5c0806979fca442b761611f”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “497551ca63159bd96d91d6fc4a65502a183f33278077156a5f3983d249f697ea”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “7dde036d397ffbbf0cf8bcb4e87717cbfa5c185d1a679d504fe22c1c4adcf71b”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “4c17828149c4ebb8535bf90916fff394af17279ff191e798f6013d4a1d8c1329”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “8491674116270e20636029be71b36a16774b05a7f5348fc1a0b40ce1e29d5313”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “eb7b8c15038a0964394098f800b10c977ada741a1a5c1dfdfa95e12b21c1abd5”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “1f1298586d75131dc6113e893abe0a7702711639384a08ad5b95176a059c05e7”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “ce3b7e8c9b2c25be0846d0c80a1a029d4b667cfe9bf25b2508443c384ecfa2f2”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-2mnj8 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “2c2179bb52259fc8bcfabfd7630d433c6181714cf3582b8b1077edd764094411”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 17m Warning FailedScheduling pod/coredns-6fc8c459c9-8dtjd 0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

kube-system 16m Normal Scheduled pod/coredns-6fc8c459c9-8dtjd Successfully assigned kube-system/coredns-6fc8c459c9-8dtjd to qc-master-01

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “768513899ffe6855bc7f04fa69d9b61dee95c85779626d058bc7b8f1e3459b97”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Normal SandboxChanged pod/coredns-6fc8c459c9-8dtjd Pod sandbox changed, it will be killed and re-created.

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “30a236cb78e063322b0f26ab91245d7200fedddda92d16cca6015f527b90c698”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “bcfce8652cd5042d8c56ef9ef85e80dd45637b887fc4ba5cea5e93a89ec6b06e”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “68458da7689ba5dfb640adb0e810d6fc5e8d209c40afc0217d710c9b7031c1d2”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “5942b42fd59f224e748fab4942761a0cb5fc89ed7712785bd0cc42594e50f030”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “f075c52cebaea5cc0529f44c5f7e9ab5cb24f300b46c685afe9b4fc4c2fa63d3”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “b91bb4021e618798569989821ead02474a0ed56a9a0738103523026f87a09989”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “77cd361d57398ccf98e64747158623310e7235d675a08b3d0969bed1d12aac35”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “0c030919c01287dc769a99d00a334ed1cab433ebc890ff88d0453122c0a6a791”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 16m Warning FailedCreatePodSandBox pod/coredns-6fc8c459c9-8dtjd (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “088955de1bb7d6026c9ee8156ab6275ed1c8bf28e5e23c1f36538f74479c0221”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

kube-system 17m Normal SuccessfulCreate replicaset/coredns-6fc8c459c9 Created pod: coredns-6fc8c459c9-2mnj8

kube-system 17m Normal SuccessfulCreate replicaset/coredns-6fc8c459c9 Created pod: coredns-6fc8c459c9-8dtjd

kube-system 17m Normal ScalingReplicaSet deployment/coredns Scaled up replica set coredns-6fc8c459c9 to 2

kube-system 17m Warning NodeNotReady pod/kube-apiserver-qc-master-01 Node is not ready

kube-system 17m Warning NodeNotReady pod/kube-controller-manager-qc-master-01 Node is not ready

kube-system 17m Normal LeaderElection lease/kube-controller-manager qc-master-01_af0fb92f-56d3-449e-a4c2-3309554e40d5 became leader

kube-system 17m Normal Scheduled pod/kube-proxy-89bfh Successfully assigned kube-system/kube-proxy-89bfh to qc-node-01

kube-system 13m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 13m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 11m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 11m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 2s Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 8m5s Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 47s Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 3m Warning FailedCreatePodSandBox pod/kube-proxy-89bfh Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 89s Warning FailedCreatePodSandBox pod/kube-proxy-89bfh (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/kube-proxy-jsfw4 Successfully assigned kube-system/kube-proxy-jsfw4 to qc-node-02

kube-system 16m Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 89s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 2m57s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 4m21s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 6m29s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 7m14s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 2m12s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 47s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 1s Warning FailedCreatePodSandBox pod/kube-proxy-jsfw4 (combined from similar events): Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/kube-proxy-r7296 Successfully assigned kube-system/kube-proxy-r7296 to qc-master-01

kube-system 17m Normal Pulled pod/kube-proxy-r7296 Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy:v1.26.12” already present on machine

kube-system 17m Normal Created pod/kube-proxy-r7296 Created container kube-proxy

kube-system 17m Normal Started pod/kube-proxy-r7296 Started container kube-proxy

kube-system 17m Normal Scheduled pod/kube-proxy-ttjvj Successfully assigned kube-system/kube-proxy-ttjvj to qc-node-03

kube-system 16m Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 3m50s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 59s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 13m Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 8m56s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 5m18s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.142.82:443: i/o timeout

kube-system 4m36s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 2m23s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 14s Warning FailedCreatePodSandBox pod/kube-proxy-ttjvj (combined from similar events): Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 17m Normal SuccessfulCreate daemonset/kube-proxy Created pod: kube-proxy-r7296

kube-system 17m Normal SuccessfulCreate daemonset/kube-proxy Created pod: kube-proxy-ttjvj

kube-system 17m Normal SuccessfulCreate daemonset/kube-proxy Created pod: kube-proxy-jsfw4

kube-system 17m Normal SuccessfulCreate daemonset/kube-proxy Created pod: kube-proxy-89bfh

kube-system 17m Warning NodeNotReady pod/kube-scheduler-qc-master-01 Node is not ready

kube-system 17m Normal LeaderElection lease/kube-scheduler qc-master-01_613edf65-128c-48ae-9663-9c1c28f801a6 became leader

kube-system 17m Normal Scheduled pod/nodelocaldns-dbkwb Successfully assigned kube-system/nodelocaldns-dbkwb to qc-node-02

kube-system 16m Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 16m Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 2m24s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 56s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 172.253.117.82:443: i/o timeout

kube-system 5m55s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 6m37s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 8m49s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 5m11s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 3m7s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 11s Warning FailedCreatePodSandBox pod/nodelocaldns-dbkwb (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.99.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/nodelocaldns-dlccq Successfully assigned kube-system/nodelocaldns-dlccq to qc-node-03

kube-system 3m46s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 16m Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 53s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 5m53s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 5m9s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.142.82:443: i/o timeout

kube-system 6m35s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 4m28s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 99s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 8s Warning FailedCreatePodSandBox pod/nodelocaldns-dlccq (combined from similar events): Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/nodelocaldns-hz8cm Successfully assigned kube-system/nodelocaldns-hz8cm to qc-node-01

kube-system 16m Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 16m Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 3m43s Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.197.82:443: i/o timeout

kube-system 13m Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 11m Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.250.107.82:443: i/o timeout

kube-system 10m Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

kube-system 9m37s Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 2m16s Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 8m11s Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 74.125.135.82:443: i/o timeout

kube-system 6s Warning FailedCreatePodSandBox pod/nodelocaldns-hz8cm (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image “registry.k8s.io/pause:3.8”: failed to pull image “registry.k8s.io/pause:3.8”: failed to pull and unpack image “registry.k8s.io/pause:3.8”: failed to resolve reference “registry.k8s.io/pause:3.8”: failed to do request: Head “https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 142.251.188.82:443: i/o timeout

kube-system 17m Normal Scheduled pod/nodelocaldns-tlqqn Successfully assigned kube-system/nodelocaldns-tlqqn to qc-master-01

kube-system 17m Normal Pulling pod/nodelocaldns-tlqqn Pulling image “registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20”

kube-system 17m Normal Pulled pod/nodelocaldns-tlqqn Successfully pulled image “registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache:1.22.20” in 14.160089272s (14.160101238s including waiting)

kube-system 17m Normal Created pod/nodelocaldns-tlqqn Created container node-cache

kube-system 17m Normal Started pod/nodelocaldns-tlqqn Started container node-cache

kube-system 17m Normal SuccessfulCreate daemonset/nodelocaldns Created pod: nodelocaldns-tlqqn

kube-system 17m Normal SuccessfulCreate daemonset/nodelocaldns Created pod: nodelocaldns-dlccq

kube-system 17m Normal SuccessfulCreate daemonset/nodelocaldns Created pod: nodelocaldns-dbkwb

kube-system 17m Normal SuccessfulCreate daemonset/nodelocaldns Created pod: nodelocaldns-hz8cm

kube-system 112s Warning FailedScheduling pod/openebs-localpv-provisioner-589cc46f59-6wqdf 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..

kube-system 17m Normal SuccessfulCreate replicaset/openebs-localpv-provisioner-589cc46f59 Created pod: openebs-localpv-provisioner-589cc46f59-6wqdf

kube-system 17m Normal ScalingReplicaSet deployment/openebs-localpv-provisioner Scaled up replica set openebs-localpv-provisioner-589cc46f59 to 1

kubesphere-system 112s Warning FailedScheduling pod/ks-installer-9fd999d6f-xnwc7 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..

kubesphere-system 17m Normal SuccessfulCreate replicaset/ks-installer-9fd999d6f Created pod: ks-installer-9fd999d6f-xnwc7

kubesphere-system 17m Normal ScalingReplicaSet deployment/ks-installer Scaled up replica set ks-installer-9fd999d6f to 1

root@qc-master-01:/opt/kubekey# crictl image list

IMAGE TAG IMAGE ID SIZE

k8s.gcr.io/pause 3.8 4873874c08efc 309kB

registry.aliyuncs.com/google_containers/pause 3.8 4873874c08efc 309kB

registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.8 4873874c08efc 309kB

registry.k8s.io/pause 3.8 4873874c08efc 309kB

swr.cn-north-4.myhuaweicloud.com/ddn-k8s/registry.k8s.io/pause 3.8 4873874c08efc 309kB

k8s.gcr.io/pause 3.9 e6f1816883972 322kB

registry.cn-beijing.aliyuncs.com/kubesphereio/pause 3.9 e6f1816883972 322kB

registry.cn-hangzhou.aliyuncs.com/google_containers/pause 3.9 e6f1816883972 322kB

registry.cn-beijing.aliyuncs.com/kubesphereio/cni v3.27.4 dc6f84c32585f 88.8MB

registry.cn-beijing.aliyuncs.com/kubesphereio/coredns 1.9.3 5185b96f0becf 14.8MB

swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/coredns/coredns 1.9.3 5185b96f0becf 14.8MB

registry.cn-beijing.aliyuncs.com/kubesphereio/k8s-dns-node-cache 1.22.20 ff71cd4ea5ae5 30.5MB

registry.cn-beijing.aliyuncs.com/kubesphereio/kube-apiserver v1.26.12 d48ca03484e3d 36.8MB

registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controller-manager v1.26.12 c957a971a5f4d 33.5MB

registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers v3.27.4 6b1e38763f401 33.5MB

registry.cn-beijing.aliyuncs.com/kubesphereio/kube-proxy v1.26.12 750c3024ec36f 23.6MB

registry.cn-beijing.aliyuncs.com/kubesphereio/kube-scheduler v1.26.12 9c7ffedd7c6d4 18MB

registry.cn-beijing.aliyuncs.com/kubesphereio/node v3.27.4 3dd4390f2a85a 117MB

swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/cni v3.27.4-linuxarm64 eaa2969f27e4f 81.3MB

swr.cn-north-4.myhuaweicloud.com/ddn-k8s/docker.io/calico/kube-controllers v3.27.4-linuxarm64 624858d5c19fe 29.9MB

Cauchy

拉不到pause镜像，如果是用kk部署k8s，可以在部署前执行 export KKZONE=cn, 机器上已经有containerd的话，可以修改containerd的配置里的 sandbox images，然后重启 containerd

“registry.k8s.io/pause:3.8”: failed to do request: Head “https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.8”: dial tcp 108.177.98.82:443: i/o timeout

SYB

Cauchy

目前可以拉那个镜像，出现新的报错：

root@qc-master-01:~# kubectl get events –sort-by=‘.metadata.creationTimestamp’ -n kubesphere-system

LAST SEEN TYPE REASON OBJECT MESSAGE

40m Normal ScalingReplicaSet deployment/ks-installer Scaled up replica set ks-installer-9fd999d6f to 1

40m Warning FailedScheduling pod/ks-installer-9fd999d6f-qtw7f 0/4 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had untolerated taint {node.kubernetes.io/not-ready: }. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling..

40m Normal SuccessfulCreate replicaset/ks-installer-9fd999d6f Created pod: ks-installer-9fd999d6f-qtw7f

40m Normal Scheduled pod/ks-installer-9fd999d6f-qtw7f Successfully assigned kubesphere-system/ks-installer-9fd999d6f-qtw7f to qc-node-03

40m Warning FailedCreatePodSandBox pod/ks-installer-9fd999d6f-qtw7f Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “c25ec03235ffed6631481b4bfad80ef183fa5ac3ab2d52af1a657e48a846d5f2”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

40m Normal SandboxChanged pod/ks-installer-9fd999d6f-qtw7f Pod sandbox changed, it will be killed and re-created.

40m Warning FailedCreatePodSandBox pod/ks-installer-9fd999d6f-qtw7f Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox “cc496fcb58f11f017050760ce8806e1811166bc7660c45d5c7357bde054df690”: plugin type=“calico” failed (add): stat /var/lib/calico/nodename: no such file or directory: check that the calico/node container is running and has mounted /var/lib/calico/

38m Normal Created pod/ks-installer-9fd999d6f-qtw7f Created container installer

38m Normal Started pod/ks-installer-9fd999d6f-qtw7f Started container installer

38m Normal Pulled pod/ks-installer-9fd999d6f-qtw7f Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.2” already present on machine

38m Warning BackOff pod/ks-installer-9fd999d6f-qtw7f Back-off restarting failed container installer in pod ks-installer-9fd999d6f-qtw7f_kubesphere-system(05f989a7-b9ac-47ef-8c18-4c99107c8368)

37m Normal Killing pod/ks-installer-9fd999d6f-qtw7f Stopping container installer

37m Normal SuccessfulCreate replicaset/ks-installer-9fd999d6f Created pod: ks-installer-9fd999d6f-zgxs7

37m Normal Scheduled pod/ks-installer-9fd999d6f-zgxs7 Successfully assigned kubesphere-system/ks-installer-9fd999d6f-zgxs7 to qc-node-03

34m Normal Started pod/ks-installer-9fd999d6f-zgxs7 Started container installer

34m Normal Pulled pod/ks-installer-9fd999d6f-zgxs7 Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.2” already present on machine

34m Normal Created pod/ks-installer-9fd999d6f-zgxs7 Created container installer

27m Warning BackOff pod/ks-installer-9fd999d6f-zgxs7 Back-off restarting failed container installer in pod ks-installer-9fd999d6f-zgxs7_kubesphere-system(8500d01c-df51-4db6-a6bc-df278cb1b9b3)

23m Normal SuccessfulCreate replicaset/ks-installer-9fd999d6f Created pod: ks-installer-9fd999d6f-rwv42

23m Normal Scheduled pod/ks-installer-9fd999d6f-rwv42 Successfully assigned kubesphere-system/ks-installer-9fd999d6f-rwv42 to qc-node-03

20m Normal Started pod/ks-installer-9fd999d6f-rwv42 Started container installer

20m Normal Created pod/ks-installer-9fd999d6f-rwv42 Created container installer

3m36s Normal Pulled pod/ks-installer-9fd999d6f-rwv42 Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/ks-installer:v3.3.2” already present on machine

8m35s Warning BackOff pod/ks-installer-9fd999d6f-rwv42 Back-off restarting failed container installer in pod ks-installer-9fd999d6f-rwv42_kubesphere-system(b3a1d3d3-558e-46a6-b4eb-4c4051dbb414)

SYB

liuzhen

提供一下ks-installer 退出前的日志

SYB

liuzhen

一直卡着在安装，发现pod有问题，后台查了events 日志如下：

LAST SEEN TYPE REASON OBJECT MESSAGE

47m Warning FailedKillPod pod/coredns-9584dbd77-7f4pb error killing pod: failed to “KillPodSandbox” for “781c5630-69f2-4f4f-8cbf-fbded5621d20” with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \“63d9312e8b935aaa98fd97108f0a7526e14a24c1f0978b11431f53ba21a54610\”: cni plugin not initialized"

45m Warning Unhealthy pod/coredns-9584dbd77-mbhdc Readiness probe failed: HTTP probe failed with statuscode: 503

58m Normal Scheduled pod/calico-kube-controllers-57db949bd8-cc6tp Successfully assigned kube-system/calico-kube-controllers-57db949bd8-cc6tp to qc-node-03

58m Normal TaintManagerEviction pod/coredns-9584dbd77-7f4pb Marking for deletion Pod kube-system/coredns-9584dbd77-7f4pb

58m Normal TaintManagerEviction pod/calico-kube-controllers-57db949bd8-wlnq7 Marking for deletion Pod kube-system/calico-kube-controllers-57db949bd8-wlnq7

58m Normal TaintManagerEviction pod/coredns-9584dbd77-lkh59 Marking for deletion Pod kube-system/coredns-9584dbd77-lkh59

58m Normal SuccessfulCreate replicaset/calico-kube-controllers-57db949bd8 Created pod: calico-kube-controllers-57db949bd8-cc6tp

43m Normal Pulled pod/calico-kube-controllers-57db949bd8-cc6tp Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4” already present on machine

57m Normal Created pod/calico-kube-controllers-57db949bd8-cc6tp Created container calico-kube-controllers

57m Normal Started pod/calico-kube-controllers-57db949bd8-cc6tp Started container calico-kube-controllers

56m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-cc6tp Readiness probe failed: initialized to false

47m Warning FailedKillPod pod/coredns-9584dbd77-lkh59 error killing pod: failed to “KillPodSandbox” for “2d6bfba7-087b-4355-942a-c2b0668c48e0” with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \“ca507d22116a3b2627d73543b121ba6ca28cf61cccda82d1d9db77c41880808e\”: cni plugin not initialized"

47m Warning FailedKillPod pod/calico-kube-controllers-57db949bd8-wlnq7 error killing pod: failed to “KillPodSandbox” for “d47e2c73-549c-4aed-a6c9-3e34778c863b” with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for sandbox \“4363f51fcc74e16e504a983d1c960692ad6c275d6c80a6c8600dd68f9f6aa3ed\”: cni plugin not initialized"

56m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-cc6tp Liveness probe failed: initialized to false

57m Normal Scheduled pod/snapshot-controller-0 Successfully assigned kube-system/snapshot-controller-0 to qc-node-02

57m Normal SuccessfulCreate statefulset/snapshot-controller create Pod snapshot-controller-0 in StatefulSet snapshot-controller successful

57m Normal Started pod/snapshot-controller-0 Started container snapshot-controller

57m Normal Created pod/snapshot-controller-0 Created container snapshot-controller

57m Normal Pulled pod/snapshot-controller-0 Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/snapshot-controller:v4.0.0” already present on machine

57m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-cc6tp Liveness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

57m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-cc6tp Readiness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

56m Normal Killing pod/calico-kube-controllers-57db949bd8-cc6tp Container calico-kube-controllers failed liveness probe, will be restarted

38m Warning BackOff pod/calico-kube-controllers-57db949bd8-cc6tp Back-off restarting failed container calico-kube-controllers in pod calico-kube-controllers-57db949bd8-cc6tp_kube-system(21d2732f-069e-41a6-a03a-5abc610abda8)

49m Normal Killing pod/calico-node-8tbmn Stopping container calico-node

49m Normal Pulled pod/calico-node-lnwcl Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

49m Normal Created pod/calico-node-lnwcl Created container upgrade-ipam

49m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-lnwcl

49m Normal Scheduled pod/calico-node-lnwcl Successfully assigned kube-system/calico-node-lnwcl to qc-node-03

49m Normal Started pod/calico-node-lnwcl Started container upgrade-ipam

49m Normal Created pod/calico-node-lnwcl Created container install-cni

49m Normal Started pod/calico-node-lnwcl Started container install-cni

49m Normal Pulled pod/calico-node-lnwcl Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

49m Normal Pulled pod/calico-node-lnwcl Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

49m Normal Created pod/calico-node-lnwcl Created container mount-bpffs

49m Normal Started pod/calico-node-lnwcl Started container mount-bpffs

49m Normal Started pod/calico-node-lnwcl Started container calico-node

49m Normal Pulled pod/calico-node-lnwcl Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

49m Normal Created pod/calico-node-lnwcl Created container calico-node

49m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

49m Normal TaintManagerEviction pod/coredns-9584dbd77-mbhdc Cancelling deletion of Pod kube-system/coredns-9584dbd77-mbhdc

49m Normal TaintManagerEviction pod/calico-kube-controllers-57db949bd8-cc6tp Cancelling deletion of Pod kube-system/calico-kube-controllers-57db949bd8-cc6tp

49m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: 2025-02-13 02:49:52.696 [INFO][215] confd/health.go 180: Number of node(s) with BGP peering established = 1…

48m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: 2025-02-13 02:50:02.704 [INFO][245] confd/health.go 180: Number of node(s) with BGP peering established = 1…

48m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: 2025-02-13 02:50:12.702 [INFO][285] confd/health.go 180: Number of node(s) with BGP peering established = 1…

46m Warning NetworkNotReady pod/snapshot-controller-0 network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

46m Warning NetworkNotReady pod/coredns-9584dbd77-d6h7l network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

46m Warning NetworkNotReady pod/openebs-localpv-provisioner-589cc46f59-x8ncg network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

46m Normal Killing pod/calico-node-2smv9 Stopping container calico-node

46m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: 2025-02-13 02:52:12.701 [INFO][704] confd/health.go 180: Number of node(s) with BGP peering established = 0…

46m Warning Unhealthy pod/calico-node-bjx8r Readiness probe failed: 2025-02-13 02:52:18.294 [INFO][10929] confd/health.go 180: Number of node(s) with BGP peering established = 0…

46m Warning Unhealthy pod/calico-node-2wqvf Readiness probe failed: 2025-02-13 02:52:20.758 [INFO][10836] confd/health.go 180: Number of node(s) with BGP peering established = 0…

46m Normal Scheduled pod/calico-node-9s5vf Successfully assigned kube-system/calico-node-9s5vf to qc-master-01

46m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-9s5vf

46m Warning Unhealthy pod/calico-node-lnwcl Readiness probe failed: 2025-02-13 02:52:22.692 [INFO][742] confd/health.go 180: Number of node(s) with BGP peering established = 0…

46m Normal Started pod/calico-node-9s5vf Started container install-cni

46m Normal Created pod/calico-node-9s5vf Created container install-cni

46m Normal Pulled pod/calico-node-9s5vf Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

46m Normal Created pod/calico-node-9s5vf Created container upgrade-ipam

46m Normal Started pod/calico-node-9s5vf Started container upgrade-ipam

46m Normal Pulled pod/calico-node-9s5vf Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Normal Created pod/calico-node-9s5vf Created container mount-bpffs

46m Normal Started pod/calico-node-9s5vf Started container mount-bpffs

46m Normal Pulled pod/calico-node-9s5vf Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Normal Created pod/calico-node-9s5vf Created container calico-node

46m Normal Started pod/calico-node-9s5vf Started container calico-node

46m Warning Unhealthy pod/calico-node-9s5vf Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

46m Normal Killing pod/calico-node-2wqvf Stopping container calico-node

46m Warning Unhealthy pod/calico-node-bjx8r Readiness probe failed: 2025-02-13 02:52:28.288 [INFO][10974] confd/health.go 180: Number of node(s) with BGP peering established = 0…

46m Warning Unhealthy pod/calico-node-9s5vf Readiness probe failed: 2025-02-13 02:52:32.172 [INFO][219] confd/health.go 180: Number of node(s) with BGP peering established = 2…

46m Normal Created pod/calico-node-dhprg Created container upgrade-ipam

46m Normal Scheduled pod/calico-node-dhprg Successfully assigned kube-system/calico-node-dhprg to qc-node-02

46m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-dhprg

46m Normal Pulled pod/calico-node-dhprg Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

46m Normal Started pod/calico-node-dhprg Started container upgrade-ipam

46m Normal Created pod/calico-node-dhprg Created container install-cni

46m Normal Started pod/calico-node-dhprg Started container install-cni

46m Normal Started pod/calico-node-dhprg Started container mount-bpffs

46m Normal Pulled pod/calico-node-dhprg Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Normal Created pod/calico-node-dhprg Created container mount-bpffs

46m Normal Killing pod/calico-node-bjx8r Stopping container calico-node

46m Normal Created pod/calico-node-dhprg Created container calico-node

46m Normal Started pod/calico-node-dhprg Started container calico-node

46m Normal Pulled pod/calico-node-dhprg Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Warning Unhealthy pod/calico-node-dhprg Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

46m Warning Unhealthy pod/calico-node-9s5vf Readiness probe failed: 2025-02-13 02:52:42.185 [INFO][254] confd/health.go 180: Number of node(s) with BGP peering established = 1…

46m Warning Unhealthy pod/calico-node-dhprg Readiness probe failed: 2025-02-13 02:52:48.861 [INFO][215] confd/health.go 180: Number of node(s) with BGP peering established = 1…

46m Normal TaintManagerEviction pod/coredns-9584dbd77-d6h7l Cancelling deletion of Pod kube-system/coredns-9584dbd77-d6h7l

46m Normal TaintManagerEviction pod/snapshot-controller-0 Cancelling deletion of Pod kube-system/snapshot-controller-0

46m Normal Scheduled pod/calico-node-4b5ls Successfully assigned kube-system/calico-node-4b5ls to qc-node-01

46m Normal Started pod/calico-node-4b5ls Started container upgrade-ipam

46m Normal Created pod/calico-node-4b5ls Created container upgrade-ipam

46m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-4b5ls

46m Normal Pulled pod/calico-node-4b5ls Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

46m Normal Created pod/calico-node-4b5ls Created container install-cni

46m Normal Pulled pod/calico-node-4b5ls Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

46m Warning Unhealthy pod/calico-node-9s5vf Readiness probe failed: 2025-02-13 02:52:52.174 [INFO][296] confd/health.go 180: Number of node(s) with BGP peering established = 2…

46m Normal Started pod/calico-node-4b5ls Started container install-cni

46m Normal Started pod/calico-node-4b5ls Started container mount-bpffs

46m Normal Pulled pod/calico-node-4b5ls Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Normal Created pod/calico-node-4b5ls Created container mount-bpffs

46m Normal Killing pod/calico-node-lnwcl Stopping container calico-node

46m Normal Pulled pod/calico-node-4b5ls Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

46m Normal Created pod/calico-node-4b5ls Created container calico-node

46m Normal Started pod/calico-node-4b5ls Started container calico-node

45m Warning Unhealthy pod/calico-node-4b5ls Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

45m Warning Unhealthy pod/calico-node-dhprg Readiness probe failed: 2025-02-13 02:52:58.932 [INFO][241] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-4b5ls Readiness probe failed: 2025-02-13 02:53:01.783 [INFO][220] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Normal Scheduled pod/calico-node-gfvxn Successfully assigned kube-system/calico-node-gfvxn to qc-node-03

45m Normal SuccessfulCreate daemonset/calico-node Created pod: calico-node-gfvxn

45m Normal Pulled pod/calico-node-gfvxn Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

45m Normal Created pod/calico-node-gfvxn Created container upgrade-ipam

45m Normal TaintManagerEviction pod/openebs-localpv-provisioner-589cc46f59-x8ncg Cancelling deletion of Pod kube-system/openebs-localpv-provisioner-589cc46f59-x8ncg

45m Normal Started pod/calico-node-gfvxn Started container upgrade-ipam

45m Normal Pulled pod/calico-node-gfvxn Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/cni:v3.27.4” already present on machine

45m Normal Created pod/calico-node-gfvxn Created container install-cni

45m Normal Started pod/calico-node-gfvxn Started container install-cni

45m Normal Created pod/calico-node-gfvxn Created container mount-bpffs

45m Normal Pulled pod/calico-node-gfvxn Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

45m Normal Started pod/calico-node-gfvxn Started container mount-bpffs

45m Normal Pulled pod/calico-node-gfvxn Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/node:v3.27.4” already present on machine

45m Normal Created pod/calico-node-gfvxn Created container calico-node

45m Normal Started pod/calico-node-gfvxn Started container calico-node

45m Warning Unhealthy pod/calico-node-gfvxn Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/calico/bird.ctl: connect: connection refused

45m Warning Unhealthy pod/calico-node-dhprg Readiness probe failed: 2025-02-13 02:53:08.852 [INFO][282] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-4b5ls Readiness probe failed: 2025-02-13 02:53:11.778 [INFO][246] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-gfvxn Readiness probe failed: 2025-02-13 02:53:14.204 [INFO][218] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-4b5ls Readiness probe failed: 2025-02-13 02:53:21.768 [INFO][284] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-gfvxn Readiness probe failed: 2025-02-13 02:53:24.206 [INFO][244] confd/health.go 180: Number of node(s) with BGP peering established = 1…

45m Warning Unhealthy pod/calico-node-gfvxn Readiness probe failed: 2025-02-13 02:53:34.206 [INFO][292] confd/health.go 180: Number of node(s) with BGP peering established = 1…

43m Normal SuccessfulCreate replicaset/coredns-9584dbd77 Created pod: coredns-9584dbd77-r2jcf

43m Normal Scheduled pod/coredns-9584dbd77-r2jcf Successfully assigned kube-system/coredns-9584dbd77-r2jcf to qc-master-01

43m Normal Started pod/coredns-9584dbd77-r2jcf Started container coredns

43m Normal Created pod/coredns-9584dbd77-r2jcf Created container coredns

43m Normal Pulled pod/coredns-9584dbd77-r2jcf Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/coredns:1.9.3” already present on machine

36m Normal Scheduled pod/calico-kube-controllers-57db949bd8-g7dml Successfully assigned kube-system/calico-kube-controllers-57db949bd8-g7dml to qc-node-03

36m Normal SuccessfulCreate replicaset/calico-kube-controllers-57db949bd8 Created pod: calico-kube-controllers-57db949bd8-g7dml

35m Normal Pulled pod/calico-kube-controllers-57db949bd8-g7dml Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4” already present on machine

35m Normal Created pod/calico-kube-controllers-57db949bd8-g7dml Created container calico-kube-controllers

35m Normal Started pod/calico-kube-controllers-57db949bd8-g7dml Started container calico-kube-controllers

31m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Readiness probe failed: initialized to false

35m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Liveness probe failed: initialized to false

35m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Readiness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

35m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Liveness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

35m Normal Killing pod/calico-kube-controllers-57db949bd8-g7dml Container calico-kube-controllers failed liveness probe, will be restarted

35m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Readiness probe errored: rpc error: code = NotFound desc = failed to exec in container: failed to load task: no running task found: task 17f8a0b9e14f7bae566853de7a9b3be7921d361bef955942b59a76192cc71deb not found: not found

26m Warning BackOff pod/calico-kube-controllers-57db949bd8-g7dml Back-off restarting failed container calico-kube-controllers in pod calico-kube-controllers-57db949bd8-g7dml_kube-system(1a78cbe7-3211-4e55-abf8-a9100754c573)

17m Normal Pulled pod/calico-kube-controllers-57db949bd8-g7dml Container image “registry.cn-beijing.aliyuncs.com/kubesphereio/kube-controllers:v3.27.4” already present on machine

21m Normal Created pod/calico-kube-controllers-57db949bd8-g7dml Created container calico-kube-controllers

21m Normal Started pod/calico-kube-controllers-57db949bd8-g7dml Started container calico-kube-controllers

21m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Readiness probe failed: initialized to false

22m Normal TaintManagerEviction pod/calico-kube-controllers-57db949bd8-g7dml Cancelling deletion of Pod kube-system/calico-kube-controllers-57db949bd8-g7dml

21m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Liveness probe failed: initialized to false

7m46s Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Readiness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

21m Warning Unhealthy pod/calico-kube-controllers-57db949bd8-g7dml Liveness probe failed: Error initializing datastore: Get “https://10.233.0.1:443/apis/crd.projectcalico.org/v1/clusterinformations/default”: dial tcp 10.233.0.1:443: i/o timeout

2m50s Warning BackOff pod/calico-kube-controllers-57db949bd8-g7dml Back-off restarting failed container calico-kube-controllers in pod calico-kube-controllers-57db949bd8-g7dml_kube-system(1a78cbe7-3211-4e55-abf8-a9100754c573)