求解容器运行Kubelet导致pvc创建的目录/var/openebs/local/pvc-x不存在，而存在再/rootfs/...中

mrz001

创建部署问题时，请参考下面模板，你提供的信息越多，越容易及时获得解答。
你只花一分钟创建的问题，不能指望别人花上半个小时给你解答。
发帖前请点击发表主题右边的预览(👀) 按钮，确保帖子格式正确。

操作系统信息
例如：虚拟机/物理机，Fedora(coreos)

Kubernetes版本信息
v1.21.7

容器运行时
使用 docker版本

[root@yxclean02-jg6fcdjppmoj-master-0 ~]# docker -v

Docker version 20.10.9, build c2ea9bc

KubeSphere版本信息
v3.0.0。离线安装已有K8s安装。

问题是什么
报错日志是什么，最好有截图。

[root@yxclean02-jg6fcdjppmoj-master-0 ~]# kubectl get pvc -A

NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE

kubesphere-logging-system data-elasticsearch-logging-data-0 Bound pvc-24133ddb-61c3-4446-87e1-1e76827c6686 20Gi RWO openebs-hostpath 36m

kubesphere-logging-system data-elasticsearch-logging-discovery-0 Bound pvc-87ae59f3-6d68-4569-badf-60601f90894d 4Gi RWO openebs-hostpath 36m

kubesphere-monitoring-system prometheus-k8s-db-prometheus-k8s-0 Bound pvc-98028a72-6c4c-468d-87a7-11f728809789 20Gi RWO openebs-hostpath 33m

kubesphere-system etcd-pvc Bound pvc-bc49005a-9cdd-4e3e-af67-d10e0e87cba6 20Gi RWO openebs-hostpath 36m

kubesphere-system mysql-pvc Bound pvc-138614b3-02e2-4b3a-ad21-45860c2bfb2c 20Gi RWO openebs-hostpath 37m

kubesphere-system openldap-pvc-openldap-0 Bound pvc-84b059e0-fd51-439a-96a2-7adb7a6d715d 2Gi RWO openebs-hostpath 37m

kubesphere-system redis-pvc Bound pvc-bafa6e09-eebd-4328-88bc-d6677f2a0511 2Gi RWO openebs-hostpath 37m

[root@yxclean02-jg6fcdjppmoj-master-0 ~]# kubectl get pods -n kubesphere-system

NAME READY STATUS RESTARTS AGE

etcd-8594bbf6cd-7h6ss 0/1 ContainerCreating 0 37m

ks-apiserver-76748bd7c8-fdw6j 0/1 CrashLoopBackOff 11 32m

ks-console-85b69475bf-dkggg 1/1 Running 9 35m

ks-controller-manager-7597df84f5-2pfss 0/1 CrashLoopBackOff 11 32m

ks-installer-8677b8cb9f-7cs8c 1/1 Running 0 38m

mysql-bfd68d977-qg78k 0/1 Init:0/1 0 37m

openldap-0 0/1 ContainerCreating 0 37m

redis-5d48d788db-pcds9 0/1 ContainerCreating 0 37m

[root@yxclean02-jg6fcdjppmoj-master-0 ~]# kubectl describe pods redis-5d48d788db-pcds9 -n kubesphere-system

Name: redis-5d48d788db-pcds9

Namespace: kubesphere-system

Priority: 0

Node: yxclean02-jg6fcdjppmoj-master-0/192.168.18.118

Start Time: Mon, 13 Dec 2021 08:39:23 +0000

Labels: app=redis

            pod-template-hash=5d48d788db

            tier=database

            version=redis-4.0

Annotations: <none>

Status: Pending

IP:

IPs: <none>

Controlled By: ReplicaSet/redis-5d48d788db

Containers:

redis:

Container ID:   

Image:          10.200.43.153/library/redis:5.0.5-alpine

Image ID:       

Port:           6379/TCP

Host Port:      0/TCP

State:          Waiting

  Reason:       ContainerCreating

Ready:          False

Restart Count:  0

Limits:

  cpu:     1

  memory:  1000Mi

Requests:

  cpu:        20m

  memory:     100Mi

Environment:  <none>

Mounts:

  /data from redis-pvc (rw,path="redis-data")

  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hkmjd (ro)

Conditions:

Type Status

Initialized True

Ready False

ContainersReady False

PodScheduled True

Volumes:

redis-pvc:

Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

ClaimName:  redis-pvc

ReadOnly:   false

kube-api-access-hkmjd:

Type:                    Projected (a volume that contains injected data from multiple sources)

TokenExpirationSeconds:  3607

ConfigMapName:           kube-root-ca.crt

ConfigMapOptional:       <nil>

DownwardAPI:             true

QoS Class: Burstable

Node-Selectors: <none>

Tolerations: CriticalAddonsOnly op=Exists

                         node-role.kubernetes.io/master:NoSchedule

                         node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                         node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

Type Reason Age From Message

Normal Scheduled 37m default-scheduler Successfully assigned kubesphere-system/redis-5d48d788db-pcds9 to yxclean02-jg6fcdjppmoj-master-0

Warning FailedMount 15m (x2 over 30m) kubelet Unable to attach or mount volumes: unmounted volumes=[redis-pvc], unattached volumes=[kube-api-access-hkmjd redis-pvc]: timed out waiting for the condition

Warning FailedMount 6m52s (x23 over 37m) kubelet MountVolume.NewMounter initialization failed for volume “pvc-bafa6e09-eebd-4328-88bc-d6677f2a0511” : path “/var/openebs/local/pvc-bafa6e09-eebd-4328-88bc-d6677f2a0511” does not exist

Warning FailedMount 95s (x13 over 35m) kubelet Unable to attach or mount volumes: unmounted volumes=[redis-pvc], unattached volumes=[redis-pvc kube-api-access-hkmjd]: timed out waiting for the condition

[root@yxclean02-jg6fcdjppmoj-master-0 ~]#

mrz001

复述一下问题：由于kubelet运行再容器内部

导致Kubelet无法访问创建的pvc宿主机/var/openebs/local/pvc-xxx目录，Kubelet容器内是这个路径/roofs/var/openebs/local/pvc-xxx

求如何解决此问题，是改Kubelet的映射路径？还是改redis的挂载路径？还是修改openebs？

ruiyaoOps

kubernetes 怎么安装的？

mrz001

ruiyaoOps podman安装的

mrz001

ruiyaoOps

[root@yxclean02-jg6fcdjppmoj-master-0 ~]# cat /etc/systemd/system/kubelet.service

[Unit]

Description=Kubelet via Hyperkube (System Container)

Wants=rpc-statd.service

[Service]

EnvironmentFile=/etc/sysconfig/heat-params

EnvironmentFile=/etc/kubernetes/config

EnvironmentFile=/etc/kubernetes/kubelet

ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d

ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests

ExecStartPre=/bin/mkdir -p /var/lib/calico

ExecStartPre=/bin/mkdir -p /var/lib/containerd

ExecStartPre=/bin/mkdir -p /var/lib/docker

ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins

ExecStartPre=/bin/mkdir -p /opt/cni/bin

ExecStartPre=-/usr/bin/podman rm kubelet

ExecStartPre=-/usr/bin/podman pull ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG}

ExecStart=/bin/bash -c '/usr/bin/podman run –name kubelet \

--privileged \\

--pid host \\

--network host \\

--entrypoint /hyperkube \\

--volume /:/rootfs:ro \\

--volume /etc/cni/net.d:/etc/cni/net.d:ro,z \\

--volume /etc/kubernetes:/etc/kubernetes:ro,z \\

--volume /usr/lib/os-release:/usr/lib/os-release:ro \\

--volume /etc/ssl/certs:/etc/ssl/certs:ro \\

--volume /lib/modules:/lib/modules:ro \\

--volume /run:/run \\

--volume /dev:/dev \\

--volume /sys/fs/cgroup:/sys/fs/cgroup:rw \\

--volume /sys/fs/cgroup/system.slice:/sys/fs/cgroup/system.slice \\

--volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \\

--volume /var/lib/calico:/var/lib/calico \\

--volume /var/lib/docker:/var/lib/docker \\

--volume /var/lib/containerd:/var/lib/containerd \\

--volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \\

--volume /var/log:/var/log \\

--volume /var/run:/var/run \\

--volume /var/run/lock:/var/run/lock:z \\

--volume /opt/cni/bin:/opt/cni/bin:z \\

--volume /etc/machine-id:/etc/machine-id \\

${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG} \\

kubelet \\

$KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBELET_ARGS'

ExecStop=-/usr/bin/podman stop kubelet

Delegate=yes

Restart=always

RestartSec=10

TimeoutStartSec=10min

[Install]

WantedBy=multi-user.target

[root@yxclean02-jg6fcdjppmoj-master-0 ~]#

mrz001

解决了，小改动解决大问题，再kubelet启动脚本中创建/var/openebs/local并且挂载到kubelet容器内部

shiqinfeng1

mrz001

请问具体怎么改啊，我是kubespary启动的k8s

mrz001

shiqinfeng1 我是再启动脚本中加上
–volume /var/openebs/local/:/var/openebs/local/

[root@yxall02-b5tgjsnlid6l-master-0 core]# cat /etc/systemd/system/kubelet.service 
[Unit]
Description=Kubelet via Hyperkube (System Container)
Wants=rpc-statd.service

[Service]
EnvironmentFile=/etc/sysconfig/heat-params
EnvironmentFile=/etc/kubernetes/config
EnvironmentFile=/etc/kubernetes/kubelet
ExecStartPre=/bin/mkdir -p /etc/kubernetes/cni/net.d
ExecStartPre=/bin/mkdir -p /etc/kubernetes/manifests
ExecStartPre=/bin/mkdir -p /var/lib/calico
ExecStartPre=/bin/mkdir -p /var/lib/containerd
ExecStartPre=/bin/mkdir -p /var/lib/docker
ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins
ExecStartPre=/bin/mkdir -p /var/openebs/local
ExecStartPre=/bin/mkdir -p /opt/cni/bin
ExecStartPre=-/usr/bin/podman rm kubelet
ExecStartPre=-/usr/bin/podman pull ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG}
ExecStart=/bin/bash -c '/usr/bin/podman run --name kubelet \
    --privileged \
    --pid host \
    --network host \
    --entrypoint /hyperkube \
    --volume /:/rootfs:ro \
    --volume /etc/cni/net.d:/etc/cni/net.d:ro,z \
    --volume /etc/kubernetes:/etc/kubernetes:ro,z \
    --volume /usr/lib/os-release:/usr/lib/os-release:ro \
    --volume /etc/ssl/certs:/etc/ssl/certs:ro \
    --volume /lib/modules:/lib/modules:ro \
    --volume /run:/run \
    --volume /dev:/dev \
    --volume /sys/fs/cgroup:/sys/fs/cgroup:rw \
    --volume /sys/fs/cgroup/system.slice:/sys/fs/cgroup/system.slice \
    --volume /etc/pki/tls/certs:/usr/share/ca-certificates:ro \
    --volume /var/lib/calico:/var/lib/calico \
    --volume /var/lib/docker:/var/lib/docker \
    --volume /var/lib/containerd:/var/lib/containerd \
    --volume /var/lib/kubelet:/var/lib/kubelet:rshared,z \
    --volume /var/log:/var/log \
    --volume /var/openebs/local/:/var/openebs/local/ \
    --volume /var/run:/var/run \
    --volume /var/run/lock:/var/run/lock:z \
    --volume /opt/cni/bin:/opt/cni/bin:z \
    --volume /etc/machine-id:/etc/machine-id \
    ${CONTAINER_INFRA_PREFIX:-${HYPERKUBE_PREFIX}}hyperkube:${KUBE_TAG} \
    kubelet \
    $KUBE_LOGTOSTDERR $KUBE_LOG_LEVEL $KUBELET_API_SERVER $KUBELET_ADDRESS $KUBELET_PORT $KUBELET_HOSTNAME $KUBELET_ARGS'
ExecStop=-/usr/bin/podman stop kubelet
Delegate=yes
Restart=always
RestartSec=10
TimeoutStartSec=10min
[Install]
WantedBy=multi-user.target