ceph一直“未挂载”，求助

xiao7cn

kevendeng

好像已经配置了 ceph.com/rbd 鸭。。

env:

name: PROVISIONER_NAME
value: ceph.com/rbd

kevendeng

xiao7cn
对，这个值已经配置了，我看错了。

那你需要看看对应provisioner pod的日志，如果它已经观察到这个PVC，那它通过Ceph创建Volume应该是出问题了。

xiao7cn

创建 provisioner deployment.yaml 超时了，但又看不到什么日志，有什么办法嘛？

Name: rbd-provisioner
Namespace: ak-prj
CreationTimestamp: Thu, 30 Sep 2021 11:07:56 +0800
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
Selector: app=rbd-provisioner
Replicas: 1 desired | 0 updated | 0 total | 0 available | 1 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: app=rbd-provisioner
Service Account: rbd-provisioner
Containers:
rbd-provisioner:
Image: quay.io/external_storage/rbd-provisioner:latest
Port: <none>
Host Port: <none>
Environment:
PROVISIONER_NAME: ceph.com/rbd
Mounts: <none>
Volumes: <none>
Conditions:
Type Status Reason

Available False MinimumReplicasUnavailable
ReplicaFailure True FailedCreate
Progressing False ProgressDeadlineExceeded
OldReplicaSets: <none>
NewReplicaSet: rbd-provisioner-76f6bc6669 (0/1 replicas created)
Events:
Type Reason Age From Message

Normal ScalingReplicaSet 14m deployment-controller Scaled up replica set rbd-provisioner-76f6bc6669 to 1

kevendeng

xiao7cn
你需要kubectl describe对应的Pod查看具体原因

xiao7cn

谢谢大哥。rbd-provisioner 我已经恢复启来了，原因是namespace找错了。

可现在我的 ceph-pvc 仍然是未挂载，waiting for a volume to be created, either by external provisioner “ceph.com/rbd” or manually created by system administrator

这可愁死我了。。

kevendeng

xiao7cn
kubectl logs检查下rbd-provisioner的日志

xiao7cn

I0930 04:12:06.129675 1 main.go:85] Creating RBD provisioner ceph.com/rbd with identity: ceph.com/rbd
I0930 04:12:06.133484 1 leaderelection.go:185] attempting to acquire leader lease ak-prj/ceph.com-rbd…
E0930 04:12:06.149721 1 event.go:259] Could not construct reference to: ‘&v1.Endpoints{TypeMeta:v1.TypeMeta{Kind:"“, APIVersion:”"}, ObjectMeta:v1.ObjectMeta{Name:“ceph.com-rbd”, GenerateName:"", Namespace:“ak-prj”, SelfLink:"", UID:“f0d42673-f645-4f4d-98f2-16daca07da76”, ResourceVersion:“726595”, Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0×0, ext:63768571926, loc🙁time.Location)(0×1bc94e0)}}, DeletionTimestamp🙁v1.Time)(nil), DeletionGracePeriodSeconds🙁int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{“control-plane.alpha.kubernetes.io/leader”:"{\“holderIdentity\”:\“rbd-provisioner-76f6bc6669-zdxvb_92b4a0d2-21a4-11ec-8f75-8a96dc33b0dd\”,\“leaseDurationSeconds\”:15,\“acquireTime\”:\“2021-09-30T04:12:06Z\”,\“renewTime\”:\“2021-09-30T04:12:06Z\”,\“leaderTransitions\”:0}“}, OwnerReferences:[]v1.OwnerReference(nil), Initializers🙁v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:”"}, Subsets:[]v1.EndpointSubset(nil)}’ due to: ’selfLink was empty, can’t make reference’. Will not report event: ‘Normal’ ‘LeaderElection’ ‘rbd-provisioner-76f6bc6669-zdxvb_92b4a0d2-21a4-11ec-8f75-8a96dc33b0dd became leader’
I0930 04:12:06.149862 1 leaderelection.go:194] successfully acquired lease ak-prj/ceph.com-rbd
I0930 04:12:06.149948 1 controller.go:631] Starting provisioner controller ceph.com/rbd_rbd-provisioner-76f6bc6669-zdxvb_92b4a0d2-21a4-11ec-8f75-8a96dc33b0dd!
I0930 04:12:06.250339 1 controller.go:680] Started provisioner controller ceph.com/rbd_rbd-provisioner-76f6bc6669-zdxvb_92b4a0d2-21a4-11ec-8f75-8a96dc33b0dd!
I0930 04:12:06.251241 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:12:06.261207 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 04:12:34.869208 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:12:34.874994 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 04:12:56.183660 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:12:56.195583 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 04:27:06.162162 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:27:06.171896 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 04:42:06.162312 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:42:06.169737 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 04:57:06.162636 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 04:57:06.170013 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 05:12:06.162662 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 05:12:06.170582 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 05:27:06.163031 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 05:27:06.168707 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 05:42:06.164851 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 05:42:06.170967 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 05:57:06.164866 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 05:57:06.171711 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:12:06.165056 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:12:06.171883 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:19:34.210310 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:19:34.215064 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:19:49.937792 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:19:49.942218 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:31:52.079460 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:31:52.088654 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:34:20.816072 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:34:20.821300 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:34:57.813141 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
E0930 06:34:57.825001 1 controller.go:1004] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: unexpected error getting claim reference: selfLink was empty, can’t make reference
I0930 06:35:15.307998 1 controller.go:987] provision “ak-prj/ceph-pvc” class “ceph-storage-class”: started
2021-09-30T06:35:15.

kevendeng

xiao7cn
原因是你所使用的rbd-provisioner太老旧了，与现在的K8s不兼容

你的K8s版本是什么，如果是v1.21及以下，可参考这个帖子修改：
https://kubesphere.com.cn/forum/d/4676-openebsnfsmonitoring-prometheus/8

否则只能更换provisioner，建议使用ceph-csi

xiao7cn

kevendeng

谢谢大哥及时回复。我是用最新的kubesphere，所以应该是1.21的。
也想用 ceph-chi，但 k8s.gcr.io/sig-storage 的各种镜像都pull不下来了。大哥有办法吗？

kevendeng

xiao7cn
可以把镜像地址改成国内的源，具体哪些源包含你所需要的镜像，就需要你自己查找了，可参考其他帖子：https://kubesphere.com.cn/forum/d/3524-kubesphere-v300-ceph-csi-rbd

或者也可以保留你目前的provisioner，按照上面贴子里的办法修改kube-apiserver的参数

xiao7cn

Client Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.4”, GitCommit:“e87da0bd6e03ec3fea7933c4b5263d151aafd07c”, GitTreeState:“clean”, BuildDate:“2021-02-18T16:12:00Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“20”, GitVersion:“v1.20.4”, GitCommit:“e87da0bd6e03ec3fea7933c4b5263d151aafd07c”, GitTreeState:“clean”, BuildDate:“2021-02-18T16:03:00Z”, GoVersion:“go1.15.8”, Compiler:“gc”, Platform:“linux/amd64”}

kevendeng

xiao7cn
你的K8s版本是v1.20，v1.21以下都可以通过这个方式解决。

xiao7cn

你好，我安装好了csi，还是报错。。

I1008 02:52:12.758589 1 leaderelection.go:248] failed to acquire lease ak-prj/rbd-csi-ceph-com

I1008 02:52:23.495393 1 leaderelection.go:346] lock is held by csi-rbdplugin-provisioner-8674ccd978-x8whp and has not yet expired

I1008 02:52:23.495431 1 leaderelection.go:248] failed to acquire lease ak-prj/rbd-csi-ceph-com

I1008 02:52:30.595614 1 leaderelection.go:346] lock is held by csi-rbdplugin-provisioner-8674ccd978-x8whp and has not yet expired

I1008 02:52:30.595640 1 leaderelection.go:248] failed to acquire lease ak-prj/rbd-csi-ceph-com

I1008 02:52:39.748068 1 leaderelection.go:346] lock is held by csi-rbdplugin-provisioner-8674ccd978-x8whp and has not yet expired

I1008 02:52:39.748107 1 leaderelection.go:248] failed to acquire lease ak-prj/rbd-csi-ceph-com

xiao7cn

I1008 03:36:00.618702 1 leaderelection.go:273] successfully renewed lease ak-prj/rbd-csi-ceph-com

I1008 03:36:03.724459 1 reflector.go:530] sigs.k8s.io/sig-storage-lib-external-provisioner/v6/controller/controller.go:869: Watch close - *v1.PersistentVolume total 0 items received

I1008 03:36:05.662502 1 leaderelection.go:273] successfully renewed lease ak-prj/rbd-csi-ceph-com

I1008 03:36:06.477120 1 controller.go:1332] provision “ak-prj/ceph-pvc” class “csi-rbd-sc”: started

I1008 03:36:06.477926 1 event.go:282] Event(v1.ObjectReference{Kind:“PersistentVolumeClaim”, Namespace:“ak-prj”, Name:“ceph-pvc”, UID:“86819f3e-ddb4-4fc4-aecf-6160fc61c136”, APIVersion:“v1”, ResourceVersion:“1130087”, FieldPath:""}): type: ‘Normal’ reason: ‘Provisioning’ External provisioner is provisioning volume for claim “ak-prj/ceph-pvc”

I1008 03:36:06.485232 1 connection.go:182] GRPC call: /csi.v1.Controller/CreateVolume

I1008 03:36:06.485269 1 connection.go:183] GRPC request: {“capacity_range”:{“required_bytes”:10737418240},“name”:“pvc-86819f3e-ddb4-4fc4-aecf-6160fc61c136”,“parameters”:{“clusterID”:“390a1622-3bdf-45a5-be3b-de494f903ee1”,“csi.storage.k8s.io/pv/name”:“pvc-86819f3e-ddb4-4fc4-aecf-6160fc61c136”,“csi.storage.k8s.io/pvc/name”:“ceph-pvc”,“csi.storage.k8s.io/pvc/namespace”:“ak-prj”,“imageFeatures”:“layering”,“pool”:“kube”},“secrets”:“stripped”,“volume_capabilities”:[{“AccessType”:{“Mount”:{“fs_type”:“xfs”,“mount_flags”:[“discard”]}},“access_mode”:{“mode”:1}}]}

I1008 03:36:06.537840 1 connection.go:185] GRPC response: {}

I1008 03:36:06.538005 1 connection.go:186] GRPC error: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

I1008 03:36:06.538106 1 controller.go:753] CreateVolume failed, supports topology = false, node selected false => may reschedule = false => state = Finished: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

I1008 03:36:06.538435 1 controller.go:1099] Final error received, removing PVC 86819f3e-ddb4-4fc4-aecf-6160fc61c136 from claims in progress

W1008 03:36:06.538509 1 controller.go:958] Retrying syncing claim “86819f3e-ddb4-4fc4-aecf-6160fc61c136”, failure 22

E1008 03:36:06.538613 1 controller.go:981] error syncing claim “86819f3e-ddb4-4fc4-aecf-6160fc61c136”: failed to provision volume with StorageClass “csi-rbd-sc”: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

I1008 03:36:06.538854 1 event.go:282] Event(v1.ObjectReference{Kind:“PersistentVolumeClaim”, Namespace:“ak-prj”, Name:“ceph-pvc”, UID:“86819f3e-ddb4-4fc4-aecf-6160fc61c136”, APIVersion:“v1”, ResourceVersion:“1130087”, FieldPath:""}): type: ‘Warning’ reason: ‘ProvisioningFailed’ failed to provision volume with StorageClass “csi-rbd-sc”: rpc error: code = Internal desc = failed to get connection: connecting failed: rados: ret=-22, Invalid argument

I1008 03:36:10.689551 1 leaderelection.go:273] successfully renewed lease ak-prj/rbd-csi-ceph-com

I1008 03:36:15.728322 1 leaderelection.go:273] successfully renewed lease ak-prj/rbd-csi-ceph-com

xiao7cn

我解决了， config信息写错了，谢谢关注！

wavemomo

[未知用户] 我也遇到了同样的问题，你所指的config是哪个呢