k8s上最小化安装成功后，开启其他组件时minio部署失败

Mmagese · 2020年6月29日

pvc如下所示

[centos@k8s-node1 k8s]$ kubectl get pvc --all-namespaces
NAMESPACE                      NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
kubesphere-monitoring-system   prometheus-k8s-db-prometheus-k8s-0                 Bound    pvc-d47ff171-bfe6-4ee3-b0f8-8d25547a5ec6   20Gi       RWO            openebs-hostpath   70m
kubesphere-monitoring-system   prometheus-k8s-system-db-prometheus-k8s-system-0   Bound    pvc-3c8d528a-7a9f-4b06-b756-f7da4c14ff37   20Gi       RWO            openebs-hostpath   70m
kubesphere-system              minio                                              Bound    pvc-8d03cf25-585d-45c5-be9a-5bb91781d35d   20Gi       RWO            openebs-hostpath   62m
kubesphere-system              openldap-pvc-openldap-0                            Bound    pvc-fe9088b1-3619-4353-a9d0-a76b57afa583   2Gi        RWO            openebs-hostpath   71m
kubesphere-system              redis-pvc                                          Bound    pvc-cf3cbeec-23ed-4e4e-85e0-95dffc55ba26   2Gi        RWO            openebs-hostpath   71m

Mmagese · 2020年6月29日

pod如下所示

[centos@k8s-node1 k8s]$ kubectl get pod --all-namespaces
NAMESPACE                      NAME                                           READY   STATUS    RESTARTS   AGE
ingress-nginx                  nginx-ingress-controller-dfbqz                 1/1     Running   0          4d7h
ingress-nginx                  nginx-ingress-controller-f622k                 1/1     Running   0          4d7h
ingress-nginx                  nginx-ingress-controller-n27zk                 1/1     Running   3          4d6h
kube-system                    coredns-9d85f5447-k4c4b                        1/1     Running   0          4d9h
kube-system                    coredns-9d85f5447-sq4lg                        1/1     Running   0          4d9h
kube-system                    etcd-k8s-node1                                 1/1     Running   0          4d9h
kube-system                    kube-apiserver-k8s-node1                       1/1     Running   0          4d9h
kube-system                    kube-controller-manager-k8s-node1              1/1     Running   1          4d9h
kube-system                    kube-flannel-ds-amd64-8sn7d                    1/1     Running   1          4d8h
kube-system                    kube-flannel-ds-amd64-j5zwb                    1/1     Running   0          4d8h
kube-system                    kube-flannel-ds-amd64-nlfvv                    1/1     Running   1          4d9h
kube-system                    kube-proxy-b4n68                               1/1     Running   0          4d8h
kube-system                    kube-proxy-r79kw                               1/1     Running   0          4d9h
kube-system                    kube-proxy-t94fr                               1/1     Running   0          4d8h
kube-system                    kube-scheduler-k8s-node1                       1/1     Running   1          4d9h
kube-system                    tiller-deploy-86445968c5-xqpv6                 1/1     Running   0          6h57m
kubesphere-controls-system     default-http-backend-5d464dd566-2tq6r          1/1     Running   0          73m
kubesphere-controls-system     kubectl-admin-6c664db975-rmfj7                 1/1     Running   0          72m
kubesphere-monitoring-system   kube-state-metrics-566cdbcb48-pzqjj            4/4     Running   0          73m
kubesphere-monitoring-system   node-exporter-4fhfp                            2/2     Running   0          73m
kubesphere-monitoring-system   node-exporter-rxxrh                            2/2     Running   0          73m
kubesphere-monitoring-system   node-exporter-tzkjc                            2/2     Running   0          73m
kubesphere-monitoring-system   prometheus-k8s-0                               3/3     Running   1          72m
kubesphere-monitoring-system   prometheus-k8s-system-0                        3/3     Running   1          72m
kubesphere-monitoring-system   prometheus-operator-6b97679cfd-jx6c2           1/1     Running   0          73m
kubesphere-system              ks-account-596657f8c6-sh22l                    1/1     Running   2          73m
kubesphere-system              ks-apigateway-78bcdc8ffc-frht7                 1/1     Running   0          73m
kubesphere-system              ks-apiserver-5b548d7c5c-jkgzg                  1/1     Running   0          73m
kubesphere-system              ks-console-78bcf96dbf-hfdfw                    1/1     Running   0          73m
kubesphere-system              ks-controller-manager-696986f8d9-pcttn         1/1     Running   0          73m
kubesphere-system              ks-installer-75b8d89dff-dfrpq                  1/1     Running   0          74m
kubesphere-system              minio-845b7bd867-cg6jf                         1/1     Running   0          65m
kubesphere-system              minio-make-bucket-job-s5n7h                    1/1     Running   2          65m
kubesphere-system              openldap-0                                     1/1     Running   0          73m
kubesphere-system              redis-6fd6c6d6f9-q6vp8                         1/1     Running   0          74m
openebs                        openebs-admission-server-5cf6864fbf-sp9dx      1/1     Running   0          29h
openebs                        openebs-apiserver-bc55cd99b-5qrqr              1/1     Running   6          29h
openebs                        openebs-localpv-provisioner-85ff89dd44-lt2r8   1/1     Running   4          29h
openebs                        openebs-ndm-7cxrh                              1/1     Running   1          29h
openebs                        openebs-ndm-operator-87df44d9-t74kz            1/1     Running   0          29h
openebs                        openebs-ndm-qx82m                              1/1     Running   0          29h
openebs                        openebs-ndm-x8dxk                              1/1     Running   0          29h
openebs                        openebs-provisioner-7f86c6bb64-btxkg           1/1     Running   6          29h
openebs                        openebs-snapshot-operator-54b9c886bf-gpnxf     2/2     Running   5          29h

Mmagese · 2020年6月29日

节点信息如下

[centos@k8s-node1 k8s]$ kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
k8s-node1   Ready    master   4d10h   v1.17.3
k8s-node2   Ready    <none>   4d9h    v1.17.3
k8s-node3   Ready    <none>   4d9h    v1.17.3

rayzhou2017 · 2020年6月30日

显示都是正常的呀，哪儿有问题？

rayzhou2017 · 2020年6月30日

哦，第二个有问题，你按照提示检查一下
“1. check the storage configuration and storage server”,
“2. make sure the DNS address in /etc/resolv.conf is available.”,
“3. execute ‘helm del –purge ks-minio && kubectl delete job -n kubesphere-system ks-minio-make-bucket-job’”,
“4. Restart the installer pod in kubesphere-system namespace”

Cauchy · 2020年6月30日

magese
可以先尝试着看下minio-make-bucket-job-s5n7h那个pod日志，一般是环境dns或者存储问题，也有可能集群节点之间时间不同步。

Mmagese · 2020年6月30日

Cauchy
job日志如下

[centos@k8s-node1 ~]$ kubectl logs minio-make-bucket-job-s5n7h -n kubesphere-system --tail=100
Connecting to Minio server: http://minio:9000
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-nhxof1bbipkq/?location=: dial tcp: i/o timeout.
"Failed attempts: 1"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-mo0x33zvocb6/?location=: dial tcp: i/o timeout.
"Failed attempts: 2"
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-wr0i4qwpswv5/?location=: dial tcp: i/o timeout.
"Failed attempts: 3"

集群节点的时间确认都是一致的。

如何确认是否为DNS问题呢？

Mmagese · 2020年6月30日

Cauchy
/etc/resolv.conf文件配置如下：

; generated by /usr/sbin/dhclient-script
search ap-east-1.compute.internal
nameserver 172.31.0.2

Mmagese · 2020年6月30日

rayzhou2017
1.存储是按照文档安装的openebs，pod都是running，上面有贴。

[centos@k8s-node1 ~]$ kubectl get sc
NAME                         PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
openebs-device               openebs.io/local                                           Delete          WaitForFirstConsumer   false                  4d19h
openebs-hostpath (default)   openebs.io/local                                           Delete          WaitForFirstConsumer   false                  4d19h
openebs-jiva-default         openebs.io/provisioner-iscsi                               Delete          Immediate              false                  4d19h
openebs-snapshot-promoter    volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  4d19h

2.DNS没有进行过修改，楼上贴了配置。

3.helm版本也换过好几次了，也执行过helm del –purge ks-minio重装很多次了，都不行。

Forest-L · 2020年6月30日

magese 172.31.0.2这个ip是通的吗？

Mmagese · 2020年6月30日

Forest-L
没有ping通…… 是要删掉么？

Mmagese · 2020年6月30日

Forest-L
/etc/resolv.conf配置文件该如何配置呢？

Mmagese · 2020年6月30日

我修改/etc/resolv.conf的配置为

nameserver 8.8.8.8

再次进行最小化安装，ks-apigateway直接起不来了。。。

[centos@k8s-node1 k8s]$ kubectl get pod -n kubesphere-system
NAME                                     READY   STATUS             RESTARTS   AGE
ks-account-596657f8c6-t6lwh              1/1     Running            2          6m7s
ks-apigateway-78bcdc8ffc-hlrdg           0/1     CrashLoopBackOff   5          6m8s
ks-apiserver-5b548d7c5c-p7bpv            1/1     Running            0          6m7s
ks-console-78bcf96dbf-xvrnk              1/1     Running            0          6m3s
ks-controller-manager-696986f8d9-4qjkx   1/1     Running            0          6m6s
ks-installer-75b8d89dff-28jz5            1/1     Running            0          7m28s
openldap-0                               1/1     Running            0          6m28s
redis-6fd6c6d6f9-vk6k6                   1/1     Running            0          6m37s

查看ks-apigateway日志如下：

[centos@k8s-node1 k8s]$ kubectl logs ks-apigateway-78bcdc8ffc-hlrdg -n kubesphere-system
2020/06/30 08:17:01 [INFO][cache:0xc00078c050] Started certificate maintenance routine
[DEV NOTICE] Registered directive 'authenticate' before 'jwt'
[DEV NOTICE] Registered directive 'authentication' before 'jwt'
[DEV NOTICE] Registered directive 'swagger' before 'jwt'
Activating privacy features... done.
E0630 08:17:06.752403       1 redis.go:51] unable to reach redis hostdial tcp: i/o timeout
2020/06/30 08:17:06 dial tcp: i/o timeout

头都要炸了，大佬们救救我吧 Forest-L @Cauchy @rayzhou2017

Cauchy · 2020年6月30日

magese ks-account已经起来了，等等ks-apigateway应该就正常了，不想等的话可以直接把那个pod删掉重新拉起。

所有节点的dns都需要有效哦

Mmagese · 2020年6月30日

妈耶，我要哭了。我又把/etc/resolv.conf修改回原来的配置，重启coredns。然鹅ks-account和ks-apigateway无限失败重启。

kubesphere-system              ks-account-596657f8c6-pklvp                    1/1     Running            4          7m55s
kubesphere-system              ks-apigateway-78bcdc8ffc-z49d6                 0/1     CrashLoopBackOff   6          7m57s
kubesphere-system              ks-apiserver-5b548d7c5c-nv2wp                  1/1     Running            0          7m56s
kubesphere-system              ks-console-78bcf96dbf-l9rz9                    1/1     Running            0          7m52s
kubesphere-system              ks-controller-manager-696986f8d9-98xp5         1/1     Running            0          7m55s
kubesphere-system              ks-installer-75b8d89dff-cd4kk                  1/1     Running            0          9m18s
kubesphere-system              openldap-0                                     1/1     Running            0          8m16s
kubesphere-system              redis-6fd6c6d6f9-g6q5b                         1/1     Running            0          8m26s

Mmagese · 2020年6月30日

Cauchy
现在ks-account和ks-apigateway一直在Error、CrashLoopBackOff、Running之间徘徊。

hongming · 2020年6月30日

magese 试试这个办法 https://github.com/kubesphere/ks-installer/issues/33

Mmagese · 2020年6月30日

Cauchy
已发送

Cauchy · 2020年6月30日

magese没收到哦

Mmagese · 2020年6月30日

hongming
试了一下kubectl -n kube-system edit configmap coredns 配置中没有proxy和upstream。安装依旧失败…

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-06-25T06:31:18Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "175"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 8fd545c4-9718-4537-a2fc-ebd6139547a

k8s上最小化安装成功后，开启其他组件时minio部署失败

Mmagese

Mmagese

Mmagese

rayzhou2017K零SK壹S

rayzhou2017K零SK壹S

CauchyK零SK壹S

Mmagese

Mmagese

Mmagese

Forest-LK零S

Mmagese

Mmagese

Mmagese

CauchyK零SK壹S

Mmagese

Mmagese

hongmingK零SK壹S

Mmagese

CauchyK零SK壹S

Mmagese