已经卡了好几天了,心累。
版本:
kubesphere 2.1.1
k8s 1.17.3
helm 2.16.2

最小化安装成功

**************************************************
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://172.31.47.250:30880
Account: admin
Password: P@88w0rd

NOTES:
  1. After logging into the console, please check the
     monitoring status of service components in
     the "Cluster Status". If the service is not
     ready, please wait patiently. You can start
     to use when all components are ready.
  2. Please modify the default password after login.

#####################################################

使用kubectl edit cm -n kubesphere-system ks-installer开启devops、notification、alerting后查看安装日志,显示部署minio超时。

TASK [common : Kubesphere | Deploy minio] **************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/usr/local/bin/helm upgrade --install ks-minio /etc/kubesphere/minio-ha -f /etc/kubesphere/custom-values-minio.yaml --set fullnameOverride=minio --namespace kubesphere-system --wait --timeout 1800\n", "delta": "0:30:25.772021", "end": "2020-06-29 15:52:09.602253", "msg": "non-zero return code", "rc": 1, "start": "2020-06-29 15:21:43.830232", "stderr": "Error: timed out waiting for the condition", "stderr_lines": ["Error: timed out waiting for the condition"], "stdout": "Release \"ks-minio\" does not exist. Installing it now.", "stdout_lines": ["Release \"ks-minio\" does not exist. Installing it now."]}
...ignoring

TASK [common : debug] **********************************************************
ok: [localhost] => {
    "msg": [
        "1. check the storage configuration and storage server", 
        "2. make sure the DNS address in /etc/resolv.conf is available.", 
        "3. execute 'helm del --purge ks-minio && kubectl delete job -n kubesphere-system ks-minio-make-bucket-job'", 
        "4. Restart the installer pod in kubesphere-system namespace"
    ]
}

TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "It is suggested to refer to the above methods for troubleshooting problems ."}

PLAY RECAP *********************************************************************
localhost                  : ok=34   changed=22   unreachable=0    failed=1    skipped=74   rescued=0    ignored=5

按照返回消息执行helm del –purge ks-minio && kubectl delete job -n kubesphere-system ks-minio-make-bucket-job后依然安装失败。

有看到说

但是没有找到地方可以修改kube_network_plugin: flannel

@Feynman @Cauchy 救救孩子吧!

  • Cauchy
    现在ks-account和ks-apigateway一直在Error、CrashLoopBackOff、Running之间徘徊。

pvc如下所示

[centos@k8s-node1 k8s]$ kubectl get pvc --all-namespaces
NAMESPACE                      NAME                                               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
kubesphere-monitoring-system   prometheus-k8s-db-prometheus-k8s-0                 Bound    pvc-d47ff171-bfe6-4ee3-b0f8-8d25547a5ec6   20Gi       RWO            openebs-hostpath   70m
kubesphere-monitoring-system   prometheus-k8s-system-db-prometheus-k8s-system-0   Bound    pvc-3c8d528a-7a9f-4b06-b756-f7da4c14ff37   20Gi       RWO            openebs-hostpath   70m
kubesphere-system              minio                                              Bound    pvc-8d03cf25-585d-45c5-be9a-5bb91781d35d   20Gi       RWO            openebs-hostpath   62m
kubesphere-system              openldap-pvc-openldap-0                            Bound    pvc-fe9088b1-3619-4353-a9d0-a76b57afa583   2Gi        RWO            openebs-hostpath   71m
kubesphere-system              redis-pvc                                          Bound    pvc-cf3cbeec-23ed-4e4e-85e0-95dffc55ba26   2Gi        RWO            openebs-hostpath   71m

pod如下所示

[centos@k8s-node1 k8s]$ kubectl get pod --all-namespaces
NAMESPACE                      NAME                                           READY   STATUS    RESTARTS   AGE
ingress-nginx                  nginx-ingress-controller-dfbqz                 1/1     Running   0          4d7h
ingress-nginx                  nginx-ingress-controller-f622k                 1/1     Running   0          4d7h
ingress-nginx                  nginx-ingress-controller-n27zk                 1/1     Running   3          4d6h
kube-system                    coredns-9d85f5447-k4c4b                        1/1     Running   0          4d9h
kube-system                    coredns-9d85f5447-sq4lg                        1/1     Running   0          4d9h
kube-system                    etcd-k8s-node1                                 1/1     Running   0          4d9h
kube-system                    kube-apiserver-k8s-node1                       1/1     Running   0          4d9h
kube-system                    kube-controller-manager-k8s-node1              1/1     Running   1          4d9h
kube-system                    kube-flannel-ds-amd64-8sn7d                    1/1     Running   1          4d8h
kube-system                    kube-flannel-ds-amd64-j5zwb                    1/1     Running   0          4d8h
kube-system                    kube-flannel-ds-amd64-nlfvv                    1/1     Running   1          4d9h
kube-system                    kube-proxy-b4n68                               1/1     Running   0          4d8h
kube-system                    kube-proxy-r79kw                               1/1     Running   0          4d9h
kube-system                    kube-proxy-t94fr                               1/1     Running   0          4d8h
kube-system                    kube-scheduler-k8s-node1                       1/1     Running   1          4d9h
kube-system                    tiller-deploy-86445968c5-xqpv6                 1/1     Running   0          6h57m
kubesphere-controls-system     default-http-backend-5d464dd566-2tq6r          1/1     Running   0          73m
kubesphere-controls-system     kubectl-admin-6c664db975-rmfj7                 1/1     Running   0          72m
kubesphere-monitoring-system   kube-state-metrics-566cdbcb48-pzqjj            4/4     Running   0          73m
kubesphere-monitoring-system   node-exporter-4fhfp                            2/2     Running   0          73m
kubesphere-monitoring-system   node-exporter-rxxrh                            2/2     Running   0          73m
kubesphere-monitoring-system   node-exporter-tzkjc                            2/2     Running   0          73m
kubesphere-monitoring-system   prometheus-k8s-0                               3/3     Running   1          72m
kubesphere-monitoring-system   prometheus-k8s-system-0                        3/3     Running   1          72m
kubesphere-monitoring-system   prometheus-operator-6b97679cfd-jx6c2           1/1     Running   0          73m
kubesphere-system              ks-account-596657f8c6-sh22l                    1/1     Running   2          73m
kubesphere-system              ks-apigateway-78bcdc8ffc-frht7                 1/1     Running   0          73m
kubesphere-system              ks-apiserver-5b548d7c5c-jkgzg                  1/1     Running   0          73m
kubesphere-system              ks-console-78bcf96dbf-hfdfw                    1/1     Running   0          73m
kubesphere-system              ks-controller-manager-696986f8d9-pcttn         1/1     Running   0          73m
kubesphere-system              ks-installer-75b8d89dff-dfrpq                  1/1     Running   0          74m
kubesphere-system              minio-845b7bd867-cg6jf                         1/1     Running   0          65m
kubesphere-system              minio-make-bucket-job-s5n7h                    1/1     Running   2          65m
kubesphere-system              openldap-0                                     1/1     Running   0          73m
kubesphere-system              redis-6fd6c6d6f9-q6vp8                         1/1     Running   0          74m
openebs                        openebs-admission-server-5cf6864fbf-sp9dx      1/1     Running   0          29h
openebs                        openebs-apiserver-bc55cd99b-5qrqr              1/1     Running   6          29h
openebs                        openebs-localpv-provisioner-85ff89dd44-lt2r8   1/1     Running   4          29h
openebs                        openebs-ndm-7cxrh                              1/1     Running   1          29h
openebs                        openebs-ndm-operator-87df44d9-t74kz            1/1     Running   0          29h
openebs                        openebs-ndm-qx82m                              1/1     Running   0          29h
openebs                        openebs-ndm-x8dxk                              1/1     Running   0          29h
openebs                        openebs-provisioner-7f86c6bb64-btxkg           1/1     Running   6          29h
openebs                        openebs-snapshot-operator-54b9c886bf-gpnxf     2/2     Running   5          29h

节点信息如下

[centos@k8s-node1 k8s]$ kubectl get nodes
NAME        STATUS   ROLES    AGE     VERSION
k8s-node1   Ready    master   4d10h   v1.17.3
k8s-node2   Ready    <none>   4d9h    v1.17.3
k8s-node3   Ready    <none>   4d9h    v1.17.3

    显示都是正常的呀,哪儿有问题?

    哦,第二个有问题,你按照提示检查一下
    “1. check the storage configuration and storage server”,
    “2. make sure the DNS address in /etc/resolv.conf is available.”,
    “3. execute ‘helm del –purge ks-minio && kubectl delete job -n kubesphere-system ks-minio-make-bucket-job’”,
    “4. Restart the installer pod in kubesphere-system namespace”

      magese
      可以先尝试着看下minio-make-bucket-job-s5n7h那个pod日志,一般是环境dns或者存储问题,也有可能集群节点之间时间不同步。

        Cauchy
        job日志如下

        [centos@k8s-node1 ~]$ kubectl logs minio-make-bucket-job-s5n7h -n kubesphere-system --tail=100
        Connecting to Minio server: http://minio:9000
        mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-nhxof1bbipkq/?location=: dial tcp: i/o timeout.
        "Failed attempts: 1"
        mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-mo0x33zvocb6/?location=: dial tcp: i/o timeout.
        "Failed attempts: 2"
        mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-wr0i4qwpswv5/?location=: dial tcp: i/o timeout.
        "Failed attempts: 3"

        集群节点的时间确认都是一致的。

        如何确认是否为DNS问题呢?

        Cauchy
        /etc/resolv.conf文件配置如下:

        ; generated by /usr/sbin/dhclient-script
        search ap-east-1.compute.internal
        nameserver 172.31.0.2

          rayzhou2017
          1.存储是按照文档安装的openebs,pod都是running,上面有贴。

          [centos@k8s-node1 ~]$ kubectl get sc
          NAME                         PROVISIONER                                                RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
          openebs-device               openebs.io/local                                           Delete          WaitForFirstConsumer   false                  4d19h
          openebs-hostpath (default)   openebs.io/local                                           Delete          WaitForFirstConsumer   false                  4d19h
          openebs-jiva-default         openebs.io/provisioner-iscsi                               Delete          Immediate              false                  4d19h
          openebs-snapshot-promoter    volumesnapshot.external-storage.k8s.io/snapshot-promoter   Delete          Immediate              false                  4d19h

          2.DNS没有进行过修改,楼上贴了配置。

          3.helm版本也换过好几次了,也执行过helm del –purge ks-minio重装很多次了,都不行。

          我修改/etc/resolv.conf的配置为

          nameserver 8.8.8.8

          再次进行最小化安装,ks-apigateway直接起不来了。。。

          [centos@k8s-node1 k8s]$ kubectl get pod -n kubesphere-system
          NAME                                     READY   STATUS             RESTARTS   AGE
          ks-account-596657f8c6-t6lwh              1/1     Running            2          6m7s
          ks-apigateway-78bcdc8ffc-hlrdg           0/1     CrashLoopBackOff   5          6m8s
          ks-apiserver-5b548d7c5c-p7bpv            1/1     Running            0          6m7s
          ks-console-78bcf96dbf-xvrnk              1/1     Running            0          6m3s
          ks-controller-manager-696986f8d9-4qjkx   1/1     Running            0          6m6s
          ks-installer-75b8d89dff-28jz5            1/1     Running            0          7m28s
          openldap-0                               1/1     Running            0          6m28s
          redis-6fd6c6d6f9-vk6k6                   1/1     Running            0          6m37s

          查看ks-apigateway日志如下:

          [centos@k8s-node1 k8s]$ kubectl logs ks-apigateway-78bcdc8ffc-hlrdg -n kubesphere-system
          2020/06/30 08:17:01 [INFO][cache:0xc00078c050] Started certificate maintenance routine
          [DEV NOTICE] Registered directive 'authenticate' before 'jwt'
          [DEV NOTICE] Registered directive 'authentication' before 'jwt'
          [DEV NOTICE] Registered directive 'swagger' before 'jwt'
          Activating privacy features... done.
          E0630 08:17:06.752403       1 redis.go:51] unable to reach redis hostdial tcp: i/o timeout
          2020/06/30 08:17:06 dial tcp: i/o timeout

          头都要炸了,大佬们救救我吧 Forest-L @Cauchy @rayzhou2017

            magese ks-account已经起来了,等等ks-apigateway应该就正常了,不想等的话可以直接把那个pod删掉重新拉起。

            所有节点的dns都需要有效哦

              妈耶,我要哭了。我又把/etc/resolv.conf修改回原来的配置,重启coredns。然鹅ks-account和ks-apigateway无限失败重启。

              kubesphere-system              ks-account-596657f8c6-pklvp                    1/1     Running            4          7m55s
              kubesphere-system              ks-apigateway-78bcdc8ffc-z49d6                 0/1     CrashLoopBackOff   6          7m57s
              kubesphere-system              ks-apiserver-5b548d7c5c-nv2wp                  1/1     Running            0          7m56s
              kubesphere-system              ks-console-78bcf96dbf-l9rz9                    1/1     Running            0          7m52s
              kubesphere-system              ks-controller-manager-696986f8d9-98xp5         1/1     Running            0          7m55s
              kubesphere-system              ks-installer-75b8d89dff-cd4kk                  1/1     Running            0          9m18s
              kubesphere-system              openldap-0                                     1/1     Running            0          8m16s
              kubesphere-system              redis-6fd6c6d6f9-g6q5b                         1/1     Running            0          8m26s

              Cauchy
              现在ks-account和ks-apigateway一直在Error、CrashLoopBackOff、Running之间徘徊。

                商业产品与合作咨询