多master集群部署2.1.1版本以后有些组件无法运行,ks-installer日志无报错

NAME                                         READY   STATUS             RESTARTS   AGE
pod/ks-account-845d86f776-qbd6f              0/1     Init:0/2           0          8m47s
pod/ks-account-845d86f776-w6d2n              0/1     Init:0/2           0          8m47s
pod/ks-account-845d86f776-x5g4b              0/1     Init:0/2           0          8m47s
pod/ks-apigateway-d6f89b96f-b5g2h            0/1     CrashLoopBackOff   6          8m50s
pod/ks-apigateway-d6f89b96f-dfmwt            0/1     CrashLoopBackOff   6          8m50s
pod/ks-apigateway-d6f89b96f-rck4k            0/1     CrashLoopBackOff   6          8m50s
pod/ks-apiserver-5dff5c594d-5njm9            1/1     Running            0          8m48s
pod/ks-apiserver-5dff5c594d-hpddr            1/1     Running            0          8m48s
pod/ks-apiserver-5dff5c594d-wfnnp            1/1     Running            0          8m48s
pod/ks-console-5c666c644-2hl7x               1/1     Running            0          8m40s
pod/ks-console-5c666c644-5zpjx               1/1     Running            0          8m40s
pod/ks-console-5c666c644-cd7pm               1/1     Running            0          8m40s
pod/ks-controller-manager-779d9c6bbf-hbjsv   1/1     Running            0          8m45s
pod/ks-controller-manager-779d9c6bbf-lbpv9   1/1     Running            0          8m45s
pod/ks-controller-manager-779d9c6bbf-zfq5n   1/1     Running            0          8m45s
pod/ks-installer-59fb465b7-qb79d             1/1     Running            0          11m
pod/openldap-0                               1/1     Running            0          9m29s
pod/openldap-1                               1/1     Running            0          8m54s


NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
service/ks-account      ClusterIP   10.107.1.107     <none>        80/TCP         8m47s
service/ks-apigateway   ClusterIP   10.96.107.153    <none>        80/TCP         8m50s
service/ks-apiserver    ClusterIP   10.108.177.151   <none>        80/TCP         8m48s
service/ks-console      NodePort    10.100.10.181    <none>        80:30880/TCP   8m37s
service/openldap        ClusterIP   None             <none>        389/TCP        9m29s


NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-account              0/3     3            0           8m47s
deployment.apps/ks-apigateway           0/3     3            0           8m50s
deployment.apps/ks-apiserver            3/3     3            3           8m48s
deployment.apps/ks-console              3/3     3            3           8m40s
deployment.apps/ks-controller-manager   3/3     3            3           8m45s
deployment.apps/ks-installer            1/1     1            1           11m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ks-account-845d86f776              3         3         0       8m47s
replicaset.apps/ks-apigateway-d6f89b96f            3         3         0       8m50s
replicaset.apps/ks-apiserver-5dff5c594d            3         3         3       8m48s
replicaset.apps/ks-console-5c666c644               3         3         3       8m40s
replicaset.apps/ks-controller-manager-779d9c6bbf   3         3         3       8m45s
replicaset.apps/ks-installer-59fb465b7             1         1         1       11m

NAME                        READY   AGE
statefulset.apps/openldap   2/2     9m29s

api-gateway的日志如下:

[DEV NOTICE] Registered directive 'authenticate' before 'jwt'
[DEV NOTICE] Registered directive 'authentication' before 'jwt'
[DEV NOTICE] Registered directive 'swagger' before 'jwt'
Activating privacy features... done.
2020/02/17 12:54:15 [INFO][cache:0xc00044c7d0] Started certificate maintenance routine
E0217 12:54:15.411202       1 redis.go:51] unable to reach redis hostdial tcp: lookup redis.kubesphere-system.svc on 10.96.0.10:53: no such host
2020/02/17 12:54:15 dial tcp: lookup redis.kubesphere-system.svc on 10.96.0.10:53: no such host

所以为什么redis服务没有运行呢?请大佬帮忙排查下,谢谢

看下ks-installer的日志

2020-02-17T12:46:07Z INFO     : shell-operator v1.0.0-beta.5
2020-02-17T12:46:07Z INFO     : HTTP SERVER Listening on 0.0.0.0:9115
2020-02-17T12:46:07Z INFO     : Use temporary dir: /tmp/shell-operator
2020-02-17T12:46:07Z INFO     : Initialize hooks manager ...
2020-02-17T12:46:07Z INFO     : Search and load hooks ...
2020-02-17T12:46:07Z INFO     : Load hook config from '/hooks/kubesphere/installRunner.py'
2020-02-17T12:46:07Z INFO     : Initializing schedule manager ...
2020-02-17T12:46:07Z INFO     : KUBE Init Kubernetes client
2020-02-17T12:46:07Z INFO     : KUBE-INIT Kubernetes client is configured successfully
2020-02-17T12:46:07Z INFO     : MAIN: run main loop
2020-02-17T12:46:07Z INFO     : MAIN: add onStartup tasks
2020-02-17T12:46:07Z INFO     : QUEUE add all HookRun@OnStartup
2020-02-17T12:46:07Z INFO     : Running schedule manager ...
2020-02-17T12:46:07Z INFO     : MSTOR Create new metric shell_operator_live_ticks
2020-02-17T12:46:07Z INFO     : MSTOR Create new metric shell_operator_tasks_queue_length
2020-02-17T12:46:07Z INFO     : GVR for kind 'ConfigMap' is /v1, Resource=configmaps
2020-02-17T12:46:07Z INFO     : EVENT Kube event 'c0311204-4add-4738-b61c-04951051388c'
2020-02-17T12:46:07Z INFO     : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2020-02-17T12:46:10Z INFO     : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2020-02-17T12:46:10Z INFO     : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [preinstall : check k8s version] ******************************************
changed: [localhost]

TASK [preinstall : init k8s version] *******************************************
ok: [localhost]

TASK [preinstall : Stop if kuernetes version is nonsupport] ********************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check helm status] ******************************************
changed: [localhost]

TASK [preinstall : Stop if Helm is not available] ******************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check storage class] ****************************************
changed: [localhost]

TASK [preinstall : Stop if StorageClass was not found] *************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check default storage class] ********************************
changed: [localhost]

TASK [preinstall : Stop if defaultStorageClass was not found] ******************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=4    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [common : Kubesphere | Check kube-node-lease namespace] *******************
changed: [localhost]

TASK [common : KubeSphere | Get system namespaces] *****************************
ok: [localhost]

TASK [common : set_fact] *******************************************************
ok: [localhost]

TASK [common : debug] **********************************************************
ok: [localhost] => {
    "msg": [
        "kubesphere-system",
        "kubesphere-controls-system",
        "kubesphere-monitoring-system",
        "kube-node-lease",
        "kubesphere-logging-system"
    ]
}

TASK [common : KubeSphere | Create kubesphere namespace] ***********************
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Labeling system-workspace] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Create ImagePullSecrets] ***************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Getting kubernetes master num] *********************
changed: [localhost]

TASK [common : KubeSphere | Setting master num] ********************************
ok: [localhost]

TASK [common : Kubesphere | Getting common component installation files] *******
changed: [localhost] => (item=common)
changed: [localhost] => (item=ks-crds)

TASK [common : KubeSphere | Create KubeSphere crds] ****************************
changed: [localhost]

TASK [common : Kubesphere | Checking openpitrix common component] **************
changed: [localhost]

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item={u'ks': u'mysql-pvc', u'op': u'openpitrix-db'})
skipping: [localhost] => (item={u'ks': u'etcd-pvc', u'op': u'openpitrix-etcd'})

TASK [common : Getting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
changed: [localhost] => (item={u'path': u'etcd', u'file': u'etcd.yaml'})
changed: [localhost] => (item={u'name': u'mysql', u'file': u'mysql.yaml'})
changed: [localhost] => (item={u'path': u'redis', u'file': u'redis.yaml'})

TASK [common : Kubesphere | Creating mysql sercet] *****************************
changed: [localhost]

TASK [common : Kubesphere | Deploying etcd and mysql] **************************
skipping: [localhost] => (item=etcd.yaml)
skipping: [localhost] => (item=mysql.yaml)

TASK [common : Kubesphere | Getting minio installation files] ******************
skipping: [localhost] => (item=minio-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-minio', u'file': u'custom-values-minio.yaml'})

TASK [common : Kubesphere | Check minio] ***************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy minio] **************************************
skipping: [localhost]

TASK [common : debug] **********************************************************
skipping: [localhost]

TASK [common : fail] ***********************************************************
skipping: [localhost]
TASK [common : Kubesphere | create minio config directory] *********************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
skipping: [localhost] => (item={u'path': u'/root/.config/rclone', u'file': u'rclone.conf'})

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item=helm)
skipping: [localhost] => (item=vmbased)

TASK [common : Kubesphere | Check ha-redis] ************************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis installation files] ******************
skipping: [localhost] => (item=redis-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-redis', u'file': u'custom-values-redis.yaml'})

TASK [common : Kubesphere | Check old redis status] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old redis svc] *******************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Creating redis migration script] *******************
skipping: [localhost] => (item={u'path': u'/etc/kubesphere', u'file': u'redisMigrate.py'})

TASK [common : Kubesphere | Check redis-ha status] *****************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old redis] *********************************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : Kubesphere | Getting openldap installation files] ***************
skipping: [localhost] => (item=openldap-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-openldap', u'file': u'custom-values-openldap.yaml'})

TASK [common : Kubesphere | Check old openldap status] *************************
skipping: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old openldap svc] ****************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap] ************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy openldap] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Load old openldap data] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap-ha status] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get openldap-ha pod list] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get old openldap data] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Migrating openldap data] ***************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old openldap] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Restart openldap] **********************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Check ha-redis] ************************************
changed: [localhost]

TASK [common : Kubesphere | Getting redis installation files] ******************
skipping: [localhost] => (item=redis-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-redis', u'file': u'custom-values-redis.yaml'})

TASK [common : Kubesphere | Check old redis status] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old redis svc] *******************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Creating redis migration script] *******************
skipping: [localhost] => (item={u'path': u'/etc/kubesphere', u'file': u'redisMigrate.py'})

TASK [common : Kubesphere | Check redis-ha status] *****************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old redis] *********************************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : Kubesphere | Getting openldap installation files] ***************
changed: [localhost] => (item=openldap-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
changed: [localhost] => (item={u'name': u'custom-values-openldap', u'file': u'custom-values-openldap.yaml'})

TASK [common : Kubesphere | Check old openldap status] *************************
changed: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old openldap svc] ****************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap] ************************************
changed: [localhost]

TASK [common : Kubesphere | Deploy openldap] ***********************************
changed: [localhost]

TASK [common : Kubesphere | Load old openldap data] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap-ha status] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get openldap-ha pod list] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get old openldap data] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Migrating openldap data] ***************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old openldap] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Restart openldap] **********************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Getting minio installation files] ******************
skipping: [localhost] => (item=minio-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-minio', u'file': u'custom-values-minio.yaml'})

TASK [common : Kubesphere | Check minio] ***************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy minio] **************************************
skipping: [localhost]

TASK [common : debug] **********************************************************
skipping: [localhost]

TASK [common : fail] ***********************************************************
skipping: [localhost]

TASK [common : Kubesphere | create minio config directory] *********************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
skipping: [localhost] => (item={u'path': u'/root/.config/rclone', u'file': u'rclone.conf'})

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item=helm)
skipping: [localhost] => (item=vmbased)

TASK [common : Kubesphere | Deploying common component] ************************
skipping: [localhost] => (item=mysql.yaml)

TASK [common : Kubesphere | Deploying common component] ************************
skipping: [localhost] => (item=etcd.yaml)

TASK [common : Setting persistentVolumeReclaimPolicy (mysql)] ******************
skipping: [localhost]

TASK [common : Setting persistentVolumeReclaimPolicy (etcd)] *******************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=21   changed=16   unreachable=0    failed=0    skipped=83   rescued=0    ignored=0

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [ks-core/prepare : KubeSphere | Create KubeSphere dir] ********************
ok: [localhost]

TASK [ks-core/prepare : KubeSphere | Getting installation init files] **********
changed: [localhost] => (item=workspace.yaml)
changed: [localhost] => (item=ks-init)

TASK [ks-core/prepare : KubeSphere | Init KubeSphere system] *******************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Creating KubeSphere Secret] ***************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Creating KubeSphere Secret] ***************
ok: [localhost]

TASK [ks-core/prepare : KubeSphere | Enable Token Script] **********************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Getting KS Token] *************************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Setting ks_token] *************************
ok: [localhost]
TASK [ks-core/prepare : KubeSphere | Creating manifests] ***********************
changed: [localhost] => (item={u'type': u'init', u'name': u'ks-account-init', u'file': u'ks-account-init.yaml'})
changed: [localhost] => (item={u'type': u'init', u'name': u'ks-apigateway-init', u'file': u'ks-apigateway-init.yaml'})
changed: [localhost] => (item={u'type': u'values', u'name': u'custom-values-istio-init', u'file': u'custom-values-istio-init.yaml'})
changed: [localhost] => (item={u'type': u'cm', u'name': u'kubesphere-config', u'file': u'kubesphere-config.yaml'})

TASK [ks-core/prepare : KubeSphere | Init KubeSphere] **************************
changed: [localhost] => (item=ks-account-init.yaml)
changed: [localhost] => (item=ks-apigateway-init.yaml)
changed: [localhost] => (item=kubesphere-config.yaml)

TASK [ks-core/prepare : KubeSphere | Getting controls-system file] *************
changed: [localhost] => (item={u'name': u'kubesphere-controls-system', u'file': u'kubesphere-controls-system.yaml'})

TASK [ks-core/prepare : KubeSphere | Installing controls-system] ***************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Create KubeSphere workspace] **************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Create KubeSphere vpa] ********************
skipping: [localhost]

TASK [ks-core/prepare : KubeSphere | Generate kubeconfig-admin] ****************
skipping: [localhost]

TASK [ks-core/prepare : Kubesphere | Checking kubesphere component] ************
changed: [localhost]

TASK [ks-core/prepare : Kubesphere | Get kubesphere component version] *********
skipping: [localhost]

TASK [ks-core/prepare : ks-upgrade | disable ks-apiserver] *********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-apiserver\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-apigateway] ********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 6, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-apigateway\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-account] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-account\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-console] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 16, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-console\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-controller-manager] ************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 21, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-controller-manager\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-apiserver] *********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-apiserver\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-apigateway] ********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 6, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-apigateway\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-account] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-account\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-console] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 16, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-console\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-controller-manager] ************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 21, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-controller-manager\n  ^ here\n"}
...ignoring

TASK [ks-core/ks-core : KubeSphere | Getting kubernetes version] ***************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Setting kubernetes version] ***************
ok: [localhost]

TASK [ks-core/ks-core : KubeSphere | Getting kubernetes master num] ************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Setting master num] ***********************
ok: [localhost]

TASK [ks-core/ks-core : ks-console | Checking ks-console svc] ******************
changed: [localhost]

TASK [ks-core/ks-core : ks-console | Getting ks-console svc port] **************
skipping: [localhost]

TASK [ks-core/ks-core : ks-console | Setting console_port] *********************
skipping: [localhost]

TASK [ks-core/ks-core : KubeSphere | Getting Ingress installation files] *******
changed: [localhost] => (item=ingress)
changed: [localhost] => (item=ks-account)
changed: [localhost] => (item=ks-apigateway)
changed: [localhost] => (item=ks-apiserver)
changed: [localhost] => (item=ks-console)
changed: [localhost] => (item=ks-controller-manager)

TASK [ks-core/ks-core : KubeSphere | Creating manifests] ***********************
changed: [localhost] => (item={u'path': u'ingress', u'type': u'config', u'file': u'ingress-controller.yaml'})
changed: [localhost] => (item={u'path': u'ks-account', u'type': u'deployment', u'file': u'ks-account.yml'})
changed: [localhost] => (item={u'path': u'ks-apigateway', u'type': u'deploy', u'file': u'ks-apigateway.yaml'})
changed: [localhost] => (item={u'path': u'ks-apiserver', u'type': u'deploy', u'file': u'ks-apiserver.yml'})
changed: [localhost] => (item={u'path': u'ks-controller-manager', u'type': u'deploy', u'file': u'ks-controller-manager.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'config', u'file': u'ks-console-config.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'deploy', u'file': u'ks-console-deployment.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'svc', u'file': u'ks-console-svc.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'deploy', u'file': u'ks-docs-deployment.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'config', u'file': u'sample-bookinfo-configmap.yaml'})

TASK [ks-core/ks-core : KubeSphere | Creating Ingress-controller configmap] ****
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Check ks-account version] *****************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Update kubectl image] *********************
skipping: [localhost]

TASK [ks-core/ks-core : KubeSphere | Creating ks-core] *************************
changed: [localhost] => (item={u'path': u'ks-apigateway', u'file': u'ks-apigateway.yaml'})
changed: [localhost] => (item={u'path': u'ks-apiserver', u'file': u'ks-apiserver.yml'})
changed: [localhost] => (item={u'path': u'ks-account', u'file': u'ks-account.yml'})
changed: [localhost] => (item={u'path': u'ks-controller-manager', u'file': u'ks-controller-manager.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-config.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'sample-bookinfo-configmap.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-deployment.yml'})

TASK [ks-core/ks-core : KubeSphere | Check ks-console svc] *********************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Creating ks-console svc] ******************
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-svc.yml'})

TASK [ks-core/ks-core : KubeSphere | Patch ks-console svc] *********************
skipping: [localhost]
localhost                  : ok=37   changed=21   unreachable=0    failed=0    skipped=10   rescued=0    ignored=10
PLAY RECAP *********************************************************************
localhost                  : ok=37   changed=21   unreachable=0    failed=0    skipped=10   rescued=0    ignored=10

Start installing logging
Start installing monitoring
Start installing metrics-server
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task logging status is successful
task metrics-server status is successful
total: 3     completed:2
**************************************************
task logging status is successful
task monitoring status is successful
task metrics-server status is successful
total: 3     completed:3
**************************************************
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://xxx.xxx.xxx.xx:30880
Account: admin
Password: P@88w0rd

NOTES:
  1. After logging into the console, please check the
     monitoring status of service components in
     the "Cluster Status". If the service is not
     ready, please wait patiently. You can start
     to use when all components are ready.
  2. Please modify the default password after login.

#####################################################

kubectl get node | awk '{if(NR>1){print $3}}' | grep master |wc -l

helm list

NAME                         	REVISION	UPDATED                 	STATUS  	CHART                       	APP VERSION   	NAMESPACE
elasticsearch-logging        	1       	Mon Feb 17 20:49:03 2020	DEPLOYED	elasticsearch-1.22.1        	6.7.0-0217    	kubesphere-logging-system
elasticsearch-logging-curator	1       	Mon Feb 17 20:49:08 2020	DEPLOYED	elasticsearch-curator-1.3.3 	5.5.4-0217    	kubesphere-logging-system
gitlab                       	7       	Fri Sep 20 18:52:28 2019	DEPLOYED	gitlab-ce-0.2.1             	              	gitlab
harbor                       	1       	Thu Aug 15 16:08:14 2019	DEPLOYED	harbor-1.1.1                	1.8.1         	harbor
ingress-nginx                	1       	Wed Aug 14 18:14:10 2019	DEPLOYED	nginx-ingress-0.9.5         	0.10.2        	kube-system
ks-openldap                  	1       	Mon Feb 17 20:47:36 2020	DEPLOYED	openldap-ha-0.1.0           	1.0           	kubesphere-system
ks-redis                     	7       	Sat Feb 15 17:33:32 2020	DEPLOYED	redis-ha-3.9.0              	5.0.5         	kubesphere-system
logging-fluentbit-operator   	1       	Mon Feb 17 20:48:53 2020	DEPLOYED	fluentbit-operator-0.1.0    	0.1.0-0217    	kubesphere-logging-system
metrics-server               	1       	Mon Feb 17 20:48:45 2020	DEPLOYED	metrics-server-2.5.0        	0.3.1-0217    	kube-system
nfs-client-provisioner       	1       	Mon Aug 12 11:31:07 2019	DEPLOYED	nfs-client-provisioner-0.1.0	v3.1.0-k8s1.11	kube-system

有ks-redis 改过副本数或者删过redis吗? 看下redis-ha(sts)和redis-ha-haproxy(deploy)的副本数 都调到3

之前部署完失败后曾经删除过ks-redis,ks-redis这个chart可能是没有删除干净。kubesphere-system命名空间看不到任何redis服务。

要删除的话 用helm del --purge ks-redis 然后重启下ks-installer

我删除了helm redis chart,重新部署ks-installer。这次有redis服务了,但是redis初始化容器有报错

Could not connect to Redis at redis-ha:26379: Name does not resolve
Initializing config..
/readonly-config/init.sh: line 84: Could not resolve the announce ip for this pod: not found

看起来是没有announce服务@Cauchy

看下pod状态 redis-ha起来了么 有redis-ha这个service么?

  • leo 回复了此帖

    Cauchy

    NAME                                         READY   STATUS                  RESTARTS   AGE
    pod/ks-account-845d86f776-7xlpd              1/1     Running                 0          11h
    pod/ks-account-845d86f776-f5jpw              1/1     Running                 0          11h
    pod/ks-account-845d86f776-xp8nw              1/1     Running                 0          11h
    pod/ks-apigateway-d6f89b96f-np9xn            0/1     CrashLoopBackOff        139        11h
    pod/ks-apigateway-d6f89b96f-svvpv            0/1     CrashLoopBackOff        139        11h
    pod/ks-apigateway-d6f89b96f-zmvwp            0/1     CrashLoopBackOff        140        11h
    pod/ks-apiserver-5dff5c594d-c22dv            1/1     Running                 0          11h
    pod/ks-apiserver-5dff5c594d-gh5qd            1/1     Running                 0          11h
    pod/ks-apiserver-5dff5c594d-qrn8m            1/1     Running                 0          11h
    pod/ks-console-5c666c644-5cttb               1/1     Running                 0          11h
    pod/ks-console-5c666c644-d4s97               1/1     Running                 0          11h
    pod/ks-console-5c666c644-k5g7p               1/1     Running                 0          11h
    pod/ks-controller-manager-779d9c6bbf-bsfgz   1/1     Running                 0          11h
    pod/ks-controller-manager-779d9c6bbf-n2sbw   1/1     Running                 0          11h
    pod/ks-controller-manager-779d9c6bbf-ts87s   1/1     Running                 0          11h
    pod/ks-installer-59fb465b7-c48fz             1/1     Running                 0          11h
    pod/openldap-0                               1/1     Running                 0          11h
    pod/openldap-1                               1/1     Running                 0          11h
    pod/redis-ha-haproxy-7d856799d5-5hnwf        1/1     Running                 0          11h
    pod/redis-ha-haproxy-7d856799d5-9cvxq        1/1     Running                 0          11h
    pod/redis-ha-haproxy-7d856799d5-g5vw4        1/1     Running                 0          11h
    pod/redis-ha-server-0                        0/2     Init:CrashLoopBackOff   140        11h
    
    
    NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
    service/ks-account      ClusterIP   10.107.110.199   <none>        80/TCP               11h
    service/ks-apigateway   ClusterIP   10.106.88.176    <none>        80/TCP               11h
    service/ks-apiserver    ClusterIP   10.97.56.222     <none>        80/TCP               11h
    service/ks-console      NodePort    10.105.78.128    <none>        80:30880/TCP         11h
    service/openldap        ClusterIP   None             <none>        389/TCP              11h
    service/redis           ClusterIP   10.103.120.41    <none>        6379/TCP             11h
    service/redis-ha        ClusterIP   None             <none>        6379/TCP,26379/TCP   11h
    
    
    NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
    deployment.apps/ks-account              3/3     3            3           11h
    deployment.apps/ks-apigateway           0/3     3            0           11h
    deployment.apps/ks-apiserver            3/3     3            3           11h
    deployment.apps/ks-console              3/3     3            3           11h
    deployment.apps/ks-controller-manager   3/3     3            3           11h
    deployment.apps/ks-installer            1/1     1            1           11h
    deployment.apps/redis-ha-haproxy        3/3     3            3           11h
    
    NAME                                               DESIRED   CURRENT   READY   AGE
    replicaset.apps/ks-account-845d86f776              3         3         3       11h
    replicaset.apps/ks-apigateway-d6f89b96f            3         3         0       11h
    replicaset.apps/ks-apiserver-5dff5c594d            3         3         3       11h
    replicaset.apps/ks-console-5c666c644               3         3         3       11h
    replicaset.apps/ks-controller-manager-779d9c6bbf   3         3         3       11h
    replicaset.apps/ks-installer-59fb465b7             1         1         1       11h
    replicaset.apps/redis-ha-haproxy-7d856799d5        3         3         3       11h
    
    NAME                               READY   AGE
    statefulset.apps/openldap          2/2     11h
    statefulset.apps/redis-ha-server   0/3     11h
    • Jeff 回复了此帖

      leo 你之前安装高可用成功过么

      没有,我的测试集群是单master,这是第一次在多master集群部署@Jeff

        没法远程,这是内网集群

          leo kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha" 试试能不能解析到redis-ha

          • leo 回复了此帖

            Cauchy

            $ kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha"
            ping: bad address 'redis-ha'
            pod "test" deleted
            pod kubesphere-system/test terminated (Error)