部署失败，无redis服务

Cauchy · 2020年2月17日

看下ks-installer的日志

Lleo · 2020年2月17日

2020-02-17T12:46:07Z INFO     : shell-operator v1.0.0-beta.5
2020-02-17T12:46:07Z INFO     : HTTP SERVER Listening on 0.0.0.0:9115
2020-02-17T12:46:07Z INFO     : Use temporary dir: /tmp/shell-operator
2020-02-17T12:46:07Z INFO     : Initialize hooks manager ...
2020-02-17T12:46:07Z INFO     : Search and load hooks ...
2020-02-17T12:46:07Z INFO     : Load hook config from '/hooks/kubesphere/installRunner.py'
2020-02-17T12:46:07Z INFO     : Initializing schedule manager ...
2020-02-17T12:46:07Z INFO     : KUBE Init Kubernetes client
2020-02-17T12:46:07Z INFO     : KUBE-INIT Kubernetes client is configured successfully
2020-02-17T12:46:07Z INFO     : MAIN: run main loop
2020-02-17T12:46:07Z INFO     : MAIN: add onStartup tasks
2020-02-17T12:46:07Z INFO     : QUEUE add all HookRun@OnStartup
2020-02-17T12:46:07Z INFO     : Running schedule manager ...
2020-02-17T12:46:07Z INFO     : MSTOR Create new metric shell_operator_live_ticks
2020-02-17T12:46:07Z INFO     : MSTOR Create new metric shell_operator_tasks_queue_length
2020-02-17T12:46:07Z INFO     : GVR for kind 'ConfigMap' is /v1, Resource=configmaps
2020-02-17T12:46:07Z INFO     : EVENT Kube event 'c0311204-4add-4738-b61c-04951051388c'
2020-02-17T12:46:07Z INFO     : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2020-02-17T12:46:10Z INFO     : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2020-02-17T12:46:10Z INFO     : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [preinstall : check k8s version] ******************************************
changed: [localhost]

TASK [preinstall : init k8s version] *******************************************
ok: [localhost]

TASK [preinstall : Stop if kuernetes version is nonsupport] ********************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check helm status] ******************************************
changed: [localhost]

TASK [preinstall : Stop if Helm is not available] ******************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check storage class] ****************************************
changed: [localhost]

TASK [preinstall : Stop if StorageClass was not found] *************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [preinstall : check default storage class] ********************************
changed: [localhost]

TASK [preinstall : Stop if defaultStorageClass was not found] ******************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=9    changed=4    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [common : Kubesphere | Check kube-node-lease namespace] *******************
changed: [localhost]

TASK [common : KubeSphere | Get system namespaces] *****************************
ok: [localhost]

TASK [common : set_fact] *******************************************************
ok: [localhost]

TASK [common : debug] **********************************************************
ok: [localhost] => {
    "msg": [
        "kubesphere-system",
        "kubesphere-controls-system",
        "kubesphere-monitoring-system",
        "kube-node-lease",
        "kubesphere-logging-system"
    ]
}

TASK [common : KubeSphere | Create kubesphere namespace] ***********************
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Labeling system-workspace] *************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Create ImagePullSecrets] ***************************
changed: [localhost] => (item=default)
changed: [localhost] => (item=kube-public)
changed: [localhost] => (item=kube-system)
changed: [localhost] => (item=kubesphere-system)
changed: [localhost] => (item=kubesphere-controls-system)
changed: [localhost] => (item=kubesphere-monitoring-system)
changed: [localhost] => (item=kube-node-lease)
changed: [localhost] => (item=kubesphere-logging-system)

TASK [common : KubeSphere | Getting kubernetes master num] *********************
changed: [localhost]

TASK [common : KubeSphere | Setting master num] ********************************
ok: [localhost]

TASK [common : Kubesphere | Getting common component installation files] *******
changed: [localhost] => (item=common)
changed: [localhost] => (item=ks-crds)

TASK [common : KubeSphere | Create KubeSphere crds] ****************************
changed: [localhost]

TASK [common : Kubesphere | Checking openpitrix common component] **************
changed: [localhost]

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item={u'ks': u'mysql-pvc', u'op': u'openpitrix-db'})
skipping: [localhost] => (item={u'ks': u'etcd-pvc', u'op': u'openpitrix-etcd'})

TASK [common : Getting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (mysql)] ***************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (mysql)] ***************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Getting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeName (etcd)] ****************************
skipping: [localhost]

TASK [common : Setting PersistentVolumeSize (etcd)] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
changed: [localhost] => (item={u'path': u'etcd', u'file': u'etcd.yaml'})
changed: [localhost] => (item={u'name': u'mysql', u'file': u'mysql.yaml'})
changed: [localhost] => (item={u'path': u'redis', u'file': u'redis.yaml'})

TASK [common : Kubesphere | Creating mysql sercet] *****************************
changed: [localhost]

TASK [common : Kubesphere | Deploying etcd and mysql] **************************
skipping: [localhost] => (item=etcd.yaml)
skipping: [localhost] => (item=mysql.yaml)

TASK [common : Kubesphere | Getting minio installation files] ******************
skipping: [localhost] => (item=minio-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-minio', u'file': u'custom-values-minio.yaml'})

TASK [common : Kubesphere | Check minio] ***************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy minio] **************************************
skipping: [localhost]

TASK [common : debug] **********************************************************
skipping: [localhost]

TASK [common : fail] ***********************************************************
skipping: [localhost]

TASK [common : Kubesphere | create minio config directory] *********************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
skipping: [localhost] => (item={u'path': u'/root/.config/rclone', u'file': u'rclone.conf'})

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item=helm)
skipping: [localhost] => (item=vmbased)

TASK [common : Kubesphere | Check ha-redis] ************************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis installation files] ******************
skipping: [localhost] => (item=redis-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-redis', u'file': u'custom-values-redis.yaml'})

TASK [common : Kubesphere | Check old redis status] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old redis svc] *******************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Creating redis migration script] *******************
skipping: [localhost] => (item={u'path': u'/etc/kubesphere', u'file': u'redisMigrate.py'})

TASK [common : Kubesphere | Check redis-ha status] *****************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old redis] *********************************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : Kubesphere | Getting openldap installation files] ***************
skipping: [localhost] => (item=openldap-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-openldap', u'file': u'custom-values-openldap.yaml'})

TASK [common : Kubesphere | Check old openldap status] *************************
skipping: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old openldap svc] ****************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap] ************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy openldap] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Load old openldap data] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap-ha status] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get openldap-ha pod list] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get old openldap data] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Migrating openldap data] ***************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old openldap] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Restart openldap] **********************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Check ha-redis] ************************************
changed: [localhost]

TASK [common : Kubesphere | Getting redis installation files] ******************
skipping: [localhost] => (item=redis-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-redis', u'file': u'custom-values-redis.yaml'})

TASK [common : Kubesphere | Check old redis status] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old redis svc] *******************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost]

TASK [common : Kubesphere | Getting redis PodIp] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Creating redis migration script] *******************
skipping: [localhost] => (item={u'path': u'/etc/kubesphere', u'file': u'redisMigrate.py'})

TASK [common : Kubesphere | Check redis-ha status] *****************************
skipping: [localhost]

TASK [common : ks-logging | Migrating redis data] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old redis] *********************************
skipping: [localhost]

TASK [common : Kubesphere | Deploying redis] ***********************************
skipping: [localhost] => (item=redis.yaml)

TASK [common : Kubesphere | Getting openldap installation files] ***************
changed: [localhost] => (item=openldap-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
changed: [localhost] => (item={u'name': u'custom-values-openldap', u'file': u'custom-values-openldap.yaml'})

TASK [common : Kubesphere | Check old openldap status] *************************
changed: [localhost]

TASK [common : KubeSphere | Shutdown ks-account] *******************************
skipping: [localhost]

TASK [common : Kubesphere | Delete and backup old openldap svc] ****************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap] ************************************
changed: [localhost]

TASK [common : Kubesphere | Deploy openldap] ***********************************
changed: [localhost]

TASK [common : Kubesphere | Load old openldap data] ****************************
skipping: [localhost]

TASK [common : Kubesphere | Check openldap-ha status] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get openldap-ha pod list] **************************
skipping: [localhost]

TASK [common : Kubesphere | Get old openldap data] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Migrating openldap data] ***************************
skipping: [localhost]

TASK [common : Kubesphere | Disable old openldap] ******************************
skipping: [localhost]

TASK [common : Kubesphere | Restart openldap] **********************************
skipping: [localhost]

TASK [common : KubeSphere | Restarting ks-account] *****************************
skipping: [localhost]

TASK [common : Kubesphere | Getting minio installation files] ******************
skipping: [localhost] => (item=minio-ha)

TASK [common : Kubesphere | Creating manifests] ********************************
skipping: [localhost] => (item={u'name': u'custom-values-minio', u'file': u'custom-values-minio.yaml'})

TASK [common : Kubesphere | Check minio] ***************************************
skipping: [localhost]

TASK [common : Kubesphere | Deploy minio] **************************************
skipping: [localhost]

TASK [common : debug] **********************************************************
skipping: [localhost]

TASK [common : fail] ***********************************************************
skipping: [localhost]

TASK [common : Kubesphere | create minio config directory] *********************
skipping: [localhost]

TASK [common : Kubesphere | Creating common component manifests] ***************
skipping: [localhost] => (item={u'path': u'/root/.config/rclone', u'file': u'rclone.conf'})

TASK [common : include_tasks] **************************************************
skipping: [localhost] => (item=helm)
skipping: [localhost] => (item=vmbased)

TASK [common : Kubesphere | Deploying common component] ************************
skipping: [localhost] => (item=mysql.yaml)

TASK [common : Kubesphere | Deploying common component] ************************
skipping: [localhost] => (item=etcd.yaml)

TASK [common : Setting persistentVolumeReclaimPolicy (mysql)] ******************
skipping: [localhost]

TASK [common : Setting persistentVolumeReclaimPolicy (etcd)] *******************
skipping: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=21   changed=16   unreachable=0    failed=0    skipped=83   rescued=0    ignored=0

[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'

PLAY [localhost] ***************************************************************

TASK [download : include_tasks] ************************************************
skipping: [localhost]

TASK [download : Download items] ***********************************************
skipping: [localhost]

TASK [download : Sync container] ***********************************************
skipping: [localhost]

TASK [kubesphere-defaults : Configure defaults] ********************************
ok: [localhost] => {
    "msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}

TASK [ks-core/prepare : KubeSphere | Create KubeSphere dir] ********************
ok: [localhost]

TASK [ks-core/prepare : KubeSphere | Getting installation init files] **********
changed: [localhost] => (item=workspace.yaml)
changed: [localhost] => (item=ks-init)

TASK [ks-core/prepare : KubeSphere | Init KubeSphere system] *******************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Creating KubeSphere Secret] ***************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Creating KubeSphere Secret] ***************
ok: [localhost]

TASK [ks-core/prepare : KubeSphere | Enable Token Script] **********************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Getting KS Token] *************************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Setting ks_token] *************************
ok: [localhost]

TASK [ks-core/prepare : KubeSphere | Creating manifests] ***********************
changed: [localhost] => (item={u'type': u'init', u'name': u'ks-account-init', u'file': u'ks-account-init.yaml'})
changed: [localhost] => (item={u'type': u'init', u'name': u'ks-apigateway-init', u'file': u'ks-apigateway-init.yaml'})
changed: [localhost] => (item={u'type': u'values', u'name': u'custom-values-istio-init', u'file': u'custom-values-istio-init.yaml'})
changed: [localhost] => (item={u'type': u'cm', u'name': u'kubesphere-config', u'file': u'kubesphere-config.yaml'})

TASK [ks-core/prepare : KubeSphere | Init KubeSphere] **************************
changed: [localhost] => (item=ks-account-init.yaml)
changed: [localhost] => (item=ks-apigateway-init.yaml)
changed: [localhost] => (item=kubesphere-config.yaml)

TASK [ks-core/prepare : KubeSphere | Getting controls-system file] *************
changed: [localhost] => (item={u'name': u'kubesphere-controls-system', u'file': u'kubesphere-controls-system.yaml'})

TASK [ks-core/prepare : KubeSphere | Installing controls-system] ***************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Create KubeSphere workspace] **************
changed: [localhost]

TASK [ks-core/prepare : KubeSphere | Create KubeSphere vpa] ********************
skipping: [localhost]

TASK [ks-core/prepare : KubeSphere | Generate kubeconfig-admin] ****************
skipping: [localhost]

TASK [ks-core/prepare : Kubesphere | Checking kubesphere component] ************
changed: [localhost]

TASK [ks-core/prepare : Kubesphere | Get kubesphere component version] *********
skipping: [localhost]

TASK [ks-core/prepare : ks-upgrade | disable ks-apiserver] *********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-apiserver\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-apigateway] ********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 6, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-apigateway\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-account] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-account\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-console] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 16, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-console\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | disable ks-controller-manager] ************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout != ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout != ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-stop.yaml': line 21, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | disable ks-controller-manager\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-apiserver] *********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 1, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-apiserver\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-apigateway] ********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 6, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-apigateway\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-account] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 11, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-account\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-console] ***********************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 16, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-console\n  ^ here\n"}
...ignoring

TASK [ks-core/prepare : ks-upgrade | restart ks-controller-manager] ************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'console_version.stdout and console_version.stdout == ks_version' failed. The error was: error while evaluating conditional (console_version.stdout and console_version.stdout == ks_version): 'dict object' has no attribute 'stdout'\n\nThe error appears to be in '/kubesphere/installer/roles/ks-core/prepare/tasks/ks-restart.yaml': line 21, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: ks-upgrade | restart ks-controller-manager\n  ^ here\n"}
...ignoring

TASK [ks-core/ks-core : KubeSphere | Getting kubernetes version] ***************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Setting kubernetes version] ***************
ok: [localhost]

TASK [ks-core/ks-core : KubeSphere | Getting kubernetes master num] ************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Setting master num] ***********************
ok: [localhost]

TASK [ks-core/ks-core : ks-console | Checking ks-console svc] ******************
changed: [localhost]

TASK [ks-core/ks-core : ks-console | Getting ks-console svc port] **************
skipping: [localhost]

TASK [ks-core/ks-core : ks-console | Setting console_port] *********************
skipping: [localhost]

TASK [ks-core/ks-core : KubeSphere | Getting Ingress installation files] *******
changed: [localhost] => (item=ingress)
changed: [localhost] => (item=ks-account)
changed: [localhost] => (item=ks-apigateway)
changed: [localhost] => (item=ks-apiserver)
changed: [localhost] => (item=ks-console)
changed: [localhost] => (item=ks-controller-manager)

TASK [ks-core/ks-core : KubeSphere | Creating manifests] ***********************
changed: [localhost] => (item={u'path': u'ingress', u'type': u'config', u'file': u'ingress-controller.yaml'})
changed: [localhost] => (item={u'path': u'ks-account', u'type': u'deployment', u'file': u'ks-account.yml'})
changed: [localhost] => (item={u'path': u'ks-apigateway', u'type': u'deploy', u'file': u'ks-apigateway.yaml'})
changed: [localhost] => (item={u'path': u'ks-apiserver', u'type': u'deploy', u'file': u'ks-apiserver.yml'})
changed: [localhost] => (item={u'path': u'ks-controller-manager', u'type': u'deploy', u'file': u'ks-controller-manager.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'config', u'file': u'ks-console-config.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'deploy', u'file': u'ks-console-deployment.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'svc', u'file': u'ks-console-svc.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'deploy', u'file': u'ks-docs-deployment.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'type': u'config', u'file': u'sample-bookinfo-configmap.yaml'})

TASK [ks-core/ks-core : KubeSphere | Creating Ingress-controller configmap] ****
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Check ks-account version] *****************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Update kubectl image] *********************
skipping: [localhost]

TASK [ks-core/ks-core : KubeSphere | Creating ks-core] *************************
changed: [localhost] => (item={u'path': u'ks-apigateway', u'file': u'ks-apigateway.yaml'})
changed: [localhost] => (item={u'path': u'ks-apiserver', u'file': u'ks-apiserver.yml'})
changed: [localhost] => (item={u'path': u'ks-account', u'file': u'ks-account.yml'})
changed: [localhost] => (item={u'path': u'ks-controller-manager', u'file': u'ks-controller-manager.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-config.yml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'sample-bookinfo-configmap.yaml'})
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-deployment.yml'})

TASK [ks-core/ks-core : KubeSphere | Check ks-console svc] *********************
changed: [localhost]

TASK [ks-core/ks-core : KubeSphere | Creating ks-console svc] ******************
changed: [localhost] => (item={u'path': u'ks-console', u'file': u'ks-console-svc.yml'})

TASK [ks-core/ks-core : KubeSphere | Patch ks-console svc] *********************
skipping: [localhost]
localhost                  : ok=37   changed=21   unreachable=0    failed=0    skipped=10   rescued=0    ignored=10
PLAY RECAP *********************************************************************
localhost                  : ok=37   changed=21   unreachable=0    failed=0    skipped=10   rescued=0    ignored=10

Start installing logging
Start installing monitoring
Start installing metrics-server
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task metrics-server status is successful
total: 3     completed:1
**************************************************
task logging status is successful
task metrics-server status is successful
total: 3     completed:2
**************************************************
task logging status is successful
task monitoring status is successful
task metrics-server status is successful
total: 3     completed:3
**************************************************
#####################################################
###              Welcome to KubeSphere!           ###
#####################################################

Console: http://xxx.xxx.xxx.xx:30880
Account: admin
Password: P@88w0rd

NOTES：
  1. After logging into the console, please check the
     monitoring status of service components in
     the "Cluster Status". If the service is not
     ready, please wait patiently. You can start
     to use when all components are ready.
  2. Please modify the default password after login.

#####################################################

Lleo · 2020年2月17日

@Cauchy 上传了完整日志

Cauchy · 2020年2月17日

kubectl get node | awk '{if(NR>1){print $3}}' | grep master |wc -l

helm list

Lleo · 2020年2月17日

Lleo · 2020年2月17日

NAME                         	REVISION	UPDATED                 	STATUS  	CHART                       	APP VERSION   	NAMESPACE
elasticsearch-logging        	1       	Mon Feb 17 20:49:03 2020	DEPLOYED	elasticsearch-1.22.1        	6.7.0-0217    	kubesphere-logging-system
elasticsearch-logging-curator	1       	Mon Feb 17 20:49:08 2020	DEPLOYED	elasticsearch-curator-1.3.3 	5.5.4-0217    	kubesphere-logging-system
gitlab                       	7       	Fri Sep 20 18:52:28 2019	DEPLOYED	gitlab-ce-0.2.1             	              	gitlab
harbor                       	1       	Thu Aug 15 16:08:14 2019	DEPLOYED	harbor-1.1.1                	1.8.1         	harbor
ingress-nginx                	1       	Wed Aug 14 18:14:10 2019	DEPLOYED	nginx-ingress-0.9.5         	0.10.2        	kube-system
ks-openldap                  	1       	Mon Feb 17 20:47:36 2020	DEPLOYED	openldap-ha-0.1.0           	1.0           	kubesphere-system
ks-redis                     	7       	Sat Feb 15 17:33:32 2020	DEPLOYED	redis-ha-3.9.0              	5.0.5         	kubesphere-system
logging-fluentbit-operator   	1       	Mon Feb 17 20:48:53 2020	DEPLOYED	fluentbit-operator-0.1.0    	0.1.0-0217    	kubesphere-logging-system
metrics-server               	1       	Mon Feb 17 20:48:45 2020	DEPLOYED	metrics-server-2.5.0        	0.3.1-0217    	kube-system
nfs-client-provisioner       	1       	Mon Aug 12 11:31:07 2019	DEPLOYED	nfs-client-provisioner-0.1.0	v3.1.0-k8s1.11	kube-system

Cauchy · 2020年2月17日

有ks-redis 改过副本数或者删过redis吗？看下redis-ha（sts）和redis-ha-haproxy（deploy）的副本数都调到3

Lleo · 2020年2月17日

之前部署完失败后曾经删除过ks-redis，ks-redis这个chart可能是没有删除干净。kubesphere-system命名空间看不到任何redis服务。

Cauchy · 2020年2月17日

要删除的话用helm del --purge ks-redis 然后重启下ks-installer

Lleo · 2020年2月17日

我删除了helm redis chart，重新部署ks-installer。这次有redis服务了，但是redis初始化容器有报错

Could not connect to Redis at redis-ha:26379: Name does not resolve
Initializing config..
/readonly-config/init.sh: line 84: Could not resolve the announce ip for this pod: not found

看起来是没有announce服务@Cauchy

Cauchy · 2020年2月18日

看下pod状态 redis-ha起来了么有redis-ha这个service么？

Lleo · 2020年2月18日

Cauchy

NAME                                         READY   STATUS                  RESTARTS   AGE
pod/ks-account-845d86f776-7xlpd              1/1     Running                 0          11h
pod/ks-account-845d86f776-f5jpw              1/1     Running                 0          11h
pod/ks-account-845d86f776-xp8nw              1/1     Running                 0          11h
pod/ks-apigateway-d6f89b96f-np9xn            0/1     CrashLoopBackOff        139        11h
pod/ks-apigateway-d6f89b96f-svvpv            0/1     CrashLoopBackOff        139        11h
pod/ks-apigateway-d6f89b96f-zmvwp            0/1     CrashLoopBackOff        140        11h
pod/ks-apiserver-5dff5c594d-c22dv            1/1     Running                 0          11h
pod/ks-apiserver-5dff5c594d-gh5qd            1/1     Running                 0          11h
pod/ks-apiserver-5dff5c594d-qrn8m            1/1     Running                 0          11h
pod/ks-console-5c666c644-5cttb               1/1     Running                 0          11h
pod/ks-console-5c666c644-d4s97               1/1     Running                 0          11h
pod/ks-console-5c666c644-k5g7p               1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-bsfgz   1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-n2sbw   1/1     Running                 0          11h
pod/ks-controller-manager-779d9c6bbf-ts87s   1/1     Running                 0          11h
pod/ks-installer-59fb465b7-c48fz             1/1     Running                 0          11h
pod/openldap-0                               1/1     Running                 0          11h
pod/openldap-1                               1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-5hnwf        1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-9cvxq        1/1     Running                 0          11h
pod/redis-ha-haproxy-7d856799d5-g5vw4        1/1     Running                 0          11h
pod/redis-ha-server-0                        0/2     Init:CrashLoopBackOff   140        11h


NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
service/ks-account      ClusterIP   10.107.110.199   <none>        80/TCP               11h
service/ks-apigateway   ClusterIP   10.106.88.176    <none>        80/TCP               11h
service/ks-apiserver    ClusterIP   10.97.56.222     <none>        80/TCP               11h
service/ks-console      NodePort    10.105.78.128    <none>        80:30880/TCP         11h
service/openldap        ClusterIP   None             <none>        389/TCP              11h
service/redis           ClusterIP   10.103.120.41    <none>        6379/TCP             11h
service/redis-ha        ClusterIP   None             <none>        6379/TCP,26379/TCP   11h


NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-account              3/3     3            3           11h
deployment.apps/ks-apigateway           0/3     3            0           11h
deployment.apps/ks-apiserver            3/3     3            3           11h
deployment.apps/ks-console              3/3     3            3           11h
deployment.apps/ks-controller-manager   3/3     3            3           11h
deployment.apps/ks-installer            1/1     1            1           11h
deployment.apps/redis-ha-haproxy        3/3     3            3           11h

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/ks-account-845d86f776              3         3         3       11h
replicaset.apps/ks-apigateway-d6f89b96f            3         3         0       11h
replicaset.apps/ks-apiserver-5dff5c594d            3         3         3       11h
replicaset.apps/ks-console-5c666c644               3         3         3       11h
replicaset.apps/ks-controller-manager-779d9c6bbf   3         3         3       11h
replicaset.apps/ks-installer-59fb465b7             1         1         1       11h
replicaset.apps/redis-ha-haproxy-7d856799d5        3         3         3       11h

NAME                               READY   AGE
statefulset.apps/openldap          2/2     11h
statefulset.apps/redis-ha-server   0/3     11h

Jeff · 2020年2月18日

leo 你之前安装高可用成功过么

Lleo · 2020年2月18日

没有，我的测试集群是单master，这是第一次在多master集群部署@Jeff

Cauchy · 2020年2月18日

leo 发个远程瞧瞧

Lleo · 2020年2月18日

没法远程，这是内网集群

Cauchy · 2020年2月18日

leo kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha" 试试能不能解析到redis-ha

Lleo · 2020年2月18日

Cauchy

$ kubectl run -it --rm test --image=alpine:3.7 --restart=Never -n kubesphere-system -- sh -c "ping redis-ha"
ping: bad address 'redis-ha'
pod "test" deleted
pod kubesphere-system/test terminated (Error)

Cauchy · 2020年2月18日

解析不到的话应该是集群的dns有问题
可以先参考下这个：https://github.com/kubesphere/ks-installer/issues/33

Lleo · 2020年2月18日

Cauchy 我用那个命令测了一下我的单master集群，也是解析失败。但是那个集群成功部署了

部署失败，无redis服务

CauchyK零SK壹S

LleoK零S

LleoK零S

CauchyK零SK壹S

LleoK零S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

JeffK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S

CauchyK零SK壹S

LleoK零S