- 已编辑
使用三台ubuntu18.04搭建了k8s和kubesphere,版本信息如下:
k8s:v1.18.8
kubesphere:v3.0.0
容器运行时:containerd
采用kk创建的集群,并安装好kubesphere,计划通过kk进行升级。升级命令为:
./kk upgrade –with-kubernetes v1.18.8 –with-kubesphere v3.1.0
v3.1.0
[master1 172.17.3.12] MSG:
namespace/kubesphere-system unchanged
namespace/kubesphere-monitoring-system unchanged
[master1 172.17.3.12] MSG:
namespace/kubesphere-system unchanged
serviceaccount/ks-installer unchanged
customresourcedefinition.apiextensions.k8s.io/clusterconfigurations.installer.kubesphere.io unchanged
clusterrole.rbac.authorization.k8s.io/ks-installer unchanged
clusterrolebinding.rbac.authorization.k8s.io/ks-installer unchanged
deployment.apps/ks-installer unchanged
Please wait for the installation to complete: >>--->
Please wait for the installation to complete: <---<<
WARN[12:29:28 CST] Task failed ...
WARN[12:29:28 CST] error: KubeSphere startup timeout.
Error: Failed to upgrade kubesphere: KubeSphere startup timeout.
Usage:
kk upgrade [flags]
Flags:
--download-cmd string The user defined command to download the necessary binary files. The first param '%s' is output path, the second param '%s', is the URL (default "curl -L -o %s %s")
-f, --filename string Path to a configuration file
-h, --help help for upgrade
--skip-pull-images Skip pre pull images
--with-kubernetes string Specify a supported version of kubernetes
--with-kubesphere Deploy a specific version of kubesphere (default v3.0.0)
Global Flags:
--debug Print detailed information (default true)
--in-cluster Running inside the cluster
Failed to upgrade kubesphere: KubeSphere startup timeout.
发现控制台启动超时,执行如下命令查看日志
kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath=‘{.items[0].metadata.name}’) -f
2021-06-24T03:22:52Z INFO : shell-operator latest
2021-06-24T03:22:52Z INFO : Use temporary dir: /tmp/shell-operator
2021-06-24T03:22:52Z INFO : Initialize hooks manager ...
2021-06-24T03:22:52Z INFO : Search and load hooks ...
2021-06-24T03:22:52Z INFO : Load hook config from '/hooks/kubesphere/installRunner.py'
2021-06-24T03:22:52Z INFO : HTTP SERVER Listening on 0.0.0.0:9115
2021-06-24T03:22:53Z INFO : Load hook config from '/hooks/kubesphere/schedule.sh'
2021-06-24T03:22:53Z INFO : Initializing schedule manager ...
2021-06-24T03:22:53Z INFO : KUBE Init Kubernetes client
2021-06-24T03:22:53Z INFO : KUBE-INIT Kubernetes client is configured successfully
2021-06-24T03:22:53Z INFO : MAIN: run main loop
2021-06-24T03:22:53Z INFO : MAIN: add onStartup tasks
2021-06-24T03:22:53Z INFO : QUEUE add all HookRun@OnStartup
2021-06-24T03:22:53Z INFO : Running schedule manager ...
2021-06-24T03:22:53Z INFO : MSTOR Create new metric shell_operator_live_ticks
2021-06-24T03:22:53Z INFO : MSTOR Create new metric shell_operator_tasks_queue_length
2021-06-24T03:22:53Z INFO : GVR for kind 'ClusterConfiguration' is installer.kubesphere.io/v1alpha1, Resource=clusterconfigurations
2021-06-24T03:22:53Z INFO : EVENT Kube event '53b1e5bc-92d5-4e2e-b32f-bb45f1153d49'
2021-06-24T03:22:53Z INFO : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2021-06-24T03:22:56Z INFO : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2021-06-24T03:22:56Z INFO : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
Delete old cluster configuration successfully
Create cluster configuration successfully
2021-06-24T03:22:56Z INFO : EVENT Kube event '53b1e5bc-92d5-4e2e-b32f-bb45f1153d49'
2021-06-24T03:22:56Z INFO : QUEUE add TASK_HOOK_RUN@KUBE_EVENTS kubesphere/installRunner.py
2021-06-24T03:22:59Z INFO : TASK_RUN HookRun@KUBE_EVENTS kubesphere/installRunner.py
2021-06-24T03:22:59Z INFO : Running hook 'kubesphere/installRunner.py' binding 'KUBE_EVENTS' ...
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [download : include_tasks] ************************************************
skipping: [localhost]
TASK [download : Downloading items] ********************************************
skipping: [localhost]
TASK [download : Synchronizing container] **************************************
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
"msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}
TASK [preinstall : KubeSphere | Checking Kubernetes version] *******************
changed: [localhost]
TASK [preinstall : KubeSphere | Initing Kubernetes version] ********************
ok: [localhost]
TASK [preinstall : KubeSphere | Stopping if Kubernetes version is nonsupport] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [preinstall : KubeSphere | Checking StorageClass] *************************
changed: [localhost]
TASK [preinstall : KubeSphere | Stopping if StorageClass was not found] ********
skipping: [localhost]
TASK [preinstall : KubeSphere | Checking default StorageClass] *****************
changed: [localhost]
TASK [preinstall : KubeSphere | Stopping if default StorageClass was not found] ***
ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
TASK [preinstall : KubeSphere | Checking KubeSphere component] *****************
changed: [localhost]
TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
changed: [localhost]
TASK [preinstall : KubeSphere | Getting KubeSphere component version] **********
skipping: [localhost] => (item=ks-openldap)
skipping: [localhost] => (item=ks-redis)
skipping: [localhost] => (item=ks-minio)
skipping: [localhost] => (item=ks-openpitrix)
skipping: [localhost] => (item=elasticsearch-logging)
skipping: [localhost] => (item=elasticsearch-logging-curator)
skipping: [localhost] => (item=istio)
skipping: [localhost] => (item=istio-init)
skipping: [localhost] => (item=jaeger-operator)
skipping: [localhost] => (item=ks-jenkins)
skipping: [localhost] => (item=ks-sonarqube)
skipping: [localhost] => (item=logging-fluentbit-operator)
skipping: [localhost] => (item=uc)
skipping: [localhost] => (item=metrics-server)
localhost : ok=9 changed=5 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0
PLAY RECAP *********************************************************************
localhost : ok=9 changed=5 unreachable=0 failed=0 skipped=6 rescued=0 ignored=0
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
PLAY [localhost] ***************************************************************
TASK [download : include_tasks] ************************************************
skipping: [localhost]
TASK [download : Downloading items] ********************************************
skipping: [localhost]
TASK [download : Synchronizing container] **************************************
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Setting images' namespace override] ***
skipping: [localhost]
TASK [kubesphere-defaults : KubeSphere | Configuring defaults] *****************
ok: [localhost] => {
"msg": "Check roles/kubesphere-defaults/defaults/main.yml"
}
TASK [metrics-server : Metrics-Server | Getting metrics-server installation files] ***
changed: [localhost]
TASK [metrics-server : Metrics-Server | Creating manifests] ********************
ok: [localhost] => (item={'file': 'metrics-server.yaml'})
TASK [metrics-server : Metrics-Server | Checking Metrics-Server] ***************
changed: [localhost]
TASK [metrics-server : Metrics-Server | Uninstalling old metrics-server] *******
changed: [localhost]
TASK [metrics-server : Metrics-Server | Installing new metrics-server] *********
changed: [localhost]
TASK [metrics-server : Metrics-Server | Waitting for metrics.k8s.io ready] *****
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (60 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (59 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (58 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (57 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (56 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (55 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (54 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (53 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (52 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (51 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (50 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (49 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (48 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (47 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (46 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (45 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (44 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (43 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (42 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (41 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (40 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (39 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (38 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (37 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (36 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (35 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (34 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (33 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (32 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (31 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (30 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (29 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (28 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (27 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (26 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (25 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (24 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (23 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (22 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (21 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (20 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (19 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (18 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (17 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (16 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (15 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (14 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (13 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (12 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (11 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (10 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (9 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (8 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (7 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (6 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (5 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (4 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (3 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (2 retries left).
FAILED - RETRYING: Metrics-Server | Waitting for metrics.k8s.io ready (1 retries left).
fatal: [localhost]: FAILED! => {"attempts": 60, "changed": true, "cmd": "/usr/local/bin/kubectl get apiservice | grep metrics.k8s.io\n", "delta": "0:00:00.150931", "end": "2021-06-24 03:33:37.107400", "rc": 0, "start": "2021-06-24 03:33:36.956469", "stderr": "", "stderr_lines": [], "stdout": "v1beta1.metrics.k8s.io kube-system/metrics-server False (MissingEndpoints) 10m", "stdout_lines": ["v1beta1.metrics.k8s.io kube-system/metrics-server False (MissingEndpoints) 10m"]}
PLAY RECAP *********************************************************************
localhost : ok=6 changed=4 unreachable=0 failed=1 skipped=4 rescued=0 ignored=0