3.0.0 apiserver CrashLoopBackOff

hqq_get

apiserver日志报错：
W0901 19:04:57.447145 1 client_config.go:543] Neither –kubeconfig nor –master was specified. Using the inClusterConfig. This might not work.
Error: failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout
2020/09/01 19:05:02 failed to connect to redis service, please check redis status, error: dial tcp: i/o timeout

请问如何解决，谢谢

Cauchy

看起来是k8s集群网络有问题。
看看kube-system下边pod状态。
kubectl get pod -n kube-system

hqq_get

这个redis是可以连接的，已经测试过了

Cauchy

hqq_get
容器里可以连吗？
pod里用集群内域名连接试试 redis.kubesphere-system.svc:6379

hqq_get

[root@shzj-o-batch-12-25 _]# kubectl get svc -n kubesphere-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ks-apiserver ClusterIP 172.29.2.12 <none> 80/TCP 159m
ks-console NodePort 172.29.1.196 <none> 80:30880/TCP 159m
ks-controller-manager ClusterIP 172.29.0.26 <none> 443/TCP 159m
openldap ClusterIP None <none> 389/TCP 159m
redis ClusterIP 172.29.3.116 <none> 6379/TCP 159m
redis-ha ClusterIP None <none> 6379/TCP,26379/TCP 159m
redis-ha-announce-0 ClusterIP 172.29.0.101 <none> 6379/TCP,26379/TCP 159m
redis-ha-announce-1 ClusterIP 172.29.1.79 <none> 6379/TCP,26379/TCP 159m
redis-ha-announce-2 ClusterIP 172.29.1.220 <none> 6379/TCP,26379/TCP 159m
[root@shzj-o-batch-12-25 _]#
[root@shzj-o-batch-12-25 _]# redis-cli -h 172.29.3.116
172.29.3.116:6379>

Cauchy

hqq_get

kubectl run -it --rm test --image=alpine:3.7 --restart=Never -- sh -c 'until nc -z redis.kubesphere-system.svc 6379; do echo "waiting for redis"; sleep 2; done;'

hqq_get

我在容器里边试了，结果是通的

Cauchy

看下那几个没起来的pod分布在哪些节点，然后在那些节点上的pod中测试下这个redis服务是否可通。

hqq_get

测试都是通的，我觉得不是网络问题

hqq_get

[root@shzj-o-batch-12-25 _]# kubectl logs -f -n kubesphere-system ks-installer-74757f9779-b6h77
2020-09-02T00:27:12+08:00 INFO : shell-operator latest
2020-09-02T00:27:12+08:00 INFO : HTTP SERVER Listening on 0.0.0.0:9115
2020-09-02T00:27:12+08:00 INFO : Use temporary dir: /tmp/shell-operator
2020-09-02T00:27:12+08:00 INFO : Initialize hooks manager …
2020-09-02T00:27:12+08:00 INFO : Search and load hooks …
2020-09-02T00:27:12+08:00 INFO : Load hook config from ‘/hooks/kubesphere/installRunner.py’
2020-09-02T00:27:15+08:00 INFO : Load hook config from ‘/hooks/kubesphere/schedule.sh’
2020-09-02T00:27:15+08:00 INFO : Initializing schedule manager …
2020-09-02T00:27:15+08:00 INFO : KUBE Init Kubernetes client
2020-09-02T00:27:15+08:00 INFO : KUBE-INIT Kubernetes client is configured successfully
2020-09-02T00:27:15+08:00 INFO : MAIN: run main loop
2020-09-02T00:27:15+08:00 INFO : MAIN: add onStartup tasks
2020-09-02T00:27:15+08:00 INFO : Running schedule manager …
2020-09-02T00:27:15+08:00 INFO : MSTOR Create new metric shell_operator_tasks_queue_length
2020-09-02T00:27:15+08:00 INFO : MSTOR Create new metric shell_operator_live_ticks
2020-09-02T00:27:15+08:00 INFO : QUEUE add all HookRun@OnStartup
2020-09-02T00:27:15+08:00 ERROR : error getting GVR for kind ‘ClusterConfiguration’: unable to retrieve the complete list of server APIs: admissionregistration.k8s.io/v1: the server could not find the requested resource, apiextensions.k8s.io/v1: the server could not find the requested resource, discovery.k8s.io/v1beta1: the server could not find the requested resource
2020-09-02T00:27:15+08:00 ERROR : Enable kube events for hooks error: unable to retrieve the complete list of server APIs: admissionregistration.k8s.io/v1: the server could not find the requested resource, apiextensions.k8s.io/v1: the server could not find the requested resource, discovery.k8s.io/v1beta1: the server could not find the requested resource
2020-09-02T00:27:18+08:00 INFO : TASK_RUN Exit: program halts.

为什么有时候重启ks-install的时候就会报这个错误

hqq_get

麻烦帮忙看一下，谢谢大佬

hongming

hqq_get K8s是什么版本?

Cauchy

kubectl get apiservice 看看

hqq_get

hongming K8S版本为v1.18.8

hqq_get

[root@shzj-o-batch-12-25 _]# kubectl get version
error: the server doesn’t have a resource type “version”

[root@shzj-o-batch-12-25 ~]# kubectl version 
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:12:48Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"9f2892aab98fe339f3bd70e3c470144299398ace", GitTreeState:"clean", BuildDate:"2020-08-13T16:04:18Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
[root@shzj-o-batch-12-25 ~]# 
[root@shzj-o-batch-12-25 ~]# 
[root@shzj-o-batch-12-25 ~]# kubectl get apiservice
NAME                                        SERVICE   AVAILABLE   AGE
v1.                                         Local     True        505d
v1.admissionregistration.k8s.io             Local     True        39h
v1.apiextensions.k8s.io                     Local     True        39h
v1.apps                                     Local     True        505d
v1.authentication.k8s.io                    Local     True        505d
v1.authorization.k8s.io                     Local     True        505d
v1.autoscaling                              Local     True        505d
v1.batch                                    Local     True        505d
v1.coordination.k8s.io                      Local     True        505d
v1.crd.projectcalico.org                    Local     True        31m
v1.monitoring.coreos.com                    Local     True        31m
v1.networking.k8s.io                        Local     True        505d
v1.rbac.authorization.k8s.io                Local     True        505d
v1.scheduling.k8s.io                        Local     True        505d
v1.storage.k8s.io                           Local     True        505d
v1.volumesnapshot.external-storage.k8s.io   Local     True        31m
v1alpha1.appcatalog.appscode.com            Local     True        31m
v1alpha1.catalog.kubedb.com                 Local     True        31m
v1alpha1.cluster.kubesphere.io              Local     True        31m
v1alpha1.devops.kubesphere.io               Local     True        31m
v1alpha1.installer.kubesphere.io            Local     True        31m
v1alpha1.kubedb.com                         Local     True        31m
v1alpha1.monitoring.kubesphere.io           Local     True        31m
v1alpha1.network.kubesphere.io              Local     True        31m
v1alpha1.notification.kubesphere.io         Local     True        31m
v1alpha1.openebs.io                         Local     True        31m
v1alpha1.pingcap.com                        Local     True        31m
v1alpha1.storage.kubesphere.io              Local     True        31m
v1alpha1.tenant.kubesphere.io               Local     True        31m
v1alpha2.iam.kubesphere.io                  Local     True        31m
v1alpha2.servicemesh.kubesphere.io          Local     True        31m
v1alpha2.tenant.kubesphere.io               Local     True        31m
v1alpha3.devops.kubesphere.io               Local     True        31m
v1beta1.admissionregistration.k8s.io        Local     True        505d
v1beta1.apiextensions.k8s.io                Local     True        505d
v1beta1.app.k8s.io                          Local     True        31m
v1beta1.authentication.k8s.io               Local     True        505d
v1beta1.authorization.k8s.io                Local     True        505d
v1beta1.batch                               Local     True        505d
v1beta1.certificates.k8s.io                 Local     True        505d
v1beta1.coordination.k8s.io                 Local     True        505d
v1beta1.discovery.k8s.io                    Local     True        39h
v1beta1.events.k8s.io                       Local     True        505d
v1beta1.extensions                          Local     True        505d
v1beta1.networking.k8s.io                   Local     True        505d
v1beta1.node.k8s.io                         Local     True        505d
v1beta1.policy                              Local     True        505d
v1beta1.rbac.authorization.k8s.io           Local     True        505d
v1beta1.scheduling.k8s.io                   Local     True        505d
v1beta1.snapshot.storage.k8s.io             Local     True        31m
v1beta1.storage.k8s.io                      Local     True        505d
v2beta1.autoscaling                         Local     True        505d
v2beta2.autoscaling                         Local     True        505d

rayzhou2017

hqq_get K8s 官网建议不要用1.18.7和1.18.8，用1.18.6

hongming

hqq_get kubectl get crds ，看看有没有clusterconfigurations.installer.kubesphere.io

hqq_get

rayzhou2017 确定是这个问题吗

rayzhou2017

hqq_get 我不知道，官网很明确用黑体字说了, Do not use these releases

hqq_get

hongming

hqq_get

hongming 是有的

Cauchy

hqq_get
k8s集群是否进行过升级？集群中有几个master呢？每个master的kube-apiserver版本是否一致？