• 开发

  • KubeSphere3.3本地开发跑不起来

在服务器有6台虚拟机，部署了集群版KubeSphere3.3.2，本地想要连上服务器进行开发，但是本地启动不起来，一直报错。

W0330 14:28:50.637818   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:28:50.637909   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:28:50.640605   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:28:50.640656   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:28:50.640604   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:28:50.640688   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
……（不断重复打印上面的错误）

KubeSphere安装是参考https://kubesphere.io/zh/docs/v3.3/installing-on-linux/high-availability-configurations/internal-ha-configuration/，安装后能够正常使用，本地可以通过浏览器访问，也可以通过本地部署的前端访问，本地的kubectl和telepresence也是能够访问到服务器。
但是本地开发就启动不起来，开发流程是参考https://github.com/kubesphere/community/blob/master/developer-guide/development/quickstart.md，在本地启动的那一步，无论是通过以下哪种方式都启动不起来

go run cmd/ks-apiserver/apiserver.go
telepresence --namespace  kubesphere-system --swap-deployment ks-apiserver --run go run apiserver.go

启动日志

❯ telepresence --namespace  kubesphere-system --swap-deployment ks-apiserver --run go run apiserver.go
Legacy Telepresence command used
Command roughly translates to the following in Telepresence:
telepresence intercept ks-apiserver --namespace kubesphere-system -- go run apiserver.go
running...
Using Deployment ks-apiserver
   Intercept name    : ks-apiserver-kubesphere-system
   State             : ACTIVE
   Workload kind     : Deployment
   Destination       : 127.0.0.1:8080
   Volume Mount Error: sshfs is not installed on your local machine
   Intercepting      : all TCP requests
W0330 14:28:48.466150   78581 metricsserver.go:238] Metrics API not available.
I0330 14:28:48.582618   78581 interface.go:50] start helm repo informer
I0330 14:28:49.021796   78581 apiserver.go:434] Start cache objects
W0330 14:28:49.282026   78581 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
W0330 14:28:49.395478   78581 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
W0330 14:28:50.458664   78581 apiserver.go:419] resource notification.kubesphere.io/v2beta2, Resource=routers not exists in the cluster
W0330 14:28:50.458737   78581 apiserver.go:419] resource notification.kubesphere.io/v2beta2, Resource=silences not exists in the cluster
W0330 14:28:50.637818   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:28:50.637909   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:28:50.640605   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:28:50.640656   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:28:50.640604   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:28:50.640688   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
W0330 14:28:51.617219   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:28:51.617268   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:28:52.025951   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:28:52.026044   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:28:52.218329   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:28:52.218396   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
W0330 14:28:54.005926   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:28:54.005965   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:28:54.075707   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:28:54.075742   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:28:54.586197   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:28:54.586402   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
W0330 14:28:58.771803   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:28:58.771846   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:28:59.202958   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:28:59.203024   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:28:59.216301   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:28:59.216373   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
W0330 14:29:06.226963   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
E0330 14:29:06.227001   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.ClusterRuleGroup: failed to list *v2beta1.ClusterRuleGroup: the server could not find the requested resource (get clusterrulegroups.alerting.kubesphere.io)
W0330 14:29:08.614509   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
E0330 14:29:08.614551   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.GlobalRuleGroup: failed to list *v2beta1.GlobalRuleGroup: the server could not find the requested resource (get globalrulegroups.alerting.kubesphere.io)
W0330 14:29:11.343240   78581 reflector.go:424] kubesphere/pkg/client/informers/externalversions/factory.go:129: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
E0330 14:29:11.343347   78581 reflector.go:140] kubesphere/pkg/client/informers/externalversions/factory.go:129: Failed to watch *v2beta1.RuleGroup: failed to list *v2beta1.RuleGroup: the server could not find the requested resource (get rulegroups.alerting.kubesphere.io)
……（不断重复打印上面的错误）

服务器环境

系统：CentOS 7.9.2009 Kernel 5.4.233-1.el7.elrepo.x86_64
k8s：1.23.10

本地环境

系统：macOS Ventura 13.2.1
k8s：1.23.10

hosts（服务器没有外网IP，本地需要使用Pritunl通过VPN访问）

192.168.0.62 lb.kubesphere.local
10.233.7.99 redis.kubesphere-system.svc

kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://lb.kubesphere.local:6443
  name: cluster.local
contexts:
- context:
    cluster: cluster.local
    user: kubernetes-admin
  name: kubernetes-admin@cluster.local
current-context: kubernetes-admin@cluster.local
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

kubesphere.yaml

kubernetes:
  kubeconfig: "/Users/hld/.kube/config"
  master: https://lb.kubesphere.local:6443
  $qps: 1e+06
  burst: 1000000
authentication:
  authenticateRateLimiterMaxTries: 10
  authenticateRateLimiterDuration: 10m0s
  loginHistoryRetentionPeriod: 168h
  maximumClockSkew: 10s
  multipleLogin: True
  kubectlImage: registry.cn-beijing.aliyuncs.com/kubesphereio/kubectl:v1.22.0
  jwtSecret: "n1TIQQYpMMluTmRLEeQL77Z2UVUpiYqU"
  oauthOptions:
    clients:
    - name: kubesphere
      secret: kubesphere
      redirectURIs:
      - '*'
ldap:
  host: openldap.kubesphere-system.svc:389
  managerDN: cn=admin,dc=kubesphere,dc=io
  managerPassword: admin
  userSearchBase: ou=Users,dc=kubesphere,dc=io
  groupSearchBase: ou=Groups,dc=kubesphere,dc=io
cache:
  type: redis
  options:
    host: redis.kubesphere-system.svc
    port: 6379
    password: nxuAiAughrRuKbD
    db: 0
s3:
  endpoint: http://minio.kubesphere-system.svc:9000
  region: us-east-1
  disableSSL: True
  forcePathStyle: True
  accessKeyID: openpitrixminioaccesskey
  secretAccessKey: openpitrixminiosecretkey
  bucket: s2i-binaries
network:
  ippoolType: none
  weaveScopeHost: weave-scope-app.weave
devops:
  host: http://devops-jenkins.kubesphere-devops-system.svc/
  username: admin
  password: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJlbWFpbCI6ImFkbWluQGt1YmVzcGhlcmUuaW8iLCJ1c2VybmFtZSI6ImFkbWluIiwidG9rZW5fdHlwZSI6InN0YXRpY190b2tlbiJ9.
  maxConnections: 100
  endpoint: http://devops-apiserver.kubesphere-devops-system:9090
servicemesh:
  istioPilotHost: http://istiod.istio-system.svc:8080/version
  jaegerQueryHost: http://jaeger-query.istio-system.svc:16686
  servicemeshPrometheusHost: http://prometheus-k8s.kubesphere-monitoring-system.svc:9090
  kialiQueryHost: http://kiali.istio-system:20001
openpitrix:
  s3:
    endpoint: http://minio.kubesphere-system.svc:9000
    region: us-east-1
    disableSSL: True
    forcePathStyle: True
    accessKeyID: openpitrixminioaccesskey
    secretAccessKey: openpitrixminiosecretkey
    bucket: app-store
multicluster:
  clusterRole: none
monitoring:
  endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
  enableGPUMonitoring: false
gpu:
  kinds:
  - resourceName: nvidia.com/gpu
    resourceType: GPU
    default: True
notification:
  endpoint: http://notification-manager-svc.kubesphere-monitoring-system.svc:19093
logging:
  host: http://elasticsearch-logging-data.kubesphere-logging-system.svc:9200
  indexPrefix: ks-logstash-log
alerting:
  prometheusEndpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
  thanosRulerEndpoint: http://thanos-ruler-operated.kubesphere-monitoring-system.svc:10902
  thanosRuleResourceLabels: thanos-ruler=kubesphere,role=alert-rules
terminal:
  image: registry.cn-beijing.aliyuncs.com/kubesphereio/alpine:3.14
  timeout: 600
gateway:
  watchesPath: /var/helm-charts/watches.yaml
  repository: registry.cn-beijing.aliyuncs.com/kubesphereio/nginx-ingress-controller
  tag: v1.1.0
  namespace: kubesphere-controls-system
16 天 后

之前用的是master分支，改成用release-3.3的分支就可以了……

    hld0763

    我从github上下载的v3.3.2的代码，结果build镜像之后跑起来提示版本是3.0.0的。。
    请问你们有遇到过这个问题吗

        felix_liban 你试下本地直接运行看看吧，我build镜像之后还没更新到服务器上，本地用Telepresence也有问题，所以看不到效果……

          写了个解决办法见https://kubesphere.io/forum/d/22104-m1mac-teng-xun-yun-qing-liang-zhu-ji-2c4gcong-ling-bu-shu-zhi-nan