安装完成，验证安装结果失败

cffycls

主机：
阿里云 2核4g 40g ubuntu18.4lts
端口确认开放了，其它检测

+---------------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+
| name                | sudo | curl | openssl | ebtables | socat | ipset | conntrack | docker | nfs client | ceph client | glusterfs client | time         |
+---------------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+
| xx-temp-server-001 | y    | y    | y       | y        | y     | y     | y         | y      | y          |             |                  | CST 14:00:58 |
+---------------------+------+------+---------+----------+-------+-------+-----------+--------+------------+-------------+------------------+--------------+

验证安装结果失败：
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

Feynman

cffycls 你看一下 installer 的日志：

kubectl logs -n kubesphere-system $(kubectl get pod -n kubesphere-system -l app=ks-install -o jsonpath='{.items[0].metadata.name}') -f

cffycls

Feynman 日志只有
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver-kubelet-client, verb=get, resource=nodes, subresource=proxy)

安装命令使用的是 ./kk create cluster --with-kubesphere v3.0.0

网络需求最下方的
ipip IPENCAP / IPIP allow calico needs to allow the ipip protocol
这个不知道怎么开放，没做处理

Cauchy

cffycls

kubectl get node -o wide
kubectl get pod -A  -o wide

Feynman

@Cauchy @Forest-L 帮忙看看，看起来还是防火墙的问题吧？

cffycls

Cauchy

root@xx-temp-server-001:~# kubectl get node -o wide
NAME                  STATUS   ROLES           AGE     VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
xx-temp-server-001   Ready    master,worker   2d19h   v1.17.9   172.19.69.100   <none>        Ubuntu 18.04.5 LTS   4.15.0-124-generic   docker://19.3.8
root@xx-temp-server-001:~# kubectl get pod -A  -o wide
NAMESPACE           NAME                                           READY   STATUS              RESTARTS   AGE     IP              NODE                  NOMINATED NODE   READINESS GATES
kube-system         calico-kube-controllers-59d85c5c84-2g2xl       0/1     ContainerCreating   0          2d19h   <none>          xx-temp-server-001   <none>           <none>
kube-system         calico-node-6rxlm                              0/1     CrashLoopBackOff    785        2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         coredns-74d59cc5c6-czvnn                       0/1     ContainerCreating   0          2d19h   <none>          xx-temp-server-001   <none>           <none>
kube-system         coredns-985b9f698-bfg72                        0/1     ContainerCreating   0          2d      <none>          xx-temp-server-001   <none>           <none>
kube-system         coredns-985b9f698-w4cvl                        0/1     ContainerCreating   0          2d      <none>          xx-temp-server-001   <none>           <none>
kube-system         kube-apiserver-xx-temp-server-001             1/1     Running             0          2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         kube-controller-manager-xx-temp-server-001    1/1     Running             0          2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         kube-proxy-dmg8d                               1/1     Running             0          2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         kube-scheduler-xx-temp-server-001             1/1     Running             0          2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         metrics-server-5ddd98b7f9-5fwtm                0/1     CrashLoopBackOff    758        2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         nodelocaldns-ww5j4                             1/1     Running             0          2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         openebs-localpv-provisioner-84956ddb89-tqnr5   0/1     Error               1          2d19h   <none>          xx-temp-server-001   <none>           <none>
kube-system         openebs-ndm-58w6w                              0/1     CrashLoopBackOff    508        2d19h   172.19.69.100   xx-temp-server-001   <none>           <none>
kube-system         openebs-ndm-operator-6896cbf7b8-kqh8d          0/1     ContainerCreating   1          2d19h   <none>          xx-temp-server-001   <none>           <none>
kubesphere-system   ks-installer-85854b8c8-sgbzb                   0/1     ContainerCreating   0          2d19h   <none>          xx-temp-server-001   <none>           <none>
root@xx-temp-server-001:~#

Cauchy

cffycls
calico不正常，导致集群不能正常工作。
这个操作系统做过什么定制化才做吗？可以看下那个calico-node为啥运行失败。

cffycls

Cauchy 装过宝塔，端口也是阿里云+宝塔双放行，其它没有操作

duanjiong

cffycls describe calico-node或者看下calico-node的log

cffycls

duanjiong

root@xx-temp-server-001:~/tmp# kubectl describe pod calico-node-6rxlm --namespace=kube-system
Name:                 calico-node-6rxlm
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 xx-temp-server-001/172.19.69.100
Start Time:           Fri, 25 Dec 2020 21:33:59 +0800
Labels:               controller-revision-hash=6b95d9c9d
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   172.19.69.100
IPs:
  IP:           172.19.69.100
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker://42e9ef40292ba8aa1a04aade6b19289ea093ef99fa5c4fc1120ffe69a069327a
    Image:         calico/cni:v3.15.1
    Image ID:      docker-pullable://calico/cni@sha256:b86711626e68a5298542efc52e2bd3c64e212a635359b3a017ee0a8cd47b0c1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Dec 2020 16:33:25 +0800
      Finished:     Sat, 26 Dec 2020 16:33:25 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-6gbmk (ro)
  install-cni:
    Container ID:  docker://b46dd0a02441942ef6bb6b667dff0a36109423f52aa61fc318e7bc2d8109b742
    Image:         calico/cni:v3.15.1
    Image ID:      docker-pullable://calico/cni@sha256:b86711626e68a5298542efc52e2bd3c64e212a635359b3a017ee0a8cd47b0c1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Dec 2020 16:33:26 +0800
      Finished:     Sat, 26 Dec 2020 16:33:27 +0800
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-6gbmk (ro)
  flexvol-driver:
    Container ID:   docker://a3c78aee0b563fd4487cdc23a69bde2fee9bf03ea0d86dea5258317a43b123d2
    Image:          calico/pod2daemon-flexvol:v3.15.1
    Image ID:       docker-pullable://calico/pod2daemon-flexvol@sha256:c2c6bbe3e10d24a01d6f3fd5b446cce6cf3e37f943960263bf6e5c458ecdeb52
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 26 Dec 2020 16:33:49 +0800
      Finished:     Sat, 26 Dec 2020 16:33:49 +0800
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-6gbmk (ro)
Containers:
  calico-node:
    Container ID:   docker://a22cb8f728bbb3eb2bbc8228fb0db8cdbfb71e35a58d8095a0d3c7a27cba4aed
    Image:          calico/node:v3.15.1
    Image ID:       docker-pullable://calico/node@sha256:b386769a293d180cb6ee208c8594030128a0810b286a93ae897a231ef247afa8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Tue, 29 Dec 2020 16:11:52 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Tue, 29 Dec 2020 16:05:40 +0800
      Finished:     Tue, 29 Dec 2020 16:06:49 +0800
    Ready:          False
    Restart Count:  1167
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               10.233.64.0/18
      CALICO_IPV4POOL_BLOCK_SIZE:         24
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-6gbmk (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-6gbmk:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-6gbmk
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     :NoSchedule
                 :NoExecute
                 CriticalAddonsOnly
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                       From                          Message
  ----     ------     ----                      ----                          -------
  Warning  Unhealthy  13m (x7045 over 2d23h)    kubelet, xx-temp-server-001  Readiness probe failed: calico/node is not ready: BIRD is not ready: Error querying BIRD: unable to connect to BIRDv4 socket: dial unix /var/run/bird/bird.ctl: connect: no such file or directory
  Warning  BackOff    3m9s (x14225 over 2d23h)  kubelet, xx-temp-server-001  Back-off restarting failed container