在使用kk安装的时候 pod calico-node 无法启动

bill-byteArk · 2020年8月26日

os : ubuntu 18.0.6

启动命令如下

./kk create cluster -f config-sample.yaml

config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha1
kind: Cluster
metadata:
  name: config-sample
spec:
  hosts:
  - {name: node1, address: 10.10.0.14, internalAddress: 10.10.0.14, user: root, password: XXXX}
  - {name: node2, address: 10.10.0.166, internalAddress: 10.10.0.166, user: root, password: XXX}
 # - {name: node3, address: 10.10.0.11, internalAddress: 10.10.0.11, user: root, password: XXXX}         
  roleGroups:
    etcd:
    - node1
    master:
    - node1
    worker:
    - node[1:2]
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    address: ""
    port: "6443"
  kubernetes:
    version: v1.18.6
    imageRepo: kubesphere
    clusterName: cluster.local
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
  registry:
    registryMirrors: []
    insecureRegistries: []
  storage:
    defaultStorageClass: localVolume
    localVolume:
      storageClassName: local


---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
  name: ks-installer
  namespace: kubesphere-system
  labels:
    version: v3.0.0
spec:
  local_registry: ""
  persistence:
    storageClass: ""
  authentication:
    jwtSecret: ""
  etcd:
    monitoring: true
    endpointIps: localhost
    port: 2379
    tlsEnable: true
  common:
    es:
      elasticsearchDataVolumeSize: 20Gi
      elasticsearchMasterVolumeSize: 4Gi
      elkPrefix: logstash
      logMaxAge: 7
    mysqlVolumeSize: 20Gi
    minioVolumeSize: 20Gi
    etcdVolumeSize: 20Gi
    openldapVolumeSize: 2Gi
    redisVolumSize: 2Gi
  console:
    enableMultiLogin: false  # enable/disable multi login
    port: 30880
  alerting:
    enabled: false
  auditing:
    enabled: false
  devops:
    enabled: false
    jenkinsMemoryLim: 2Gi
    jenkinsMemoryReq: 1500Mi
    jenkinsVolumeSize: 8Gi
    jenkinsJavaOpts_Xms: 512m
    jenkinsJavaOpts_Xmx: 512m
    jenkinsJavaOpts_MaxRAM: 2g
  events:
    enabled: false
    ruler:
      enabled: true
      replicas: 2
  logging:
    enabled: false
    logsidecarReplicas: 2
  metrics_server:
    enabled: true
  monitoring:
    prometheusMemoryRequest: 400Mi
    prometheusVolumeSize: 20Gi
  multicluster:
    clusterRole: host  # host | member | none
  networkpolicy:
    enabled: false
  notification:
    enabled: false
  openpitrix:
    enabled: false
  servicemesh:
    enabled: false

kubectl get po -A
NAMESPACE                      NAME                                               READY   STATUS    RESTARTS   AGE
kube-federation-system         kubefed-admission-webhook-66bbdff47d-tdlvb         1/1     Running   0          4h5m
kube-federation-system         kubefed-controller-manager-54b459dfc4-qmlsj        1/1     Running   0          4h5m
kube-system                    calico-kube-controllers-578894d4cd-6lfvg           1/1     Running   0          4h10m
kube-system                    calico-node-48mw7                                  0/1     Running   0          4h10m
kube-system                    calico-node-tjf4v                                  0/1     Running   0          4h10m
kube-system                    coredns-6dd6674597-llfs7                           1/1     Running   0          4h10m
kube-system                    coredns-6dd6674597-s59qz                           1/1     Running   0          4h10m
kube-system                    kube-apiserver-node1                               1/1     Running   0          4h10m
kube-system                    kube-controller-manager-node1                      1/1     Running   0          4h10m
kube-system                    kube-proxy-2xs2s                                   1/1     Running   0          4h10m
kube-system                    kube-proxy-rm8bt                                   1/1     Running   0          4h10m
kube-system                    kube-scheduler-node1                               1/1     Running   0          4h10m
kube-system                    metrics-server-f47758d8f-9hcrx                     1/1     Running   0          4h7m
kube-system                    nodelocaldns-75vf4                                 1/1     Running   0          4h10m
kube-system                    nodelocaldns-9cpjl                                 1/1     Running   0          4h10m
kube-system                    openebs-localpv-provisioner-84446d4bd7-8l6k5       1/1     Running   0          4h8m
kube-system                    openebs-ndm-6v5dr                                  1/1     Running   0          4h8m
kube-system                    openebs-ndm-jgchw                                  1/1     Running   0          4h8m
kube-system                    openebs-ndm-operator-6456dc9db-qsnbs               1/1     Running   1          4h8m
kube-system                    snapshot-controller-0                              1/1     Running   0          4h6m
kubesphere-controls-system     default-http-backend-857d7b6856-qg4j7              1/1     Running   0          4h5m
kubesphere-controls-system     kubectl-admin-58f985d8f6-j2vhm                     1/1     Running   0          165m
kubesphere-monitoring-system   alertmanager-main-0                                2/2     Running   0          3h47m
kubesphere-monitoring-system   kube-state-metrics-95c974544-bznv8                 3/3     Running   0          4h4m
kubesphere-monitoring-system   node-exporter-26fcw                                2/2     Running   0          4h4m
kubesphere-monitoring-system   node-exporter-dwzrq                                2/2     Running   0          4h4m
kubesphere-monitoring-system   notification-manager-deployment-7c8df68d94-ftdfm   1/1     Running   0          3h48m
kubesphere-monitoring-system   notification-manager-operator-7b78b76c96-rr4b8     2/2     Running   0          4h4m
kubesphere-monitoring-system   prometheus-k8s-0                                   3/3     Running   1          3h47m
kubesphere-monitoring-system   prometheus-operator-84d58bf775-tgpz2               2/2     Running   0          4h4m
kubesphere-system              ks-apiserver-5b88998cd5-s66jj                      1/1     Running   1          3h47m
kubesphere-system              ks-console-b4df86d6f-gznfg                         1/1     Running   0          4h5m
kubesphere-system              ks-controller-manager-595677fd9c-5g56r             1/1     Running   0          3h47m
kubesphere-system              ks-installer-74757f9779-x2zmf                      1/1     Running   0          4h8m
kubesphere-system              openldap-0                                         1/1     Running   0          4h6m
kubesphere-system              redis-644bc597b9-782sl                             1/1     Running   0          4h6m
kubesphere-system              tower-5f64d8bbd8-bxtjm                             1/1     Running   0          4h5m

$ kubectl describe pod  calico-node-48mw7 -n kube-system
Name:                 calico-node-48mw7
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 node2/10.10.0.166
Start Time:           Wed, 26 Aug 2020 05:58:47 +0000
Labels:               controller-revision-hash=fc89bc6dd
                      k8s-app=calico-node
                      pod-template-generation=1
Annotations:          <none>
Status:               Running
IP:                   10.10.0.166
IPs:
  IP:           10.10.0.166
Controlled By:  DaemonSet/calico-node
Init Containers:
  upgrade-ipam:
    Container ID:  docker://714b95ec298c4fab9490d5b10f770e7a6ec8eeea3c3e0a0a5041695b3c17310a
    Image:         calico/cni:v3.15.1
    Image ID:      docker-pullable://calico/cni@sha256:b86711626e68a5298542efc52e2bd3c64e212a635359b3a017ee0a8cd47b0c1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /opt/cni/bin/calico-ipam
      -upgrade
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 26 Aug 2020 05:58:50 +0000
      Finished:     Wed, 26 Aug 2020 05:58:50 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      KUBERNETES_NODE_NAME:        (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:  <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
    Mounts:
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/lib/cni/networks from host-local-net-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-j29zs (ro)
  install-cni:
    Container ID:  docker://0af3f84231f675932e46c80a57237c413e65fd427e881d18e21796784debbf23
    Image:         calico/cni:v3.15.1
    Image ID:      docker-pullable://calico/cni@sha256:b86711626e68a5298542efc52e2bd3c64e212a635359b3a017ee0a8cd47b0c1e
    Port:          <none>
    Host Port:     <none>
    Command:
      /install-cni.sh
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 26 Aug 2020 05:58:51 +0000
      Finished:     Wed, 26 Aug 2020 05:58:52 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      CNI_CONF_NAME:         10-calico.conflist
      CNI_NETWORK_CONFIG:    <set to the key 'cni_network_config' of config map 'calico-config'>  Optional: false
      KUBERNETES_NODE_NAME:   (v1:spec.nodeName)
      CNI_MTU:               <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      SLEEP:                 false
    Mounts:
      /host/etc/cni/net.d from cni-net-dir (rw)
      /host/opt/cni/bin from cni-bin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-j29zs (ro)
  flexvol-driver:
    Container ID:   docker://0d18f4c02e1b2e9de9eb6ff1b59955ac6ec99d624c40f49bc4829e20441a054f
    Image:          calico/pod2daemon-flexvol:v3.15.1
    Image ID:       docker-pullable://calico/pod2daemon-flexvol@sha256:c2c6bbe3e10d24a01d6f3fd5b446cce6cf3e37f943960263bf6e5c458ecdeb52
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 26 Aug 2020 05:58:52 +0000
      Finished:     Wed, 26 Aug 2020 05:58:52 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /host/driver from flexvol-driver-host (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-j29zs (ro)
Containers:
  calico-node:
    Container ID:   docker://02f07a11f2490c56904ded628b857255c40167da556a9af8f7e9a897f4bde764
    Image:          calico/node:v3.15.1
    Image ID:       docker-pullable://calico/node@sha256:b386769a293d180cb6ee208c8594030128a0810b286a93ae897a231ef247afa8
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Wed, 26 Aug 2020 05:58:53 +0000
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      250m
    Liveness:   exec [/bin/calico-node -felix-live -bird-live] delay=10s timeout=1s period=10s #success=1 #failure=6
    Readiness:  exec [/bin/calico-node -felix-ready -bird-ready] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      DATASTORE_TYPE:                     kubernetes
      WAIT_FOR_DATASTORE:                 true
      NODENAME:                            (v1:spec.nodeName)
      CALICO_NETWORKING_BACKEND:          <set to the key 'calico_backend' of config map 'calico-config'>  Optional: false
      CLUSTER_TYPE:                       k8s,bgp
      IP:                                 autodetect
      CALICO_IPV4POOL_IPIP:               Always
      CALICO_IPV4POOL_VXLAN:              Never
      FELIX_IPINIPMTU:                    <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_VXLANMTU:                     <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      FELIX_WIREGUARDMTU:                 <set to the key 'veth_mtu' of config map 'calico-config'>  Optional: false
      CALICO_IPV4POOL_CIDR:               10.233.64.0/18
      CALICO_IPV4POOL_BLOCK_SIZE:         24
      CALICO_DISABLE_FILE_LOGGING:        true
      FELIX_DEFAULTENDPOINTTOHOSTACTION:  ACCEPT
      FELIX_IPV6SUPPORT:                  false
      FELIX_LOGSEVERITYSCREEN:            info
      FELIX_HEALTHENABLED:                true
    Mounts:
      /lib/modules from lib-modules (ro)
      /run/xtables.lock from xtables-lock (rw)
      /var/lib/calico from var-lib-calico (rw)
      /var/run/calico from var-run-calico (rw)
      /var/run/nodeagent from policysync (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from calico-node-token-j29zs (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  lib-modules:
    Type:          HostPath (bare host directory volume)
    Path:          /lib/modules
    HostPathType:  
  var-run-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/calico
    HostPathType:  
  var-lib-calico:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/calico
    HostPathType:  
  xtables-lock:
    Type:          HostPath (bare host directory volume)
    Path:          /run/xtables.lock
    HostPathType:  FileOrCreate
  cni-bin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /opt/cni/bin
    HostPathType:  
  cni-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/cni/net.d
    HostPathType:  
  host-local-net-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/cni/networks
    HostPathType:  
  policysync:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/nodeagent
    HostPathType:  DirectoryOrCreate
  flexvol-driver-host:
    Type:          HostPath (bare host directory volume)
    Path:          /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
    HostPathType:  DirectoryOrCreate
  calico-node-token-j29zs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  calico-node-token-j29zs
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     :NoSchedule
                 :NoExecute
                 CriticalAddonsOnly
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:
  Type     Reason     Age                      From            Message
  ----     ------     ----                     ----            -------
  Warning  Unhealthy  3m47s (x1462 over 4h7m)  kubelet, node2  (combined from similar events): Readiness probe failed: 2020-08-26 10:03:57.161 [INFO][16418] confd/health.go 180: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.21.0.1

Quinton · 2020年8月26日

me too…

Cauchy · 2020年8月26日

看下calico的日志

bill-byteArk · 2020年8月26日

Cauchy 感谢回复

kubectl logs  calico-node-48mw7 -n kube-system
...
2020-08-26 14:47:02.074 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:02.074 [INFO][52] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family="inet"
2020-08-26 14:47:02.074 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:02.075 [INFO][52] felix/ipsets.go 306: Resyncing ipsets with dataplane. family="inet"
2020-08-26 14:47:02.083 [INFO][52] felix/ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=8.299009ms
2020-08-26 14:47:02.084 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=9.470970000000001
2020-08-26 14:47:12.340 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:12.340 [INFO][52] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family="inet"
2020-08-26 14:47:12.340 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:12.340 [INFO][52] felix/ipsets.go 306: Resyncing ipsets with dataplane. family="inet"
2020-08-26 14:47:12.347 [INFO][52] felix/ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=6.452743ms
2020-08-26 14:47:12.347 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=7.138739
2020-08-26 14:47:23.015 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:23.015 [INFO][52] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family="inet"
2020-08-26 14:47:23.015 [INFO][52] felix/ipsets.go 306: Resyncing ipsets with dataplane. family="inet"
2020-08-26 14:47:23.015 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:23.022 [INFO][52] felix/ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=6.842824ms
2020-08-26 14:47:23.023 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=7.43691
2020-08-26 14:47:26.655 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:26.655 [INFO][52] felix/route_table.go 399: Queueing a resync of routing table. ifaceRegex="^cali.*" ipVersion=0x4
2020-08-26 14:47:26.655 [INFO][52] felix/wireguard.go 534: Queueing a resync of wireguard configuration
2020-08-26 14:47:26.655 [INFO][52] felix/route_table.go 399: Queueing a resync of routing table. ifaceRegex="^wireguard.cali$" ipVersion=0x4
2020-08-26 14:47:26.656 [INFO][52] felix/route_rule.go 172: Queueing a resync of routing rules. ipVersion=4
2020-08-26 14:47:26.656 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:26.662 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=7.07819
2020-08-26 14:47:27.172 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:27.172 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:27.172 [INFO][52] felix/table.go 877: Invalidating dataplane cache ipVersion=0x4 reason="refresh timer" table="filter"
2020-08-26 14:47:27.173 [INFO][52] felix/table.go 564: Loading current iptables state and checking it is correct. ipVersion=0x4 table="filter"
2020-08-26 14:47:27.180 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=7.9616370000000005
2020-08-26 14:47:33.585 [INFO][52] felix/int_dataplane.go 1245: Applying dataplane updates
2020-08-26 14:47:33.585 [INFO][52] felix/ipsets.go 223: Asked to resync with the dataplane on next update. family="inet"
2020-08-26 14:47:33.585 [INFO][52] felix/wireguard.go 578: Wireguard is not enabled
2020-08-26 14:47:33.585 [INFO][52] felix/ipsets.go 306: Resyncing ipsets with dataplane. family="inet"
2020-08-26 14:47:33.588 [INFO][52] felix/ipsets.go 356: Finished resync family="inet" numInconsistenciesFound=0 resyncDuration=2.604319ms
2020-08-26 14:47:33.588 [INFO][52] felix/int_dataplane.go 1259: Finished applying updates to dataplane. msecToApply=2.9547839999999996

bill-byteArk · 2020年8月27日

我在阿里云集群上（1master+1worker）使用上面同样的配置文件部署成功，两者唯一的区别是上面的集群没有外网IP，而阿里云集群有外网IP

bill-byteArk · 2020年8月27日

还有阿里云部署的 kk 二进制文件是在本地编译，而非直接下载

3344089386 · 2020年12月24日

我也遇到了，有大神解决下吗
Cauchy

Cauchy · 2020年12月24日

344089386

机器是否为多网卡？机器之间是否存在防火墙或者安全组，如果机器挂了公网ip，应该是会默认挂一个防火墙或者安全组。

在使用kk安装的时候 pod calico-node 无法启动

bill-byteArkK零S

Quinton

CauchyK零SK壹S

bill-byteArkK零S

bill-byteArkK零S

bill-byteArkK零S

3344089386K零S

CauchyK零SK壹S