guoh1988K零S- 已编辑
### 1.k8s版本
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T14:00:52Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.10", GitCommit:"f3add640dbcd4f3c33a7749f38baaac0b3fe810d", GitTreeState:"clean", BuildDate:"2020-05-20T13:51:56Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}1个管理节点3个工作节点
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 22d v1.16.10
master2 Ready <none> 22d v1.16.10
master3 Ready <none> 22d v1.16.10
worker1 Ready <none> 21d v1.16.102.安装了KubeSphere 2.1.1的所有组件
helm list
NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE
elasticsearch-logging 1 Wed Nov 11 18:24:06 2020 DEPLOYED elasticsearch-1.22.1 6.7.0-0217 kubesphere-logging-system
elasticsearch-logging-curator 1 Wed Nov 11 18:24:09 2020 DEPLOYED elasticsearch-curator-1.3.3 5.5.4-0217 kubesphere-logging-system
istio 1 Wed Nov 11 18:33:32 2020 DEPLOYED istio-1.3.3 1.3.3 istio-system
istio-init 1 Wed Nov 11 18:32:48 2020 DEPLOYED istio-init-1.3.2 1.3.2 istio-system
jaeger-operator 1 Wed Nov 11 18:34:13 2020 DEPLOYED jaeger-operator-2.9.0 1.13.1 istio-system
ks-jenkins 4 Mon Nov 23 09:50:01 2020 DEPLOYED jenkins-0.19.0 2.121.3-0217 kubesphere-devops-system
ks-minio 1 Wed Nov 11 18:22:25 2020 DEPLOYED minio-2.5.16 RELEASE.2019-08-07T01-59-21Z kubesphere-system
ks-openldap 1 Wed Nov 11 18:17:00 2020 DEPLOYED openldap-ha-0.1.0 1.0 kubesphere-system
ks-openpitrix 1 Wed Nov 11 18:23:59 2020 DEPLOYED openpitrix-0.1.0 v0.4.8 openpitrix-system
logging-fluentbit-operator 1 Wed Nov 11 18:24:03 2020 DEPLOYED fluentbit-operator-0.1.0 0.1.0-0217 kubesphere-logging-system
metrics-server 1 Wed Nov 11 18:21:46 2020 DEPLOYED metrics-server-2.5.0 0.3.1-0217 kube-system
uc 1 Wed Nov 11 18:31:00 2020 DEPLOYED jenkins-update-center-0.8.0 2.1.1 kubesphere-devops-system 3.运行情况良好，有一天一个节点突然宕机，当时运行在上面的metrics-server，ks-apiserver发生迁移ks-apiserver一直无法启动
节点丢失
kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 22d v1.16.10
master2 Ready <none> 22d v1.16.10
master3 Ready <none> 22d v1.16.10
worker1 NotReady <none> 21d v1.16.10
ks-apiserver 无法启动
kubectl get pods -n kubesphere-system -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
etcd-5769d4997f-8vkxc 1/1 Running 0 3h16m 10.244.180.48 master2 <none> <none>
ks-account-78dd6486bf-s5w8x 1/1 Running 0 10d 10.244.137.69 master1 <none> <none>
ks-apigateway-764d86967d-7qrkh 1/1 Running 0 10d 10.244.137.86 master1 <none> <none>
ks-apiserver-6b75dfdf4-gjxpr 0/1 Error 1 10d 10.244.137.83 master1 <none> <none>
ks-console-7fd5b7d47-rkwmv 1/1 Running 0 10d 10.244.137.82 master1 <none> <none>
ks-controller-manager-5cd6ff58b7-5cg8p 1/1 Running 1 10d 10.244.137.68 master1 <none> <none>
ks-installer-7d9fb945c7-ld466 1/1 Running 1 20d 10.244.136.13 master3 <none> <none>
minio-845b7bd867-r762v 1/1 Running 1 20d 10.244.136.32 master3 <none> <none>
mysql-66df969d-b9dzk 1/1 Running 1 20d 10.244.136.11 master3 <none> <none>
openldap-0 1/1 Running 5 22d 10.244.137.78 master1 <none> <none>
redis-6fd6c6d6f9-jtxzs 1/1 Running 4 22d 10.244.137.75 master1 <none> <none>
### 4.查看ks-apiserver 的日志，确认是连接kube-apiserver的时候出错
kubectl logs -f ks-apiserver-6b75dfdf4-gjxpr -n kubesphere-system
W1203 10:26:44.173078 1 client_config.go:549] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1203 10:26:44.174200 1 server.go:179] Start cache objects
Error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request
Usage:
ks-apiserver [flags]
Flags:
--add-dir-header If true, adds the file directory to the header
--alsologtostderr log to standard error as well as files
--bind-address string server bind address (default "0.0.0.0")
--elasticsearch-host string ElasticSearch logging service host. KubeSphere is using elastic as log store, if this filed left blank, KubeSphere will use kubernetes builtin log API instead, and the following elastic search options will be ignored.
--elasticsearch-version string ElasticSearch major version, e.g. 5/6/7, if left blank, will detect automatically.Currently, minimum supported version is 5.x
-h, --help help for ks-apiserver
--index-prefix string Index name prefix. KubeSphere will retrieve logs against indices matching the prefix. (default "fluentbit")
--insecure-port int insecure port number (default 9090)
--istio-pilot-host string istio pilot discovery service url
--jaeger-query-host string jaeger query service url
--jenkins-host string Jenkins service host address. If left blank, means Jenkins is unnecessary.
--jenkins-max-connections int Maximum allowed connections to Jenkins. (default 100)
--jenkins-password string Password for access to Jenkins service, used pair with username.
--jenkins-username string Username for access to Jenkins service. Leave it blank if there isn't any.
--kubeconfig string Path for kubernetes kubeconfig file, if left blank, will use in cluster way.
--log-backtrace-at traceLocation when logging hits line file:N, emit a stack trace (default :0)
--log-dir string If non-empty, write log files in this directory
--log-file string If non-empty, use this log file
--log-file-max-size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
--logtostderr log to standard error instead of files (default true)
--master string Used to generate kubeconfig for downloading, if not specified, will use host in kubeconfig.
--mysql-host string MySQL service host address. If left blank, the following related mysql options will be ignored.
--mysql-max-connection-life-time duration Maximum connection life time allowed to connecto to mysql. (default 10s)
--mysql-max-idle-connections int Maximum idle connections allowed to connect to mysql. (default 100)
--mysql-max-open-connections int Maximum open connections allowed to connect to mysql. (default 100)
--mysql-password string Password for access to mysql, should be used pair with password.
--mysql-username string Username for access to mysql service.
--openpitrix-app-manager-endpoint string OpenPitrix app manager endpoint
--openpitrix-attachment-manager-endpoint string OpenPitrix attachment manager endpoint
--openpitrix-category-manager-endpoint string OpenPitrix category manager endpoint
--openpitrix-cluster-manager-endpoint string OpenPitrix cluster manager endpoint
--openpitrix-repo-indexer-endpoint string OpenPitrix repo indexer endpoint
--openpitrix-repo-manager-endpoint string OpenPitrix repo manager endpoint
--openpitrix-runtime-manager-endpoint string OpenPitrix runtime manager endpoint
--prometheus-endpoint string Prometheus service endpoint which stores KubeSphere monitoring data, if left blank, will use builtin metrics-server as data source.
--prometheus-secondary-endpoint string Prometheus secondary service endpoint, if left empty and endpoint is set, will use endpoint instead.
--s3-access-key-id string access key of s2i s3 (default "AKIAIOSFODNN7EXAMPLE")
--s3-bucket string bucket name of s2i s3 (default "s2i-binaries")
--s3-disable-SSL disable ssl (default true)
--s3-endpoint string Endpoint to access to s3 object storage service, if left blank, the following options will be ignored.
--s3-force-path-style force path style (default true)
--s3-region string Region of s3 that will access to, like us-east-1. (default "us-east-1")
--s3-secret-access-key string secret access key of s2i s3 (default "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY")
--s3-session-token string session token of s2i s3
--secure-port int secure port number
--servicemesh-prometheus-host string prometheus service for servicemesh
--skip-headers If true, avoid header prefixes in the log messages
--skip-log-headers If true, avoid headers when opening log files
--sonarqube-host string Sonarqube service address, if left empty, following sonarqube options will be ignored.
--sonarqube-token string Sonarqube service access token.
--stderrthreshold severity logs at or above this threshold go to stderr (default 2)
--tls-cert-file string tls cert file
--tls-private-key string tls private key
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
2020/12/03 10:26:44 unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request5.查看kube-apiserver中的日志发现连接metrics.k8s.io/v1beta1 时候出错
kubectl logs -f kube-apiserver-master1 -n kube-system
.....
E1203 10:22:44.410647 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:22:49.410972 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:22:54.411326 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:22:59.411622 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:04.411972 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:09.412390 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:14.412799 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:19.413131 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:21.758943 1 timeout.go:132] net/http: abort Handler
E1203 10:23:24.413409 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:29.413710 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:34.414094 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:23:39.534172 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I1203 10:23:40.410119 1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
W1203 10:23:40.410178 1 handler_proxy.go:99] no RequestInfo found in the context
E1203 10:23:40.410216 1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I1203 10:23:40.410223 1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1203 10:24:04.246085 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:24:09.246520 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:24:34.246193 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E1203 10:24:39.246591 1 available_controller.go:416] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: Get https://10.101.186.48:443/apis/metrics.k8s.io/v1beta1: net/http: request canceled (Client.Timeout exceeded while awaiting headers)6.查看metrics-server 服务运行情况，确认已经在其他节点上重建
kubectl get pods -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-bbdc58449-cgsbt 1/1 Running 5 22d 10.244.137.79 master1 <none> <none>
calico-node-2qk2b 1/1 NodeLost 5 21d 192.168.210.74 worker1 <none> <none>
calico-node-gjdrw 1/1 Running 5 22d 192.168.210.73 master3 <none> <none>
calico-node-jwbnz 1/1 Running 4 22d 192.168.210.72 master2 <none> <none>
calico-node-svmv7 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
coredns-85d448b787-8nks7 1/1 Running 5 22d 10.244.137.87 master1 <none> <none>
coredns-85d448b787-lpxtt 1/1 Running 5 22d 10.244.137.80 master1 <none> <none>
etcd-master1 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
kube-apiserver-master1 1/1 Running 2 7d8h 192.168.210.71 master1 <none> <none>
kube-controller-manager-master1 1/1 Running 9 22d 192.168.210.71 master1 <none> <none>
kube-proxy-7wkbr 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
kube-proxy-8d7dj 1/1 Running 4 22d 192.168.210.72 master2 <none> <none>
kube-proxy-nhdsn 1/1 NodeLost 4 21d 192.168.210.74 worker1 <none> <none>
kube-proxy-sfbjm 1/1 Running 4 22d 192.168.210.73 master3 <none> <none>
kube-scheduler-master1 1/1 Running 9 22d 192.168.210.71 master1 <none> <none>
metrics-server-8b7689b66-xm6mf 1/1 Running 0 36s 10.244.180.55 master2 <none> <none>
metrics-server-8b7689b66-z9hk9 1/1 Unknown 0 3m58s 10.244.235.186 worker1 <none> <none>
tiller-deploy-5fd994b8f-twpn2 1/1 Running 0 3h13m 10.244.180.23 master2 <none> <none>7.查看api-resources情况
kubectl api-resources
NAME SHORTNAMES APIGROUP NAMESPACED KIND
bindings true Binding
componentstatuses cs false ComponentStatus
configmaps cm true ConfigMap
endpoints ep true Endpoints
events ev true Event
limitranges limits true LimitRange
namespaces ns false Namespace
nodes no false Node
persistentvolumeclaims pvc true PersistentVolumeClaim
persistentvolumes pv false PersistentVolume
pods po true Pod
podtemplates true PodTemplate
replicationcontrollers rc true ReplicationController
resourcequotas quota true ResourceQuota
secrets true Secret
serviceaccounts sa true ServiceAccount
services svc true Service
mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration
validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration
customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition
apiservices apiregistration.k8s.io false APIService
applications app.k8s.io true Application
controllerrevisions apps true ControllerRevision
daemonsets ds apps true DaemonSet
deployments deploy apps true Deployment
replicasets rs apps true ReplicaSet
statefulsets sts apps true StatefulSet
meshpolicies authentication.istio.io false MeshPolicy
policies authentication.istio.io true Policy
tokenreviews authentication.k8s.io false TokenReview
localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview
selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview
selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview
subjectaccessreviews authorization.k8s.io false SubjectAccessReview
horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler
cronjobs cj batch true CronJob
jobs batch true Job
certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest
adapters config.istio.io true adapter
attributemanifests config.istio.io true attributemanifest
handlers config.istio.io true handler
httpapispecbindings config.istio.io true HTTPAPISpecBinding
httpapispecs config.istio.io true HTTPAPISpec
instances config.istio.io true instance
quotaspecbindings config.istio.io true QuotaSpecBinding
quotaspecs config.istio.io true QuotaSpec
rules config.istio.io true rule
templates config.istio.io true template
leases coordination.k8s.io true Lease
bgpconfigurations crd.projectcalico.org false BGPConfiguration
bgppeers crd.projectcalico.org false BGPPeer
blockaffinities crd.projectcalico.org false BlockAffinity
clusterinformations crd.projectcalico.org false ClusterInformation
felixconfigurations crd.projectcalico.org false FelixConfiguration
globalnetworkpolicies gnp crd.projectcalico.org false GlobalNetworkPolicy
globalnetworksets crd.projectcalico.org false GlobalNetworkSet
hostendpoints crd.projectcalico.org false HostEndpoint
ipamblocks crd.projectcalico.org false IPAMBlock
ipamconfigs crd.projectcalico.org false IPAMConfig
ipamhandles crd.projectcalico.org false IPAMHandle
ippools crd.projectcalico.org false IPPool
kubecontrollersconfigurations crd.projectcalico.org false KubeControllersConfiguration
networkpolicies crd.projectcalico.org true NetworkPolicy
networksets crd.projectcalico.org true NetworkSet
s2ibinaries devops.kubesphere.io true S2iBinary
s2ibuilders s2ib devops.kubesphere.io true S2iBuilder
s2ibuildertemplates s2ibt devops.kubesphere.io false S2iBuilderTemplate
s2iruns s2ir devops.kubesphere.io true S2iRun
events ev events.k8s.io true Event
ingresses ing extensions true Ingress
jaegers jaegertracing.io true Jaeger
fluentbits logging.kubesphere.io true FluentBit
alertmanagers monitoring.coreos.com true Alertmanager
podmonitors monitoring.coreos.com true PodMonitor
prometheuses monitoring.coreos.com true Prometheus
prometheusrules monitoring.coreos.com true PrometheusRule
servicemonitors monitoring.coreos.com true ServiceMonitor
destinationrules dr networking.istio.io true DestinationRule
envoyfilters networking.istio.io true EnvoyFilter
gateways gw networking.istio.io true Gateway
serviceentries se networking.istio.io true ServiceEntry
sidecars networking.istio.io true Sidecar
virtualservices vs networking.istio.io true VirtualService
ingresses ing networking.k8s.io true Ingress
networkpolicies netpol networking.k8s.io true NetworkPolicy
runtimeclasses node.k8s.io false RuntimeClass
poddisruptionbudgets pdb policy true PodDisruptionBudget
podsecuritypolicies psp policy false PodSecurityPolicy
clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding
clusterroles rbac.authorization.k8s.io false ClusterRole
rolebindings rbac.authorization.k8s.io true RoleBinding
roles rbac.authorization.k8s.io true Role
authorizationpolicies rbac.istio.io true AuthorizationPolicy
clusterrbacconfigs rbac.istio.io false ClusterRbacConfig
rbacconfigs rbac.istio.io true RbacConfig
servicerolebindings rbac.istio.io true ServiceRoleBinding
serviceroles rbac.istio.io true ServiceRole
priorityclasses pc scheduling.k8s.io false PriorityClass
servicepolicies servicemesh.kubesphere.io true ServicePolicy
strategies servicemesh.kubesphere.io true Strategy
csidrivers storage.k8s.io false CSIDriver
csinodes storage.k8s.io false CSINode
storageclasses sc storage.k8s.io false StorageClass
volumeattachments storage.k8s.io false VolumeAttachment
workspaces tenant.kubesphere.io false Workspace
error: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request8.查看conntrack 连接情况发现kube-apiserver 在连接丢失的pod
查看metrics-server 的service
kubectl get svc -n kube-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
etcd ClusterIP None <none> 2379/TCP 21d
kube-controller-manager-headless ClusterIP None <none> 10252/TCP 22d
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 22d
kube-scheduler-headless ClusterIP None <none> 10251/TCP 22d
kubelet ClusterIP None <none> 10250/TCP 22d
metrics-server ClusterIP **10.101.186.48** <none> 443/TCP 21d
tiller-deploy ClusterIP 10.103.19.75 <none> 44134/TCP 22d查看metrics-server 运行情况及podIP
kubectl get pods -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-bbdc58449-cgsbt 1/1 Running 5 22d 10.244.137.79 master1 <none> <none>
calico-node-2qk2b 1/1 NodeLost 5 21d 192.168.210.74 worker1 <none> <none>
calico-node-gjdrw 1/1 Running 5 22d 192.168.210.73 master3 <none> <none>
calico-node-jwbnz 1/1 Running 4 22d 192.168.210.72 master2 <none> <none>
calico-node-svmv7 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
coredns-85d448b787-8nks7 1/1 Running 5 22d 10.244.137.87 master1 <none> <none>
coredns-85d448b787-lpxtt 1/1 Running 5 22d 10.244.137.80 master1 <none> <none>
etcd-master1 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
kube-apiserver-master1 1/1 Running 2 7d8h 192.168.210.71 master1 <none> <none>
kube-controller-manager-master1 1/1 Running 9 22d 192.168.210.71 master1 <none> <none>
kube-proxy-7wkbr 1/1 Running 5 22d 192.168.210.71 master1 <none> <none>
kube-proxy-8d7dj 1/1 Running 4 22d 192.168.210.72 master2 <none> <none>
kube-proxy-nhdsn 1/1 NodeLost 4 21d 192.168.210.74 worker1 <none> <none>
kube-proxy-sfbjm 1/1 Running 4 22d 192.168.210.73 master3 <none> <none>
kube-scheduler-master1 1/1 Running 9 22d 192.168.210.71 master1 <none> <none>
metrics-server-8b7689b66-xm6mf 1/1 Running 0 36s **10.244.180.55** master2 <none> <none>
metrics-server-8b7689b66-z9hk9 1/1 Unknown 0 3m58s **10.244.235.186** worker1 <none> <none>
tiller-deploy-5fd994b8f-twpn2 1/1 Running 0 3h13m 10.244.180.23 master2 <none> <none>观察连接情况
conntrack -L | grep 10.101.186.48
tcp 6 278 ESTABLISHED src=10.101.186.48 dst=**10.101.186.48** sport=45842 dport=443 src=**10.244.235.186** dst=192.168.210.71 sport=443 dport=19158 [ASSURED] mark=0 use=1
tcp 6 298 ESTABLISHED src=10.101.186.48 dst=**10.101.186.48** sport=45820 dport=443 src=**10.244.235.186** dst=**192.168.210.71** sport=443 dport=15276 [ASSURED] mark=0 use=2这个bug只有在节点突然宕机或者拔掉网线的情况下出现，要经过10min才能切换，或者重启kube-apiserver，正常关闭节点没有
