使用日志报错

hongming · 2020年9月9日

@hetao kube-system 和kubesphere-system 下的组件都还是正常的，再看看 ks-apigateway/ks-apiserver/ks-account 这几个组件的日志，我看到上面贴的日志有大量的 connection refused 和 connection reset by peer ，检查一下节点网络是否正常

Hhetao · 2020年9月9日

hongming

# kubectl logs ks-apigateway-94687746b-89h9n -n kubesphere-system |tail -n 100 |grep ERROR
2020/09/09 08:35:22 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:33476: use of closed network connection
2020/09/09 08:35:25 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:33700: use of closed network connection
2020/09/09 08:35:27 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:33834: use of closed network connection
2020/09/09 08:36:26 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:33946: use of closed network connection
2020/09/09 08:36:28 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:35558: use of closed network connection
2020/09/09 08:36:34 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:35630: use of closed network connection
2020/09/09 08:37:26 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:35804: use of closed network connection
2020/09/09 08:37:30 [ERROR] failed to copy buffer:  read tcp 10.233.96.121:2018->10.233.96.201:37170: use of closed network connection
2020/09/09 08:38:30 [ERROR] failed to copy buffer:  readfrom tcp 10.233.96.121:2018->10.233.96.201:37306: read tcp 10.233.96.121:35332->10.233.0.1:443: use of closed network connection

# # kubectl logs ks-apiserver-74b4876f95-fcc52 -n kubesphere-system |tail -n 100 
E0909 07:11:12.649654       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.650768       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha2.ServicePolicy: Get https://10.233.0.1:443/apis/servicemesh.kubesphere.io/v1alpha2/servicepolicies?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.651777       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Role: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.652874       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.Ingress: Get https://10.233.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.653897       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.654998       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ControllerRevision: Get https://10.233.0.1:443/apis/apps/v1/controllerrevisions?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.656008       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.DaemonSet: Get https://10.233.0.1:443/apis/apps/v1/daemonsets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.657119       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.RoleBinding: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.658163       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Deployment: Get https://10.233.0.1:443/apis/apps/v1/deployments?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.659282       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: Get https://10.233.0.1:443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.660340       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha2.Strategy: Get https://10.233.0.1:443/apis/servicemesh.kubesphere.io/v1alpha2/strategies?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.661385       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ClusterRoleBinding: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.662483       1 reflector.go:134] sigs.k8s.io/application/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1beta1.Application: Get https://10.233.0.1:443/apis/app.k8s.io/v1beta1/applications?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.663487       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: Get https://10.233.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.664595       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha1.Workspace: Get https://10.233.0.1:443/apis/tenant.kubesphere.io/v1alpha1/workspaces?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.665609       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Secret: Get https://10.233.0.1:443/api/v1/secrets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.666674       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha1.S2iBinary: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2ibinaries?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:12.667738       1 reflector.go:134] github.com/kubesphere/s2ioperator/pkg/client/informers/externalversions/factory.go:116: Failed to list *v1alpha1.S2iBuilderTemplate: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2ibuildertemplates?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.624755       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: Get https://10.233.0.1:443/apis/apps/v1/replicasets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.638788       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.CronJob: Get https://10.233.0.1:443/apis/batch/v1beta1/cronjobs?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.639815       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v2beta2.HorizontalPodAutoscaler: Get https://10.233.0.1:443/apis/autoscaling/v2beta2/horizontalpodautoscalers?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.640842       1 reflector.go:134] github.com/kubesphere/s2ioperator/pkg/client/informers/externalversions/factory.go:116: Failed to list *v1alpha1.S2iBuilder: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2ibuilders?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.641864       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ResourceQuota: Get https://10.233.0.1:443/api/v1/resourcequotas?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.642908       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StatefulSet: Get https://10.233.0.1:443/apis/apps/v1/statefulsets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.643965       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Job: Get https://10.233.0.1:443/apis/batch/v1/jobs?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.645002       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ClusterRole: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterroles?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.646053       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Namespace: Get https://10.233.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.647095       1 reflector.go:134] github.com/kubesphere/s2ioperator/pkg/client/informers/externalversions/factory.go:116: Failed to list *v1alpha1.S2iRun: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2iruns?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.648199       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.PersistentVolumeClaim: Get https://10.233.0.1:443/api/v1/persistentvolumeclaims?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.649208       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Pod: Get https://10.233.0.1:443/api/v1/pods?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.650336       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ConfigMap: Get https://10.233.0.1:443/api/v1/configmaps?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.651384       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha2.ServicePolicy: Get https://10.233.0.1:443/apis/servicemesh.kubesphere.io/v1alpha2/servicepolicies?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.652397       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Role: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/roles?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.653470       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1beta1.Ingress: Get https://10.233.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.654498       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.655559       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ControllerRevision: Get https://10.233.0.1:443/apis/apps/v1/controllerrevisions?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.656641       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.DaemonSet: Get https://10.233.0.1:443/apis/apps/v1/daemonsets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.657724       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.RoleBinding: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/rolebindings?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.658774       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Deployment: Get https://10.233.0.1:443/apis/apps/v1/deployments?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.659837       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.StorageClass: Get https://10.233.0.1:443/apis/storage.k8s.io/v1/storageclasses?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.660900       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha2.Strategy: Get https://10.233.0.1:443/apis/servicemesh.kubesphere.io/v1alpha2/strategies?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.661974       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ClusterRoleBinding: Get https://10.233.0.1:443/apis/rbac.authorization.k8s.io/v1/clusterrolebindings?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.663035       1 reflector.go:134] sigs.k8s.io/application/pkg/client/informers/externalversions/factory.go:117: Failed to list *v1beta1.Application: Get https://10.233.0.1:443/apis/app.k8s.io/v1beta1/applications?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.664104       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Node: Get https://10.233.0.1:443/api/v1/nodes?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.665148       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha1.Workspace: Get https://10.233.0.1:443/apis/tenant.kubesphere.io/v1alpha1/workspaces?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.666184       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.Secret: Get https://10.233.0.1:443/api/v1/secrets?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.667289       1 reflector.go:134] kubesphere.io/kubesphere/pkg/client/informers/externalversions/factory.go:120: Failed to list *v1alpha1.S2iBinary: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2ibinaries?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:13.668351       1 reflector.go:134] github.com/kubesphere/s2ioperator/pkg/client/informers/externalversions/factory.go:116: Failed to list *v1alpha1.S2iBuilderTemplate: Get https://10.233.0.1:443/apis/devops.kubesphere.io/v1alpha1/s2ibuildertemplates?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E0909 07:11:17.080510       1 reflector.go:134] k8s.io/client-go/informers/factory.go:132: Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kubesphere-system:kubesphere" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0909 07:31:01.680206       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34036-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:32:09.264097       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34034-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:32:42.992128       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34038-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:33:09.232241       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34040-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:33:43.088089       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34026-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:34:53.168116       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34032-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:37:50.540461       1 v2.go:105] websocket: close 1001 (going away)
W0909 07:37:58.542511       1 terminal.go:133] 1Process exited
E0909 07:42:15.021883       1 v2.go:105] websocket: close 1001 (going away)
W0909 07:42:23.023352       1 terminal.go:133] 1Process exited
E0909 07:43:35.536174       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34030-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:45:26.320262       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:34028-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:46:26.352092       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:38520-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:47:32.272127       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:40288-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:52:01.072128       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:47680-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:52:53.328232       1 v2.go:105] websocket: close 1006 (abnormal closure): unexpected EOF
E0909 07:53:01.104153       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:40838-\u003e10.233.70.66:389: read: connection reset by peer"
}
W0909 07:53:01.329598       1 terminal.go:133] 1Process exited
E0909 07:57:38.032086       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:49300-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 07:59:38.736128       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:47676-\u003e10.233.70.66:389: read: connection reset by peer"
}
E0909 08:00:38.704101       1 metrics.go:706] status: 500,message: {
 "message": "unable to read LDAP response packet: read tcp 10.233.96.232:49298-\u003e10.233.70.66:389: read: connection reset by peer"
}

# kubectl logs ks-account-7f67d5966d-6td8z -n kubesphere-system |tail -n 100 
W0909 07:13:07.174268       1 client_config.go:549] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0909 07:13:07.789468       1 server.go:113] Server listening on 0.0.0.0:9090 
E0909 07:31:01.679962       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34036->10.233.70.66:389: read: connection reset by peer
E0909 07:31:01.680002       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34036->10.233.70.66:389: read: connection reset by peer
E0909 07:32:09.263903       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34034->10.233.70.66:389: read: connection reset by peer
E0909 07:32:09.263925       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34034->10.233.70.66:389: read: connection reset by peer
E0909 07:32:42.991906       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34038->10.233.70.66:389: read: connection reset by peer
E0909 07:32:42.991930       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34038->10.233.70.66:389: read: connection reset by peer
E0909 07:33:09.231945       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34040->10.233.70.66:389: read: connection reset by peer
E0909 07:33:09.231974       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34040->10.233.70.66:389: read: connection reset by peer
E0909 07:33:43.087867       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34026->10.233.70.66:389: read: connection reset by peer
E0909 07:33:43.087890       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34026->10.233.70.66:389: read: connection reset by peer
E0909 07:34:53.167920       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34032->10.233.70.66:389: read: connection reset by peer
E0909 07:34:53.167942       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34032->10.233.70.66:389: read: connection reset by peer
E0909 07:43:35.535953       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34030->10.233.70.66:389: read: connection reset by peer
E0909 07:43:35.535978       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34030->10.233.70.66:389: read: connection reset by peer
E0909 07:45:26.320046       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:34028->10.233.70.66:389: read: connection reset by peer
E0909 07:45:26.320069       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:34028->10.233.70.66:389: read: connection reset by peer
E0909 07:46:26.351887       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:38520->10.233.70.66:389: read: connection reset by peer
E0909 07:46:26.351907       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:38520->10.233.70.66:389: read: connection reset by peer
E0909 07:47:32.271931       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:40288->10.233.70.66:389: read: connection reset by peer
E0909 07:47:32.271955       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:40288->10.233.70.66:389: read: connection reset by peer
E0909 07:52:01.071933       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:47680->10.233.70.66:389: read: connection reset by peer
E0909 07:52:01.071954       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:47680->10.233.70.66:389: read: connection reset by peer
E0909 07:53:01.103948       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:40838->10.233.70.66:389: read: connection reset by peer
E0909 07:53:01.103968       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:40838->10.233.70.66:389: read: connection reset by peer
E0909 07:57:38.031909       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:49300->10.233.70.66:389: read: connection reset by peer
E0909 07:57:38.031925       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:49300->10.233.70.66:389: read: connection reset by peer
E0909 07:59:38.735929       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:47676->10.233.70.66:389: read: connection reset by peer
E0909 07:59:38.735951       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:47676->10.233.70.66:389: read: connection reset by peer
E0909 08:00:38.703909       1 im.go:586] search user unable to read LDAP response packet: read tcp 10.233.96.232:49298->10.233.70.66:389: read: connection reset by peer
E0909 08:00:38.703932       1 im.go:312] unable to read LDAP response packet: read tcp 10.233.96.232:49298->10.233.70.66:389: read: connection reset by peer

我今天下午的操作顺序：
重启master的docker，失败，然后重启master服务器，发现日志和istio还是异常，于是执行

kubectl edit cm -n kubesphere-system ks-installer

关掉了日志和istio

hongming · 2020年9月9日

hetao kubernetes.default.svc 不通会影响到其他的服务

 kubectl get svc 
 kubectl get ep
 kubectl -n kube-system logs -l k8s-app=kube-proxy

Hhetao · 2020年9月9日

hongming

# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.233.0.1   <none>        443/TCP   288d

# kubectl get ep
NAME         ENDPOINTS          AGE
kubernetes   192.168.8.4:6443   288d

# kubectl -n kube-system logs -l k8s-app=kube-proxy
I0909 07:11:35.527138       1 conntrack.go:52] Setting nf_conntrack_max to 524288
I0909 07:11:35.584437       1 conntrack.go:83] Setting conntrack hashsize to 131072
I0909 07:11:35.585194       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0909 07:11:35.585227       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0909 07:11:35.585401       1 config.go:313] Starting service config controller
I0909 07:11:35.585378       1 config.go:131] Starting endpoints config controller
I0909 07:11:35.585415       1 shared_informer.go:197] Waiting for caches to sync for service config
I0909 07:11:35.585416       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0909 07:11:35.685527       1 shared_informer.go:204] Caches are synced for endpoints config 
I0909 07:11:35.685527       1 shared_informer.go:204] Caches are synced for service config 
E0909 07:11:17.085669       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:node-proxier" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
I0909 07:20:33.648788       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:20:33.648851       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:32:33.650351       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.62:15010
I0909 07:32:33.650394       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.77:15010
I0909 07:58:33.653548       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:33.653613       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:33.666599       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:33.666644       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 09:00:33.666677       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010
E0909 07:11:12.624946       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: Get https://192.168.8.4:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
E0909 07:11:13.624657       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: Get https://192.168.8.4:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
E0909 07:11:13.625574       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: Get https://192.168.8.4:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
I0909 07:20:41.786721       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:20:41.786781       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:58:41.791489       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:41.791532       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:41.798039       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:41.798092       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010
I0909 09:00:41.798122       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 07:20:18.789498       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:20:18.789575       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:32:18.791772       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.62:15010
I0909 07:35:18.792536       1 graceful_termination.go:93] lw: remote out of the list: 10.233.49.137:8010/TCP/10.233.96.61:80
I0909 07:56:18.796131       1 graceful_termination.go:93] lw: remote out of the list: 10.233.8.125:8020/TCP/10.233.96.130:8020
I0909 07:58:18.796769       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:18.796834       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:18.807518       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:18.807584       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 09:00:18.807625       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010

hongming · 2020年9月9日

hetao 你这个集群是怎么升级的?

192.168.8.4:6443 这个端口连通性应该是有问题的，另外缺失了一些clusterrole

 [clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:node-proxier" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]

查看一下网络问题，把clusterrole 补上重启一下 kube-proxy 应该就可以了

Hhetao · 2020年9月9日

hongming
官网下载的升级包，然后修改完common.yaml文件，执行的scripts下的upgrade.sh。我k8s不太懂，clusterrole怎么补啊？

hongming · 2020年9月9日

hetao kubectl apply -f 一下下面这个yaml，然后重启kube-proxy，另外192.168.8.4:6443这个端口是通的吗

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:public-info-viewer
    resourceVersion: "48"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Apublic-info-viewer
    uid: f59e529b-c472-4deb-ad5c-ff4b2c5d904c
  rules:
  - nonResourceURLs:
    - /healthz
    - /livez
    - /readyz
    - /version
    - /version/
    verbs:
    - get
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:node-proxier
    resourceVersion: "72"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Anode-proxier
    uid: 14a3acb5-6a37-4bef-92c0-1ce392a28dc0
  rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    verbs:
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    - events.k8s.io
    resources:
    - events
    verbs:
    - create
    - patch
    - update
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:discovery
    resourceVersion: "46"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Adiscovery
    uid: 10407319-a9af-4625-8aca-d524b39ae14b
  rules:
  - nonResourceURLs:
    - /api
    - /api/*
    - /apis
    - /apis/*
    - /healthz
    - /livez
    - /openapi
    - /openapi/*
    - /readyz
    - /version
    - /version/
    verbs:
    - get
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:basic-user
    resourceVersion: "47"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Abasic-user
    uid: 2e5f0e28-9471-48fd-8e81-5dddc07ca389
  rules:
  - apiGroups:
    - authorization.k8s.io
    resources:
    - selfsubjectaccessreviews
    - selfsubjectrulesreviews
    verbs:
    - create
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Hhetao · 2020年9月9日

hongming
在node节点上telnet 192.168.8.4 6443 端口都是通的

# kubectl apply -f role.yaml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:public-info-viewer": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:node-proxier": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:discovery": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:basic-user": the object has been modified; please apply your changes to the latest version and try again

执行的时候报错了
实在抱歉，打扰您这么久，明天您方便远程给看一下吗？

hongming · 2020年9月9日

hetao 可以的，远程方式发送到kubesphere@yunify.com就可以

Hhetao · 2020年9月9日

hongming
好的，万分感谢！！！

Zzhanglihao · 2022年7月5日

hongming

大佬可以帮我看下日志服务没有收集日志的问题么，看了很久不清楚怎么排查，所有服务感觉都是在正常运行，没有特别明显的报错

DehaoCheng · 2022年7月5日

zhanglihao 要不你新开个帖子？贴一下具体信息？

Zzhanglihao · 2022年7月5日

DehaoCheng 好的

poo0054 · 2023年11月21日

请问下日志里面的es怎么设置账号和密码

使用日志报错

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

Zzhanglihao

DehaoChengK零S

Zzhanglihao

poo0054