使用日志报错

hongming · 2020年9月9日

hetao kubernetes.default.svc 不通会影响到其他的服务

 kubectl get svc 
 kubectl get ep
 kubectl -n kube-system logs -l k8s-app=kube-proxy

Hhetao · 2020年9月9日

hongming

# kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.233.0.1   <none>        443/TCP   288d

# kubectl get ep
NAME         ENDPOINTS          AGE
kubernetes   192.168.8.4:6443   288d

# kubectl -n kube-system logs -l k8s-app=kube-proxy
I0909 07:11:35.527138       1 conntrack.go:52] Setting nf_conntrack_max to 524288
I0909 07:11:35.584437       1 conntrack.go:83] Setting conntrack hashsize to 131072
I0909 07:11:35.585194       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0909 07:11:35.585227       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0909 07:11:35.585401       1 config.go:313] Starting service config controller
I0909 07:11:35.585378       1 config.go:131] Starting endpoints config controller
I0909 07:11:35.585415       1 shared_informer.go:197] Waiting for caches to sync for service config
I0909 07:11:35.585416       1 shared_informer.go:197] Waiting for caches to sync for endpoints config
I0909 07:11:35.685527       1 shared_informer.go:204] Caches are synced for endpoints config 
I0909 07:11:35.685527       1 shared_informer.go:204] Caches are synced for service config 
E0909 07:11:17.085669       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: endpoints is forbidden: User "system:serviceaccount:kube-system:kube-proxy" cannot list resource "endpoints" in API group "" at the cluster scope: RBAC: [clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:node-proxier" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]
I0909 07:20:33.648788       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:20:33.648851       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:32:33.650351       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.62:15010
I0909 07:32:33.650394       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.77:15010
I0909 07:58:33.653548       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:33.653613       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:33.666599       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:33.666644       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 09:00:33.666677       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010
E0909 07:11:12.624946       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: Get https://192.168.8.4:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
E0909 07:11:13.624657       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Service: Get https://192.168.8.4:6443/api/v1/services?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
E0909 07:11:13.625574       1 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1.Endpoints: Get https://192.168.8.4:6443/api/v1/endpoints?labelSelector=%21service.kubernetes.io%2Fheadless%2C%21service.kubernetes.io%2Fservice-proxy-name&limit=500&resourceVersion=0: dial tcp 192.168.8.4:6443: connect: connection refused
I0909 07:20:41.786721       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:20:41.786781       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:58:41.791489       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:41.791532       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:41.798039       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:41.798092       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010
I0909 09:00:41.798122       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 07:20:18.789498       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.130:15010
I0909 07:20:18.789575       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.95:15010
I0909 07:32:18.791772       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.62:15010
I0909 07:35:18.792536       1 graceful_termination.go:93] lw: remote out of the list: 10.233.49.137:8010/TCP/10.233.96.61:80
I0909 07:56:18.796131       1 graceful_termination.go:93] lw: remote out of the list: 10.233.8.125:8020/TCP/10.233.96.130:8020
I0909 07:58:18.796769       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.50:15010
I0909 07:58:18.796834       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.101:15010
I0909 09:00:18.807518       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.26:15010
I0909 09:00:18.807584       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.96.228:15010
I0909 09:00:18.807625       1 graceful_termination.go:93] lw: remote out of the list: 10.233.41.255:15010/TCP/10.233.90.120:15010

hongming · 2020年9月9日

hetao 你这个集群是怎么升级的?

192.168.8.4:6443 这个端口连通性应该是有问题的，另外缺失了一些clusterrole

 [clusterrole.rbac.authorization.k8s.io "system:public-info-viewer" not found, clusterrole.rbac.authorization.k8s.io "system:node-proxier" not found, clusterrole.rbac.authorization.k8s.io "system:discovery" not found, clusterrole.rbac.authorization.k8s.io "system:basic-user" not found]

查看一下网络问题，把clusterrole 补上重启一下 kube-proxy 应该就可以了

Hhetao · 2020年9月9日

hongming
官网下载的升级包，然后修改完common.yaml文件，执行的scripts下的upgrade.sh。我k8s不太懂，clusterrole怎么补啊？

hongming · 2020年9月9日

hetao kubectl apply -f 一下下面这个yaml，然后重启kube-proxy，另外192.168.8.4:6443这个端口是通的吗

apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:public-info-viewer
    resourceVersion: "48"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Apublic-info-viewer
    uid: f59e529b-c472-4deb-ad5c-ff4b2c5d904c
  rules:
  - nonResourceURLs:
    - /healthz
    - /livez
    - /readyz
    - /version
    - /version/
    verbs:
    - get
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:node-proxier
    resourceVersion: "72"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Anode-proxier
    uid: 14a3acb5-6a37-4bef-92c0-1ce392a28dc0
  rules:
  - apiGroups:
    - ""
    resources:
    - endpoints
    - services
    verbs:
    - list
    - watch
  - apiGroups:
    - ""
    resources:
    - nodes
    verbs:
    - get
    - list
    - watch
  - apiGroups:
    - ""
    - events.k8s.io
    resources:
    - events
    verbs:
    - create
    - patch
    - update
  - apiGroups:
    - discovery.k8s.io
    resources:
    - endpointslices
    verbs:
    - list
    - watch
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:discovery
    resourceVersion: "46"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Adiscovery
    uid: 10407319-a9af-4625-8aca-d524b39ae14b
  rules:
  - nonResourceURLs:
    - /api
    - /api/*
    - /apis
    - /apis/*
    - /healthz
    - /livez
    - /openapi
    - /openapi/*
    - /readyz
    - /version
    - /version/
    verbs:
    - get
- apiVersion: rbac.authorization.k8s.io/v1
  kind: ClusterRole
  metadata:
    annotations:
      rbac.authorization.kubernetes.io/autoupdate: "true"
    creationTimestamp: "2020-06-18T07:35:32Z"
    labels:
      kubernetes.io/bootstrapping: rbac-defaults
    name: system:basic-user
    resourceVersion: "47"
    selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/system%3Abasic-user
    uid: 2e5f0e28-9471-48fd-8e81-5dddc07ca389
  rules:
  - apiGroups:
    - authorization.k8s.io
    resources:
    - selfsubjectaccessreviews
    - selfsubjectrulesreviews
    verbs:
    - create
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Hhetao · 2020年9月9日

hongming
在node节点上telnet 192.168.8.4 6443 端口都是通的

# kubectl apply -f role.yaml 
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:public-info-viewer": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:node-proxier": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:discovery": the object has been modified; please apply your changes to the latest version and try again
Error from server (Conflict): Operation cannot be fulfilled on clusterroles.rbac.authorization.k8s.io "system:basic-user": the object has been modified; please apply your changes to the latest version and try again

执行的时候报错了
实在抱歉，打扰您这么久，明天您方便远程给看一下吗？

hongming · 2020年9月9日

hetao 可以的，远程方式发送到kubesphere@yunify.com就可以

Hhetao · 2020年9月9日

hongming
好的，万分感谢！！！

Zzhanglihao · 2022年7月5日

hongming

大佬可以帮我看下日志服务没有收集日志的问题么，看了很久不清楚怎么排查，所有服务感觉都是在正常运行，没有特别明显的报错

DehaoCheng · 2022年7月5日

zhanglihao 要不你新开个帖子？贴一下具体信息？

Zzhanglihao · 2022年7月5日

DehaoCheng 好的

poo0054 · 2023年11月21日

请问下日志里面的es怎么设置账号和密码

使用日志报错

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

hongmingK零SK壹S

HhetaoK零S

Zzhanglihao

DehaoChengK零S

Zzhanglihao

poo0054