不是改 configmap,要改cc 才能生效的
kubectl get cc ks-installer -n kubesphere-system -o yaml
看看你cc文件里改了没
ks3.2版本工具箱中没有日志查询和审计功能
- 已编辑
ruiyaoOps
改了的,我上面也说过了,我删除了status里的相关部分,重启ks-installer跟ks-apiserver,都没有用。
[root@kc21m01 ~]# kubectl get cc ks-installer -n kubesphere-system -o yaml
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{“apiVersion”:“installer.kubesphere.io/v1alpha1”,“kind”:“ClusterConfiguration”,“metadata”:{“annotations”:{},“labels”:{“version”:“v3.2.0”},“name”:“ks-installer”,“namespace”:“kubesphere-system”},“spec”:{“alerting”:{“enabled”:true},“auditing”:{“enabled”:true},“authentication”:{“jwtSecret”:“khUff6ToH2b5EVHTR2a8kPEKWkHpBeBj”},“common”:{“core”:{“console”:{“enableMultiLogin”:true,“port”:30880,“type”:“NodePort”}},“es”:{“basicAuth”:{“enabled”:true,“password”:“esTic456”,“username”:“elastic”},“elkPrefix”:“logstash”,“externalElasticsearchPort”:“9200”,“externalElasticsearchUrl”:“192.168.120.85”,“logMaxAge”:7},“gpu”:{“kinds”:[{“default”:true,“resourceName”:“nvidia.com/gpu”,“resourceType”:“GPU”}]},“minio”:{“volumeSize”:“20Gi”},“monitoring”:{“GPUMonitoring”:{“enabled”:true},“endpoint”:“http://prometheus-operated.kubesphere-monitoring-system.svc:9090”},“openldap”:{“enabled”:true,“volumeSize”:“2Gi”},“redis”:{“enabled”:true,“volumeSize”:“2Gi”}},“devops”:{“enabled”:true,“jenkinsJavaOpts_MaxRAM”:“2g”,“jenkinsJavaOpts_Xms”:“512m”,“jenkinsJavaOpts_Xmx”:“512m”,“jenkinsMemoryLim”:“2Gi”,“jenkinsMemoryReq”:“1500Mi”,“jenkinsVolumeSize”:“8Gi”},“etcd”:{“endpointIps”:“localhost”,“monitoring”:false,“port”:2379,“tlsEnable”:true},“events”:{“enabled”:true},“kubeedge”:{“cloudCore”:{“cloudHub”:{“advertiseAddress”:[""],“nodeLimit”:“100”},“cloudhubHttpsPort”:“10002”,“cloudhubPort”:“10000”,“cloudhubQuicPort”:“10001”,“cloudstreamPort”:“10003”,“nodeSelector”:{“node-role.kubernetes.io/worker”:""},“service”:{“cloudhubHttpsNodePort”:“30002”,“cloudhubNodePort”:“30000”,“cloudhubQuicNodePort”:“30001”,“cloudstreamNodePort”:“30003”,“tunnelNodePort”:“30004”},“tolerations”:[],“tunnelPort”:“10004”},“edgeWatcher”:{“edgeWatcherAgent”:{“nodeSelector”:{“node-role.kubernetes.io/worker”:""},“tolerations”:[]},“nodeSelector”:{“node-role.kubernetes.io/worker”:""},“tolerations”:[]},“enabled”:false},“local_registry”:"",“logging”:{“containerruntime”:“containerd”,“enabled”:true,“logsidecar”:{“enabled”:true,“replicas”:2}},“metrics_server”:{“enabled”:true},“monitoring”:{“gpu”:{“nvidia_dcgm_exporter”:{“enabled”:false}},“storageClass”:“managed-nfs-storage”},“multicluster”:{“clusterRole”:“member”},“network”:{“ippool”:{“type”:“none”},“networkpolicy”:{“enabled”:false},“topology”:{“type”:“none”}},“openpitrix”:{“store”:{“enabled”:false}},“persistence”:{“storageClass”:“managed-nfs-storage”},“servicemesh”:{“enabled”:true}}}
creationTimestamp: “2021-11-08T02:09:45Z”
generation: 47
labels:
version: v3.2.0
name: ks-installer
namespace: kubesphere-system
resourceVersion: “8400030”
selfLink: /apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations/ks-installer
uid: ddf6e2b0-dd0f-4345-9937-4f1b69585da5
spec:
alerting:
enabled: true
auditing:
enabled: true
authentication:
jwtSecret: khUff6ToH2b5EVHTR2a8adsakPEKWkHpBeBj
common:
core:
console:
enableMultiLogin: true
port: 30880
type: NodePort
es:
basicAuth:
enabled: true
password: esTic456
username: elastic
elkPrefix: logstash
externalElasticsearchPort: “9200”
externalElasticsearchUrl: 192.168.120.85
logMaxAge: 7
gpu:
kinds:
- default: true
resourceName: nvidia.com/gpu
resourceType: GPU
minio:
volumeSize: 20Gi
monitoring:
GPUMonitoring:
enabled: true
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
openldap:
enabled: true
volumeSize: 2Gi
redis:
enabled: true
volumeSize: 2Gi
devops:
enabled: true
jenkinsJavaOpts_MaxRAM: 2g
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
etcd:
endpointIps: localhost
monitoring: false
port: 2379
tlsEnable: true
events:
enabled: true
kubeedge:
cloudCore:
cloudHub:
advertiseAddress:
- ""
nodeLimit: “100”
cloudhubHttpsPort: “10002”
cloudhubPort: “10000”
cloudhubQuicPort: “10001”
cloudstreamPort: “10003”
nodeSelector:
node-role.kubernetes.io/worker: ""
service:
cloudhubHttpsNodePort: “30002”
cloudhubNodePort: “30000”
cloudhubQuicNodePort: “30001”
cloudstreamNodePort: “30003”
tunnelNodePort: “30004”
tolerations: []
tunnelPort: “10004”
edgeWatcher:
edgeWatcherAgent:
nodeSelector:
node-role.kubernetes.io/worker: ""
tolerations: []
nodeSelector:
node-role.kubernetes.io/worker: ""
tolerations: []
enabled: false
local_registry: ""
logging:
containerruntime: docker
enabled: true
logsidecar:
enabled: true
replicas: 2
metrics_server:
enabled: true
monitoring:
gpu:
nvidia_dcgm_exporter:
enabled: false
storageClass: managed-nfs-storage
multicluster:
clusterRole: member
network:
ippool:
type: none
networkpolicy:
enabled: false
topology:
type: none
openpitrix:
store:
enabled: false
persistence:
storageClass: managed-nfs-storage
servicemesh:
enabled: true
status:
alerting:
enabledTime: 2021-11-08T11:32:23CST
status: enabled
auditing:
enabledTime: 2021-11-08T11:29:19CST
status: enabled
clusterId: 34b14c4b-2834-463c-8d6a-1b1ca6571013-1636342382
core:
enabledTime: 2021-11-08T11:27:17CST
status: enabled
version: v3.2.0
devops:
enabledTime: 2021-11-08T11:30:48CST
status: enabled
events:
enabledTime: 2021-11-08T11:29:53CST
status: enabled
fluentbit:
enabledTime: 2021-11-08T11:26:15CST
status: enabled
logging:
enabledTime: 2021-11-08T11:30:06CST
status: enabled
metricsServer:
enabledTime: 2021-11-08T11:24:43CST
status: enabled
minio:
enabledTime: 2021-11-08T11:25:57CST
status: enabled
monitoring:
enabledTime: 2021-11-08T11:32:20CST
status: enabled
openldap:
enabledTime: 2021-11-08T11:25:44CST
status: enabled
redis:
enabledTime: 2021-11-08T11:25:35CST
status: enabled
servicemesh:
enabledTime: 2021-11-08T11:30:36CST
status: enabled
- 已编辑
kubectl rollout restart deploy ks-installer -n kubesphere-system
把 ks-installer 重启然后看看日志,看下执行过程
真的很郁闷,同样的环境,3.1.1的工具中是没有问题的,升级到3.2.0各种问题,然后干脆卸载了3.2,重新安装,还是一样的。还有其它的问题,比如开启了auditing,pod "kube-auditing-webhook-deploy-xxxx"中也能看到auditing日志,但是es中就是生成不了index。这些小问题排查了好多天了,头痛,让我不敢轻易上生产。官方文档很简单,就是改几个参数,查都没法查。
DehaoChengK零S
- 已编辑
morriszs 重新安装的集群仍然有问题吗?
你这个估计得找人远程看看,有买qingcloud 的服务么,在上面提个工单可以找人看看。
DehaoCheng
是的,仍然有问题,重装无数次了。各种日志翻遍了,找不到头绪。
DehaoChengK零S
morriszs 重新安装后具体是什么问题呢
WwanjunleiK零S
审计日志的问题,可以看看这个
https://kubesphere.com.cn/forum/d/5971-v320/2
DehaoChengK零S
我的集群如下:
下面的参数也是在登陆host主集群的情况下打印出来的,我想参数应该也是host集群的,跟member集群没关系。
member集群目前用默认密码登陆不上,这又是另外一个悲伤的问题了:
后台报错:
- 已编辑
DehaoCheng 不是docker环境啊,我的member集群的k8s版本是:v1.21.4,没有docker,用的containerd。这就是我为什么要升级到3.2.0版本的原因,因为它支持containerd的text日志格式和流水线。
DehaoChengK零S
morriszs 那containerruntime: docker就应该填containerd而不是docker啊。
- 已编辑
DehaoCheng
初始安装时,我是改成了containerd,但ks-installer始终报错,一直安装不上,没办法才改成了默认的docker,这才安装上了。我翻看了官方的github,对这个containerruntime参数没有任何解释,也没提供任何可选参数,默认就是docker。
DehaoChengK零S
morriszs 那你改为containerd重新安装ks-installer呢?如果有什么报错请贴出来。
DehaoCheng
改成containerd以后,ks-installer日志没有报错:
PLAY RECAP *********************************************************************
localhost : ok=28 changed=17 unreachable=0 failed=0 skipped=11 rescued=0 ignored=0
Start installing monitoring
Start installing multicluster
Start installing openpitrix
Start installing network
Start installing alerting
Start installing auditing
Start installing devops
Start installing events
Start installing logging
Start installing servicemesh
Waiting for all tasks to be completed …
task openpitrix status is successful (⅒)
task alerting status is successful (2/10)
task network status is successful (3/10)
task multicluster status is successful (4/10)
task auditing status is successful (5/10)
task events status is successful (6/10)
task logging status is successful (7/10)
task servicemesh status is successful (8/10)
task devops status is successful (9/10)
task monitoring status is successful (10/10)
Collecting installation results …
#####################################################
Welcome to KubeSphere!
#####################################################
但是,fluentbit报错了,跟以前一样:
DehaoChengK零S
morriszs 这个错误应该在于containerd的问题,找不到crictl,你可以执行
kubectl edit fluentbits.logging.kubesphere.io -n kubesphere-logging-system
添加containerLogRealPath即可