创建DevOps项目失败

344089386

shaowenchen

好像不一样

shaowenchen

344089386

dns 服务异常了。看看异常日志，另外应该有异常的 pod 才对

kubectl logs -l k8s-app:kube-dns -n kube-system -f

kubectl get pod –all-namespaces -o wide

344089386

shaowenchen

344089386

shaowenchen yunkunrao 大佬帮忙看下，花儿都谢了。。

yunkunrao

截图里的cmd有误，试一下kubectl get pod --all-namespaces -o wide

344089386

yunkunrao

shaowenchen

344089386

https://kubernetes.io/zh/docs/tasks/administer-cluster/dns-debugging-resolution/

DNS问题，按照这个文档排查下

344089386

shaowenchen

我的DNS应该是正常的啊？

344089386

shaowenchen
大佬不要放弃我。。我把远程发给你你直接上去帮我看下？

Forest-L

344089386
通过分析可知：你们环境的cpu不够，导致devops项目创建不了。

344089386

Forest-L

总共8核 32G 都不够吗

Forest-L

344089386 cpu已经使用了95%了，加到8核吧，内存是够的。

344089386

Forest-L
我之前啥都没部署的时候，占用很低也不行。。

mrz001

使用demo用户登录，打开DevOps工程流水线页面提示dns错误，如图，烦请告知下如何处理，谢谢。

Forest-L

mrz001 检查下dns服务是否正常。检查下jenkins服务是否正常运行。

mrz001

Forest-L
DNS貌似是好的。Jenkins服务我去看下。

mrz001

貌似都是好的Forest-L

Forest-L

mrz001 ks-jenkins看下，然后看下对应pod的日志。

mrz001

`Start installing monitoring
Start installing multicluster
Start installing devops
Start installing openpitrix
Start installing servicemesh

task monitoring status is running
task multicluster status is successful
task devops status is running
task openpitrix status is running
task servicemesh status is running
total: 5 completed:1

task monitoring status is running
task multicluster status is successful
task devops status is running
task openpitrix status is successful
task servicemesh status is running
total: 5 completed:2

task monitoring status is running
task multicluster status is successful
task devops status is failed
task openpitrix status is successful
task servicemesh status is running
total: 5 completed:3

task monitoring status is successful
task multicluster status is successful
task devops status is failed
task openpitrix status is successful
task servicemesh status is running
total: 5 completed:4

task monitoring status is successful
task multicluster status is successful
task devops status is failed
task openpitrix status is successful
task servicemesh status is successful
total: 5 completed:5

Task ‘devops’ failed:

{
“counter”: 63,
“created”: “2020-12-21T07:20:52.732509”,
“end_line”: 63,
“event”: “runner_on_failed”,
“event_data”: {
“duration”: 56.096621,
“end”: “2020-12-21T07:20:52.731993”,
“event_loop”: null,
“host”: “localhost”,
“ignore_errors”: null,
“play”: “localhost”,
“play_pattern”: “localhost”,
“play_uuid”: “0a580a64-0053-52f6-5b1e-000000000005”,
“playbook”: “/kubesphere/playbooks/devops.yaml”,
“playbook_uuid”: “8c299fb4-184a-44a7-9873-5cc339ab6b68”,
“remote_addr”: “127.0.0.1”,
“res”: {
“ansible_no_log”: false,
“changed”: true,
“cmd”: “/bin/bash /kubesphere/kubesphere/s2i/certs.sh –service webhook-server-service –namespace kubesphere-devops-system –secret webhook-server-secret \n”,
“delta”: “0:00:55.395342”,
“end”: “2020-12-21 07:20:52.684270”,
“invocation”: {
“module_args”: {
“raw_params”: “/bin/bash /kubesphere/kubesphere/s2i/certs.sh –service webhook-server-service –namespace kubesphere-devops-system –secret webhook-server-secret \n”,
“_uses_shell”: true,
“argv”: null,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“stdin”: null,
“stdin_add_newline”: true,
“strip_empty_ends”: true,
“warn”: true
}
},
“msg”: “non-zero return code”,
“rc”: 1,
“start”: “2020-12-21 07:19:57.288928”,
“stderr”: "Generating RSA private key, 2048 bit long modulus (2 primes)\n…………………+++++\n………+++++\ne is 65537 (0×010001)\nERROR: After approving csr webhook-server-service.kubesphere-de
vops-system, the signed certificate did not appear on the resource. Giving up after 10 attempts.", “stderr_lines”: [
“Generating RSA private key, 2048 bit long modulus (2 primes)”,
“…………………+++++”,
“………+++++”,
“e is 65537 (0×010001)”,
“ERROR: After approving csr webhook-server-service.kubesphere-devops-system, the signed certificate did not appear on the resource. Giving up after 10 attempts.”
],
“stdout”: "creating certs in tmpdir /tmp/tmp.CIljAd \ncertificatesigningrequest.certificates.k8s.io \“webhook-server-service.kubesphere-devops-system\” deleted\ncertificatesigningrequest.certificates.k8s
.io/webhook-server-service.kubesphere-devops-system created\nNAME AGE REQUESTOR CONDITION\nwebhook-server-service.kubesphere-devops-system 0s system:serviceaccount:kubesphere-system:ks-installer Pending\ncertificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system approved", “stdout_lines”: [
“creating certs in tmpdir /tmp/tmp.CIljAd ”,
"certificatesigningrequest.certificates.k8s.io \“webhook-server-service.kubesphere-devops-system\” deleted",
“certificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system created”,
“NAME AGE REQUESTOR CONDITION”,
“webhook-server-service.kubesphere-devops-system 0s system:serviceaccount:kubesphere-system:ks-installer Pending”,
“certificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system approved”
]
},
“role”: “ks-devops/s2i”,
“start”: “2020-12-21T07:19:56.635372”,
“task”: “S2I | Creating S2I secret(2)”,
“task_action”: “command”,
“task_args”: "",
“task_path”: “/kubesphere/installer/roles/ks-devops/s2i/tasks/main.yaml:25”,
“task_uuid”: “0a580a64-0053-52f6-5b1e-00000000002c”,
“uuid”: “42f9737b-4a62-40fb-9ef8-d9ed7ff02bb7”
},
“parent_uuid”: “0a580a64-0053-52f6-5b1e-00000000002c”,
“pid”: 14883,
“runner_ident”: “devops”,
“start_line”: 62,
“stdout”: "fatal: [localhost]: FAILED! => {\“changed\”: true, \“cmd\”: \"/bin/bash /kubesphere/kubesphere/s2i/certs.sh –service webhook-server-service –namespace kubesphere-devops-system –secret webhook-s
erver-secret \n\", \“delta\”: \“0:00:55.395342\”, \“end\”: \“2020-12-21 07:20:52.684270\”, \“msg\”: \“non-zero return code\”, \“rc\”: 1, \“start\”: \“2020-12-21 07:19:57.288928\”, \“stderr\”: \“Generating RSA private key, 2048 bit long modulus (2 primes)\n…………………+++++\n………+++++\ne is 65537 (0×010001)\nERROR: After approving csr webhook-server-service.kubesphere-devops-system, the signed certificate did not appear on the resource. Giving up after 10 attempts.\”, \“stderr_lines\”: [\“Generating RSA private key, 2048 bit long modulus (2 primes)\”, \“…………………+++++\”, \“………+++++\”, \“e is 65537 (0×010001)\”, \“ERROR: After approving csr webhook-server-service.kubesphere-devops-system, the signed certificate did not appear on the resource. Giving up after 10 attempts.\”], \“stdout\”: \"creating certs in tmpdir /tmp/tmp.CIljAd \ncertificatesigningrequest.certificates.k8s.io \\“webhook-server-service.kubesphere-devops-system\\” deleted\ncertificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system created\nNAME AGE REQUESTOR CONDITION\nwebhook-server-service.kubesphere-devops-system 0s system:serviceaccount:kubesphere-system:ks-installer Pending\ncertificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system approved\", \“stdout_lines\”: [\“creating certs in tmpdir /tmp/tmp.CIljAd \”, \"certificatesigningrequest.certificates.k8s.io \\“webhook-server-service.kubesphere-devops-system\\” deleted\", \“certificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system created\”, \“NAME AGE REQUESTOR CONDITION\”, \“webhook-server-service.kubesphere-devops-system 0s system:serviceaccount:kubesphere-system:ks-installer Pending\”, \“certificatesigningrequest.certificates.k8s.io/webhook-server-service.kubesphere-devops-system approved\”]}", “uuid”: “42f9737b-4a62-40fb-9ef8-d9ed7ff02bb7”
}

E1221 07:24:31.362198 1 reflector.go:284] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to watch *unstructured.Unstructured: Get "https://10.254.0.1:443/
apis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&resourceVersion=861583&timeoutSeconds=463&watch=true": dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:24:32.368375 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refused
E1221 07:26:17.516765 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:18.518051 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:19.519610 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:20.520772 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:21.522038 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:22.523218 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: connect: connection refusedE1221 07:26:53.524498 1 reflector.go:131] pkg/mod/k8s.io/client-go@v0.0.0-20190411052641-7a6b4715b709/tools/cache/reflector.go:99: Failed to list *unstructured.Unstructured: Get "https://10.254.0.1:443/a
pis/installer.kubesphere.io/v1alpha1/namespaces/kubesphere-system/clusterconfigurations?fieldSelector=metadata.name%3Dks-installer&limit=500&resourceVersion=0″: dial tcp 10.254.0.1:443: i/o timeout~ `
ks-install报这个错误,DevOps没安装成功

Forest-L

mrz001 机器防火墙关了没

mrz001

Forest-L
修改kube-controller-manager相关配置文件，增加如下两个配置
–cluster-signing-cert-file=/opt/kubernetes/ssl/ca.pem \
–cluster-signing-key-file=/opt/kubernetes/ssl/ca-key.pem \