devops 执行到部署到kubernetes 报错PEM is invalid: no begin marker

LLLDDD

devops 执行到部署到kubernetes 报错

Starting Kubernetes deployment
ERROR: ERROR: java.lang.RuntimeException: java.io.IOException: PEM is invalid: no begin marker
hudson.remoting.ProxyException: java.lang.RuntimeException: java.io.IOException: PEM is invalid: no begin marker
	at io.kubernetes.client.util.credentials.ClientCertificateAuthentication.provide(ClientCertificateAuthentication.java:46)
	at io.kubernetes.client.util.credentials.KubeconfigAuthentication.provide(KubeconfigAuthentication.java:39)
	at io.kubernetes.client.util.ClientBuilder.build(ClientBuilder.java:300)
	at io.kubernetes.client.util.Config.fromConfig(Config.java:91)
	at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.<init>(KubernetesClientWrapper.java:171)
	at com.microsoft.jenkins.kubernetes.KubernetesDeployContext$ClientWrapperFactoryImpl.buildClient(KubernetesDeployContext.java:482)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:143)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
	at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
	at java.lang.Thread.run(Thread.java:748)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 20.20.5.74/20.20.5.74:39832
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:957)
		at hudson.FilePath.act(FilePath.java:1160)
		at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
		at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
		at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
		at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
		at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
		at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
		at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
		at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
		at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
		... 1 more
Caused by: hudson.remoting.ProxyException: java.io.IOException: PEM is invalid: no begin marker
	at io.kubernetes.client.util.SSLUtils.decodePem(SSLUtils.java:179)
	at io.kubernetes.client.util.SSLUtils.loadKey(SSLUtils.java:113)
	at io.kubernetes.client.util.SSLUtils.createKeyStore(SSLUtils.java:152)
	at io.kubernetes.client.util.SSLUtils.createKeyStore(SSLUtils.java:92)
	at io.kubernetes.client.util.SSLUtils.keyManagers(SSLUtils.java:72)
	at io.kubernetes.client.util.credentials.ClientCertificateAuthentication.provide(ClientCertificateAuthentication.java:37)
	... 17 more
Kubernetes deployment ended with HasError

这是我的kubeconfig

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR1RENDQXFDZ0F3SUJBZ0lVR1FXTmoxSW5JcjkzaTNOcldjb3ZFMSsxb0Zrd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1lURUxNQWtHQTFVRUJoTUNRMDR4RVRBUEJnTlZCQWdUQ0VoaGJtZGFhRzkxTVFzd0NRWURWUVFIRXdKWQpVekVNTUFvR0ExVUVDaE1EYXpoek1ROHdEUVlEVlFRTEV3WlRlWE4wWlcweEV6QVJCZ05WQkFNVENtdDFZbVZ5CmJtVjBaWE13SUJjTk1Ua3hNakUyTURjek16QXdXaGdQTWpFeE9URXhNakl3TnpNek1EQmFNR0V4Q3pBSkJnTlYKQkFZVEFrTk9NUkV3RHdZRFZRUUlFd2hJWVc1bldtaHZkVEVMTUFrR0ExVUVCeE1DV0ZNeEREQUtCZ05WQkFvVApBMnM0Y3pFUE1BMEdBMVVFQ3hNR1UzbHpkR1Z0TVJNd0VRWURWUVFERXdwcmRXSmxjbTVsZEdWek1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTd4T2pmUkNrK3JkOWZoZ0J4QTNvcmpXZ1Z0bEIKOWc4a1BpU3cxTFdCRVhPb001aUZLRHpxRk5aZDNNVzd3VDZMcWFRMllEUmhZYnlBL3UwMXJhTzlLY2R3bHJWMQpkVGxMT2hoU2JQWFBwNGlUMVBseUNlejJiWVBnRzZXcDJDNVh3emJoZlhJalhnM0lmTnVsdnRJUFl2c1U1ZWRGCmJuNExuTHJjRGpHZ2kyeDNpbFJyWkYwSDNsSkU4T1VFRHBLOEpIZThRY3llK0tBN1VhL0ZaekZVWjNJMWRNcFUKUlJZTXBRaU9GZzFobFZHS2Q0dHFxVVI0cDRYemlrRTVpM1kwUUhoL3MyWEVnL2hNWldkSTdJOTBnTUV6My9EYwpnUE9hNXBQT204dlE0RFhhR0dOTXhrNmxLUzV2TndWRWU3U1ZGR3NtOFp3WUJGdDg5UXluUHNEMzN3SURBUUFCCm8yWXdaREFPQmdOVkhROEJBZjhFQkFNQ0FRWXdFZ1lEVlIwVEFRSC9CQWd3QmdFQi93SUJBakFkQmdOVkhRNEUKRmdRVWozRlpEblJJREtkVDlUMk5LblFROVhnajZ0UXdId1lEVlIwakJCZ3dGb0FVajNGWkRuUklES2RUOVQyTgpLblFROVhnajZ0UXdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQnV1MldEVmtoRVFpRk9ZaGdWcUoycG1oZ0tSCnkwM1k0SGlzdmE4N05NYmRMbWJtcnZPYmsrWXo3T016aWRKRjZFeVJGbVJnSUFQZHdIMjR6RmpOenpRREYvVUsKdjlmQ2lqVWQzdjFwTkNGNkt6aG9QQVpQMElPUGlZUVVralBlZVM1ZHVFd3NOOWFzWTNxRDBZekNneDVYVDhrbgpCOWRpZG1jeUJDdTNGWmJiVHRaaVhYaVE2ZDREaXFzQWtwSmI2aWFaRVJPSUVsVEJKSjdGNk1LRVVGMUJWMEh4CkpZbDJ1YklRMDc2WFdwUEVjaC9xMkh2eGE4bHhFYlovY1UxNEY1VkpMSERwWi94QzZRMXdZV2N1WjJwWW96bHoKeGxIK3FmYjBQNWMzSVMyNUEzZ05BVXB0TWRjRSt2b2V4ME0vVUNMaGVoKzloaVNXam42QzhHZE5QZEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    server: https://10.88.0.1:443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: admin
    namespace: default
  name: admin@kubernetes
current-context: admin@kubernetes
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate-data: ""
    client-key-data: ""

我把那个证书解码
—–BEGIN CERTIFICATE—–
MIIDuDCCAqCgAwIBAgIUGQWNj1InIr93i3NrWcovE1+1oFkwDQYJKoZIhvcNAQEL
BQAwYTELMAkGA1UEBhMCQ04xETAPBgNVBAgTCEhhbmdaaG91MQswCQYDVQQHEwJY
UzEMMAoGA1UEChMDazhzMQ8wDQYDVQQLEwZTeXN0ZW0xEzARBgNVBAMTCmt1YmVy
bmV0ZXMwIBcNMTkxMjE2MDczMzAwWhgPMjExOTExMjIwNzMzMDBaMGExCzAJBgNV
BAYTAkNOMREwDwYDVQQIEwhIYW5nWmhvdTELMAkGA1UEBxMCWFMxDDAKBgNVBAoT
A2s4czEPMA0GA1UECxMGU3lzdGVtMRMwEQYDVQQDEwprdWJlcm5ldGVzMIIBIjAN
BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7xOjfRCk+rd9fhgBxA3orjWgVtlB
9g8kPiSw1LWBEXOoM5iFKDzqFNZd3MW7wT6LqaQ2YDRhYbyA/u01raO9KcdwlrV1
dTlLOhhSbPXPp4iT1PlyCez2bYPgG6Wp2C5XwzbhfXIjXg3IfNulvtIPYvsU5edF
bn4LnLrcDjGgi2×3ilRrZF0H3lJE8OUEDpK8JHe8Qcye+KA7Ua/FZzFUZ3I1dMpU
RRYMpQiOFg1hlVGKd4tqqUR4p4XzikE5i3Y0QHh/s2XEg/hMZWdI7I90gMEz3/Dc
gPOa5pPOm8vQ4DXaGGNMxk6lKS5vNwVEe7SVFGsm8ZwYBFt89QynPsD33wIDAQAB
o2YwZDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIBAjAdBgNVHQ4E
FgQUj3FZDnRIDKdT9T2NKnQQ9Xgj6tQwHwYDVR0jBBgwFoAUj3FZDnRIDKdT9T2N
KnQQ9Xgj6tQwDQYJKoZIhvcNAQELBQADggEBABuu2WDVkhEQiFOYhgVqJ2pmhgKR
y03Y4Hisva87NMbdLmbmrvObk+Yz7OMzidJF6EyRFmRgIAPdwH24zFjNzzQDF/UK
v9fCijUd3v1pNCF6KzhoPAZP0IOPiYQUkjPeeS5duEwsN9asY3qD0YzCgx5XT8kn
B9didmcyBCu3FZbbTtZiXXiQ6d4DiqsAkpJb6iaZEROIElTBJJ7F6MKEUF1BV0Hx
JYl2ubIQ076XWpPEch/q2Hvxa8lxEbZ/cU14F5VJLHDpZ/xC6Q1wYWcuZ2pYozlz
xlH+qfb0P5c3IS25A3gNAUptMdcE+voex0M/UCLheh+9hiSWjn6C8GdNPdI=
—–END CERTIFICATE—–
是有开始标记的
然后我也比对了工具箱的kubeconfig

是一模一样的数据,且工具箱的kubectl也是可以正常使用的

我在devops的凭证信息

content是自动加载IDE,比对过是一致的
我的devops配置

runzexia

LLLDDD 这个问题是kubeconfig生成错误，在web控制台里可以用应该是设置了service account的缘故，可以把这个贴到一台主机试一下，应该是没办法使用的。现在可以临时切换另外一个账号的再生成kubeconfig @hongming ？

hongming

这里admin 的 kubeconfig 的序列化输出有问题，可以从kubesphere-controls-system这个项目下kubeconfig-admin 这个 configmap 中直接复制出来

LLLDDD

hongming 好的我试试

Lee

hongming
复制出来的后的，也保错，能帮忙看看吗？
Starting Kubernetes deployment
Loading configuration: /home/jenkins/agent/workspace/Ao_codedevops-java-sample_master/deploy/dev-ol/devops-sample-svc.yaml
ERROR: ERROR: java.lang.RuntimeException: io.kubernetes.client.ApiException: Not Found
hudson.remoting.ProxyException: java.lang.RuntimeException: io.kubernetes.client.ApiException: Not Found
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager.handleApiException(ResourceManager.java:193)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.createResource(V1ResourceManager.java:453)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.createResource(V1ResourceManager.java:371)
at com.microsoft.jenkins.kubernetes.wrapper.ResourceManager$ResourceUpdater.createOrApply(ResourceManager.java:100)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.handleResource(KubernetesClientWrapper.java:289)
at com.microsoft.jenkins.kubernetes.wrapper.KubernetesClientWrapper.apply(KubernetesClientWrapper.java:256)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.doCall(DeploymentCommand.java:172)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:124)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand$DeploymentTask.call(DeploymentCommand.java:106)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.233.99.15/10.233.99.15:59338
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:957)
at hudson.FilePath.act(FilePath.java:1160)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:68)
at com.microsoft.jenkins.kubernetes.command.DeploymentCommand.execute(DeploymentCommand.java:45)
at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
at com.microsoft.jenkins.kubernetes.KubernetesDeploy.perform(KubernetesDeploy.java:42)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:54)
at com.microsoft.jenkins.azurecommons.command.SimpleBuildStepExecution.run(SimpleBuildStepExecution.java:35)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
… 1 more
Caused by: hudson.remoting.ProxyException: io.kubernetes.client.ApiException: Not Found
at io.kubernetes.client.ApiClient.handleResponse(ApiClient.java:886)
at io.kubernetes.client.ApiClient.execute(ApiClient.java:802)
at io.kubernetes.client.apis.CoreV1Api.createNamespacedServiceWithHttpInfo(CoreV1Api.java:9415)
at io.kubernetes.client.apis.CoreV1Api.createNamespacedService(CoreV1Api.java:9397)
at com.microsoft.jenkins.kubernetes.wrapper.V1ResourceManager$ServiceUpdater.createResource(V1ResourceManager.java:450)
… 16 more
Api call failed with code 404, detailed message: {
“kind”: “Status”,
“apiVersion”: “v1″,
“metadata”: {

},
“status”: “Failure”,
“message”: "namespaces \“kubesphere-sample-dev\” not found",
“reason”: “NotFound”,
“details”: {
“name”: “kubesphere-sample-dev”,
“kind”: “namespaces”
},
“code”: 404
}
Kubernetes deployment ended with HasError

hongming

Lee 错误提示也很明确： “namespaces \“kubesphere-sample-dev\” not found”，需要参照着文档，先把namespace 创建出来