pod/openpitrix-iam-service 起不来

mars

pod/openpitrix-iam-service-deployment-864df9fb6f-sdw5z 0/1 CrashLoopBackOff

logs是这样的:

kubectl -n openpitrix-system logs openpitrix-iam-service-deployment-864df9fb6f-sdw5z
2019-10-16 03:06:24.17648 -INFO- Release OpVersion: [v0.3.5] (grpc_server.go:79)
2019-10-16 03:06:24.17651 -INFO- Git Commit Hash: [9333eae7354fc6634785df7a2fef35e74ed8dce5] (grpc_server.go:79)
2019-10-16 03:06:24.17652 -INFO- Build Time: [2018-11-22 06:37:13] (grpc_server.go:79)
2019-10-16 03:06:24.17653 -INFO- Service [iam-service] start listen at port [9115] (grpc_server.go:81)
2019-10-16 03:06:24.17898 -ERROR- Error 1146: Table 'iam.user' doesn't exist (event.go:36)
2019-10-16 03:06:24.17901 -ERROR- dbr.select.load.query: map[sql:SELECT user_id, username, password, email, role, description, status, create_time, update_time, status_time FROM user WHERE (`email` = 'admin@op.com')] (event.go:37)
2019-10-16 03:06:24.18013 -ERROR- Error 1146: Table 'iam.user_client' doesn't exist (event.go:36)
panic: Error 1146: Table 'iam.user' doesn't exist
goroutine 32 [running]:
openpitrix.io/openpitrix/pkg/service/iam.initIAMAccount()
        /go/src/openpitrix.io/openpitrix/pkg/service/iam/init.go:78 +0xdc3
created by openpitrix.io/openpitrix/pkg/service/iam.Serve
        /go/src/openpitrix.io/openpitrix/pkg/service/iam/server.go:25 +0x5f

但是 Job/openpitrix-db-init-job 是completed 的, 所以这里是不是db init执行不对呢?
如果是的话,我能否把openpitrix-system这个namespace删掉,然后镜像里面的ansible只保留openpitrix这个role,再执行一遍installer?
不太确定这样能不能得到预期的效果, 拜托帮忙看一下,thx

zheng1

你好，可以检查一下 openpitrix-iam-db-ctrl-job-xxxxxx 这个pod的日志，看看错误的原因。

mars

zheng1 这边看是pull image失败了
Failed to pull image "openpitrix/openpitrix:flyway-0.3.5": rpc error: code = Unknown desc = failed to resolve image "docker.io/openpitrix/openpitrix:flyway-0.3.5": no available registry endpoint: docker.io/openpitrix/openpitrix:flyway-0.3.5 not found
部署的0.3.5的版本

ks的版本跟openpitrix有对应关系吗？我可以直接布最新版openpitrix试一下吗？

mars

zheng1 我这儿单独部署了openpitrix，最新的版本，但是在kubesphere点击进到openpitrix时候右上角报“认证失败”，我是需要改动哪儿吗？

qcloud

mars 请教一下，你自己单独部署openpitrix在KubeSphere2.1.1上，是否成功？我在部署KubeSphere2.1.1时，只能设置openpitrix为false时，才能部署成功。
现在也想单独部署openpitrix，不知道会遇到哪些坑？

hongming

@mars kubesphere中集成的应用商店功能，关联了固定的 openpitrix 版本， kubesphere 2.0.2 暂时只能对接 openpitrix 0.3.5，kubesphere 2.1+ 则可以适配 openpitrix 0.4+

Feynman

qcloud 安装后在 configmap 里开启 openpitrix 就可以自动安装，你是遇到了什么问题？

qcloud

Feynman 在KubeSphere2.1.1安装中openpitrix必须设置为false，才能安装成功在这个单子上有详细描述

qcloud

遇到的问题，就是发现该minio-845b7bd867-z5×77 pod是CrashLoopBackOff和running之间不断切换

qcloud

能远程看看是啥原因吗？

zheng1

qcloud 你应该看一下minio这个pod的日志，看看为什么会crash。

OpenPitrix可以和KubeSphere共存，在部署OpenPitrix需要指定另外的namespace，不能直接使用openpitrix-system

qcloud

zheng1 那我已经安装上了openpitrix，可所有服务都异常，查看namespace也确实是openpitrix-system，那请教接下来怎么操作？

zheng1

qcloud 可以先尝试清除openpitrix， https://openpitrix.io/install#%E6%B8%85%E7%90%86%E7%8E%AF%E5%A2%83
然后重新部署到新的namespace。部署完成之后再回到ks-installer启用KubeSphere内置的应用商店

qcloud

zheng1 我想分析一下，通过KubeSphere2.1.1安装后在 configmap 里开启 openpitrix 就可以自动安装，可查看pods

openpitrix-system hyperpitrix-release-app-job-499h7 0/1 Init:0/2 0 45h
openpitrix-system openpitrix-hyperpitrix-deployment-6df877b655-z9tbv 0/1 Init:0/11 0 45h
openpitrix-system openpitrix-rp-kubernetes-deployment-cbb6786f7-mlg9p 0/1 Init:½ 0 45h
[root@k8sphere01 _]#

[root@k8sphere01 _]# kubectl describe pods -n openpitrix-system hyperpitrix-release-app-job-499h7
Name: hyperpitrix-release-app-job-499h7
Namespace: openpitrix-system
Priority: 0
Node: k8sphere01/192.168.108.72
Start Time: Sat, 30 May 2020 11:20:41 +0800
Labels: app=hyperpitrix
controller-uid=c40a1994-d2a2-4015-8ec1-92b62822a65d
job=hyperpitrix-release-app
job-name=hyperpitrix-release-app-job
version=v0.4.8
Annotations: <none>
Status: Pending
IP: 10.233.118.30
IPs:
IP: 10.233.118.30
Controlled By: Job/hyperpitrix-release-app-job
Init Containers:
wait-app-manager:
Container ID: docker://bee6e8209a1b9f264ad534d3ad948be0021e926898e6d54cd8334c43ef8060d1
Image: alpine:3.10.4
Image ID: docker://sha256:af341ccd2df8b0e2d67cf8dd32e087bfda4e5756ebd1c76bbf3efa0dc246590e
Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z openpitrix-app-manager.openpitrix-system.svc 9102; do echo “waiting for app-manager”; sleep 2; done;
State: Running
Started: Sat, 30 May 2020 11:20:42 +0800
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
wait-attachment-manager:
Container ID:

Image: alpine:3.10.4
Image ID:

Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z openpitrix-attachment-manager.openpitrix-system.svc 9122; do echo “waiting for attachment-manager”; sleep 2; done;
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
Containers:
hyperpitrix-release-app-job:
Container ID:

Image: openpitrix/release-app:v0.4.3
Image ID:

Port: <none>
Host Port: <none>
Command:
release-app
-path
/data/helm-pkg/
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-cbzwb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cbzwb
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events: <none>
[root@k8sphere01 _]#

这种问题怎么去分析原因，并解决

zheng1

qcloud describe一下openpitrix-hyperpitrix-deployment-6df877b655-z9tbv 这个pod，需要知道它为什么没起来

qcloud

zheng1 一直在等待 wait-mysql

qcloud

[root@k8sphere01 conf]# kubectl describe pods -n openpitrix-system openpitrix-hyperpitrix-deployment-75d7664679-v7lgf
Name: openpitrix-hyperpitrix-deployment-75d7664679-v7lgf
Namespace: openpitrix-system
Priority: 0
Node: k8sphere01/192.168.108.72
Start Time: Mon, 01 Jun 2020 10:15:34 +0800
Labels: app=openpitrix
component=openpitrix-hyperpitrix
pod-template-hash=75d7664679
version=v0.4.8
Annotations: kubectl.kubernetes.io/restartedAt: 2020-06-01T10:15:33+08:00
Status: Pending
IP: 10.233.118.35
IPs:
IP: 10.233.118.35
Controlled By: ReplicaSet/openpitrix-hyperpitrix-deployment-75d7664679
Init Containers:
wait-mysql:
Container ID: docker://23a1708902c267442d8f0f8f79a9f4385284a1fad7a6d3edb0d3393df804a28e
Image: alpine:3.10.4
Image ID: docker://sha256:af341ccd2df8b0e2d67cf8dd32e087bfda4e5756ebd1c76bbf3efa0dc246590e
Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z mysql.kubesphere-system.svc 3306; do echo “waiting for mysql”; sleep 2; done;
State: Running
Started: Mon, 01 Jun 2020 10:15:39 +0800
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
wait-etcd:
Container ID:

Image: alpine:3.10.4
Image ID:

Port: <none>
Host Port: <none>
Command:
sh
-c
until nc -z etcd.kubesphere-system.svc 2379; do echo “waiting for etcd”; sleep 2; done;
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-db-init:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/sql/ddl/ddl_init.sh
-hmysql.kubesphere-system.svc
-uroot
–connect-timeout=5
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-app-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/app
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/app
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-attachment-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/attachment
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/attachment
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-cluster-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/cluster
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/cluster
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-isv-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/isv
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/isv
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-job-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/job
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/job
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-repo-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/repo
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/repo
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-runtime-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/runtime
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/runtime
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
openpitrix-task-db-ctrl:
Container ID:

Image: openpitrix/openpitrix:flyway-v0.4.8
Image ID:

Port: <none>
Host Port: <none>
Command:
/flyway/flyway
-X
-url=jdbc:mysql://mysql.kubesphere-system.svc/task
-user=root
-validateOnMigrate=false
-locations=filesystem:/flyway/sql/task
migrate
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
FLYWAY_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
Containers:
hyperpitrix:
Container ID:

Image: openpitrix/openpitrix:v0.4.8
Image ID:

Ports: 9100/TCP, 9102/TCP, 9108/TCP, 9122/TCP, 9104/TCP, 9118/TCP, 9106/TCP, 9101/TCP, 9108/TCP, 9121/TCP, 9103/TCP, 9107/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
Command:
hyperpitrix
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Limits:
cpu: 500m
memory: 500Mi
Requests:
cpu: 100m
memory: 100Mi
Environment:
OPENPITRIX_GRPC_SHOW_ERROR_CAUSE: true
OPENPITRIX_LOG_LEVEL: debug
OPENPITRIX_ETCD_ENDPOINTS: etcd.kubesphere-system.svc:2379
OPENPITRIX_MYSQL_HOST: mysql.kubesphere-system.svc
OPENPITRIX_ATTACHMENT_ENDPOINT: http://minio.kubesphere-system.svc:9000
OPENPITRIX_ATTACHMENT_BUCKET_NAME: openpitrix-attachment
OPENPITRIX_MYSQL_PASSWORD: password
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-cbzwb (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-cbzwb:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-cbzwb
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled <unknown> default-scheduler Successfully assigned openpitrix-system/openpitrix-hyperpitrix-deployment-75d7664679-v7lgf to k8sphere01
Normal Pulled 23m kubelet, k8sphere01 Container image “alpine:3.10.4” already present on machine
Normal Created 23m kubelet, k8sphere01 Created container wait-mysql
Normal Started 23m kubelet, k8sphere01 Started container wait-mysql
[root@k8sphere01 conf]#

zheng1

qcloud 检查下kubesphere这个namespace下面的mysql是否正常

qcloud

zheng1 怎么检查？能远程看一下不？

qcloud

zheng1 谢谢已解决该问题。确实是mysql的问题，但因其状态是running，就忽略掉这点而没有去看详细信息。

qcloud

qcloud 已发邮件