Deploy minio FAILED!

humphery755 · 2020年9月17日

of ks-install logs 日志一直停留在这个位置
`
TASK [common : Kubesphere | Deploy minio] **************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “/usr/local/bin/helm upgrade –install ks-minio /kubesphere/kubesphere/minio-ha -f /kubesphere/kubesphere/custom-values-minio.yaml –set fullnameOverride=minio –namespace kubesphere-system –wait –timeout 1800s\n”, “delta”: “0:00:19.928550”, “end”: “2020-09-16 08:34:00.494830”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2020-09-16 08:33:40.566280”, “stderr”: "Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade minio/templates/post-install-create-bucket-job.yaml failed: jobs.batch \“minio-make-bucket-job\” already exists", “stderr_lines”: ["Error: UPGRADE FAILED: post-upgrade hooks failed: warning: Hook post-upgrade minio/templates/post-install-create-bucket-job.yaml failed: jobs.batch \“minio-make-bucket-job\” already exists"], “stdout”: "", “stdout_lines”: []}
…ignoring

TASK [common : debug] **********************************************************
ok: [localhost] => {
“msg”: [
“1. check the storage configuration and storage server”,
“2. make sure the DNS address in /etc/resolv.conf is available”,
“3. execute ‘kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job’ to watch logs”,
“4. execute ‘helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job’”,
“5. Restart the installer pod in kubesphere-system namespace”
]
}

TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “It is suggested to refer to the above methods for troubleshooting problems .”}
kubectl get all --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kubesphere-system pod/ks-installer-744f4574f6-2dnq8 1/1 Running 0 3d23h
kubesphere-system pod/minio-749d86bd7d-9skpx 1/1 Running 0 32h
kubesphere-system pod/openldap-0 1/1 Running 1 3d22h
kubesphere-system pod/redis-696b58877-k89q8 1/1 Running 1 3d22h

NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubesphere-system service/minio ClusterIP 10.233.38.26 <none> 9000/TCP 32h
`

Forest-L · 2020年9月17日

humphery755 可以先按上面的方法排查下

“1. check the storage configuration and storage server”,
“2. make sure the DNS address in /etc/resolv.conf is available”,
“3. execute ‘kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job’ to watch logs”,
“4. execute ‘helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job’”,
“5. Restart the installer pod in kubesphere-system namespace”

2、kubectl logs看下minio pod的日志。

humphery755 · 2020年9月17日

Forest-L

`
helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job
/usr/local/bin/helm upgrade –install ks-minio /kubesphere/kubesphere/minio-ha -f /kubesphere/kubesphere/custom-values-minio.yaml –set fullnameOverride=minio –namespace kubesphere-system –wait –timeout 1800s

Release “ks-minio” does not exist. Installing it now.
进入ks-installer-744f4574f6-2dnq8重新安装后,一直卡在helm upgrade --install这句，过了1800s后
Error: failed post-install: timed out waiting for the condition
of pod/minio-make-bucket-job-ql6ch logs
mc: <ERROR> Unable to initialize new config from the provided credentials. Get http://minio:9000/probe-bucket-sign-udemsclvbdgc/?location=: dial tcp: lookup minio on 169.254.25.10:53: no such host.
minio的dns未能正确解析，发现coredns在work节点CrashLoopBackOff
kube-system pod/nodelocaldns-lf9pt 0/1 CrashLoopBackOff 38 175m
of pod/nodelocaldns-lf9pt logs
2020/09/17 13:52:19 [INFO] Using Corefile /etc/coredns/Corefile
2020/09/17 13:52:19 [ERROR] Failed to read node-cache coreFile /etc/coredns/Corefile.base - open /etc/coredns/Corefile.base: no such file or directory
2020/09/17 13:52:19 [ERROR] Failed to sync kube-dns config directory /etc/kube-dns, err: lstat /etc/kube-dns: no such file or directory
Listen: listen tcp 169.254.25.10:53: bind: address already in use
`
删除master和work上的nodelocaldns重建后，问题依旧

humphery755 · 2020年9月17日

humphery755

NAME STATUS ROLES AGE VERSION
master1 Ready master 4d3h v1.18.6
node1 Ready worker 4d3h v1.18.6

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/nodelocaldns-9hh9d 1/1 Running 0 116s
kube-system pod/nodelocaldns-zx5c2 0/1 CrashLoopBackOff 3 109s

dongweibh · 2020年9月21日

离线安装3.0，也遇到了这个问题
TASK [common : Kubesphere | Check minio] ***************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: "/usr/local/bin/helm list -n kubesphere-system | grep \“ks-minio\”\n", “delta”: “0:00:00.139724″, “end”: “2020-09-21 06:45:08.530778″, “msg”: “non-zero return code”, “rc”: 1, “start”: “2020-09-21 06:45:08.391054″, “stderr”: "", “stderr_lines”: [], “stdout”: "", “stdout_lines”: []}
…ignoring

TASK [common : Kubesphere | Deploy minio] **************************************
fatal: [localhost]: FAILED! => {“changed”: true, “cmd”: “/usr/local/bin/helm upgrade –install ks-minio /kubesphere/kubesphere/minio-ha -f /kubesphere/kubesphere/custom-values-minio.yaml –set fullnameOverride=minio –namespace kubesphere-system –wait –timeout 1800s\n”, “delta”: “0:30:00.885918”, “end”: “2020-09-21 07:15:09.927753”, “msg”: “non-zero return code”, “rc”: 1, “start”: “2020-09-21 06:45:09.041835”, “stderr”: “Error: timed out waiting for the condition”, “stderr_lines”: [“Error: timed out waiting for the condition”], “stdout”: "Release \“ks-minio\” does not exist. Installing it now.", “stdout_lines”: ["Release \“ks-minio\” does not exist. Installing it now."]}
…ignoring

TASK [common : debug] **********************************************************
ok: [localhost] => {
“msg”: [
“1. check the storage configuration and storage server”,
“2. make sure the DNS address in /etc/resolv.conf is available”,
“3. execute ‘kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job’ to watch logs”,
“4. execute ‘helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job’”,
“5. Restart the installer pod in kubesphere-system namespace”
]
}

TASK [common : fail] ***********************************************************
fatal: [localhost]: FAILED! => {“changed”: false, “msg”: “It is suggested to refer to the above methods for troubleshooting problems .”}

PLAY RECAP *********************************************************************
localhost

Lliziyang · 2020年9月27日

离线的helm install 安装的minio失败了，可以手动创建几个，把下面的工作负载导进去，报错的话再建立minio的存储卷，然后再重启ks-install容器就好了

kind: Deployment
apiVersion: apps/v1
metadata:
name: minio
namespace: kubesphere-system
labels:
app: minio
app.kubernetes.io/managed-by: Helm
chart: minio-2.5.16
heritage: Helm
release: ks-minio
annotations:
deployment.kubernetes.io/revision: ‘1’
meta.helm.sh/release-name: ks-minio
meta.helm.sh/release-namespace: kubesphere-system
spec:
replicas: 1
selector:
matchLabels:
app: minio
release: ks-minio
template:
metadata:
name: minio
creationTimestamp: null
labels:
app: minio
release: ks-minio
annotations:
checksum/config: c6cc7f4b40064dffd59b339e133fa4819f787573ee18e1d001435aa4daff8ba2
checksum/secrets: f9625c177e0e74a3b9997c3c65189ebffcfbde7aaa910de0ba38b48b032c1a96
spec:
volumes:
- name: export
persistentVolumeClaim:
claimName: minio
- name: minio-user
secret:
secretName: minio
defaultMode: 420
- name: minio-config-dir
emptyDir: {}
containers:
- name: minio
image: ‘minio/minio:RELEASE.2019-08-07T01-59-21Z’
command:
- /bin/sh
- ‘-ce’
- /usr/bin/docker-entrypoint.sh minio -C /root/.minio/ server /data
ports:
- name: service
containerPort: 9000
protocol: TCP
env:
- name: MINIO_ACCESS_KEY
valueFrom:
secretKeyRef:
name: minio
key: accesskey
- name: MINIO_SECRET_KEY
valueFrom:
secretKeyRef:
name: minio
key: secretkey
- name: MINIO_BROWSER
value: ‘on’
resources:
requests:
cpu: 250m
memory: 256Mi
volumeMounts:
- name: export
mountPath: /data
- name: minio-config-dir
mountPath: /root/.minio/
livenessProbe:
httpGet:
path: /minio/health/live
port: service
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 30
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
path: /minio/health/ready
port: service
scheme: HTTP
initialDelaySeconds: 5
timeoutSeconds: 1
periodSeconds: 15
successThreshold: 1
failureThreshold: 3
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: IfNotPresent
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: ks-minio
serviceAccount: ks-minio
securityContext: {}
schedulerName: default-scheduler
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 100%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600

sunguosheng · 2020年10月13日

你解决了没？？

Hhf12 · 2020年12月10日

53端口被占用，看看是什么其他程序在占着

3344089386 · 2020年12月24日

遇到同样的问题。。up
humphery755 你解决了吗

hrdhrdhrdhrd · 2021年6月30日

你好，同问

Llingyun · 2021年6月30日

我是执行了这两步就可以了

execute ‘helm -n kubesphere-system uninstall ks-minio && kubectl -n kubesphere-system delete job minio-make-bucket-job’”,
Restart the installer pod in kubesphere-system namespace”

YYSH · 2021年7月25日

lingyun bro我跟你一模一样的问题，请问你这两步咋排查的

YYSH · 2021年7月25日

lingyun

Llingyun · 2021年7月26日

YSH 就是先卸载，然后restart

f327888576 · 2021年8月18日

YSH 同样问题

YYSH · 2021年8月26日

f327888576 我是整个流程重新部署就好了，可能之前中间步骤出现了问题

Zzhuyifei-ruichuang · 10 天前

Forest-L

我遇到了一样的问题，排查到第三个步骤时，看到如下信息

root@master:~# kubectl logs -n kubesphere-system -l job-name=minio-make-bucket-job
No resources found in kubesphere-system namespace.

Deploy minio FAILED!

humphery755

Forest-LK零S

humphery755

humphery755

dongweibhK零S

Lliziyang

sunguosheng

Hhf12

3344089386K零S

hrdhrdhrdhrd

Llingyun

YYSH

YYSH

Llingyun

f327888576

YYSH

Zzhuyifei-ruichuang