操作系统信息
虚拟机,Centos7.9,8C/16G
Kubernetes版本信息
v1.21.14
容器运行时
docker 20.10.8
KubeSphere版本信息
v3.3.2。使用kk脚本在线安装,kk版本3.0.7
问题是什么
Kubesphere默认使用glusterfs存储,今天发现elasticsearch的存储卷空间不大够,就对data-elasticsearch-logging-data-0,1,2三个存储卷进行了扩展。扩展过程中导致2个node进入了NotReady状态,恢复后所有容器重启,elasticsearch启动失败,报错日志如下:
OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=1
[2023-03-30T08:27:07,924][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-logging-data-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Failed to create node environment
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:163) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.8.22.jar:6.8.22]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) ~[elasticsearch-6.8.22.jar:6.8.22]
Caused by: java.lang.IllegalStateException: Failed to create node environment
at org.elasticsearch.node.Node.<init>(Node.java:299) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.22.jar:6.8.22]
... 6 more
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/nodes/0
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:388) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:273) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:206) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:270) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.node.Node.<init>(Node.java:296) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) ~[elasticsearch-6.8.22.jar:6.8.22]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) ~[elasticsearch-6.8.22.jar:6.8.22]
... 6 more