创建部署问题时,请参考下面模板,你提供的信息越多,越容易及时获得解答。如果未按模板创建问题,管理员有权关闭问题。
确保帖子格式清晰易读,用 markdown code block 语法格式化代码块。
你只花一分钟创建的问题,不能指望别人花上半个小时给你解答。

操作系统信息
例如:虚拟机/物理机,Centos7.5/Ubuntu18.04,4C/8G

Kubernetes版本信息
kubectl version 命令执行结果贴在下方

容器运行时
containerd

KubeSphere版本信息
例如:v3.4.0。离线安装 在已有K8s上安装

问题是什么

启用了altering 配置 后 altermanager 一直有这个报错。

[root@k8s-master01 ~]# kubectl logs alertmanager-main-0 -n kubesphere-monitoring-system -f

level=info ts=2023-11-27T07:31:21.768Z caller=main.go:225 msg=“Starting Alertmanager” version=“(version=0.23.0, branch=HEAD, revision=61046b17771a57cfd4c4a51be370ab930a4d7d54)”

level=info ts=2023-11-27T07:31:21.768Z caller=main.go:226 build_context=“(go=go1.16.7, user=root@e21a959be8d2, date=20210825-10:48:55)”

level=warn ts=2023-11-27T07:31:24.868Z caller=cluster.go:251 component=cluster msg=“failed to join cluster” err=“3 errors occurred:\n\t* Failed to join 10.244.1.3: dial tcp 10.244.1.3:9094: connect: no route to host\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=info ts=2023-11-27T07:31:24.868Z caller=cluster.go:253 component=cluster msg=“will retry joining cluster every 10s”

level=warn ts=2023-11-27T07:31:24.868Z caller=main.go:316 msg=“unable to join gossip mesh” err=“3 errors occurred:\n\t* Failed to join 10.244.1.3: dial tcp 10.244.1.3:9094: connect: no route to host\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=info ts=2023-11-27T07:31:24.868Z caller=cluster.go:671 component=cluster msg=“Waiting for gossip to settle…” interval=2s

level=info ts=2023-11-27T07:31:25.055Z caller=coordinator.go:113 component=configuration msg=“Loading configuration file” file=/etc/alertmanager/config/alertmanager.yaml

level=info ts=2023-11-27T07:31:25.055Z caller=coordinator.go:126 component=configuration msg=“Completed loading of configuration file” file=/etc/alertmanager/config/alertmanager.yaml

level=info ts=2023-11-27T07:31:25.058Z caller=main.go:518 msg=Listening address=:9093

level=info ts=2023-11-27T07:31:25.058Z caller=tls_config.go:191 msg=“TLS is disabled.” http2=false

level=info ts=2023-11-27T07:31:26.869Z caller=cluster.go:696 component=cluster msg=“gossip not settled” polls=0 before=0 now=1 elapsed=2.00072561s

level=info ts=2023-11-27T07:31:28.855Z caller=coordinator.go:113 component=configuration msg=“Loading configuration file” file=/etc/alertmanager/config/alertmanager.yaml

level=info ts=2023-11-27T07:31:28.856Z caller=coordinator.go:126 component=configuration msg=“Completed loading of configuration file” file=/etc/alertmanager/config/alertmanager.yaml

level=info ts=2023-11-27T07:31:34.872Z caller=cluster.go:688 component=cluster msg=“gossip settled; proceeding” elapsed=10.004086522s

level=warn ts=2023-11-27T07:31:39.881Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:31:49.881Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=10.244.2.249:9094 err=“1 error occurred:\n\t* Failed to join 10.244.2.249: dial tcp 10.244.2.249:9094: i/o timeout\n\n”

level=warn ts=2023-11-27T07:31:54.877Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:31:54.880Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-2.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:09.877Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:09.880Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-2.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:24.878Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:24.882Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-2.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:39.906Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-1.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-1.alertmanager-operated:9094: lookup alertmanager-main-1.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

level=warn ts=2023-11-27T07:32:39.909Z caller=cluster.go:461 component=cluster msg=refresh result=failure addr=alertmanager-main-2.alertmanager-operated:9094 err=“1 error occurred:\n\t* Failed to resolve alertmanager-main-2.alertmanager-operated:9094: lookup alertmanager-main-2.alertmanager-operated on 10.96.0.10:53: no such host\n\n”

    qing_tian
    可以尝试重启下服务,这里是 alertmanager 因为网络原因失败创建集群了