notifivation-manager接入微信报警收不到信息

wanjunlei

# kubectl logs notification-manager-deployment-5675dc9d7-8ts8f  -n kubesphere-monitoring-system  --tail 500 
level=info ts=2021-03-11T14:55:40.804736133+08:00 caller=main.go:100 msg="Starting notification manager..." addr=:19093 timeout=5s
level=info ts=2021-03-11T14:55:43.523676264+08:00 caller=config.go:340 msg="Setting up informers successfully"
level=info ts=2021-03-11T14:56:01.080585465+08:00 caller=config.go:679 msg="resource change" op=add type=wechat name=default-wechat-config namespace=kubesphere-monitoring-system
level=info ts=2021-03-11T14:56:01.094736472+08:00 caller=config.go:679 msg="resource change" op=add type=wechat name=global-wechat-receiver namespace=kubesphere-monitoring-system

# kubectl logs notification-manager-operator-dd4c6fd6f-pqtrc  -n kubesphere-monitoring-system -c notification-manager-operator  --tail 500 
2021-03-11T14:55:21.665+0800	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": "127.0.0.1:8080"}
2021-03-11T14:55:21.666+0800	INFO	setup	starting manager
I0311 14:55:21.666557       1 leaderelection.go:242] attempting to acquire leader lease  kubesphere-monitoring-system/7b8d27e6.kubesphere.io...
2021-03-11T14:55:21.670+0800	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
I0311 14:55:38.534004       1 leaderelection.go:252] successfully acquired lease kubesphere-monitoring-system/7b8d27e6.kubesphere.io
2021-03-11T14:55:38.534+0800	DEBUG	controller-runtime.manager.events	Normal	{"object": {"kind":"ConfigMap","namespace":"kubesphere-monitoring-system","name":"7b8d27e6.kubesphere.io","uid":"cee2a831-d2e0-45de-87af-b4813c744676","apiVersion":"v1","resourceVersion":"904756"}, "reason": "LeaderElection", "message": "notification-manager-operator-dd4c6fd6f-pqtrc_e30b19a1-2d0c-4038-a604-3c23f7ea9158 became leader"}
2021-03-11T14:55:38.536+0800	INFO	controller-runtime.controller	Starting EventSource	{"controller": "notificationmanager", "source": "kind source: /, Kind="}
2021-03-11T14:55:38.837+0800	INFO	controller-runtime.controller	Starting EventSource	{"controller": "notificationmanager", "source": "kind source: /, Kind="}
2021-03-11T14:55:38.837+0800	INFO	controller-runtime.controller	Starting Controller	{"controller": "notificationmanager"}
2021-03-11T14:55:38.837+0800	INFO	controller-runtime.controller	Starting workers	{"controller": "notificationmanager", "worker count": 1}
2021-03-11T14:55:38.964+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "notificationmanager", "request": "kubesphere-monitoring-system/notification-manager"}
2021-03-11T14:55:39.159+0800	ERROR	controllers.NotificationManager	Failed to CreateOrUpdate deployment	{"NotificationManager Operator": "kubesphere-monitoring-system/notification-manager", "result": "unchanged", "error": "Operation cannot be fulfilled on deployments.apps \"notification-manager-deployment\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
github.com/kubesphere/notification-manager/pkg/controllers.(*NotificationManagerReconciler).Reconcile
	/workspace/pkg/controllers/notificationmanager_controller.go:94
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:256
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88
2021-03-11T14:55:39.160+0800	ERROR	controller-runtime.controller	Reconciler error	{"controller": "notificationmanager", "request": "kubesphere-monitoring-system/notification-manager", "error": "Operation cannot be fulfilled on deployments.apps \"notification-manager-deployment\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:258
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:232
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.2/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.17.2/pkg/util/wait/wait.go:88
2021-03-11T14:55:40.188+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "notificationmanager", "request": "kubesphere-monitoring-system/notification-manager"}
2021-03-11T14:55:41.420+0800	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "notificationmanager", "request": "kubesphere-monitoring-system/notification-manager"}

wanjunlei
我在ui界面重新部署了下alertmanager，日志如下：

# kubectl logs --tail 500 alertmanager-main-0  -n kubesphere-monitoring-system  -c alertmanager
level=info ts=2021-03-11T07:58:45.069Z caller=main.go:216 msg="Starting Alertmanager" version="(version=0.21.0, branch=HEAD, revision=4c6c03ebfe21009c546e4d1e9b92c371d67c021d)"
level=info ts=2021-03-11T07:58:45.069Z caller=main.go:217 build_context="(go=go1.14.4, user=root@dee35927357f, date=20200617-08:54:02)"
level=info ts=2021-03-11T07:58:45.477Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2021-03-11T07:58:45.562Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml
level=info ts=2021-03-11T07:58:45.664Z caller=main.go:485 msg=Listening address=:9093

wanjunlei
可以发送，如下：

 # curl -X POST -H "Content-Type: application/json" -d @test.json http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts
{"Status":200,"Message":"Notification request accepted"}

这样是不是说am到nm是没有问题的吧，是nm到微信是不通的？

wanjunlei
没有收到通知，日志跟上面我贴的一样，没有产生新的日志信息

wanjunlei
已经升级了，使用json测试，终端打印200，但是nm日志还是没有

我发现升级nm到0.7时，出现个奇怪的现象：nm能apply创建成功，但是kubectl get pod -n xxx时竟然没有，什么错误信息都没有~

wanjunlei
老师，使用json测试，终端打印200，但是nm日志已经有了，日志过来了~
现在nm到微信好像不通，我刚新建个cm，微信上并没有信息出来

wanjunlei
我当前使用的版本是0.7.0~
我在终端执行：

# curl -X POST -H "Content-Type: application/json" -d @test02.json http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts

nm的日志：

level=debug ts=2021-03-11T17:45:46.091975534+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:45:46.09207367+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:45:46.09211236+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:45:46.092926795+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:45:46.092977108+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:45:46.092991445+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:45:46.093001959+08:00 caller=handler.go:176 msg="Worker exit after 812.211µs"

wanjunlei
我在ui上创建/删除cm，nm能打印出响应的信息，但是微信没有收到~

level=info ts=2021-03-11T17:48:28.740767061+08:00 caller=config.go:677 msg="resource change" op=add type=wechat name=global-wechat-receiver namespace=kubesphere-monitoring-system
level=debug ts=2021-03-11T17:48:35.013114851+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:48:35.013443668+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:48:35.013744399+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:48:35.01458076+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:48:35.014787087+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:48:35.014936298+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:48:35.015021244+08:00 caller=handler.go:176 msg="Worker exit after 1.168177ms"
level=debug ts=2021-03-11T17:49:11.074751414+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:49:11.07485059+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:49:11.075250662+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:49:11.076171668+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:49:11.076350858+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:49:11.076455036+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:49:11.076556322+08:00 caller=handler.go:176 msg="Worker exit after 1.108427ms"