• 告警通知
  • notifivation-manager接入微信报警收不到信息

wanjunlei
老师,使用json测试,终端打印200,但是nm日志已经有了,日志过来了~
现在nm到微信好像不通,我刚新建个cm,微信上并没有信息出来

wanjunlei
我当前使用的版本是0.7.0~
我在终端执行:

# curl -X POST -H "Content-Type: application/json" -d @test02.json http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts

nm的日志:

level=debug ts=2021-03-11T17:45:46.091975534+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:45:46.09207367+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:45:46.09211236+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:45:46.092926795+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:45:46.092977108+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:45:46.092991445+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:45:46.093001959+08:00 caller=handler.go:176 msg="Worker exit after 812.211µs"

wanjunlei
我在ui上创建/删除cm,nm能打印出响应的信息,但是微信没有收到~

level=info ts=2021-03-11T17:48:28.740767061+08:00 caller=config.go:677 msg="resource change" op=add type=wechat name=global-wechat-receiver namespace=kubesphere-monitoring-system
level=debug ts=2021-03-11T17:48:35.013114851+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:48:35.013443668+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:48:35.013744399+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:48:35.01458076+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:48:35.014787087+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:48:35.014936298+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:48:35.015021244+08:00 caller=handler.go:176 msg="Worker exit after 1.168177ms"
level=debug ts=2021-03-11T17:49:11.074751414+08:00 caller=handler.go:63 msg="Acquired worker queue lock..."
level=debug ts=2021-03-11T17:49:11.07485059+08:00 caller=handler.go:214 msg="Notification request accepted"
level=debug ts=2021-03-11T17:49:11.075250662+08:00 caller=handler.go:159 msg="Begins to send notification..."
level=debug ts=2021-03-11T17:49:11.076171668+08:00 caller=handler.go:142 msg="Worker: notification sent"
level=debug ts=2021-03-11T17:49:11.076350858+08:00 caller=handler.go:150 msg="Worker: exiting"
level=debug ts=2021-03-11T17:49:11.076455036+08:00 caller=handler.go:153 msg="Worker: exit"
level=debug ts=2021-03-11T17:49:11.076556322+08:00 caller=handler.go:176 msg="Worker exit after 1.108427ms"

你这是没有找到receiver,你检查你的receiver和config的crd,看看有没有问题,需要是global的receiver和default的config

    wanjunlei
    我就按照官网上写的:
    Deploy the default WechatConfig and a global WechatReceivers:

    cat <<EOF | kubectl apply -f -
    apiVersion: notification.kubesphere.io/v1alpha1
    kind: WechatConfig
    metadata:
      name: default-wechat-config
      namespace: kubesphere-monitoring-system
      labels:
        app: notification-manager
        type: default
    spec:
      wechatApiUrl: < wechat-api-url >
      wechatApiSecret:
        key: wechat
        name: < wechat-api-secret >
      wechatApiCorpId: < wechat-api-corp-id >
      wechatApiAgentId: < wechat-api-agent-id >
    ---
    apiVersion: notification.kubesphere.io/v1alpha1
    kind: WechatReceiver
    metadata:
      name: global-wechat-receiver
      namespace: kubesphere-monitoring-system
      labels:
        app: notification-manager
        type: global 
    spec:
      # wechatConfigSelector needn't to be configured for a global receiver
      # optional
      # One of toUser, toParty, toParty should be specified.
      toUser: < wechat-user >
      toParty: < wechat-party >
      toTag: < wechat-tag >
    ---
    apiVersion: v1
    data:
      wechat: dGVzdA==
    kind: Secret
    metadata:
      labels:
        app: notification-manager
      name: < wechat-api-secret >
      namespace: kubesphere-monitoring-system
    type: Opaque
    EOF

    toUser: < wechat-user >这个地方我写的是wechatApiCorpId

    wanjunlei
    非常感谢老师您的远程指导,我现在已经调通,使用的是0.7.0这个版本,上面的微信不报警的原因也找到了,是之前您回答的一个提问给出的答案:

    kubectl edit notificationmanagers.notification.kubesphere.io -n kubesphere-monitoring-system notification-manager
    把image的tag改成v0.1.1

    这个我不太明白为什么这样做,还有就是我根据官网使用0.8.0版本进行操作,发现nm不能正常使用~

    0.1.1修复了0.1.0的一个bug,这个在0.6.0和0.7.0中应该是不存在的,我再确认下吧
    0.7到0.8notification manager有了很大的修改,crd变成cluster scope的,所以需要重新部署,

      wanjunlei
      好的,感谢老师的答复~
      如有结果请告诉我下,我也学习和摸索下,万分感谢~

      wanjunlei
      还有一点,我使用notification-manager v0.1.1后,发现不用添加如下代码也可以收到信息:

      "receivers":
           - "name": "notification-manager"
             "webhook_configs":
             - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"

      请问这个在v0.1.1是正常的吗?然后升级到0.6版本后才添加的吗

        "webhook_configs":
        - "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
      - "name": "event"
        "webhook_configs":
        - "send_resolved": false
          "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"
      - "name": "auditing"
        "webhook_configs":
        - "send_resolved": false
          "url": "http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts"

      默认是这样的,

        wanjunlei 老师请教个问题我使用kubesphere可插拔安装grafana,使用kubesphere中的prometheus,发现grafana中没有数据显示,我查看了定义的变量,如我定义个变量,label_values(node_uname_info,job),发现获取不到job值请问这个是什么情况呢?是不是需要修改下node_uname_info这个指标呢?

        3 个月 后

        @wanjunlei 我这边也遇到类似的情况,调用http://notification-manager-svc.kubesphere-monitoring-system.svc:19093/api/v2/alerts可以返回成功,但是企业微信收不到告警,nm使用的是1.0.0版本;

        notification-manager-deployment日志:

        config和receiver配置:

        @wanjunlei 我是按项目配置告警的,请问下有什么问题吗

        看看notification manager的日志,有没有报错
        看看config和receiver定义的是否正确
        确认下企业微信发送消息是不是达到限制了

          wanjunlei 另外alertmanager日志一直没有新增,不知道是不是alertmanager到notification manager不通?

          这是alertmanager的配置

          配置的是租户级别的通知渠道,只会接收租户所在的namespace下的告警信息,如果要接收全部的告警信息,需要配置全局的通知渠道

            wanjunlei 请问下config和receiver配置的type和user以及toUser是否正确?