Grafana alarm access to Feishu notification

Grafana series of articles, version: OOS v9.3.1

  1. Introduction and installation of Grafana
  2. Introduction to configuration parameters of Grafana monitoring large screen (1)
  3. Introduction to configuration parameters of Grafana monitoring large screen (2)
  4. Grafana monitors large-screen visualization charts
  5. Grafana query data and transform data
  6. Introduction to Grafana Alarm Module
  7. Grafana alarm access to Feishu notification

foreword

We already know that Grafana already supports DingTalk, email and other alarms. DingTalk's ability to enter Grafana is also related to Ali's emphasis on open source. On Grafana's GitHub, someone mentioned such a PR very early.

They hoped that Grafana could integrate Feishu, but was rejected by the Grafana team because the demand for Feishu notifications was not strong enough at that time.

But it’s okay, there is no native integration, we can also use the universal webHook to convert the parameters through the transfer service and then send them to Feishu.

Create alert rules

For testing, the data source uses Grafana's Test DB, and a new rule is created.

Query data from the Random Walk type chart. When the maximum value of the data is greater than 10, an alarm will be generated and evaluated every 1 minute. If the alarm condition is met for 2 minutes, an alarm will be generated.

This indicates the large screen and chart panel associated with the alarm, and there is also a description field

Configure WebHook Notifications

Here, for testing purposes, only one receiving address is filled in, without authentication, the external network environment must be authenticated

In addition, because Grafana is deployed using Docker, the transit service is deployed on the local physical machine. If you fill in 127.0.0.1, you cannot access it inside Docker, and you need to fill in the LAN IP of the physical machine.

Configure alert policy

Finally, you need to configure an alarm strategy. Here, for testing, the time is set relatively short. Mute timings is not an alarm time configuration, which can be left unset.

Deploy transit service

The following paragraph is the content of the webHook alarm received by the transit service.

{
	"receiver": "feishu",
	"status": "resolved",
	"alerts": [{
		"status": "resolved",
		"labels": {
			"alertname": "Simple dummy streaming example",
			"grafana_folder": "test"
		},
		"annotations": {
			"description": "Test the large screen alarm"
		},
		"startsAt": "2023-01-12T14:46:00Z",
		"endsAt": "2023-01-12T15:07:00Z",
		"generatorURL": "http://localhost:3000/alerting/grafana/zgbSUF24z/view?orgId=1",
		"fingerprint": "09b095c0ac5a31bc",
		"silenceURL": "http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DSimple+dummy+streaming+example\u0026matcher=grafana_folder%3Dtest",
		"dashboardURL": "http://localhost:3000/d/TXSTREZ?orgId=1",
		"panelURL": "http://localhost:3000/d/TXSTREZ?orgId=1\u0026viewPanel=2",
		"values": null,
		"valueString": ""
	}],
	"groupLabels": {
		"alertname": "Simple dummy streaming example"
	},
	"commonLabels": {
		"alertname": "Simple dummy streaming example",
		"grafana_folder": "test"
	},
	"commonAnnotations": {
		"description": "Test the large screen alarm"
	},
	"externalURL": "http://localhost:3000/",
	"version": "1",
	"groupKey": "{}/{}:{alertname=\"Simple dummy streaming example\"}",
	"truncatedAlerts": 0,
	"orgId": 1,
	"title": "[RESOLVED] Simple dummy streaming example (test)",
	"state": "ok",
	"message": "**Resolved**\n\nValue: [no value]\nLabels:\n - alertname = Simple dummy streaming example\n - grafana_folder = test\nAnnotations:\n - description = Test the large screen alarm\nSource: http://localhost:3000/alerting/grafana/zgbSUF24z/view?orgId=1\nSilence: http://localhost:3000/alerting/silence/new?alertmanager=grafana\u0026matcher=alertname%3DSimple+dummy+streaming+example\u0026matcher=grafana_folder%3Dtest\nDashboard: http://localhost:3000/d/TXSTREZ?orgId=1\nPanel: http://localhost:3000/d/TXSTREZ?orgId=1\u0026viewPanel=2\n"
}
copy

The above fields are also easy to understand. We parse and encapsulate the above fields into the notification format of Feishu to send notifications to Feishu.

Let's simply adjust the package based on the previous code.

AlarmMessage.java alarm interface

public interface AlarmMessage {

    /**
     * Send text alert
     * @param content
     */
    void sendData(String content);
}
copy

FeiShuNotifier.java Feishu sends alarm code

@Service
public class FeiShuNotifier implements AlarmMessage {

	private static final String DEFAULT_MESSAGE = " Alert title:#{title} \n Alarm description:#{description} \n panel URL: #{panelUrl} \n Time of occurrence:#{startTime}";

	private final SpelExpressionParser parser = new SpelExpressionParser();

	@Value("${feishu.webhook-url}")
	private String webhookUrl;

	@Value("${feishu.secret}")
	private String secret;

	private Expression message = parser.parseExpression(DEFAULT_MESSAGE, ParserContext.TEMPLATE_EXPRESSION);

	private RestTemplate restTemplate = new RestTemplate();

	@Override
	public void sendData(String content) {
		Map<String, Object> message = createMessage(getText(content));
		HttpHeaders headers = new HttpHeaders();
		headers.setContentType(MediaType.APPLICATION_JSON);
		restTemplate.postForEntity(webhookUrl, new HttpEntity<>(message, headers), Void.class);
	}


	/**
	 * Build a notification message
	 * @param content
	 * @return
	 */
	protected Map<String, Object> createMessage(String content) {
		Map<String, Object> messageJson = new HashMap<>();
		messageJson.put("msg_type", "text");

		Map<String, Object> text = new HashMap<>();
		text.put("text", content);
		messageJson.put("content", text);
		Long timestamp = System.currentTimeMillis() / 1000;
		messageJson.put("timestamp", timestamp);
		messageJson.put("sign", getSign(timestamp));

		return messageJson;
	}

	/**
	 * Build text content
	 * @param body
	 * @return
	 */
	private String getText(String body) {
		JSONObject json = JSON.parseObject(body);
		Map<String, Object> root = new HashMap<>();
		root.put("title", json.getString("title"));
		root.put("description", json.getJSONObject("commonAnnotations").getString("description"));
		JSONObject alertObject = json.getJSONArray("alerts").getJSONObject(0);
		root.put("panelUrl", alertObject.getString("panelURL"));
		root.put("startTime", DateUtil.parseUTC(alertObject.getString("startsAt")).toString(TimeZone.getDefault()));
		StandardEvaluationContext context = new StandardEvaluationContext(root);
		context.addPropertyAccessor(new MapAccessor());
		return message.getValue(context, String.class);
	}

	private String getSign(Long timestamp) {
		try {
			String stringToSign = timestamp + "\n" + secret;
			Mac mac = Mac.getInstance("HmacSHA256");
			mac.init(new SecretKeySpec(stringToSign.getBytes(StandardCharsets.UTF_8), "HmacSHA256"));
			byte[] signData = mac.doFinal(new byte[]{});
			return new String(Base64.encodeBase64(signData, false));
		}
		catch (Exception ex) {
			ex.printStackTrace();
		}
		return "";
	}

}
copy

webhook interface code

@RestController
public class WebhookController {

    @Resource
    private FeiShuNotifier feiShuNotifier;

    @RequestMapping("/webhook/receive")
    public void receive(@RequestBody String message) {
        feiShuNotifier.sendData(message);
    }
}
copy

The following is the alert received by Feishu:

In addition, if you need to access other chat tools such as corporate WeChat, you can use the open source PrometheusAlert, see its GitHub for the usage plan. https://github.com/feiyu563/PrometheusAlert

Author's other articles:

  1. Spring Boot Admin Reference Guide
  2. The problem that the SpringBoot Admin service is offline and does not display health information
  3. Loading of Spring Boot Admin2 @EnableAdminServer
  4. Detailed Explanation of Spring Boot Admin2 AdminServerAutoConfiguration
  5. Detailed Explanation of Spring Boot Admin2 Instance Status Monitoring
  6. Spring Boot Admin2 custom JVM monitoring notification
  7. Spring Boot Admin2 custom exception monitoring
  8. Spring Boot Admin monitoring indicators connected to Grafana visualization

Tags: Java Docker Spring github Spring Boot

Posted by PHPoracle on Sat, 14 Jan 2023 12:18:16 +1030