skip.link.title

How to send AWS CloudWatch Alarms to Slack? (Terraform Included)

  • You can find the source code for this video in my GitHub Repo.

Intro

In this video, I'll show you how to send CloudWatch alarms to the Slack channel. When we create a new CloudWatch alarm, we will get a Slack message that the new alarm was registered. To test integration, we will configure this CloudWatch alarm to fire up when the CPU usage of the EC2 instance exceeds 80%. We can also send the message to Slack when the alarm is resolved.

The way it's going to work is the following. When the CloudWatch alarm fires up, it will send a message to the SNS topic. Then this message will trigger the lambda function. It will parse the payload with an alarm message and send a custom message to the Slack channel based on the current and previous state of the alarm.

To create this notification pipeline, we will use the AWS console as well as I'll show you how to reproduce this setup using the terraform code.

Create SNS Topic

First of all, we need to create an SNS topic. CloudWatch will use it to publish messages based on the alarm conditions that we will define later. It can be a standard SNS topic with a name alarms.

Image title

When you are just getting started, I would suggest that you enable logging for your topic. It can save you a lot of time when you develop your notification pipeline with the lambda function.

As with any other AWS service, SNS would need a set of permissions to push logs to the CloudWatch. Let's create an IAM role for the SNS topic first. Search for SNS under use cases for other AWS services, then select SNS.

Image title

Let's call it sns-logs.

Image title

Return to the alarms SNS topic, select Delivery status logging, and click Edit.

Image title

We only need to choose AWS Lambda protocol since that's the only service that will be subscribed to this topic. Only for testing, set to log every single message. For production, set it to 0 or close to it; otherwise, you'll get a large AWS bill. Then enter the ARN of the sns-logs IAM role for both successful and failed deliveries.

Image title

Create Slack App

Next, we need to create a Slack channel to receive CloudWatch alarms and Slack app. Give it a name for the channel alarms.

Image title

To create a Slack app, go to https://api.slack.com/apps and click create an app. Let's call it CloudWatch Alarms and put it in your workspace.

Image title

We're going to be using webhooks to receive events from CloudWatch. Go ahead and activate incoming webhooks and add it to the alarms Slack channel. Slack will generate a URL that we can use from Lambda to publish messages.

Image title

Optionally you can add an app icon and a short description, something like Sends AWS Alerts to Slack.

Create Lambda Function

The next step is to create an AWS Lambda function. If you decide to create this function from the console, AWS can generate an IAM role for you, which is fine when you are just getting started, but the better approach is to create those IAM roles and policies yourself. Select Lambda from the common use cases.

Image title

Then we need to grant permissions for our Lambda to write logs to CloudWatch. For that, you can use one of the predefined IAM policies called AWSLambdaBasicExecutionRole. Call this role send-cloudwatch-alarms-to-slack.

Image title

Now we can create Lambda itself. Select create a function from scratch. Give it a name send-cloudwatch-alarms-to-slack. For the runtime, choose the last available Python version; in my case, it's Python 3.9. Since we created the IAM role, specify it as well.

Image title

Then the function itself. In a nutshell, it sends a message to Slack when you register a new CloudWatch Alarm. When the alarm is triggered, you'll get a warning to the channel. Also, when the alarm is resolved, you'll get a message to Slack.

Don't forget to replace the slack_url variable with yours. You can find it under Webhook URL.

Image title

functions/send-cloudwatch-alarms-to-slack/function.py
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
import urllib3
import json


slack_url = "https://hooks.slack.com/services/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"
http = urllib3.PoolManager()


def get_alarm_attributes(sns_message):
    alarm = dict()

    alarm['name'] = sns_message['AlarmName']
    alarm['description'] = sns_message['AlarmDescription']
    alarm['reason'] = sns_message['NewStateReason']
    alarm['region'] = sns_message['Region']
    alarm['instance_id'] = sns_message['Trigger']['Dimensions'][0]['value']
    alarm['state'] = sns_message['NewStateValue']
    alarm['previous_state'] = sns_message['OldStateValue']

    return alarm


def register_alarm(alarm):
    return {
        "type": "home",
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": ":warning: " + alarm['name'] + " alarm was registered"
                }
            },
            {
                "type": "divider"
            },
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": "_" + alarm['description'] + "_"
                },
                "block_id": "text1"
            },
            {
                "type": "divider"
            },
            {
                "type": "context",
                "elements": [
                    {
                        "type": "mrkdwn",
                        "text": "Region: *" + alarm['region'] + "*"
                    }
                ]
            }
        ]
    }


def activate_alarm(alarm):
    return {
        "type": "home",
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": ":red_circle: Alarm: " + alarm['name'],
                }
            },
            {
                "type": "divider"
            },
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": "_" + alarm['reason'] + "_"
                },
                "block_id": "text1"
            },
            {
                "type": "divider"
            },
            {
                "type": "context",
                "elements": [
                    {
                        "type": "mrkdwn",
                        "text": "Region: *" + alarm['region'] + "*"
                    }
                ]
            }
        ]
    }


def resolve_alarm(alarm):
    return {
        "type": "home",
        "blocks": [
            {
                "type": "header",
                "text": {
                    "type": "plain_text",
                    "text": ":large_green_circle: Alarm: " + alarm['name'] + " was resolved",
                }
            },
            {
                "type": "divider"
            },
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": "_" + alarm['reason'] + "_"
                },
                "block_id": "text1"
            },
            {
                "type": "divider"
            },
            {
                "type": "context",
                "elements": [
                    {
                        "type": "mrkdwn",
                        "text": "Region: *" + alarm['region'] + "*"
                    }
                ]
            }
        ]
    }


def lambda_handler(event, context):
    sns_message = json.loads(event["Records"][0]["Sns"]["Message"])
    alarm = get_alarm_attributes(sns_message)

    msg = str()

    if alarm['previous_state'] == "INSUFFICIENT_DATA" and alarm['state'] == 'OK':
        msg = register_alarm(alarm)
    elif alarm['previous_state'] == 'OK' and alarm['state'] == 'ALARM':
        msg = activate_alarm(alarm)
    elif alarm['previous_state'] == 'ALARM' and alarm['state'] == 'OK':
        msg = resolve_alarm(alarm)

    encoded_msg = json.dumps(msg).encode("utf-8")
    resp = http.request("POST", slack_url, body=encoded_msg)
    print(
        {
            "message": msg,
            "status_code": resp.status,
            "response": resp.data,
        }
    )

Subscribe Lambda to SNS Topic

If you want to subscribe your Lambda to an SNS topic using the console, it's very easy. Add a trigger and select the SNS topic. Behind the scene, AWS will add a resource-based policy to the function and add a subscription to the topic.

Image title

Create CloudWatch Alarm

Finally, let's create a CloudWatch alarm. We can only specify a single metric

Image title

Then the condition, if the CPU utilization for the EC2 instance is equal or greater than 80, trigger the alarm.

Image title

To get notifications when the alarm is resolved, add another OK notification to the same SNS topic.

Image title

You need to provide both alarm name and description since our Lambda expects both attributes; otherwise, Lambda will fail.

Image title

To test the CPU utilization, you can use the stress Linux utility. To install it, run sudo apt install stress, and then to test, stress --cpu 2.

Image title

top.title