How to send AWS CloudWatch Alarms to Slack? (Terraform Included)¶
- You can find the source code for this video in my GitHub Repo.
In this video, I'll show you how to send CloudWatch alarms to the Slack channel. When we create a new CloudWatch alarm, we will get a Slack message that the new alarm was registered. To test integration, we will configure this CloudWatch alarm to fire up when the CPU usage of the EC2 instance exceeds 80%. We can also send the message to Slack when the alarm is resolved.
The way it's going to work is the following. When the CloudWatch alarm fires up, it will send a message to the SNS topic. Then this message will trigger the lambda function. It will parse the payload with an alarm message and send a custom message to the Slack channel based on the current and previous state of the alarm.
To create this notification pipeline, we will use the AWS console as well as I'll show you how to reproduce this setup using the terraform code.
Create SNS Topic¶
When you are just getting started, I would suggest that you enable logging for your topic. It can save you a lot of time when you develop your notification pipeline with the lambda function.
As with any other AWS service, SNS would need a set of permissions to push logs to the CloudWatch. Let's create an IAM role for the SNS topic first. Search for SNS under use cases for other AWS services, then select SNS.
Let's call it
Return to the
alarms SNS topic, select Delivery status logging, and click Edit.
We only need to choose
AWS Lambda protocol since that's the only service that will be subscribed to this topic.
Only for testing, set to log every single message. For production, set it to
0 or close to it; otherwise, you'll get a large AWS bill.
Then enter the ARN of the
sns-logs IAM role for both successful and failed deliveries.
Create Slack App¶
Next, we need to create a Slack channel to receive CloudWatch alarms and Slack app. Give it a name for the channel
To create a Slack app, go to https://api.slack.com/apps and click create an app. Let's call it
CloudWatch Alarms and put it in your workspace.
We're going to be using webhooks to receive events from CloudWatch. Go ahead and activate incoming webhooks and add it to the
alarms Slack channel.
Slack will generate a URL that we can use from Lambda to publish messages.
Optionally you can add an app icon and a short description, something like
Sends AWS Alerts to Slack.
Create Lambda Function¶
The next step is to create an AWS Lambda function. If you decide to create this function from the console, AWS can generate an IAM role for you, which is fine when you are just getting started, but the better approach is to create those IAM roles and policies yourself. Select Lambda from the common use cases.
Then we need to grant permissions for our Lambda to write logs to CloudWatch. For that, you can use one of the predefined IAM policies called
Call this role
Now we can create Lambda itself. Select create a function from scratch. Give it a name
For the runtime, choose the last available Python version; in my case, it's
Python 3.9. Since we created the IAM role, specify it as well.
Then the function itself. In a nutshell, it sends a message to Slack when you register a new CloudWatch Alarm. When the alarm is triggered, you'll get a warning to the channel. Also, when the alarm is resolved, you'll get a message to Slack.
Don't forget to replace the
slack_url variable with yours. You can find it under Webhook URL.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
Subscribe Lambda to SNS Topic¶
If you want to subscribe your Lambda to an SNS topic using the console, it's very easy. Add a trigger and select the SNS topic. Behind the scene, AWS will add a resource-based policy to the function and add a subscription to the topic.
Create CloudWatch Alarm¶
Finally, let's create a CloudWatch alarm. We can only specify a single metric
Then the condition, if the CPU utilization for the EC2 instance is equal or greater than 80, trigger the alarm.
To get notifications when the alarm is resolved, add another OK notification to the same SNS topic.
You need to provide both alarm name and description since our Lambda expects both attributes; otherwise, Lambda will fail.
To test the CPU utilization, you can use the stress Linux utility. To install it, run
sudo apt install stress, and then to test,
stress --cpu 2.