Create AWS CloudWatch Alarm on the Number of Running EC2 Instances

One of the first thing an attacker would likely do if they ever got access to your AWS secret keys would be to spin up as many servers as possible - probably large ones as well! So to help prevent this kind of attack from getting out of hand, we will be creating a CloudWatch alarm to trigger when the number of EC2 instances is above a certain threshold.

The basic items we need to create are as follows:

• Lambda function to get the total number of EC2 instances running in our account
• Custom CloudWatch metric with the the current instance count
• CloudWatch alarm to trigger on the EC2 instance count metric

Python Lambda function

This function will be used to store the instance count within CloudWatch Metrics

import boto3

ec2 = boto3.client('ec2');
cloudwatch = boto3.client('cloudwatch');

def lambda_handler(event, context):
    count = 0
    response = ec2.describe_instances()
    for r in response['Reservations']:
        count += len(r['Instances'])

    cloudwatch.put_metric_data(
        Namespace='EC2',
        MetricData=[
            {
                'MetricName': 'InstanceCount',
                'StatisticValues': {
                    'SampleCount': count,
                    'Sum': count,
                    'Minimum': count,
                    'Maximum': count
                },
                'Unit': 'Count'
            },
        ]
    )

The function basically counts all your instances and stores them in CloudWatch under a custom metric with the namespace EC2 and metric InstanceCount.

Sample Lambda IAM Role

The Lambda function only needs the ability to read all the instances and put a custom CloudWatch metric. Below is an example of the IAM role needed. You’ll likely want to add the AWSLambdaBasicExecutionRole policy to this to ensure normal CloudWatch logging.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "cloudwatch:PutMetricData"
      ],
      "Resource": "*"
    }
  ]
}

Schedule Lambda Function

To trigger this Lambda function, you’ll need to setup a CloudWatch Event. I’m currently using every 15 minutes but feel free to adjust the frequency to fit your needs and level of paranoia.

Creating CloudWatch Alarm

After you setup the Lambda function and run it successfully at least once, you should be able to setup the alarm. Within AWS go to the CloudWatch & Alarms page and press “Create Alarm”.

Towards the bottom of the popup screen, under Custom Metrics select EC2

From the next screen choose “InstanceCount” then “Next” at the bottom.

Then give the alarm a name and optional description. For my case I’m running 8 instances, so I set the threshold to > 10 which give me room to launch one more without triggering an alarm. So adjust that value to be the number of instances to fit your infrastructure needs.

The most import part is to set the Statistic value to Maximum and choose a notification action. Otherwise the alarm data will not be correct and you’ll never get notified.

I know I left off a lot of the small details, so please leave a message below if you need any further explanation on anything.