One of the first thing an attacker would likely do if they ever got access to your AWS secret keys would be to spin up as many servers as possible - probably large ones as well! So to help prevent this kind of attack from getting out of hand, we will be creating a CloudWatch alarm to trigger when the number of EC2 instances is above a certain threshold.

The basic items we need to create are as follows:

1. Lambda function to get the total number of EC2 instances running in our account
2. Custom CloudWatch metric with the the current instance count
3. CloudWatch alarm to trigger on the EC2 instance count metric

Python Lambda function to get and store the instance count

import boto3

ec2 = boto3.client('ec2');
cloudwatch = boto3.client('cloudwatch');

def lambda_handler(event, context):
    count = 0
    response = ec2.describe_instances()
    for r in response['Reservations']:
        count += len(r['Instances'])

    cloudwatch.put_metric_data(
        Namespace='EC2',
        MetricData=[
            {
                'MetricName': 'InstanceCount',
                'StatisticValues': {
                    'SampleCount': count,
                    'Sum': count,
                    'Minimum': count,
                    'Maximum': count
                },
                'Unit': 'Count'
            },
        ]
    )
    

The function basically counts all your instances and stores them in CloudWatch under a custom metric with the namespace "EC2" and metric "InstanceCount".

Sample Lambda IAM Role

The Lambda function only needs the ability to read all the instances and put a custom CloudWatch metric. Below is an example of the IAM role needed. You'll likely want to add the AWSLambdaBasicExecutionRole policy to this to ensure normal CloudWatch logging.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "cloudwatch:PutMetricData"
            ],
            "Resource": "*"
        }
    ]
}

Schedule Lambda Function

To trigger this Lambda function, you'll need to setup a CloudWatch Event. I'm currently using every 15 minutes but feel free to adjust the frequency to fit your needs and level of paranoia.

Creating CloudWatch Alarm

After you setup the Lambda function and run it successfully at least once, you should be able to setup the alarm. Within AWS go to the CloudWatch > Alarms page and press "Create Alarm".

Towards the bottom of the popup screen, under "Custom Metrics" select "EC2"

From the next screen choose "InstanceCount" then "Next" at the bottom.

Then give the alarm a name and optional description. For my case I'm running 8 instances, so I set the threshold to >= 10 which give me room to launch one more without triggering an alarm. So adjust that value to be the number of instances to fit your infrastructure needs.

The most import part is to set the "Statistic" value to "Maximum" and choose a notification action. Otherwise the alarm data will not be correct and you'll never get notified.

Sample alarm screen

I know I left off a lot of the small details, so please leave a message below if you need any further explanation on anything.