Saving energy and money: stopping EC2 instances when not in use

Most AWS users have some EC2 instances running which are sporadically used; for example staging/test environments, CI/CD runners, bastion hosts, etc. They consume power, and although AWS claims to use renewable energy, there isn’t much solar power during the night… So in reality these instances run on the power source the grid has to offer (a bit of wind, but mostly gas/coal/nuclear). Stopping EC2 instances when not in use, will save you money and help with energy reduction. When EC2 instances are stopped, AWS will only charge you for the storage costs, but that’s a fraction of running the EC2 instances.

Photo by Jasmin Sessler on Unsplash

Automatically stopping EC2 instances sounds easy, but can be a bit difficult to implement. You don’t want your services to be degraded. As re-starting a stopped instances can take 30 seconds, production environments are probably not an option. Also think about how to start the instance when needed. For staging/acceptance environments you could start it on weekdays at 7am. Although that doesn’t seem to help much, limiting running times to weekdays 7am-6pm will reduce your costs by 2/3!

Starting an instance

For CI/CD jobs you can check if the instance is running, and if not start it. For example:

START_OUTPUT=$(aws ec2 start-instances --instance-ids $AWS_RUNNER_INSTANCE_ID)
- if [[ "$START_OUTPUT" == *"pending"* ]]; then echo "Waiting for start instance to complete"; sleep 30; fi

This uses the AWS CLI to start the EC2 instance $AWS_RUNNER_INSTANCE_ID and store the result in START_OUTPUT . If the result contains *pending* you know it was off and is starting. In that case, sleep for 30 seconds, to give the instance enough time to start.

Options to stop an instance

Now that you have an idea how to start your instance, it’s time to stop it automatically. There are four ways to do this:

1. Use EventBridge with a cron expression, to turn it off at a specified time, and use the target AWS Service: EC2 Stop Instances API call . Unfortunately, this is only configurable in the AWS console (see docs), and not in CloudFormation nor other automated ways (CLI/API).
2. AWS offers a Scheduler service, which can turn on and off EC2 instances based on tags. A pretty nifty solution, but sometimes a bit too complex, and running it still costs $10 / month as a Lambda runs every minute.
3. CPU based: using a CloudWatch alarm.
4. Time based: using a CloudWatch cron, which triggers a Lambda function to turn off the EC2 instance.

As I’m a firm believer in scripting infrastructure (otherwise things will break or environments will get differences), so we’re going for option 3 and 4. CloudFormation is a great solution to script infrastructure. But if CloudFormation is not your thing, check out this AWS example, which describes the manual configuration.

CPU based stopping

The easiest way is to let CloudWatch check the CPU levels of the EC2 instance (or some other metric), and turn it off when it’s not being used. This can be useful in case the EC2 instance is used for CI/CD pipelines or other infrequent jobs, and it’s clear by looking at the CPU when it’s not doing anything anymore. It might not be the best solution for staging environments, which needs to be up and running during office hours (see next section).

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  EC2InstanceId:
    Type: 'String'
    Description: 'EC2 Instance ID, for example i-1234567890abcdef0'
Resources:
  # Optional CloudWatch alarm when CPU < 20% for 50 minutes
  CPUAlarm:
    Type: AWS::CloudWatch::Alarm
    Properties:
      AlarmActions:
        - !Sub "arn:aws:automate:${AWS::Region}:ec2:stop"
      AlarmDescription: CPU alarm for unused EC2 instance
      ComparisonOperator: LessThanThreshold
      DatapointsToAlarm: 10
      EvaluationPeriods: 12
      MetricName: CPUUtilization
      Namespace: AWS/EC2
      Period: 300
      Statistic: Maximum
      Threshold: 20
      Dimensions:
        - Name: InstanceId
          Value: !Ref EC2InstanceId

Some interesting aspects:

• You can configure the EC2 instance ID in a paramater, which will also be asked by AWS when uploading this file in CloudFormation.
• We only need define in the AlarmActions that the EC2 instance needs to stop; the EC2 instance ID is defined in the Dimensions and CloudWatch is smart enough to know which instance to stop.
• We’re using CPU here, but you could use network traffic or other metrics.

Time based stopping

An alternative, but less efficient method, is turning the EC2 instance off at a specific time. A CloudWatch cron triggers a Lambda function to turn off the EC2 instance.

Let’s start with the Lambda function to turn off the EC2 instance. It also needs an IAM role, to have privileges to turn off the EC2 instance.

AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  EC2InstanceId:
    Type: 'String'
    Description: 'EC2 Instance ID, for example i-1234567890abcdef0'
Resources:
  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Description: Stop GitLab runner EC2 instance
      Code:
        ZipFile: !Sub |
          import boto3
          region = '${AWS::Region}'
          instances = ['${EC2InstanceId}']
          ec2 = boto3.client('ec2', region_name=region)
          
          def lambda_handler(event, context):
            ec2.stop_instances(InstanceIds=instances)
            print('stopped your instances: ' + str(instances))
      Handler: index.lambda_handler
      MemorySize: 128
      Timeout: 10
      Role: !Sub ${LambdaExecutionRole.Arn}
      Runtime: python3.9
  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub '${AWS::StackName}-lambda-role'
      Description: !Sub 'Instance Profile for Lambda stopping EC2 instance'
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: StopEC2Instance
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action: 'ec2:StopInstances'
                Resource: !Sub 'arn:aws:ec2:${AWS::Region}:${AWS::AccountId}:instance/${EC2InstanceId}'

There are a couple of things to notice here:

• The Lambda function runs Python code, which uses the AWS SDK to stop the EC2 instance.
• Because this function normally shouldn’t have access to stop the EC2 instance, we create an IAM role for it, and assign the role to the Lambda function (Role: !Sub ${LambdaExecutionRole.Arn}). This role is using the default AWSLambdaBasicExecutionRole, to be able to execute the function, as well as the ec2:StopInstances action for this specific instance.

Now the Lambda function is described, we can start with the CloudWatch cron to trigger the Lambda function.

LambdaSchedule:
  Type: AWS::Events::Rule
  Properties:
    Description: Schedule for stopping EC2 instance end of day
    ScheduleExpression: cron(0 18 * * ? *)
    State: ENABLED
    Targets:
      - Arn: !Sub ${LambdaFunction.Arn}
        Id: LambdaSchedule
LambdaSchedulePermission:
  Type: AWS::Lambda::Permission
  Properties:
    Action: 'lambda:InvokeFunction'
    FunctionName: !Sub ${LambdaFunction.Arn}
    Principal: 'events.amazonaws.com'
    SourceArn: !Sub ${LambdaSchedule.Arn}

Pretty straightforward, just don’t forget to add the permissions, otherwise the Lambda won’t be called.

If you want to start an instance in the beginning of the day, just create a similar rule, reconfigure the Lambda to start the instance as well, and assign the necessary permissions.

The entire CloudFormation script can be found on GitHub.

What do you think? Did you use it, or do you have alternative solutions? Let me know in the comments.