CloudWatch Agent on EC2 with Terraform

Although Serverless applications become very popular nowadays, EC2 still commonly used in some uses cases. When coming to EC2 monitoring, AWS comes with native CloudWatch metric for some basic metrics, but exclude memory and disk usage metric. This article going to show you how we can simply enable CloudWatch agents to monitor memory and disk usage metric using Terraform. To make things simple we will provision 1 EC2 using Terraform.

Image credit to AWS

Requirements

If you want to follow this guide, please make sure you have the following tools installed and configure:-

Terraform

Visual Studio Code

AWS Account

AWS CLI

Configure AWS Credential on your machine

Prepare Main Terraform Script

Use Visual Studio Code, Create a new working folder and create main.tf , with fill in the following detail:-

The Terraform script above do 3 things:-

1. Create a local variable to load User_data using templatefile function.
2. Create one EC2 resource with ami from ap-southeast-1 , configure instance profile, and user_data from the local variables.
3. Create SSM Parameter resource, and load its value from file cw_agent_config.json

Prepare Instance Profile Terraform Script

To make things easier, we will not configure the private key for EC2, but instead, we use Session Manager to login. Session Manager is a fully managed system manager that lets you control your EC2 instance, read here for more detail.

In the same folder create a file instance_profie.tf and fill in the following content:-

The Terraform script above does 4 things:-

1. Create one instance profile, the reference name must be the same as the previous Terraform script.
2. Create policy attachment that uses AmazonEC2RoleForSSM that allows EC2 to talk to SSM service, and CloudWatchAgentServerPolicy that allows EC2 to talk to CloudWatch service.
3. Create a custom role policy that will allow EC2 to make API call ssm:GetParameter , the main reason we need to allow this permission again is that we will need CloudWatch agent to load the configuration from SSM service, and the action is using this permission which not include in AmazonEC2RoleForSSM .
4. Lastly, the script will create assume role policy for ec2.amazonaws.com .

Create the Cloudwatch Agent Configuration file

At this step, we will create the Cloudwatch Agent Configuration file, the config will instruct the agent on how to pull the logs and metric.

Create a file cw_agent_config.json under root folder with the following content:-

The JSON above configure the agent to collect the metric for every 10 seconds and collect disk_used_percent metric, mem_acailable_percent metric, and aggregate the metric with few dimensions. You can read here for more on how to configure the Cloudwatch agent.

Create User Data to initialize the EC2

Lastly, we will create a user data file that initializes the EC2 set up, create a file user_data.sh under root folder with this content:-

What the script above does a few simple steps as below:-

1. Output all log to the specified location
2. Upgrade the machine with yum command
3. Configure Cloudwatch agent with download package from AWS and install it
4. Lastly run the cloud watch agent with configuring it to use config from SSM, as if you aware of the first Terraform script, we use a template file to load the file, so we can replace ours SSM config value.

Run and verify your

At this step will be simple execute the following command

terraform init
terraform plan
terraform apply --auto-approve

And you should see its provision one EC2 successfully in your account. Let the EC2 run for a few minutes to generate enough metric, and navigate to AWS Console, go to Cloudwatch, Metrics, you should see custom metric being populated as below:-

Try to navigate the Namespaces, and browse through the metric, you will get mem_available_percent metric like below:-

And disk_used_percent metrics.

Now you try login to the EC2 using the session manager as below:-

1. Click on the Connect button in EC2 console

2. Connect to your instance with Session Manager

3. Input command cat /var/log/user-data.log into browser SSH terminal and you will see the Cloudwatch setup log.

Lastly, run the command terraform destroy --auto-approve to clean up your environment.

Summary

In this article we use Terraform to build EC2 with configuring Session Manager permission and Cloudwatch Agent to collect other non-default metrics, with extra configuration, we can use Cloudwatch Agent to push our custom log to Cloudwatch Log Group . When you configure Cloudwatch Agent , suggest only collect metrics that important to you because it does not come with free of charge, please refer here to understand the Cloudwatch pricing.

If you want to clone the complete source, feel free to get it from here.