Automate your Cloud Operations Part 1: AWS CloudFormation

Operations

What is Operations?

In the IT world, Operations refers to a team or department within IT which is responsible for the running of a business’ IT systems and infrastructure.

So what kind of activities this team perform on day to day basis?

Building, modifying, provisioning, updating systems, software and infrastructure to keep them available, performing and secure which ensures that users can be as productive as possible.

When moving to public cloud platforms the areas of focus for Operations are:

  • Cost reduction: if we design it properly and apply good practices when managing it (scale down / switch off)
  • Smarter operation: Use of Automation and APIs
  • Agility: faster in provisioning infrastructure or environments by Automating the everything
  • Better Uptime: Plan for failover, and design effective DR solutions more cost effectively.

If Cloud is the new normal then Automation is the new normal.

For this blog post we will focus on automation using AWS CloudFormation. The template I will use for this post for educational purposes only and may not be suitable for production workloads :).

AWS CloudFormation

AWS CloudFormation provides developers and system administrators DevOps an easy way to create and manage a collection of related AWS resources, including provisioning and updating them in an orderly and predictable fashion. AWS provides various CloudFormation templates, snippets and reference implementations.

Let’s talk about versioning before diving deeper into CloudFormation. It is extremely important to version your AWS infrastructure in the same way as you version your software. Versioning will help you to track change within your infrastructure by identifying:

  • What changed?
  • Who changed it?
  • When was it changed?
  • Why was it changed?

You can tie this version to a service management or project delivery tools if you wish.

You should also put your templates into source control. Personally I am using Github to version my infrastructure code, but any system such as Team Foundation Server (TFS) will do.

AWS Infrastructure

The below diagram illustrates the basic AWS infrastructure we will build and automate for this blog post:

CloudFormation1

Initial Stack

Firstly we will create the initial stack. Below are the components for the initial stack:

  • A VPC with CIDR block of 192.168.0.0/16 : 65,543 IPs
  • Three Public Subnets across 3 Availability Zones : 192.168.10.0/24, 192.168.11.0/24,  192.168.12.0/24
  • An Internet Gateway attached to the VPC to allow public Internet access. This is a routing construct for VPC and not an EC2 instance
  • Routes and Route tables for three public subnets so EC2 instances in those public subnets can communicate
  • Default Network ACLs to allow all communication inside of the VPC.

Below is the CloudFormation template to build the initial stack.


{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Builds a VPC w/ INET Gateway and 3 public subnets. **WARNING** This template creates Amazon EC2 instance(s). You will be billed for the AWS resources used if you create a stack from this template.",
"Resources" : {
"VPC" : {
"Type" : "AWS::EC2::VPC",
"Properties" : {
"CidrBlock" : "192.168.0.0/16",
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"PublicSubnetA" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"CidrBlock" : "192.168.10.0/24",
"AvailabilityZone" : { "Fn::Select": [ "0", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]},
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"PublicSubnetB" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"CidrBlock" : "192.168.11.0/24",
"AvailabilityZone" : { "Fn::Select": [ "1", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]},
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"PublicSubnetC" : {
"Type" : "AWS::EC2::Subnet",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"CidrBlock" : "192.168.12.0/24",
"AvailabilityZone" : { "Fn::Select": [ "2", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]},
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"InternetGateway" : {
"Type" : "AWS::EC2::InternetGateway",
"Properties" : {
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"AttachGateway" : {
"Type" : "AWS::EC2::VPCGatewayAttachment",
"Properties" : {
"VpcId" : { "Ref" : "VPC" },
"InternetGatewayId" : { "Ref" : "InternetGateway" }
}
},
"PublicRouteTable" : {
"Type" : "AWS::EC2::RouteTable",
"Properties" : {
"VpcId" : {"Ref" : "VPC"},
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Public" }
]
}
},
"PublicRoute" : {
"Type" : "AWS::EC2::Route",
"Properties" : {
"RouteTableId" : { "Ref" : "PublicRouteTable" },
"DestinationCidrBlock" : "0.0.0.0/0",
"GatewayId" : { "Ref" : "InternetGateway" }
}
},
"PublicSubnetRouteTableAssociationA" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "PublicSubnetA" },
"RouteTableId" : { "Ref" : "PublicRouteTable" }
}
},
"PublicSubnetRouteTableAssociationB" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "PublicSubnetB" },
"RouteTableId" : { "Ref" : "PublicRouteTable" }
}
},
"PublicSubnetRouteTableAssociationC" : {
"Type" : "AWS::EC2::SubnetRouteTableAssociation",
"Properties" : {
"SubnetId" : { "Ref" : "PublicSubnetC" },
"RouteTableId" : { "Ref" : "PublicRouteTable" }
}
}
},
"Outputs" : {
"VpcId" : {
"Value" : {"Ref" : "VPC"},
"Description" : "VPC ID of newly created VPC"
},
"PublicSubnetA" : {
"Value" : {"Ref" : "PublicSubnetA"},
"Description" : "Public Subnet in AZ A"
},
"PublicSubnetB" : {
"Value" : {"Ref" : "PublicSubnetB"},
"Description" : "Public Subnet in AZ B"
},
"PublicSubnetC" : {
"Value" : {"Ref" : "PublicSubnetC"},
"Description" : "Public Subnet in AZ C"
}
}
}

The template can be downloaded here: https://s3-ap-southeast-2.amazonaws.com/andreaswasita/cloudformation_template/demo/lab1-vpc_ELB_combined.template

I put together the following video on how to use the template:

Understanding a CloudFormation template

AWS CloudFormation is pretty neat and FREE. You only need to pay for the AWS resources provisioned by the CloudFormation template.

The next bit is understanding the Structure of the template. Typically CloudFormation template will have 5 sections:

  • Headers
  • Parameters
  • Mappings
  • Resources
  • Outputs

Headers: Example:


"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Builds a VPC w/ INET Gateway and 3 public subnets. **WARNING** This template creates Amazon EC2 instance(s). You will be billed for the AWS resources used if you create a stack from this template.",

view raw

JSON_Headers

hosted with ❤ by GitHub

Parameters: Provision-time spec command-line options. Example:


"Parameters" : {
"KeyName" : {
"Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances",
"Type" : "String",
"MinLength": "1",
"MaxLength": "64",
"AllowedPattern" : "[-_ a-zA-Z0-9]*",
"ConstraintDescription" : "can contain only alphanumeric characters, spaces, dashes and underscores."
},
"VpcId" : {
"Type" : "String",
"Description" : "VpcId of your existing Virtual Private Cloud (VPC)"
},
"SubnetId" : {
"Type" : "String",
"Description" : "SubnetId of an existing Public facing subnet in your Virtual Private Cloud (VPC)"
}
},

view raw

Parameters_JSON

hosted with ❤ by GitHub

Mappings: Conditionals Case Statements. Example:


"Mappings" : {
"AWSNATAMI": {
"us-east-1": {"AMI": "ami-6e9e4b06"},
"us-west-2": {"AMI": "ami-8b6912bb"},
"us-west-1": {"AMI": "ami-1d2b2958"},
"eu-west-1": {"AMI": "ami-14913f63"},
"ap-northeast-1": {"AMI": "ami-27d6e626"}
}
},

view raw

Mappings_JSON

hosted with ❤ by GitHub

Resources: All resources to be provisioned. Example:


"Resources" : {
"NATIPAddress" : {
"Type" : "AWS::EC2::EIP",
"Properties" : {
"Domain" : "vpc",
"InstanceId" : { "Ref" : "NATDevice" }
}
},
"NATDevice" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"InstanceType" : "m3.medium",
"KeyName" : { "Ref" : "KeyName" },
"SubnetId" : { "Ref" : "SubnetId" },
"SourceDestCheck" : "false",
"ImageId" : { "Fn::FindInMap" : [ "AWSNATAMI", { "Ref" : "AWS::Region" }, "AMI" ]},
"SecurityGroupIds" : [{ "Ref" : "NATSecurityGroup" }]
}
},
"NATSecurityGroup" : {
"Type" : "AWS::EC2::SecurityGroup",
"Properties" : {
"GroupDescription" : "Enable internal access to the NAT device",
"VpcId" : { "Ref" : "VpcId" },
"SecurityGroupIngress" : [
{ "IpProtocol" : "tcp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" },
{ "IpProtocol" : "udp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" } ,
{ "IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "192.168.0.0/16" }
],
"SecurityGroupEgress" : [
{ "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0" },
{ "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0" },
{ "IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "0.0.0.0/0" },
{ "IpProtocol" : "tcp", "FromPort" : "9418", "ToPort" : "9418", "CidrIp" : "0.0.0.0/0" },
{ "IpProtocol" : "tcp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" },
{ "IpProtocol" : "udp", "FromPort" : "123", "ToPort" : "123", "CidrIp" : "0.0.0.0/0" },
{ "IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "0.0.0.0/0" }
]
}
},
"PrivateRouteTable" : {
"Type" : "AWS::EC2::RouteTable",
"Properties" : {
"VpcId" : {"Ref" : "VpcId"},
"Tags" : [
{"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} },
{"Key" : "Network", "Value" : "Private Route" }
]
}
},
"PrivateRoute" : {
"Type" : "AWS::EC2::Route",
"Properties" : {
"RouteTableId" : { "Ref" : "PrivateRouteTable" },
"DestinationCidrBlock" : "0.0.0.0/0",
"InstanceId" : { "Ref" : "NATDevice" }
}
}
},

view raw

Resources_JSON

hosted with ❤ by GitHub

Outputs: Example:


"Outputs" : {
"PrivateRouteTableId" : {
"Value" : {"Ref" : "PrivateRouteTable"},
"Description" : "Private Route Table ID"
}
}
}

view raw

Outputs_JSON

hosted with ❤ by GitHub

Note: Not all AWS Resources can be provisioned using AWS CloudFormation and it has the following limitations.

In Part 2 we will deep dive further on AWS CloudFormation and automating the EC2 including the configuration for NAT and Bastion Host instance.

http://www.wasita.net

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.