Operations
What is Operations?
In the IT world, Operations refers to a team or department within IT which is responsible for the running of a business’ IT systems and infrastructure.
So what kind of activities this team perform on day to day basis?
Building, modifying, provisioning, updating systems, software and infrastructure to keep them available, performing and secure which ensures that users can be as productive as possible.
When moving to public cloud platforms the areas of focus for Operations are:
- Cost reduction: if we design it properly and apply good practices when managing it (scale down / switch off)
- Smarter operation: Use of Automation and APIs
- Agility: faster in provisioning infrastructure or environments by Automating the everything
- Better Uptime: Plan for failover, and design effective DR solutions more cost effectively.
If Cloud is the new normal then Automation is the new normal.
For this blog post we will focus on automation using AWS CloudFormation. The template I will use for this post for educational purposes only and may not be suitable for production workloads :).
AWS CloudFormation
AWS CloudFormation provides developers and system administrators DevOps an easy way to create and manage a collection of related AWS resources, including provisioning and updating them in an orderly and predictable fashion. AWS provides various CloudFormation templates, snippets and reference implementations.
Let’s talk about versioning before diving deeper into CloudFormation. It is extremely important to version your AWS infrastructure in the same way as you version your software. Versioning will help you to track change within your infrastructure by identifying:
- What changed?
- Who changed it?
- When was it changed?
- Why was it changed?
You can tie this version to a service management or project delivery tools if you wish.
You should also put your templates into source control. Personally I am using Github to version my infrastructure code, but any system such as Team Foundation Server (TFS) will do.
AWS Infrastructure
The below diagram illustrates the basic AWS infrastructure we will build and automate for this blog post:
Initial Stack
Firstly we will create the initial stack. Below are the components for the initial stack:
- A VPC with CIDR block of 192.168.0.0/16 : 65,543 IPs
- Three Public Subnets across 3 Availability Zones : 192.168.10.0/24, 192.168.11.0/24, 192.168.12.0/24
- An Internet Gateway attached to the VPC to allow public Internet access. This is a routing construct for VPC and not an EC2 instance
- Routes and Route tables for three public subnets so EC2 instances in those public subnets can communicate
- Default Network ACLs to allow all communication inside of the VPC.
Below is the CloudFormation template to build the initial stack.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "AWSTemplateFormatVersion" : "2010-09-09", | |
| "Description" : "Builds a VPC w/ INET Gateway and 3 public subnets. **WARNING** This template creates Amazon EC2 instance(s). You will be billed for the AWS resources used if you create a stack from this template.", | |
| "Resources" : { | |
| "VPC" : { | |
| "Type" : "AWS::EC2::VPC", | |
| "Properties" : { | |
| "CidrBlock" : "192.168.0.0/16", | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "PublicSubnetA" : { | |
| "Type" : "AWS::EC2::Subnet", | |
| "Properties" : { | |
| "VpcId" : { "Ref" : "VPC" }, | |
| "CidrBlock" : "192.168.10.0/24", | |
| "AvailabilityZone" : { "Fn::Select": [ "0", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]}, | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "PublicSubnetB" : { | |
| "Type" : "AWS::EC2::Subnet", | |
| "Properties" : { | |
| "VpcId" : { "Ref" : "VPC" }, | |
| "CidrBlock" : "192.168.11.0/24", | |
| "AvailabilityZone" : { "Fn::Select": [ "1", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]}, | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "PublicSubnetC" : { | |
| "Type" : "AWS::EC2::Subnet", | |
| "Properties" : { | |
| "VpcId" : { "Ref" : "VPC" }, | |
| "CidrBlock" : "192.168.12.0/24", | |
| "AvailabilityZone" : { "Fn::Select": [ "2", {"Fn::GetAZs": {"Ref": "AWS::Region"}} ]}, | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "InternetGateway" : { | |
| "Type" : "AWS::EC2::InternetGateway", | |
| "Properties" : { | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "AttachGateway" : { | |
| "Type" : "AWS::EC2::VPCGatewayAttachment", | |
| "Properties" : { | |
| "VpcId" : { "Ref" : "VPC" }, | |
| "InternetGatewayId" : { "Ref" : "InternetGateway" } | |
| } | |
| }, | |
| "PublicRouteTable" : { | |
| "Type" : "AWS::EC2::RouteTable", | |
| "Properties" : { | |
| "VpcId" : {"Ref" : "VPC"}, | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Public" } | |
| ] | |
| } | |
| }, | |
| "PublicRoute" : { | |
| "Type" : "AWS::EC2::Route", | |
| "Properties" : { | |
| "RouteTableId" : { "Ref" : "PublicRouteTable" }, | |
| "DestinationCidrBlock" : "0.0.0.0/0", | |
| "GatewayId" : { "Ref" : "InternetGateway" } | |
| } | |
| }, | |
| "PublicSubnetRouteTableAssociationA" : { | |
| "Type" : "AWS::EC2::SubnetRouteTableAssociation", | |
| "Properties" : { | |
| "SubnetId" : { "Ref" : "PublicSubnetA" }, | |
| "RouteTableId" : { "Ref" : "PublicRouteTable" } | |
| } | |
| }, | |
| "PublicSubnetRouteTableAssociationB" : { | |
| "Type" : "AWS::EC2::SubnetRouteTableAssociation", | |
| "Properties" : { | |
| "SubnetId" : { "Ref" : "PublicSubnetB" }, | |
| "RouteTableId" : { "Ref" : "PublicRouteTable" } | |
| } | |
| }, | |
| "PublicSubnetRouteTableAssociationC" : { | |
| "Type" : "AWS::EC2::SubnetRouteTableAssociation", | |
| "Properties" : { | |
| "SubnetId" : { "Ref" : "PublicSubnetC" }, | |
| "RouteTableId" : { "Ref" : "PublicRouteTable" } | |
| } | |
| } | |
| }, | |
| "Outputs" : { | |
| "VpcId" : { | |
| "Value" : {"Ref" : "VPC"}, | |
| "Description" : "VPC ID of newly created VPC" | |
| }, | |
| "PublicSubnetA" : { | |
| "Value" : {"Ref" : "PublicSubnetA"}, | |
| "Description" : "Public Subnet in AZ A" | |
| }, | |
| "PublicSubnetB" : { | |
| "Value" : {"Ref" : "PublicSubnetB"}, | |
| "Description" : "Public Subnet in AZ B" | |
| }, | |
| "PublicSubnetC" : { | |
| "Value" : {"Ref" : "PublicSubnetC"}, | |
| "Description" : "Public Subnet in AZ C" | |
| } | |
| } | |
| } |
The template can be downloaded here: https://s3-ap-southeast-2.amazonaws.com/andreaswasita/cloudformation_template/demo/lab1-vpc_ELB_combined.template
I put together the following video on how to use the template:
Understanding a CloudFormation template
AWS CloudFormation is pretty neat and FREE. You only need to pay for the AWS resources provisioned by the CloudFormation template.
The next bit is understanding the Structure of the template. Typically CloudFormation template will have 5 sections:
- Headers
- Parameters
- Mappings
- Resources
- Outputs
Headers: Example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "AWSTemplateFormatVersion" : "2010-09-09", | |
| "Description" : "Builds a VPC w/ INET Gateway and 3 public subnets. **WARNING** This template creates Amazon EC2 instance(s). You will be billed for the AWS resources used if you create a stack from this template.", |
Parameters: Provision-time spec command-line options. Example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "Parameters" : { | |
| "KeyName" : { | |
| "Description" : "Name of an existing EC2 KeyPair to enable SSH access to the instances", | |
| "Type" : "String", | |
| "MinLength": "1", | |
| "MaxLength": "64", | |
| "AllowedPattern" : "[-_ a-zA-Z0-9]*", | |
| "ConstraintDescription" : "can contain only alphanumeric characters, spaces, dashes and underscores." | |
| }, | |
| "VpcId" : { | |
| "Type" : "String", | |
| "Description" : "VpcId of your existing Virtual Private Cloud (VPC)" | |
| }, | |
| "SubnetId" : { | |
| "Type" : "String", | |
| "Description" : "SubnetId of an existing Public facing subnet in your Virtual Private Cloud (VPC)" | |
| } | |
| }, |
Mappings: Conditionals Case Statements. Example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "Mappings" : { | |
| "AWSNATAMI": { | |
| "us-east-1": {"AMI": "ami-6e9e4b06"}, | |
| "us-west-2": {"AMI": "ami-8b6912bb"}, | |
| "us-west-1": {"AMI": "ami-1d2b2958"}, | |
| "eu-west-1": {"AMI": "ami-14913f63"}, | |
| "ap-northeast-1": {"AMI": "ami-27d6e626"} | |
| } | |
| }, |
Resources: All resources to be provisioned. Example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "Resources" : { | |
| "NATIPAddress" : { | |
| "Type" : "AWS::EC2::EIP", | |
| "Properties" : { | |
| "Domain" : "vpc", | |
| "InstanceId" : { "Ref" : "NATDevice" } | |
| } | |
| }, | |
| "NATDevice" : { | |
| "Type" : "AWS::EC2::Instance", | |
| "Properties" : { | |
| "InstanceType" : "m3.medium", | |
| "KeyName" : { "Ref" : "KeyName" }, | |
| "SubnetId" : { "Ref" : "SubnetId" }, | |
| "SourceDestCheck" : "false", | |
| "ImageId" : { "Fn::FindInMap" : [ "AWSNATAMI", { "Ref" : "AWS::Region" }, "AMI" ]}, | |
| "SecurityGroupIds" : [{ "Ref" : "NATSecurityGroup" }] | |
| } | |
| }, | |
| "NATSecurityGroup" : { | |
| "Type" : "AWS::EC2::SecurityGroup", | |
| "Properties" : { | |
| "GroupDescription" : "Enable internal access to the NAT device", | |
| "VpcId" : { "Ref" : "VpcId" }, | |
| "SecurityGroupIngress" : [ | |
| { "IpProtocol" : "tcp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" }, | |
| { "IpProtocol" : "udp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" } , | |
| { "IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "192.168.0.0/16" } | |
| ], | |
| "SecurityGroupEgress" : [ | |
| { "IpProtocol" : "tcp", "FromPort" : "22", "ToPort" : "22", "CidrIp" : "0.0.0.0/0" }, | |
| { "IpProtocol" : "tcp", "FromPort" : "80", "ToPort" : "80", "CidrIp" : "0.0.0.0/0" }, | |
| { "IpProtocol" : "tcp", "FromPort" : "443", "ToPort" : "443", "CidrIp" : "0.0.0.0/0" }, | |
| { "IpProtocol" : "tcp", "FromPort" : "9418", "ToPort" : "9418", "CidrIp" : "0.0.0.0/0" }, | |
| { "IpProtocol" : "tcp", "FromPort" : "0", "ToPort" : "65535", "CidrIp" : "192.168.0.0/16" }, | |
| { "IpProtocol" : "udp", "FromPort" : "123", "ToPort" : "123", "CidrIp" : "0.0.0.0/0" }, | |
| { "IpProtocol" : "icmp", "FromPort" : "-1", "ToPort" : "-1", "CidrIp" : "0.0.0.0/0" } | |
| ] | |
| } | |
| }, | |
| "PrivateRouteTable" : { | |
| "Type" : "AWS::EC2::RouteTable", | |
| "Properties" : { | |
| "VpcId" : {"Ref" : "VpcId"}, | |
| "Tags" : [ | |
| {"Key" : "Application", "Value" : { "Ref" : "AWS::StackName"} }, | |
| {"Key" : "Network", "Value" : "Private Route" } | |
| ] | |
| } | |
| }, | |
| "PrivateRoute" : { | |
| "Type" : "AWS::EC2::Route", | |
| "Properties" : { | |
| "RouteTableId" : { "Ref" : "PrivateRouteTable" }, | |
| "DestinationCidrBlock" : "0.0.0.0/0", | |
| "InstanceId" : { "Ref" : "NATDevice" } | |
| } | |
| } | |
| }, |
Outputs: Example:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| "Outputs" : { | |
| "PrivateRouteTableId" : { | |
| "Value" : {"Ref" : "PrivateRouteTable"}, | |
| "Description" : "Private Route Table ID" | |
| } | |
| } | |
| } |
Note: Not all AWS Resources can be provisioned using AWS CloudFormation and it has the following limitations.
In Part 2 we will deep dive further on AWS CloudFormation and automating the EC2 including the configuration for NAT and Bastion Host instance.
Andreas Wasita 