Manual configuration of public and private subnet, routes, security groups, nat gateway and bastion server via de AWS console
Automation, Ansible, AWS
In our previous two parts of our blog Harden your application security on AWS, (part I and part II) we explained that service platforms like AWS, Microsoft Azure and Google Cloud Platform are as smart as their users since the required architecture still has to be created by people who have understanding of areas like security, networking etc. Also we saw that although the AWS console is very user friendly, we have to go through many manual steps to define the architecture we like to achieve and since there is no graphical representation with an overview of the already defined architecture we can easily loose track when we are not planning or drawing our draft achitecture upfront.
Due to the fact that the architecture is created by defining services in different tables with relations and many manual steps are required, building up or breaking down such an architecture is very time consuming and does not give us the flexibility we need for our systems to serve our business scenarios. Luckily AWS has a great API that gives us the power to automate our architectures on AWS, via tools like Ansible, Chef, Puppet etc. we can leverage the AWS API.
In this blog we will be looking into Ansible and show how to automate the architecture we described in our previous blogs about Harden your application security on AWS.
Ansible is an powerful IT automation engine that lets you deliver your applications faster. Since every business today is a digital business and technology is your innovation engine delivering your applications/solutions faster it helps your organisation win by reducing the time being spend on repetitive tasks.
Ansible lets you avoid repetitive tasks by automating them, this frees up admins to focus on efforts that help deliver, and building a culture of success. Ultimately it will deliver something you can get never enough of: time. Allowing smart people to concentrate on smart things. Ansible is a simple automation language that can perfectly describe an IT application infrastructure. It is easy to learn, self documenting, and doesn't require high level computer science skills.
Ansible has support for a broad range of different modules to be automated: Cloud Modules, Clustering Modules, Command Modules, Crypto Modules, Database Modules, File Modules, Identity Modules, Inventory Modules, Messaging Modules, Network Modules, Packaging Modules, Remote Management Modules, Source Control Modules, Storage Modules, Utilities Modules, Web Infrastructure Modules, Windows Modules.
We will be looking into the Ansible Cloud Module Amazon EC2 to automate repetitive tasks where we used the AWS Console before, giving us more control and flexibility. Ansible is delivering a large list of methods to be used on EC2, see below overview (we are going to make use of some of them in our example underlined in red).
Playbooks are Ansible's configuration, deployment and orchestration language to remote machines, in our case mutli-machine deployment system on AWS. Playbooks can be kept under source control systems like Git etc. Playbooks are expressed in the YAML format and have a minimum of syntax, which intentionally tries to not be a progamming language or script, but rather a model of configuration or a process.
Each playbook is composed of one or more 'plays' in a list. The goal of a play is to map a group of hosts to dome well defined roles, represented by Ansible tasks. At a basic level, a task is nothing more than a call to an Ansible module like the AWS modules presented above.
By composing a Playbook of multiple 'plays', it is possible to orchestrate multi-machine deployments.
You can find more details on Playbooks, Handlers, Tasks, Roles and Variables in the documentation of Ansible which can be found following the url: http://docs.ansible.com/ansible/latest/playbooks.html
Our example architecture consists of a VPC, a Private and Public subnet, a Internet Gateway, a NAT Gateway, a Bastion Server, Route tables and Security Groups. The example is not meant to be as a complete production ready solution but more as an introduction how you can automate repetitive tasks on AWS via Ansible. You can can optimize this example to your own needs.
First we created a structure on disk for our Playbook that is shown in below picture.
Ansible can work against multiple systems in your infrastructure at the same time, it does this by selecting portions of systems listed in the inventory file. You can specify in this file Hosts and Groups, Host Variables, Group Variables etc. In our case this file only contains the section local like shown in below picture meaning that the Ansible playbook will be run from the local machine.
Next we defined global variables to be used in our playbook in the vars.yml file as shown below. These variables will be adressed in Ansible tasks.
As we can see we provided our AWS credentials (access_key and secret_key) as global variables since we need to authorize ourselves with AWS if we call the API methods via the Ansible Cloud Module Amazon. Also we can see that we specified an aws_region in which we like to work as well as the ip addresses of the VPC, Public and Private Subnet as well as the AMI to be used for the Bastion Server. Further we specified our own ip-address my_ip to be able to connect to our AWS instances.
In the playbook.yml we specified the following, meaning that the role vpc will be executed on the local machine. (The tasks for the vpc role are defined under the subdirectory /roles/vpc/tasks)
In the tasks subfolder we will find the file main.yml which is containing the tasks to be executed by Ansible, we did make some variable fields on purpose gray so you can try to complete the architecture yourself as a practise understanding exactly what you are doing instead of copying the example.
From the example we can see that we proceeding the following steps:
Now that we have defined the Ansible script to deploy the architecture, we can excute it by the following command:
ansible-playbook playbook.yml -i inventory -e @vars.yml from the ansible playbook directory vpctest, for this we created a shell script deploy.sh so we can execute it via ./deploy.sh
We will execute this script you will see the status of your Ansible tasks scroll over your screen, below you see the latest steps of our script:
In the AWS console we can check now the result, it should be something like below:
Virtual Private Network:
Public and Private subnet:
Public and Private Route Tables:
Public and Private Security Group:
Next we can make our Ansible script to break down this architecture, you can try this yourself.
By using Ansible and leveraging the AWS API we can automate repeating tasks, it allows for templating and gives us much more flexibility in creating our infrastructure than using the AWS console in building, spinning up and breaking down our architectures on the fly.
The only thing that is also missing in Ansible is a graphical overview of the tasks being executed by the playbook. If you like to work graphically with Amazon using building blocks with a drag and drop interface you can for example have a look at VisualOps http://www.visualops.io, VisualOps is also leveraging the AWS API and can read your architecture from AWS into VisualOps or deploy your created architecture in VisualOps to AWS.
When using VisualOps we encountered that it didn't support the latest building blocks (services) of Amazon like the NAT Gateway, further then that it is very user friendly in building, deploying and destroying your architecture on AWS.
On the other hand if you know Ansible works you are probably better of when using Ansible since VisualOps only works on AWS and Ansible is a multi-purpose automating tool that can be used also for other systems, for example your own Linux or Windows systems.
We hoped you enjoyed this three part blog about Harden your application security on AWS.