Why You Should Be Using an AMI Bakery
When operating in either a multi-account or multi-cloud environment it can be hard to keep track of the operating systems that are being used, if they are hardened, and if they contain vulnerabilities. To assist in managing these concerns we began to use what we call the “AMI Bakery”. This concept isn’t exclusive to AWS, so really you could call this an “image bakery”, however, the concepts we will cover target AWS resources.
What is an AMI Bakery?
An AMI (Amazon Machine Image) Bakery is an automated and repeatable process to create standardized, hardened, operating system images. In the context of this document an image is a snapshot of an operating system with the desired installed and configured software. Images can be shared and reused by other engineers.
Why use an AMI Bakery?
An AMI Bakery can offer many benefits. One of the clear benefits of this is the ability to create consistent images through a repeatable process. We are also able to track vulnerabilities that exist within images in our environment. Below are other reasons on why you should consider using an AMI Bakery.
- Repeatability of image creation in a standardized way.
- Enforcing instance hardening at the image level.
- Enforcing logging configurations at the image level.
- Enforcing custom software installations (such as an anti-virus).
- Auditability of how instances were configured.
- Tracking of vulnerabilities within an environment.
- Enforcement of standardized images.
How does an AMI Bakery work?
The AMI Bakery itself could be a script, however I suggest using a CI/CD pipeline such as GitlabCI or Jenkins, this will be the magic sauce for automating image creation. The CI/CD pipeline includes steps for each operating system we wish to provision, with the ability to initiate the creation of all images or manually choose which ones we’d like to create. This allows us to scale as new operating system variants come out, as well as add in custom images that software engineers may want to use.
Each operating system is stored in AWS Parameter store with a unique name that can be referenced. This way the engineer needs only to know the parameter store name to get the latest AMI they should be using. AWS Parameter store will keep track of the different versions of the AMI and historical metadata.
Instance Creation
When an image creation step is initiated an Ansible playbook is kicked off. This playbook creates an instance based off of the AWS marketplace AMI for that variant. Again, the tool used here is pretty fluid, you may decide that something such as Packer would be more suitable.
Instance Configuration
Once the instance has been created we must now manipulate it to our desired state. This step also uses Ansible, but like before you may choose a tool you’re more comfortable with such as Chef or Puppet.
Our Gitlab runner connects to the instance via Ansible and runs multiple playbooks that harden, patch and install custom software. One of the components of custom software worth mentioning here is the AWS Inspector agent. This will allow us to scan the image for vulnerabilities once the image is baked. You can easily substitute this with a vulnerability scanner of your choosing.
Baking the Image
After all of our playbooks have run the instance should be configured and ready to be baked. Ansible calls the AWS API to create a new image from the instance we just configured.Once the image is created following calls are executed to update AWS Parameter store with metadata about the image and most importantly the AMI ID. Another call is then executed that shares the gold image out to all of the AWS accounts within our organization.
Vulnerability Scanning
After the new image has been created we initiate vulnerability scanning. A fresh instance is launched from the AMI after which a vulnerability scanning job is executed by AWS inspector. The instance runs for the duration of the vulnerability scan and is then terminated. The vulnerability information collected by AWS inspector is gathered and sent to AWS Security Hub. Here we can keep track of all the images that are created from the bakery and what vulnerabilities exist.
It would behoove you to have another vulnerability scanning pipeline for images in use. Over time the images will gather vulnerabilities and should be scanned regularly. A weekly job that spins up images in use, scans them and sends the data to security hub should be included in tandem to your bakery process. This way you are able to quickly identify AMI’s that contain critical vulnerabilities and need rotated as soon as possible.
An important thing to note is that if an image is a “pet server” it should have vulnerability management handled separately. Pet servers will often have non-standardized software installed after the instance is launched. The strength of the AMI Bakery and vulnerability management comes into play with immutable environments that are able to be destroyed and created rapidly.
AMI Enforcement
You might be wondering how you can force engineers to use AMI’s created from your bakery. To do this will require either a Lambda function or the use of AWS Config. A custom AWS Config rule that references the gold standard images can validate that any image that is created exists as a gold standard image. If an image that is created does not match an AMI ID of a gold standard image we can automatically power off the instance or terminate it. The same concept can be accomplished with a Lambda function. You can monitor instances in the account by looking at each instances image ID. Again if the ID does not match you are able to power off or terminate the instance.