Next-Generation Bastion: SSH Tunneling to RDS & EC2 without Public Facing Infrastructure

In this post we will be covering how you can use AWS SSM Session Manager and an SSH proxy to perform tunneling. This will allow you to connect to RDS and EC2 instances without the requirement of public facing infrastructure.

The Old

If you have designed your AWS network according to best practices you likely use a bastion server to connect to resources within a private subnet. In terms of this blog a bastion server is a specially hardened server that will act as a single point of ingress into an application VPC. Bastion servers are often internet facing in a DMZ but can be locked down with security groups. 

A secure network using bastions may look something like below. 

Standard Network

Here the end administrator is using an SSH tunnel to connect through the bastion to the RDS instance located in the Application VPC’s Private Subnet A. This is a pretty common configuration and when using network ACL’s and security groups this is quite secure. However, you must still have a public IP address that is accessible from the internet.  Misconfigurations of Network ACLs and security groups could lead to exposure resulting in brute force attacks and pesty port-probes.

The New

The problems of the past have been resolved! Well… kind of. You are still required to have a bastion server, however this bastion server does not need to live in the DMZ and within a separate VPC.  Instead we can leverage AWS Systems Manager Session Manager to tunnel to resources such as RDS and other EC2 instances.  Ideally this would be entirely through the AWS SSM client, however until then, here is how you can do it with Session Manager and an internal bastion server. 

With this new configuration using AWS SSM Session Manager I’m able to strip out the entire bastion VPC. The network architecture then looks similar to below.

SSM Bastion Network

The Configuration

To get started using AWS SSM Session Manager with an internal bastion we will need to configure some stuff. Really there are three components that require configuration, Session Manager, the client ~/.ssh/config file, and the internal bastion.

For this example we will assume that some fundamental configurations exist. For further instructions on how to configure the resources below, please refer to the AWS documentation. 

Assumptions

Internal Bastion Configuration

After you have launched your new internal bastion server with the AWS SSM agent installed and the appropriate IAM role associated with the instance we can begin configuring users.

Configuring users on this instance will be no different than configuring them on a regular Linux server. However later we will touch on some enhancements & automations that can help streamline the use of this technology. 

Connect to the newly configured internal bastion server by running 

aws --profile <your aws profile>  ssm start-session --target <your internal bastion instance-id>

This will drop you in as a default user with the ability to escalate to root privileges. This will be useful for our initial configuration in this example. However later on in this blog post we will be covering how to revoke those permissions. 

On the internal bastion host create the necessary users with their ssh keys in their respective authorized_keys files. If you are unfamiliar with adding SSH users to a linux instance see the document below. 

https://aws.amazon.com/premiumsupport/knowledge-center/new-user-accounts-linux-instance/

Once the users have been added exit the bastion server. 

Session Manager Configurations

Navigate to AWS Systems Manager > Session Manager > Preferences.

Here we will “Enable Run As support for Linux instances” and enter an arbitrary username that you would like session manager users to connect as. This is recommended because if not selected the default user has the ability to escalate to root. 

You may enable KMS encryption to encrypt session data. If you do not select this session data will still be encrypted but with only TLS 1.2. Enabling KMS encryption will use both KMS to wrap the session data and TLS 1.2. 

Furthermore we have chosen to send our session outputs to CloudWatch logs. This is optional but recommended if you wish to have an audit trail. 

SSH Configuration

The key to being able to access other resources within the VPC once connected to the internal bastion is to use the internal bastion as a proxy. To do this we will configure our ssh config file, on linux machines this is located ~/.ssh/config .

Open ~/.ssh/config with the text editor of your choosing and add the following entry. 

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws --profile default ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

This entry will configure your ssh client in a way that anytime you initiate an ssh session to a host beginning with “i-” or “mi-” the “aws ssm start-session” command will be executed as well. 

You may note that we have –profile default configured in here. In this example I’m using my default aws profile, if you are performing this example in an environment other than your default one, replace default with your aws profile name. 

You might wonder what to do if you have multiple AWS profiles. If that is the case you can copy and paste the entry in your config file replacing “i-*” with the specific bastion instance-id for the respective environment.

Alternatively you might want to use an environment variable that you export such as.

# SSH over Session Manager
host i-* mi-*
    ProxyCommand sh -c "aws --profile $AWSPROFILE ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

This requires you to run “export AWSPROFILE=your_aws_profile_name” whenever you wish to connect to the desired AWS environments bastion.

Connecting

Now that you have configured everything we can attempt our first connection. 

I will assume you have either logged into AWS SSO and got your temporary STS tokens or are using AWS access keys. Whichever approach the user connecting must have the ability to perform ssm:StartSession and ssm:TerminateSession. 

You will now be able to connect to your internal bastion server by running the following command.

ssh <username>@<instance-id>

For example.

ssh blogexample@i-000000000

If the username on your local machine is the same as the username on the bastion along with the configured ssh key, you will be able to just run ssh and the instance id of your internal bastion. 

ssh i-0000000

You should note that this has dropped you into your home directory as the user we configured earlier.

[blogexample@i-000000000 ~]$ whoami
blogexample
[blogexample@i-000000000 ~]$ pwd
/home/blogexample

If you wish to port-forward to an RDS instance all that is required is that the internal bastion’s security group is able to communicate with RDS and you have the necessary RDS credentials. To initiate the connection you can run:

ssh blogexample@i-000000000 -L 3306:<your-rds-datbase-name>.us-east-1.rds.amazonaws.com:3306

You then are able to login to the RDS instance from your client machine via localhost:3306.

This port-forwarding can be applied to other resources within your environment. For example if you have another instance running in the private subnet with a private IP 10.0.2.60 website running on port 8080, you could connect to it like so.

ssh blogexample@i-000000000 -L 8080:10.0.2.60:8080

You would then be able to open up your client browser to localhost:8080 and access the website. 

Take-aways

Here we provided a working example of how you can eliminate any public facing instance. We also proved that you are able to connect to other resources within side your VPC by leveraging an internal bastion. 

It is also important to note that even if a user SSH key is somehow forgotten and left on the bastion host, the user will not be able to connect to the bastion unless they have an AWS IAM account or are an authorized federated user. 

I realize the example I have provided isn’t a perfect solution. There are actually several enhancements that you can add to make this solution more streamlined. We will cover this in the enhancements section. 

Enhancements 

Automated User Provisioning

As you can imagine provisioning users on the internal bastion can be cumbersome. However, this is easily solved through automation. You may use a pre-baked AMI that has the SSM agent installed and a user you can connect to that has permissions to provision users. Furthermore you might consider leveraging Ansible, Chef, or Puppet to add and remove users from the internal bastion.

This is easily done using AWS OpsWorks. You can create a new stack for your bastion and leverage OpsWorks to add or remove users. I have found this to be the simplest and most streamlined approach when operating within AWS. 

Removing the ssh-user

Multiple users connecting to the bastion host and using the ssh-user account doesn’t seem desirable for an audit trail. What we have discovered through feature or bug is that the user we provisioned “ssh-user” can actually be removed from the bastion host. Authenticated users then do not have the ability to gain a shell without a profile on the internal bastion system. 

This is desirable because it only leverages SSM for initial authentication and initialization of the proxy and then leverages SSH for authentication on the internal bastion. Again it is important to note that removal of a user from AWS IAM or in the case of them being federated, the IDP, they will not be able to login to the bastion server from outside the VPC. You must have both access keys/ federation AND user profile on the bastion server.

By removing this user you will not be able to connect to SSM from the AWS Console.

Tags & Policies

You can further enhance your environment’s flexibility by leveraging tags. By adding a tag with the key value pair “AccessGranted”:”true” and “ServerType”:”bastion”. We are able to create policies that limit the scope of what SSM can connect to. For example users that you wish to only be able to leverage SSM to connect to the bastion you can add the following to their IAM role. 

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": [
                "arn:aws:ec2:*:*:instance/*"
            ],
            "Condition": {
                "StringLike": {
                    "ssm:resourceTag/ServerType": [
                        "bastion"
                    ],
                    "ssm:resourceTag/AccessGranted": [
                        "true"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:StartSession"
            ],
            "Resource": "arn:aws:ssm:us-east-1:*:document/AWS-StartSSHSession"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ssm:TerminateSession"
            ],
            "Resource": [
                "arn:aws:ssm:*:*:session/${aws:username}-*"
            ]
        }
    ]
}

If we then set the value for “AccessGranted” to false, users with this policy attached to their role will not even be able to connect to the bastion. This also prevents them from accessing other resources within your environment that might have the SSM agent and not be a bastion server.

I will warn that this isn’t to replace any security controls you might have in place within your environment. This can just add a layer of difficulty so a user doesn’t accidentally connect to a resource and do something to mess it up. That being said, continue to control user access as necessary and don’t rely on the above policy as a reason to give carte blanche access to users.

Why you should be using an AMI Bakery

When operating in either a multi-account or multi-cloud environment it can be hard to keep track of the operating systems that are being used, if they are hardened, and if they contain vulnerabilities. To assist in managing these concerns we use what we call the “AMI Bakery”. This concept isn’t exclusive to AWS, so really you could call this an “image bakery”, however, the concepts we cover will target AWS resources.

What is an AMI Bakery?

An AMI (Amazon Machine Image) Bakery is an automated and repeatable process to create standardized, hardened, operating system images. In the context of this document an image is a snapshot of an operating system with the desired installed and configured software. Images can be shared and reused by other engineers.

Why use an AMI Bakery?

An AMI Bakery can offer many benefits. One of the clear benefits of this is the ability to create consistent images through a repeatable process. We are also able to track vulnerabilities that exist within images in our environment. Below are other reasons on why you should consider using an AMI Bakery.  

  • Repeatability of image creation in a standardized way.
  • Enforcing instance hardening at the image level. 
  • Enforcing logging configurations at the image level.
  • Enforcing custom software installations (such as an anti-virus). 
  • Auditability of how instances were configured.
  • Tracking of vulnerabilities within an environment. 
  • Enforcement of standardized images.


How Does an AMI Bakery Work?

AMI Bakery on AWS.

The AMI Bakery itself could be a script, however I suggest using a CI/CD pipeline such as GitlabCI or Jenkins, this will be the magic sauce for automating image creation. The CI/CD pipeline includes steps for each operating system we wish to provision, with the ability to initiate the creation of all images or manually choose which ones we’d like to create. This allows us to scale as new operating system variants come out, as well as add in custom images that software engineers may want to use.

Each operating system is stored in AWS Parameter store with a unique name that can be referenced. This way the engineer only needs to know the parameter store name to get the latest AMI that they should be using. AWS Parameter store will keep track of the different versions of the AMI and historical metadata.

Instance Creation

When an image creation step is initiated an Ansible playbook is kicked off. This playbook creates an instance based off of the AWS marketplace AMI for that variant. Again, the tool used here is pretty fluid, you may decide that something such as Packer would be more suitable.

Instance Configuration

Once the instance has been created we must now manipulate it to our desired state. This step also uses Ansible, but like before you may choose a tool you’re more comfortable with such as Chef or Puppet.

Our Gitlab runner connects to the instance via Ansible and runs multiple playbooks that harden, patch and install custom software. One of the components of custom software worth mentioning here is the AWS Inspector agent. This will allow us to scan the image for vulnerabilities once the image is baked. You can easily substitute this with a vulnerability scanner of your choosing.

Baking the Image

After all of our playbooks have run the instance should be configured and ready to be baked. Ansible calls the AWS API to create a new image from the instance we just configured.Once the image is created following calls are executed to update AWS Parameter store with metadata about the image and most importantly the AMI ID. Another call is then executed that shares the gold image out to all of the AWS accounts within our organization.

Vulnerability Scanning

After the new image has been created we initiate vulnerability scanning. A fresh instance is launched from the AMI after which a vulnerability scanning job is executed by AWS inspector. The instance runs for the duration of the vulnerability scan and is then terminated. The vulnerability information collected by AWS inspector is gathered and sent to AWS Security Hub. Here we can keep track of all the images that are created from the bakery and what vulnerabilities exist.

It would behoove you to have another vulnerability scanning pipeline for images in use. Over time the images will gather vulnerabilities and should be scanned regularly. A weekly job that spins up images in use, scans them and sends the data to security hub should be included in tandem to your bakery process. This way you are able to quickly identify AMI’s that contain critical vulnerabilities and need rotated as soon as possible.

An important thing to note is that if an image is a “pet server” it should have vulnerability management handled separately. Pet servers will often have non-standardized software installed after the instance is launched. The strength of the AMI Bakery and vulnerability management comes into play with immutable environments that are able to be destroyed and created rapidly.

AMI Enforcement

You might be wondering how you can force engineers to use AMI’s created from your bakery. To do this will require either a Lambda function or the use of AWS Config. A custom AWS Config rule that references the gold standard images can validate that any image that is created exists as a gold standard image. If an image that is created does not match an AMI ID of a gold standard image we can automatically power off the instance or terminate it. The same concept can be accomplished with a Lambda function. You can monitor instances in the account by looking at each instances image ID. Again if the ID does not match you are able to power off or terminate the instance.

Minikube, Kubernetes on Windows 10

Today I did something that I did not think I would ever do.. I installed Minikube …on Windows 10. I know what you’re probably thinking, “Why would you use/learn Kubernetes on Windows?”. Well, quite honestly, it is really not that bad either. 

In this post I’ll show you how to get Minikube up and running on Windows 10 with kubectl in a relatively short amount of time. So, if you are comfortable with Windows you can dive right into learning k8’s (Kubernetes). I typically have only seen Minikube deployed on Linux and OSX. There is a lot of content online to get Minikube up and running on those platforms so I’ll strictly be covering Windows in this post.

What is Minikube? Minikube will allow you to run a single node k8’s cluster locally on a virtual machine, this is especially useful if you would like to learn k8’s or you would like to have a development environment that you can use without spinning up an entire k8’s cluster.

Prerequisites:

  • Windows 10
  • Virtualbox

Installing Minikube:

Releases for Minikube can be found on the official Kubernetes Github page:

https://github.com/kubernetes/minikube/releases

Install the file called “minikube-installer.exe” and run the installer.  As noted in the README.md it will automatically setup the path for you. This method is listed as experimental but honestly, I had no issues using it.

Once the installation has been complete open up Windows Powershell.  (Windows > Windows Powershell)

You should be able to run the following command now and have the same content returned:

minikube help


Lets start Minikube, when we start Minikube it will automatically configure the server, the certificates, and the kube config file.

minikube start

You might have noticed something else, it says that it is pointing to a minikube-vm.  If you navigate to your Virtual Box application you will notice that a new VM has been provisioned.

Now that we have our k8’s environment setup on Minikube we need a way to communicate with it. To do this we will need Kubectl. You can find the installation guide for Kubectl located here: https://kubernetes.io/docs/tasks/tools/install-kubectl/

We can download the binary for Windows by opening up a browser and navigating to the following URL:

https://storage.googleapis.com/kubernetes-release/release/v1.13.0/bin/windows/amd64/kubectl.exe

This will download the executable to your default Downloads directory.  I suggest moving this to a location other than that.  I put my installation at C:\kubernetes by creating a directory called “kubernetes” on C:\.

Once the file has been moved there we will need to add it to our PATH. We can do this by going to Windows > Advanced System Settings > Advanced > Environment Variables > Path > Edit

You will now need to add the location of the binaries directory, in my case this was C:\kubernetes

Click,  OK > OK > OK

Open up another Power Shell (Windows > Windows Powershell)

You should now notice that your kubectl command will work if you run:

kubectl

Now that we have validated kubectl is working based off our path we can check the status of our cluster. Go ahead and execute the following commands:

kubectl get namespaces
kubectl get pods -o wide –all-namespaces

You should see something similar:

This displays all of the namespaces in the k8’s cluster that exists as well as the pods that are running in the cluster.

Congratulations you now have a Kubernetes cluster running on Minikube that you can communicate with and learn to use!

To stop your Minikube it is as simple as running:

minikube stop

Ansible OpenSCAP Remediation for CentOS 7

In the previous blog post we initiated an OpenSCAP assessment with the DISA STIG profile. What we are going to do is use the GUI of scap-workbench to create an Ansible playbook that we can use to remediate the findings on the CentOS 7 system. The requirements to perform this is a Linux system with a GUI. In this blog post I will be using Fedora (my primary OS) to generate the Ansible playbook used to remediate findings.  

Note that this should not be done on a production system since the fixes can be invasive, proceed at your own risk.

Generating The Playbook

First, we must install the needed package(s). Open up a terminal and run:

sudo yum install scap-workbench ansible -y

Now lets start the scap-workbench:

scap-workbench

This should bring up a window like so:

Change the drop down to the appropriate operating system and click “load content”.  For mine I’m using CentOS 7.

Now you will see a fancy tool called SCAP Workbench.  We will need to set the profile, so click the drop down and select the profile that applies.  For this post I will be using the “DISA STIG for Red Hat Enterprise Linux 7(243)”

As you might have guessed you can run the scan through the SCAP Workbench to produce the same results we did via the command line in the previous blog post. You also have the ability to run the scan against a remote host.  Ignore that aspect of the tool for now and click the drop down “Generate remediation role”.

Select “ansible” which will open up a file explorer, save the file to where you know you will be able to find it again.

What just happened?

SCAP Workbench generated a remediation role for Ansible against the DISA STIG profile. If we view the file, we can see exactly what it is doing and will apply when we run the playbook.

Running The Playbook

For the sake of this example I am going to modify the playbook to ignore any errors that are found.  If you do not do this the playbook will fail out instead of skipping over any missing files.

Using Vim at the beginning of the file I modified the it to include “ignore_errors: yes” under my “- hosts: all” like so:

Save and quit the file (if using Vim that is :wq)

Now let us create a hosts file that we will use as an inventory.  In the same location as your remediation.yml lets create a file called “inventory”.

You can get really advanced with your inventory files, but for this example we are keeping it simple. Add the IP address of the CentOS server you would like to harden.

So now this is what my inventory file looks like, really simple just one line with “192.168.1.39” :

Side note, you are also able to leverage dynamic inventories.  For example if you are using AWS and have ec2 instances tagged as something you are able to use the dynamic inventory with the tags instead of specifying IP addresses, more on that here

https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html

Now back at the terminal we can execute the playbook.  What this command is doing below is telling ansible to use the switch -i which passes a file called “inventory”, –user is run the ssh connection as the user “spencer” , -k which is ask for the ssh password, and -K which is ask for the sudo password, –become which means become root by default (you can become other users but the default is root), and -v for verbose.

ansible-playbook -i inventory --user=spencer -k -K --become remediation.yml -v

You will now see the playbook executing and changing the system.  By us making the change to the playbook ignore_errors: yes, it will skip over any problematic errors (which consequently means those mitigation’s won’t be applied).

This playbook will take some time so let it run.  Upon completion you should see something like this:

If you login to the system you will be able to validate that everything that has been marked as “changed” has been modified on the system. You will notice that all the hardening paramaters now exist on the system we ran the playbook against.

Assessing CentOS 7 with OpenSCAP

In this post I’ll be using a tool called OpenSCAP (Details can be found here, check it out https://www.open-scap.org/) to assess a CentOS 7 system. If you have never heard of OpenSCAP before but have had to perform a hardening assessment of a system, OpenSCAP will be a life saver. You can use OpenSCAP with different profiles aligned with different standards such as PCI-DSS. In this example I will be using the DISA STIG (security technical implementation guides) profile which is quite stringent. Exercise with caution when using the suggested hardening parameters, many of them are invasive changes that could impact applications running.

Typically, a method that has been helpful is to bake an image with the hardened parameters that will be used, update the image with newly hardened parameters as they come out. This isn’t necessarily required but it can help speed things up, also tools like Ansible, Chef, and Puppet can help automate the hardening.

Open up a shell to the system you wish to assess and install the necessary packages:

sudo yum install openscap scap-security-guide -y

This will install oscap server/client needed to run the scan as well as the security content (STIGs) for each operating system. You can find this information located on your system at:

/usr/share/xml/scap/ssg/content/

To see the different profiles that can be executed you can run the following command to output information for your operating system. Since ours is CentOS 7 I selected that, if you are using RHEL you would select that profile. This will list all the profiles you can run your scan against, we are going to use the DISA STIG profile as mentioned earlier on.

oscap info /usr/share/xml/scap/ssg/content/ssg-centos7-ds.xml

We are selecting the profile:

Id: xccdf_org.ssgproject.content_profile_stig-rhel7-disa

To run the scan against the DISA STIG we execute the following command.  This command will output an html report to /tmp/report.html , this report allows you to visually see the findings.

sudo oscap xccdf eval --profile xccdf_org.ssgproject.content_profile_stig-rhel7-disa --report /tmp/report.html /usr/share/xml/scap/ssg/content/ssg-centos7-ds.xml

You will begin to see the scan execute, with results similar to those below, the scan takes time to complete, this is expected.

View the output file /tmp/report.html once the scan has been completed.

The OpenSCAP evaluation report provides a summary of findings.  Navigate to “Compliance and Scoring” here you will see the summary of results, in this case out of the box my CentOS 7 system is 53.9% compliant.

For this score to increase you would need to remediate the findings and rerun the scan. Again be careful, many of the findings and implementations could be dangerous to enable in your environment. Always remediate with caution when hardening your system.

Welcome.

So I have decided to create a blog like everyone else in the universe… However, I hope you find the topics that I write about useful. There might be times where I write a blog post ranting about some inane topic in the industry, but I will try to keep most posts to tutorials and demonstrations (videos too?).

I plan on focusing my energy primarily on DevSecOps, with an even more specific scope of cloud security. This isn’t to say that posts related to development and operations won’t be published. When it comes to cloud I will primarily be using AWS. I also am a Linux fan boy so beware.

During the creation of blog posts please feel free to comment on or contact me if it pertains to one of the following:
– A better way to do something than I am showing.
– Questions.
– Future blog post recommendations.
– Discussions

~ Spencer