AWS EC2 is one of the most commonly used service for hosting instances on AWS cloud. When you have multiple team members, you have to handle headaches of managing their SSH acces. Well... What if I tell you that now :

• You can manage EC2 ssh access by adding just AWS IAM user policy. Oh yes! You read that right!
• NO need to add SSH keys and manage those on your own for providing access to users in the organization
• NO need to release production instance updates to add or remove new users and their SSH keys
• NO need to worry about the key rotation, key management done by AWS automatically
• All connection requests using EC2 Instance Connect are logged to AWS CloudTrail so that you can audit connection requests
• Optionally, If your instance is publically accessible, you can SSH directly from an AWS browser shell window without even doing SSH locally
• Last but not the least, you can set this up entirely on an instance under just 15 minutes

This is all possible now, thanks to the new AWS service called EC2 instance connect. If this does not convince you to explore more about this, nothing else will!

AWS Instance Connect :

This is by far one of my favorite AWS service announcement of 2019. We all have had our headaches to manage SSH keys on EC2 instances, add new keys for new users, rotate the keys to add more security, remove the keys for users who left the company. Oh man, give me more tissues!

But the new AWS EC2 connect is the ultimate saviour.

AWS EC2 Instance Connect provides a simple and secure way to connect to your instances using Secure Shell (SSH). With EC2 Instance Connect, you use AWS Identity and Access Management (IAM) policies and principals to control SSH access to your instances, removing the need to share and manage SSH keys. All connection requests using EC2 Instance Connect are logged to AWS CloudTrail so that you can audit connection requests.

You can use Instance Connect to connect to your Linux instances using one or all of below :

• A browser-based client
• The Amazon EC2 Instance Connect CLI
• The SSH client of your choice.

The cheery on the top is temporary nature of the SSH public key. When you connect to an instance using EC2 Instance Connect, the instance connect API pushes a one time use SSH public key to the instance which is getting connected. It uses instance metadata for the same. And it key just stays there for 60 seconds, so that SSH is done correctly. That key is then removed and for next connection the same process is done internally by AWS EC2 instance connect.

An IAM policy attached to your IAM user authorizes your IAM user to push the public key to the instance metadata and for the users who do not have this policy attached, they can not connect to the instance.

The SSH daemon internally uses AuthorizedKeysCommand and AuthorizedKeysCommandUser, which are configured when Instance Connect is installed, to look up the public key from the instance metadata for authentication, and connects you to the instance.

Isn't that awesome!

A heads up on instance OS support :

The EC2 instance connect is currently supported for :

• Amazon Linux 2 having any version
• Ubuntu 16.04 or later versions

Hoping that AWS will add support for the other operating systems!

Step 1 - Spinning up the EC2 server to dryrun ECV2 Instance Connect :

I would highly recommend to NOT try installing anything on your working instance. It is always safe to try it out on a small vanilla instance, know the gotchas and then be prepared to do it on production or any of your existing instances. So for the dryrun we will spin up a test EC2 instance.

• Login to your AWS Console and go to the region you want the instance to be in
• Select EC2 service and click on Launch to spin up a new instance
• The EC2 launch wizard will be shown, search for ubuntu and press enter
• It will show number of ubuntu AMIs. Let's choose Ubuntu Server 18.04 LTS
• Now click on select
• Click on continue and choose instance type as t2.micro as it is inside the free tier usage
• Click on Next: Configure Instance Details
• In this step you dont need to worry about VPC and instance role as this instance is just a test one. You can keep the default settings
• Click on Next: Add Storage
• Keep the default storage as it is as we won't need any extra storage space
• Click on Next: Add Tags and add the tags you need for this instance
• Click on Next: Configure Security Group
• You need to select Create a new security group. Add security group name as Temporary EC2 Connect SG. And add just a rule for port 22(SSH) open to all with value 0.0.0.0/0
• Click on Review and Launch
• Verify the details once in this final summary screen and click on Launch
• It will ask you to select a key pair create a new one as EC2-Connect-Dryrun-key-pair and download it.
• Now finally.. Click on Launch Instances

You are done with launching the instance..

Step 2 : Install EC2 Instance Connect on the instance :

Now, we will install EC2 instance connect service on our new EC2 instance with AWS CLI. You can SSH to your instance using following command :

ssh -i EC2-Connect-Dryrun-key-pair.pem ubuntu@x.x.x.x

Where x.x.x.x is the new instance IP address. Now you can use below set of commands to install AWS CLI and then EC2 instance connect :

# Update the packages as we are logging to the instance for the first time
sudo apt-get -y update
# Install python 3 and pip3 which is needed to install AWS CLI
sudo add-apt-repository ppa:ubuntu-toolchain-r/ppa
sudo apt install -y python3.7 python3-pip
# Install AWS CLI
sudo pip3 install awscli --upgrade --user
# Install EC2 instance connect
sudo apt-get install -y ec2-instance-connect
# Make sure instance connect files are created properly
ls -1  /usr/share/ec2-instance-connect/

You can verify the instance connect has updated the ssh setting by running follwing command :

sudo cat /lib/systemd/system/ssh.service.d/ec2-instance-connect.conf

This should give output containing following strings :

AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys %u %f
AuthorizedKeysCommandUser ec2-instance-connect

Now, you are done with the setup on your instance. Believe it or not, that is it for the instance changes!

Step 3 : Create policy to allow SSH Connect Access :

We will create a new IAM policy which we can attach to a user so that user can SSH to the instance. Below is the policy JSON :

{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "ec2-instance-connect:SendSSHPublicKey",
        "Resource": [
            "arn:aws:ec2:region:account-id:instance/i-1234567890abcdef0",
            "arn:aws:ec2:region:account-id:instance/i-0598c7d356eba48d7"
        ],
        "Condition": {
            "StringEquals": {
                "ec2:osuser": "ubuntu"
            }
        }
      },
      {
        "Effect": "Allow",
        "Action": "ec2:DescribeInstances",
        "Resource": "*"
      }
    ]
}

Make sure you replace the resources with their respective arns. e.g. arn:aws:ec2:us-west-1:12341274:instance/i-1234567890abcdef0 where region is us-west-1 and account id is 12341274.

Let's now understand this policy. This policy will allow user to send a new SSH public key to the instances specified in the resources array. This pushed public key will allow the SSH connection to the instance.

The ec2:osuser as ubuntu is the SSH connection linux user. Realistically ubuntu is the main super user. We will change this on the last part of this article where we can have different linux user having less priviledges on the instance. For now, let's keep going.

The second policy rule ec2:DescribeInstances is used by the user to get instance details from its instance id. We will be using instance id instead of public or private DNS or IP. So this rule is required in the policy so that part of the process works.

• Login to your AWS Console
• Select IAM service and click on Policies tab on the left menu
• Click on Create Policy
• Click on JSON tab and enter above policy JSON in it. Make sure you replace the instance ARN as per your instance metadata details
• Click on Review Policy
• Give policy name as EC2-SSH-Access and enter description you need
• Click on Create

Now we have the policy ready which can give access to any user to connect to the instance via Secure Shell (SSH).

Step 4 : Create a new User and attach policy :

We will create a new user as user-with-ssh-access and attach above policy.

• Login to your AWS Console
• Select IAM service and click on Users tab on the left menu
• Click on Add User
• Give user name as user-with-ssh-access
• In Select AWS access type choose Programmatic access
• Click on Next: Permissions
• In set permiossions tab choose Attach existing policies directly
• Search our policy EC2-SSH-Access and select it
• Click on Next: Tags and add tags as per your need
• Click on Next which will review your user
• Click on Create

Now, we will need to add AWS credentials to this user so that we can use those to connect.

• Login to your AWS Console
• Select IAM service and click on Users tab on the left menu
• Search our user user-with-ssh-access and click on it
• User Summary page will appear, click on Security credentials tab
• In Access Keys section click on Create access key
• It will give you Access Key Id and Secret Access Key. Make sure you save it locally in notepad or somewhere safe as we will need it to ssh to the instance

Step 5 : Setting up local system for Secure Shell(SSH) connection :

Locally you need to have AWS CLI installed. If you do not have it, make sure you install it. Click Here to folow the steps.

Once install run following on your terminal :

aws configure

If you get error like aws command not found find where the aws service is installed on your computer by doing which aws. Sometimes you need to add full path like /usr/local/bin/aws configure

It will ask you details like below :

AWS Access Key ID : ********************
AWS Secret Access Key : ****************************
Default region name [None]: us-west-1
Default output format [None]:

Make sure you add the access key and secret access key we got from earlier step and specify the region in which your EC2 instance resides.

Now, you need to also install EC2 Instance Connect on your local system as it is used to ssh to the instance. Use following command to do the same :

sudo pip install ec2instanceconnectcli

And done! Your local system is ready to SSH connect!

Step 5 : Connect to the instance via Secure Shell(SSH) :

After all above steps are done, we can now SSH to our instance. To ssh we will use mssh command which ships with EC2 instance connect service.

mssh ubuntu@i-xxxxxxxxxxxxxxx

Where i-xxxxxxxxxxxxxxx is the instance id of our test instance we spun up in step 1. You would be able to SSH to the instance if you have followed above steps correctly.

Real world use cases for SSH users :

In above setup, we used ubuntu user to SSH connect. However, in real world we can divide our organizational users who need access into groups based on access they need. For example :

• You can create a user named administrator on your EC2 instance and then use that as ec2:osuser in our policy for system administrators.
• You can create a user named developer on your EC2 instance and then use that as ec2:osuser in our policy for application team which might have access to just the application codebase.

Please note that what access and permission the connected user must have is in your hands. The responsibility of EC2 instance connect is just to allow the IAM user to SSH to the instance with allowed SSH user.

Click Here to know more about quick steps to add new user to the instance.

Managing the ssh access :

You can create multiple policies based on the resources and os username the IAM user can SSH to. If you want to give a new user ssh access, you can just attach respective policy to the user. Similarly, you can just remove an existing ssh cononect policy to remove the SSH access from an existing user.

This is a HUGEEEE plus point of this service.

Accessing the connection logs :

EC2 connect pushes the SSH connection logs to AWS Cloudtrail. To view the logs follow below steps :

• Login to your AWS Console
• Select Clourtrail service and go to the region you have EC2 instance connect resources in
• Click on Event History tab on the left menu
• Addd Event source as the filter type and enter ec2-instance-connect.amazonaws.com as it's value

You would be able to see all the SSH conection logs. Beautiful fact is that, even if you have 10 different IAM users using same SSH user, let's say developer, in the logs, you would be able to differentiate which user connected to SSH via developer connection user.