So in previous sections we talked about how SSH keys were far more preferable to passwords, and considerably more secure. But how do you get these onto your EC2 instances?
We saw that when you launch an instance, you can – in fact should – assign an AWS generated key to them. But what if you dont want to use that key – wouldnt it be nice to get a key already installed on the EC2 instance when it starts up? This is perfectly possible, along with other things, and we shall use this to demonstrate a very useful tool called Packer…
First we need to get access to the AWS command line. If you dont yet have the AWS CLI installed on your box, be that Windows Mac or Linux, then go and get that downloaded and installed now from here.. https://aws.amazon.com/cli/
I make no apologies for doing all this from a Windows machine btw – because that is what most people will probably start by using. I would urge everyone to start moving towards a linux world though, simply because it is the most flexible if a little more awkward to learn…
Once you have the CLI installed you need to get access to it programatically. This needs access keys, so fire up the AWS console, log in and go to IAM – Identity and Access Management.
In here, there is an option to add access keys to an IAM user. If you don’t have an IAM user set up and are still using root – then stop and get a user added first.
From the user section, add an access key for CLI access. This will give you an access key, and a secret key. Make very careful note of them – again in a password safe like https://keepass.info/ is an ideal place.
You can now configure the aws cli to use these keys to access your AWS account. Get to the command line, and enter aws configure
This will ask for the access and secret key, as well as your default region and output options. You can pick the defaults for the latter two – I would suggest <none> for output format unless you want JSON for some reason…
C:\Users\cs\git\packer>aws configure
AWS Access Key ID [****************KPMR]:
AWS Secret Access Key [****************P1ku]:
Default region name [eu-north-1]:
Default output format [None]:
C:\Users\cs\git\packer>If you now type aws s3 ls from the command line it should just return a list of all the S3 buckets on the account – or a blank line if you don’t have any. If you havent set up the keys or entered them correctly it will give you an access error instead which will need troubleshooting.
Assuming all is well, then we have access to the account from the CLI.
What we are going to do now is create an new Amazon Machine Image. You have probably seen these before – they are the AMI’s that are used to create EC2 instances. As well as the several hundred, if not thousands that you are provided with you can also roll your own. We are going to build one with our keys built into it, so we can connect directly.
Packer is a tool from Hashicorp that will let you build images of, amongst other things Amazon Machine Images, AMI’s. You can get a download from here… https://developer.hashicorp.com/packer/install?product_intent=packer
Once you have it downloaded, you will have a single file – that’s all no dependencies. Stick it somewhere in your file structure – I prefer to have a folder c:\program files\hashicorp and all their tooling such as Packer and Terraform goes in there. I then just add it to the path, so I can run it from anywhere.
Lets do a dry run. In a new folder, create a file called packer.pkr.hcl and enter the following…
packer {
required_plugins {
amazon = {
version = ">= 1.2.8"
source = "github.com/hashicorp/amazon"
}
}
}
source "amazon-ebs" "amazon-linux" {
ami_name = "packer-example-2"
instance_type = "t3.micro"
region = "eu-north-1"
source_ami_filter {
filters = {
image-id = "ami-0d74f1e79c38f2933"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["137112412989"]
}
ssh_username = "ec2-user"
}
build {
name = "SSH-keys-loaded"
sources = [
"source.amazon-ebs.amazon-linux"
]
}It’s quite a lot to take in, so lets look at what it is going to do…
The first part sets up the plugins and requirements by downloading the latest shim from gitlab to enable the packer tool to interface correctly to AWS.
The second part – starting with source “amazon-ebs” “amazon-linux” defines the source image that you are going to use to build your image. Here, the packer tool will download and create an ami image called “packer-example-2” with the defined instance types, here it is a t3.micro etc. It does this from the source image that is defined in the source_ami_filter.
You have to pick an AMI image that is available in your region (remember the default region you picked when setting up the CLI?) An AMI image for Amazon-Linux2 in say, us-west-1 is NOT the same as one in eu-north-1 despite being exactly the same, they have a different AMI number. As newer versions are released and updated, the AMI number changes as well. You need to got to the EC2 instance page, and list the AMI’s available from the AMI section on the left toolbar. Find an Amazon-Linux2 image, and copy the entire AMI number (including the AMI- bit at the beginning)
You now need the owner ID, which you can find by listing all the available AMI’s – stick in the AMI id and it will filter down to give you just one line. You will also want the Owner ID – this is the AWS account that owns that image, in this case oddly enough it is Amazon.
If you are runing on en-north-1 (Stockholm) and it’s not too far into the future, the supplied values I gave in the file will work. Otherwise plug your details in and save the file.
Now we can run Packer against the file and it will do stuff….
Firstly, you must run packer init packer.pkr.hcl to download the relevant plugins that are needed. You only have to do this once per folder that you use.
C:\Users\cs\git\packer>packer init packer.pkr.hcl
Installed plugin github.com/hashicorp/amazon v1.3.1 in "C:/Users/cs/AppData/Roaming/packer.d/plugins/github.com/hashicorp/amazon/packer-plugin-amazon_v1.3.1_x5.0_windows_amd64.exe"Once init is done, then you can run against the main file with packer build packer.pkr.hcl
C:\Users\cs\git\packer>packer build packer.pkr.hcl
SSH-keys-loaded.amazon-ebs.amazon-linux: output will be in this color.
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Prevalidating any provided VPC information
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Prevalidating AMI Name: packer-example
SSH-keys-loaded.amazon-ebs.amazon-linux: Found Image ID: ami-0d74f1e79c38f2933
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating temporary keypair: packer_6609bc15-904d-2bb3-1580-c0f46b08246d
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating temporary security group for this instance: packer_6609bc16-6bef-fc19-83e6-eef8816d6119
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Launching a source AWS instance...
SSH-keys-loaded.amazon-ebs.amazon-linux: Instance ID: i-0f1346d8d379362b8
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for instance (i-0f1346d8d379362b8) to become ready...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Using SSH communicator to connect: 13.51.162.255
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for SSH to become available...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Connected to SSH!
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Stopping the source instance...
SSH-keys-loaded.amazon-ebs.amazon-linux: Stopping instance
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for the instance to stop...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating AMI packer-example from instance i-0f1346d8d379362b8
SSH-keys-loaded.amazon-ebs.amazon-linux: AMI: ami-09cdb78d2a743a9a4
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for AMI to become ready...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Skipping Enable AMI deprecation...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Terminating the source AWS instance...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Cleaning up any extra volumes...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: No volumes to clean up, skipping
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Deleting temporary security group...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Deleting temporary keypair...
Build 'SSH-keys-loaded.amazon-ebs.amazon-linux' finished after 2 minutes 57 seconds.
==> Wait completed after 2 minutes 57 seconds
==> Builds finished. The artifacts of successful builds are:
--> SSH-keys-loaded.amazon-ebs.amazon-linux: AMIs were created:
eu-north-1: ami-09cdb78d2a743a9a4If you watch the EC2 instances window in the console, you will see that an instance starts on it’s own, and it is not started by the AWS “launch-wizard-2”. Instead it is launched by the Packer program with packer named security groups as you can see here. It is using your access keys on the AWS cli that you set up earlier to do this.
What Packer is doing, is launching an AMI that it controls, and making a copy of it. It takes that launched image, clones it, generates a new AMI and then uploads that to your AWS account. This is then your image and no one else can use it, unless you decide to publish it to the world. However it is possible for you to use it to launch EC2 instances. The very last thing that Packer does is show you the AMI ID… eu-north-1: ami-09cdb78d2a743a9a4
You can see if you go to the AWS console, into EC2 and then the AMI list, that Packer has cloned an AMI instance, and added it into here – your own list of AMI’s…. you can see here that it is owned by me, not by Amazon, or Canonical or anyone else.
On the face of it thats not that useful. If you were to start up an EC2 instance from this, then it would not do anything different from the Amazon Linux master AMI it was copied from. What we need to do is change it.
Take a look at the following build block. We have added a provisioner section to it – go ahead and paste that part into your working file now.
build {
name = "SSH-keys-loaded"
sources = [
"source.amazon-ebs.amazon-linux"
]
provisioner "shell" {
inline = [
"cd /home/ec2-user",
"echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGva38OqNvc1FZgsv3OP2dk9HjdM+482iMwXR1lqenx7 cs' >> .ssh/authorized_keys",
]
}This instructs Packer to build onto the source that it already has… the source is the Amazon provided AMI and this builder changes that AMI and makes it more useful to us… It does so by means of a provisioner. You can see that the first line references the source so we make sure we are building on the correct thing. It then starts the provisioner – in this case a shell that runs inline with the AMI. In essence, it starts the Amazon linux image up, runs some shell commands, shuts down when they are completed and then packages up the AMI.
The commands are quite simple – it goes the the ec2-users home directory, echos a new authorized SSH key into the .ssh folder and appends it and then finishes. We – of course – can choose any SSH key we want there and clearly you can pick the one that would be used by your local client to connect to the EC2 instances.
To get the key for the shell provisioner, you need to go to the linux box that will connect to this EC2 instance….
┌─[✓]─[cs@rootca]─[~]
└─cat .ssh/id_ed25519.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGva38OqNvc1FZgsv3OP2dk9HjdM+482iMwXR1lqenx7 cs@rootca
┌─[✓]─[cs@rootca]─[~]
└─
All you are doing is listing the contents of the public key used for SSH connections. The three line block line starting with “ssh-ed” is in fact just one line. That is pasted into the packer file above, as can be seen.
So now we have the key in place, let us or rather let Packer build it…
C:\Users\cs\git\packer>packer build packer.pkr.hcl
SSH-keys-loaded.amazon-ebs.amazon-linux: output will be in this color.
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Prevalidating any provided VPC information
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Prevalidating AMI Name: packer-example
SSH-keys-loaded.amazon-ebs.amazon-linux: Found Image ID: ami-0d74f1e79c38f2933
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating temporary keypair: packer_6609bf5b-e4a3-ce11-c840-1824b58cc7bd
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating temporary security group for this instance: packer_6609bf5c-7dfe-655b-a95e-1e3d3c063d2b
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Authorizing access to port 22 from [0.0.0.0/0] in the temporary security groups...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Launching a source AWS instance...
SSH-keys-loaded.amazon-ebs.amazon-linux: Instance ID: i-022160480d5269485
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for instance (i-022160480d5269485) to become ready...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Using SSH communicator to connect: 13.51.163.233
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for SSH to become available...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Connected to SSH!
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Provisioning with shell script: C:\Users\cs\AppData\Local\Temp\packer-shell1025933176
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Stopping the source instance...
SSH-keys-loaded.amazon-ebs.amazon-linux: Stopping instance
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for the instance to stop...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Creating AMI packer-example from instance i-022160480d5269485
SSH-keys-loaded.amazon-ebs.amazon-linux: AMI: ami-0eb503b7ffff18ef2
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Waiting for AMI to become ready...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Skipping Enable AMI deprecation...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Terminating the source AWS instance...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Cleaning up any extra volumes...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: No volumes to clean up, skipping
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Deleting temporary security group...
==> SSH-keys-loaded.amazon-ebs.amazon-linux: Deleting temporary keypair...
Build 'SSH-keys-loaded.amazon-ebs.amazon-linux' finished after 2 minutes 58 seconds.
==> Wait completed after 2 minutes 58 seconds
==> Builds finished. The artifacts of successful builds are:
--> SSH-keys-loaded.amazon-ebs.amazon-linux: AMIs were created:
eu-north-1: ami-0eb503b7ffff18ef2
On a first glance this looks no different from the previous packer run, however, there is an extra line about halfway down… ==> SSH-keys-loaded.amazon-ebs.amazon-linux: Provisioning with shell script: C:\Users\cs\AppData\Local\Temp\packer-shell1025933176 It is clear that the shell provisioner has run, and done it’s magic.
If we now start up a new EC2 instance from our AMI, and – quite deliberatly – we do not give it any SSH keys, we would expect to find it quite difficult to log in – assuming all was normal… So let’s fire up a new EC2 instance from our newly created AMI – you can find it by selecting the “My AMIs” and “Owned by me” options
We will be sure to supply no key at all..
Now the image is running, we can find it’s IP address from the EC2 instances dashboard in the usual fashion. All the old instances you see here and what are left over from my creating AMI images with Packer and test building them…
The IP address is 13.60.27.58… so we go to the linux box that has the public/private key pair that we installed using packer and log on…
As you can see, the system logs in after checking the host key. No password needed, no AWS supplied key needs to be installed.
Cleanup
When you have finished, be sure to shut down the machines that are created otherwise they will eat into the free tier, and perhaps you will be charged for them.
Also – it does cost money to keep an AMI image lurking around in your private store. The actual AMI itself is free, but it is based on an EBS snapshot and that is covered to a point by the free tier – however I would still deRegister (not deactivate) the AMI to be sure there are not unexpected charges later on.
Recap
We took an existing Amazon-linx AMI and instructed Packer to do the following.
- Spin up a copy on our account, using our CLI credentials
- Modify that copy, by means of a builder which made use of a shell provider to add in the public key from our local linux box
- Packer made the modification, and then copied the changed machine making a new Amazon Machine Image which it placed in our private store and then gave us the new AMI ID
- We then started an EC2 image from that new AMI and were able to log in directly without using AWS supplied keys
On the face of it this is not that useful – we could just as well keep a key in AWS and use that instead. The principle however of creating significantly modified images with for example custom software loads, configurations, etc is clear to see. There is no need to build a webserver onto every EC2 instance that you stand up, if you can use Packer to install and configure, and then load the website onto a custom image for you.