If you launch an EC2 instance, come on, weve all done it – launched the thing and not provided any keys for it and you cannot connect. I’ve heard this time and time again – why can’t we use a password?
Because passwords suck thats why. https://xkcd.com/936/
AWS uses SSH keys, and if you have never used these before then there is a lot more to them than the first glance you will have seen with EC2 instances. Let’s take a look at them in a lab environment….
How Do SSH Keys Work?
SSH can authenticate clients using several different means. The most basic and familiar perhaps is username and password authentication. Although passwords are sent to the host over the network in a secure manner, it is possible to attack them by various means. Brute force attacks, shoulder surfing, and even keylogging devices, malware or cameras can all reveal passwords. Also, humans are really bad at picking passwords, and they can often be guessed. Despite there being sticking plasters for this, SSH keys prove to be a reliable and much more secure alternative.
An SSH keypair is made of two linked keys. There is a public key, which can be sent to anyone without security being comprimised, and a private key that must remain, as they name says, utterly private. The usual names for these keys are id_rsa and id_rsa.pub
The private key should remain on the client machine that will use it, and should never be moved anywhere else. If it does, anyone with access will also be able to access remote hosts as if they were the owner. As a precaution, the privatekey can be encrypted on disk with a passphrase.
The public key can be shared freely without consequence. The public key is used to encrypt messages, and due to the design, you cannot decrypt them with the public key. You are only able to decrypt them by using the private key. Consequently, this can be used as a way of proving that you possess the private key, if someone else has the public key. They send you a message encrypted with the public key – you decrypt with the private key and send it back to them.
For use with SSH, your public key is uploaded to a remote host you wish to connect to. The key is appended to a file that contains all the public keys that are allowed to connect to the server, known as the authorized keys file, which usually is found at ~/.ssh/authorized_keys.
When an SSH session takes place, the client connects to the remote host and presents the fingerprint of the public key. If the remote host has that public key it challenges the client to prove it has the private key by encrypting a message. If the client can decrypt the challenge, and prove it has the private key, then it is allowed to connect, an appropriate shell session is spawned, and the client connects.
Homelab
This is easy to investigate in a homelab if you are running something like Proxmox, VMware Workstation, or similar….
Here we have a few servers in my home lab. Host1, Host2, and client, all are running Debian 11. If we connect to the client machine and then try to SSH to another – say host1 we get a response about the authenticity of the host not being established…..
This is the Trust on first use or TOFU problem. You may have no idea if the server you are connecting to is actually the one you expect and so the system asks you if you are really sure that you want to connect to it. Type yes, and it says something about adding a key, and then it lets you login…
If you subsequently try again then it connects straight away – without the message as can be seen here. I disconnect from the far end server and then reconnect and it goes in without the message.
Let us have a look at just what SSH is doing here….
On the client machine, there is a hidden folder in the users home directory called .ssh Inside this, there may well be a number of files, but the one we are interested in is called known_hosts
The known_hosts file contains a list of the host keys that it uses to identify hosts that it has sucessfully connected to. If you look in the /etc/ssh folder on a server you will see a number of files.
┌─[✓]─[root@host2]─[~]
└─cd /etc/ssh
┌─[✓]─[root@host2]─[/etc/ssh]
└─ls
moduli sshd_config.d ssh_host_ed25519_key.pub
ssh_config ssh_host_ecdsa_key ssh_host_rsa_key
ssh_config.d ssh_host_ecdsa_key.pub ssh_host_rsa_key.pub
sshd_config ssh_host_ed25519_keyThe ssh_host_ecdsa_key.pub is presented from the server to the client during the initial signon. Once successfully signed on you can see the key appear on the client known_hosts file. Looking at the example screens below you can clearly see the key appearing in the clients known_hosts file – a matching section has been highlighted.
The key comes as two parts – there is a public and private part. When log on the server sends the public part of it’s key and asks you to trust it. Assuming you do, then when you next connect you can use the public part of the key to send an encrypted challenge, that only the server with the private part can reply to. Hence trust is assured.
EC2 keys
When you log on to an AWS EC2 instance, you don’t use a username and password. Instead, you supply a keypair. This keypair acts as a logon credential, being something that you – and hopefully you alone possess. So how does it work.
Lets look at the lab environment again.
Host1 is set up to permit password based authentication – the default in SSH is to permit this and unless specifically set otherwise it will always allow a username and password to be supplied.
cat sshd_config
...
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
...If you set this to PasswordAuthentication no then you cannot login with a password – it will simply refuse and if you have no other way of getting in, then you are stuck! If you want to try this, open TWO sessions to the host and make the change and then reconnect one of them. The other session will remain open so you can revert the change. Remember that you need to restart the sshd service after making the change with a systemctl restart sshd command.
The result is clear… on host1 we turn off password authentication
┌─[✓]─[root@host1]─[/etc/ssh]
└─cat sshd_config | grep PasswordAuthentication
PasswordAuthentication no
Connections from the client machine are now refused.
┌─[✓]─[cs@client]─[~]
└─ssh cs@host1
cs@host1: Permission denied (publickey).
┌─[✗]─[cs@client]─[~]
└─In order to logon, we need some way other than a username and password now. I’ll show you know how you can do this with SSH keys.
To generate a public key pair we use ssh-keygen. This will create a public and a private keypair. If you allow it to use the defaults it will work, but we want to tweak them a little. By setting a couple of options, we generate RSA keys, and we have a keysize of 4096 bytes, or 4k.
This will give us a good old fashioned, but perfectly secure pair of keys that should work on just about anything. They will if created appear in the hidden .ssh directory in the home folder.
┌─[✗]─[cs@client]─[~]
└─ssh-keygen -t rsa -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/cs/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/cs/.ssh/id_rsa
Your public key has been saved in /home/cs/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:a7Wukl39ArAdZlwPAMexB1IB6mG5/zgulRcnp1YLqJQ cs@client.chris-street.test
The key's randomart image is:
+---[RSA 4096]----+
| +=B+ |
| o o.oo |
| =. o...o |
| oEoo O.+ . |
| .o.SB.@ . |
| ..+oB.o |
| +++.. . |
| +.o+ . . |
| ++oo . |
+----[SHA256]-----+
┌─[✓]─[cs@client]─[~]
└─ls .ssh
id_rsa id_rsa.pub known_hosts
┌─[✓]─[cs@client]─[~]
└─
We now have the means of identifying ourselves to another server by something other than a password. We can have the remote host that we are connecting to send us a challenge, which we can prove with the private half of the keypair. This is something that only we (in theory!) know, so the host knows it is us.
For this to work though – we have to have the public part of the key on the remote server. So that needs to be copied over there, like this. If you haven’t yet set the host back to accepting password logins, then you need to do this first.
Most SSH based systems provide the ssh-copy-id tool which will be used to copy the key over to the host. It does require that you have a valid username and password. However below I will show the process manually so you can more easily see what is going on, with an example of ssh-copy-id given at the end.
┌─[✓]─[cs@client]─[~]
└─scp .ssh/id_rsa.pub cs@host1:/home/cs
cs@host1's password:
id_rsa.pub 100% 753 1.4MB/s 00:00
┌─[✓]─[cs@client]─[~]
└─ssh cs@host1
cs@host1's password:
Linux host1.chris-street.test 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64
┌─[✓]─[cs@host1]─[~]
└─ls
id_rsa.pub
┌─[✓]─[cs@host1]─[~]
└─ls .ssh
host1_rsa_key host1_rsa_key-cert.pub host1_rsa_key.pub
┌─[✓]─[cs@host1]─[~]
└─rm -rf .ssh
┌─[✓]─[cs@host1]─[~]
└─ls
id_rsa.pub
┌─[✓]─[cs@host1]─[~]
└─ls .ssh
ls: cannot access '.ssh': No such file or directory
┌─[✗]─[cs@host1]─[~]
└─mkdir .ssh
┌─[✓]─[cs@host1]─[~]
└─ls -la
total 36
drwxr-xr-x 4 cs cs 4096 Jan 23 18:51 .
drwxr-xr-x 3 root root 4096 Jan 15 19:26 ..
-rw------- 1 cs cs 202 Jan 23 02:48 .bash_history
-rw-r--r-- 1 cs cs 220 Jan 15 19:26 .bash_logout
-rw-r--r-- 1 cs cs 3930 Jan 16 03:21 .bashrc
-rw-r--r-- 1 cs cs 753 Jan 23 18:40 id_rsa.pub
drwxr-xr-x 3 cs cs 4096 Jan 16 03:21 .local
-rw-r--r-- 1 cs cs 807 Jan 15 19:26 .profile
drwxr-xr-x 2 cs cs 4096 Jan 23 18:51 .ssh
┌─[✓]─[cs@host1]─[~]
└─chmod 700 .ssh
┌─[✓]─[cs@host1]─[~]
└─cat id_rsa.pub >> .ssh/authorized_keys
┌─[✓]─[cs@host1]─[~]
└─ls -la .ssh
total 12
drwx------ 2 cs cs 4096 Jan 23 18:52 .
drwxr-xr-x 4 cs cs 4096 Jan 23 18:51 ..
-rw-r--r-- 1 cs cs 753 Jan 23 18:52 authorized_keys
┌─[✓]─[cs@host1]─[~]
└─chmod 600 .ssh/authorized_keys
┌─[✓]─[cs@host1]─[~]
└─ls -la .ssh
total 12
drwx------ 2 cs cs 4096 Jan 23 18:52 .
drwxr-xr-x 4 cs cs 4096 Jan 23 18:51 ..
-rw------- 1 cs cs 753 Jan 23 18:52 authorized_keys
┌─[✓]─[cs@host1]─[~]
└─The stages of the process are…
- SecureCoPy the public key from the client to the host using the SCP command.
- Logon to the host with username/password combo.
- Verify that the id_rsa.pub file has arrived
- Check and if necessary create the hidden ~/.ssl directory
- Make sure that the permissions for ~/.ssl are set to 700 or it will not work
- The contents of the public key file are then listed and appended with the double redirects to the end of the authorized_keys file. If you use a single redirect, then it will overwrite the existing list of keys! This is why this method is not recommended as it is too easy to break something catastrophically.
- Once the key has been appended, ensure that the authorized_keys file has a permission of 600 – again if this is incorrect then it will not work.
Once the clients public key is on the host then it can use this to be sure that the client is authorised to connect – because it can verify that the public key is present in it’s authorized_key files. The host can then send an encrypted challenge to the client and get a response back – proving that the client does have the private half of the key.
If once set up correctly you try to connect, SSH will use your local id_rsa keypair automatically, and you will find that it will just connect without requesting any further information.
┌─[✓]─[cs@client]─[~]
└─ssh cs@host1
Linux host1.chris-street.test 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64
┌─[✓]─[cs@host1]─[~]
└─Well done. You now have ssh key based authentication to allow you to connect.
To set this up with no risk of breaking things ssh-copy-id should be used. It is a simple one line command and it will automatically pick the key for you, and send it to far end if you provide username and password.
┌─[✓]─[cs@client]─[~]
└─ssh-copy-id cs@host1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cs/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
cs@host1's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'cs@host1'"
and check to make sure that only the key(s) you wanted were added.
┌─[✓]─[cs@client]─[~]
└─ssh cs@host1
Linux host1.chris-street.test 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64
┌─[✓]─[cs@host1]─[~]
└─
It carries out the same manual process as above, but far more effectively.
This is the same process that AWS EC2 uses – we have simply replicated what the process is. In the next post I will look at the common problems that are found with this, and the varying ways that these can be overcome.