I was faced with an interesting one a couple of days back that needed a little time to find a solution. We’ve all probably been there, developers leaving cruft behind, hopefully in QA that is no longer needed.
We’d had a check show up some security groups that were basically wide open and permitting access to 0.0.0.0/0 -not the ideal solution. We needed to remove, or close them down, and the trick was to see if they were actually used.
Naturally none of them were convenient and defined in the Terraform back end, these had been added through the console.
It should be clear that if a SG isn’t actually attached to a network interface then it’s probably not doing very much and can be removed. With the CLI you can easily query network interfaces to see what groups are associated with them…
aws ec2 --profile qa describe-network-interfacesBecasue I have a number of accounts to look at, my credentials file has a whole series of different sections for each of the accounts.. I can therefore pick which one commands will run against with a –profile switch…
.aws/credentials
[qa]
aws_access_key_id = KEYIDWOULDBEHERE
aws_secret_access_key = secrestsfilewouldbeherebutimnotthatstupid
[dev]
aws_access_key_id = KEYIDWOULDBEHERE
aws_secret_access_key = secrestsfilewouldbeherebutimnotthatstupid
[staging_limefieldhouse]
aws_access_key_id = KEYIDWOULDBEHERE
aws_secret_access_key = secrestsfilewouldbeherebutimnotthatstupid
[prod_telehouse1]
aws_access_key_id = KEYIDWOULDBEHERE
aws_secret_access_key = secrestsfilewouldbeherebutimnotthatstupid
[prod_telehouse2]
aws_access_key_id = KEYIDWOULDBEHERE
aws_secret_access_key = secrestsfilewouldbeherebutimnotthatstupid
.aws/config
[profile qa]
region = eu-west-2
output = json
[profile dev]
region = eu-west-2
output = json
..etcYou will note that there is not a default section – so if I miss a profile out then it will error. This stops me inadvertently running the command against the default account, which might be the wrong one and could cause serious issues…
The output from the describe-networks shows the network connection in full. It’s fairly easy therefore to see if there is an attached security group…
┌─[✗]─[cs@box3]─[~]
└─aws ec2 --profile qa describe-network-interfaces
{
"NetworkInterfaces": [
{
"Attachment": {
"AttachmentId": "ela-attach-123456",
"DeleteOnTermination": false,
"DeviceIndex": 1,
"InstanceOwnerId": "amazon-aws",
"Status": "attached"
},
"AvailabilityZone": "eu-west-2a",
"Description": "VPC Endpoint Interface vpce-123412341234",
"Groups": [
{
"GroupName": "SG-endpoints-commonqa",
"GroupId": "sg-2134afe234234"
}
],
"InterfaceType": "vpc_endpoint",
"Ipv6Addresses": [],With a little poking its then easy to take an existing SG id and use that to query the network interfaces list to see if your SG is attached to any network interface.
aws ec2 --profile qa describe-network-interfaces --filters Name=group-id,Values='sg-011101234edd342e'Lets look at the command. The first part lists all the network interfaces in the account described in the profile. The second part filters the output.
AWS cli has two ways of searching for things – the server side –filters and the client side –query. Confusingly, client side querying isn’t really querying as such – it’s more selecting a set of fields to return. It is usually easier to filter server side where you can search for specific values.
Looking at the raw JSON output from the first command, you can see the values we are interested in here..
"Groups": [
{
"GroupName": "SG-endpoints-commonqa",
"GroupId": "sg-2134afe234234"Aggravatingly, the –filters option does not use these – if you were to filter the key “GroupId” it will fail. The filter values are instead found in the AWS cli reference documentation found here…
Now we can see that we can filter on the group-id (which is case sensitive – don’t call it Group-Id) and we also specify a Values field, which can contain more than one value to filter on. Here though we only look for one value.
┌─[✓]─[cs@box3]─[~]
└─aws ec2 --profile smi-qa describe-network-interfaces --filters Name=group-id,Values='sg-0510c5afb3d8XXX2e'
{
"NetworkInterfaces": []
}
The query returns an empty value – this security group is not bound to any network interface and can probably be removed as being unused.
This works fine of course, assuming that you know the ID for the group in question. If however you have no idea of which groups to even start looking at, then it becomes a much more problematic situation as even a small account usually has a large number of security groups to deal with.
We can use a related aws cli command called describe-security-groups to obtain all of the security group ID’s in an account. Let us have a look at this…
aws --profile=qa ec2 describe-security-groupsThis lists all of the groups information – which isn’t quite what we want. By using a client side query, we can now narrow the field down. We are going to select all of the contents of the top level key SecurityGroups by setting that as a wildcard, and then tell it to extract what is in each GroupId subkey.
─[✓]─[cs@box3]─[~]
└─aws --profile=qa ec2 describe-security-groups --no-paginate --query 'SecurityGroups[*].[GroupId]'
[
[
"sg-1234df4325a213244"
],
[
"sg-123234def1a213244"
],
[
"sg-1a3324fd25a213244"
],
This is the information that we need, but we cannot easily use this or parse it for another command. However, if we choose text output instead of JSON….
┌─[✓]─[cs@box3]─[~]
└─aws --profile=qa ec2 describe-security-groups --no-paginate --query 'SecurityGroups[*].[GroupId]' --output text
sg-1234df4325a213244
sg-123234def1a213244
sg-1a3324fd25a213244
Now we have a nice clean list of ID’s with nothing else on them. This is perfect for parsing with a tool called xargs. This will work on linux and Mac, but you will need something like cygwin or gnu bash for this to work on Windows.
If you pass the previous command into a pipe and send it the AWS command tools, the entire block of ID’s will arrive as one big blob. This is not what we want. xargs will split up the incoming blob, usually when it finds a whitespace character and then use just the split parts to call whatever command you want. Let’s have a look at how we are going to do this…
aws --profile=qa ec2 describe-security-groups --no-paginate --query 'SecurityGroups[*].[GroupId]' --output text | xargs -I % aws --profile=smi-qa ec2 describe-network-interfaces --filters Name=group-id,Values=%The first part before the pipe is just the code we saw that gives us a clean list of ID’s. That output is piped to the input of xargs.
The -I flag tells xargs to split on linefeed, rather than looking for whitespace. Lets have an example.
the ls -1 command will show you a directory listing on a single column, like this..
┌─[✓]─[cs@box3]─[/]
└─ls -1
Applications
Library
System
Users
Volumes
bin
cores
dev
etc
home
opt
private
sbin
tmp
usr
var
If we feed this output and pipe it to xargs, we can tell it by means of the -I flag to split on each line, and store the value in the ‘replace string’ symbol which is defined immediately after the -I flag. We can then use that ‘replace string’ value in a command. Xargs will take the first line of the directory listing, store it its value, and then call a command where we can use this value.
A simple example, using echo… we get the listing, process it and call echo to prefix each line before we read it back out again.
┌─[✓]─[cs@box3]─[/]
└─ls -1 | xargs -I % echo 'Directory name of %'
Directory name of Applications
Directory name of Library
Directory name of System
Directory name of Users
Directory name of Volumes
Directory name of bin
Directory name of cores
Directory name of dev
Directory name of etc
Directory name of home
Directory name of opt
Directory name of private
Directory name of sbin
Directory name of tmp
Directory name of usr
Directory name of varWe can now build a command to read all the security group id’s, split them into a series of individual ID’s and iterate over the list to see if that ID is in fact attached to a network interface….
aws --profile=qa ec2 describe-security-groups --no-paginate --query 'SecurityGroups[*].[GroupId]' --output text | xargs -I % aws --profile=qa ec2 describe-network-interfaces --filters Name=group-id,Values=% > sg.txt Reading through the command, the first part before the pipe is our old aws command that generates a list of ID’s one per line. This is then fed to xargs, which is told to split on each new line, and place the first value in the ‘%’ string. It then calls the second aws command to query if there is a network interface which has that security group ID attached. Once the command finishes, xargs then pops off the next ID, and runs the command again, and repeats until the list is finished. The entire output is then redirected to a disk file for easier processing.
Unfortunately the output is not actually of any use…. Examination of the file shows that the output is dumped but because there is no record returned we cannot find out which security groups result in an empty response. The file section is shown below… (you will excuse my editing of internal naming structures for security reasons..)
{
"NetworkInterfaces": [
{
"Attachment": {
"AttachmentId": "ela-attach-XXXXXXXXXXX",
"DeleteOnTermination": false,
"DeviceIndex": 1,
"InstanceOwnerId": "amazon-aws",
"Status": "attached"
},
"AvailabilityZone": "eu-west-2a",
"Description": "AWS Lambda VPC ",
"Groups": [
{
"GroupName": "A-QA-group-name",
"GroupId": "sg-02df223453edf2"
}
],
.
.
.
.
.
.
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-03ed2323edfa34",
"TagSet": [],
"VpcId": "vpc-234edf324edfe2342"
}
]
}
{
"NetworkInterfaces": []
}
{
"NetworkInterfaces": [
{
"Attachment": {
"AttachmentId": "ela-attach-0df324233453458c",
"DeleteOnTermination": false,
"DeviceIndex": 1,
"InstanceOwnerId": "amazon-aws",
"Status": "attached"
You can see that in the first case, the information returned contains the GroupId of the security group, the second one simply returns a completely empty null value. "NetworkInterfaces": [] It clearly shows that a group exists with no attachments to a network interface – but which group ID was it?
The easiest solution is to modify the command called by xargs to echo the contents of the ‘%’ to the output before calling the aws describe-network-interfaces command. This is easily done by wrapping the two commands as a string, and getting xargs to invoke a shell to process it.
aws --profile=qa ec2 describe-security-groups --no-paginate --query 'SecurityGroups[*].[GroupId]' --output text | xargs -I % sh -c 'echo %; aws --profile=qa ec2 describe-network-interfaces --filters Name=group-id,Values=%' > sg.txt The command that xargs runs is now just sh for the comamnd shell, passing it a command string with the -c flag. It echos the contents of ‘%’ to stout, calls the aws describe-network-interfaces and then takes the stout and redirects to a text file. The results are below.
sg-02df223453edf2
{
"NetworkInterfaces": [
{
"Attachment": {
"AttachmentId": "ela-attach-XXXXXXXXXXX",
"DeleteOnTermination": false,
"DeviceIndex": 1,
"InstanceOwnerId": "amazon-aws",
"Status": "attached"
},
"AvailabilityZone": "eu-west-2a",
"Description": "AWS Lambda VPC ",
"Groups": [
{
"GroupName": "A-QA-group-name",
"GroupId": "sg-02df223453edf2"
}
],
.
.
.
.
.
.
"SourceDestCheck": true,
"Status": "in-use",
"SubnetId": "subnet-03ed2323edfa34",
"TagSet": [],
"VpcId": "vpc-234edf324edfe2342"
}
]
}
sg-ImatestSG34
{
"NetworkInterfaces": []
}
sg-23ed455634dea3452
{
"NetworkInterfaces": [
{
"Attachment": {
"AttachmentId": "ela-attach-0df324233453458c",
"DeleteOnTermination": false,
"DeviceIndex": 1,
"InstanceOwnerId": "amazon-aws",
"Status": "attached"Yes, one of the offending groups was called ImatestSG34 – but there were about ten others with “normal” names out of about 800 in total. The output file provides a useful record and it’s easy to search it with a text editor for the "NetworkInterfaces": [] lines.
Unlike say Powershell, the AWS cli doesn’t have great support for pipelining. I hope though that this shows you with a little ingenuity you can build up powerful single line commands to solve problems in the CLI