Why You Should Secure Your Server (part 1)

If you run a personal server that is accessible on the Internet, I want to share reasons why it is important to keep it secure.

I ran an experiment, analyzed the server logs, and sharing the first part of the results.

You will have a better understanding of what might be happening to your server on the Internet and how to analyze SSH log files.

This article will take you about seven minutes to read.

source and more information

Introduction

I love to do test driven development. Too many times in my career, I got “bit” by a small error in my code that just could be avoided and the stress of hunting down that bug under pressure wasn’t worth it.

I always test first.

When I was configuring my server, an AWS EC2 instance, I noticed I could add two factor authentication to it. Just like in gmail, you can scan a QR code into the Authenticator app to generate codes for you. Cool, I’ve stepped up security in my server from using SSH keys to another level using a two factor authentication method.

But - did this security change make my life more secure? As far as I know, no one was accessing my server. Why? I never gave out details.

Or is the server discoverable by others??

How do I know I was the only person accessing my server? I never tested the security of my server. I had at the time was my server felt right, I could access it and my files were there.

This brought up a question: how do I test the security of my server?

First question: is anyone else accessing my server? How do I know?

Experiment

I ran an experiment: I had an AWS EC2 instance running Ubuntu 14.04 for about one month with port 22 open (the default for SSH). I only accessed this server for this experiment (i.e. I accessed it less than ten times during this period.)

The operating system is Ubuntu 14.04 as this was the long term supported version at the time of the experiment.

I also configured SSH to use SSH keys and two factor authentication to prevent access from anyone else.

I gathered the results from the SSH logs and did some analysis on them.

Analysis tools

The SSH log had over 50k entries. This was a bit more than what I was expecting. Instead of writing my own custom code and testing it out, I used tried and true UNIX tools such as:

• cat
• | (UNIX pipe)
• grep
• awk
• sed
• wc
• sort
• uniq
• geoiplookup

The only command that is not part of the standard UNIX system is geoiplookup, which is a tool to look up geography information about an IP address and used in the next article on this topic.

All commands have documentation in UNIX under man:

man <command>

The log files in this article are available here so you can check my work or if you derive more insights, let me know!

Question 1: Purity

Was there anyone else accessing the server other than me?

If the answer is: “yes”, then it would compromise the experiment.

The auth log recorded seven times there was a successful SSH:

cat auth.log | grep sshd | grep 'session opened' | wc -l

The IP address that had those logins:

cat auth.log | grep -oE '.*Accepted publickey.*' | awk '{ print $11 }' | sort | uniq

• 127.213.25.139
• 208.167.254.47
• 54.196.23.221
• 54.88.109.46

Because these IPs are different, how do I know they were me?

Verification

If I look at the full log entries, of which there are only seven:

Jan 11 01:56:37 ip-172-31-1-163 sshd[1303]: Accepted publickey for ubuntu from 127.213.25.139 port 65087 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Jan 11 12:07:15 ip-172-31-1-163 sshd[2434]: Accepted publickey for ubuntu from 208.167.254.47 port 49268 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Jan 13 02:31:48 ip-172-31-1-163 sshd[4944]: Accepted publickey for ubuntu from 54.196.23.221 port 46494 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Jan 16 16:19:25 ip-172-31-1-163 sshd[11978]: Accepted publickey for ubuntu from 54.196.23.221 port 46515 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Feb  1 13:23:24 ip-172-31-1-163 sshd[26207]: Accepted publickey for ubuntu from 127.213.25.139 port 57738 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Feb  1 20:24:21 ip-172-31-1-163 sshd[29424]: Accepted publickey for ubuntu from 127.213.25.139 port 61530 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
Feb  4 14:44:38 ip-172-31-1-163 sshd[5620]: Accepted publickey for ubuntu from 54.88.109.46 port 39387 ssh2: RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c
a

The most important part of the line is at the end, starting with RSA:

RSA 0a:78:92:3c:c8:27:13:d3:f7:ee:d5:ac:75:45:31:5c

This means all successful entries used all the same SSH key. Of which, I know I only generated once.

The most likely reason why there would be multiple IPs with the same SSH key: I accessed this server from different locations: home, the office, and phone, which has different IPs.

Whew - OK, at least this experiment is valid.

Question 2: Logins

What are all these other entries in the auth log file?? Seven logins does not generate 50k worth of entries. If they did, a real system with hundreds of users would run out of storage space just from the SSH logs! (Or every storage business’ dream!)

Login Attempts

Seven successful login attempts, which of those were unsuccessful login attempts were there? 50k?

Looking at the logs, every time there was an Invalid User entry:

Jan 16 10:01:06 ip-172-31-1-163 sshd[11555]: Invalid user jay from 113.5.255.22

means there was a login attempt. Number of attempts? Using:

cat auth.log | grep 'Invalid user' | wc -l

There was 11k login attempts over a month. This means there was about 20 attempts per hour or one every three minutes! In the time it takes you to read this article, there could be at least six attempts to login to your server.

Login Names

The logs also provide the names used in a login attempt, in this case: jay

Jan 16 10:01:06 ip-172-31-1-163 sshd[11555]: Invalid user jay from 113.5.255.22

Of those, the number of variations of login names? To find out, I used:

cat auth.log | grep -oE 'Invalid user.*' | awk '{ print $3 }' | sort | uniq | wc -l

Results around 1600 different login names used. To sort these out, I used:

cat auth.log | grep -oE 'Invalid user.*' | awk '{ print $3 }' | sort | uniq -c | sort

The list of the top 10:

Wow, who is this admin person? I don’t know anyone named: admin, do you?? Is this person a celebrity I haven’t heard about or something??? :-)

Question 3: Server Scans

One popular way to check if a server is an “easy target” is to scan what open ports it has. Port 22 would be hard (it’s dedicated to SSH) - port 23 would be easy, it’s dedicated to telnet. source

Looking at the logs, the value that shows up for each port scan is:

Jan 11 08:34:21 ip-172-31-1-163 sshd[2245]: Received disconnect from 221.194.44.195: 11: [preauth]

Using:

cat auth.log | grep -oE 'Received disconnect.*' | wc -l

There were over 14k port scans done on the EC2 server in a month. Every hour the server was active, it would receive 20 scans. In the time you finished reading this article, there could be six port scans on a server.

What were the number of people/unique IPs were port scanning? 14k? 1? We can find out using:

cat auth.log | grep -oE 'Received disconnect.*' | awk '{ print $4 }' | sort | uniq | wc -l

Results in 156 different IPs. Not 14k (thank god!) and not 1 (who are you?!)

Of the IPs doing port scans, what were the number of attempts each one did? I used:

cat auth.log | grep -oE 'Received disconnect.*' | awk '{ print $4 }' | sort | uniq -c | sort -n

It’s not that each attacker did 93 port scans each, it seems a small number of attackers performed an extraodinary number of port scans. The top two attackers accounted for about 33% of the total port scans done.

Conclusion

From just a cursory analysis of my SSH logs on a server with just SSH port open, I am not the only one trying to access my server.

The number of and different method of attempts lead me to believe these attempts were not an “accident”.

In the next part, I dive a bit deeper into the logs and use a tool to lookup where these attempts came from.