Parsing SSH logs - Intermediate

This is part two of four in a series of articles about security.

1. Text Processing Tools in UNIX
2. Parsing SSH Logs
   • Basics
   • Intermediate
3. Configuring fail2ban
4. Configuring mail (and Slack) for SSH Notifications

This time, I will explore how many times SSH daemon’s own defenses stopped login attempts and use geoiplookup to lookup physical location data on IP addresses found in the log.

SSH daemon’s Defense

SSH itself is not completely defenseless against multiple login attempts in a short period of time. The SSH daemon will start to reject login attempts if the same IP address has multiple consecutive failures.

Total Rejections

From the log, the total number of times SSH disconnected a client because of too many failed attempts:

$ cat megaauth.log | grep -oE 'maximum authentication attempts.*' | wc -l`

SSH log shows there were 895 instances of too many failed attempts. This might not seem like a lot, but these are only the number of times the SSH daemon saw there was more than six failed attempts in a row.

Total Rejections for ‘root’

Number of Maximum Authentication attempts using ‘root’ as the username:

cat megaauth.log | grep -oE 'maximum authentication attempts.*for root' | wc -l

SSH rejected ‘root’ login 614 times in 24 days. This is a very good reason to disable root login on any openly accessible server. ‘root’ is the default administrative username for any UNIX system.

Total Rejections for other names

Number of Maximum Authentication attempts using name other than ‘root’:

cat megaauth.log | grep -oE 'maximum authentication attempts.*for invalid.*' | wc -l

278 login attempts were made using a username other than ‘root’. The most frequently used non-‘root’ username:

cat megaauth.log | grep -oE 'maximum authentication attempts.*for invalid.*' | awk '{ print $8 }' | sort | uniq -c | sort -n

There is some overlap with the most frequently used login name, but it is not as frequent, due to SSH only counting these after the maximum number of attempts have been made.

Using geoiplookup to map Location

Let’s start with the most frequently seen IP address in the logs:

cat megaauth.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | sort | uniq -c | sort -n | tail

Countries where these IP address are located:

cat megaauth.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | awk '{ system("geoiplookup " $1) }' | awk '{ $1="";$2="";$3=""; print $0 }' | sort | uniq -c | sort -n | tail

For the next section, to improve processing, creating an intermediate file, country_ip.log that maps IP address with its associated country.

cat auth.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | awk '{ print $0 system("geoiplookup " $1) }' | tr '\n' '|' | xargs -n 2 -d '|' | sed -n -e 's/0$//p' | sed -n -e 's/GeoIP Country Edition: //p' | awk '{ printf("%s %s\n", $(NF), $0 ) } END { print }' | awk '{ $(NF)=""; print $0 }' | head -n -1 > country_ip.log

note: this is pretty messy, so if there’s a better way to get this done, let me know!

How many Different Countries?

cat country_ip.log | awk '{ $1=""; print $0 }' | sort | uniq | wc -l

There were a total of 67 difference countries that showed up in the SSH log. There are 193 UN countries in the world. So, almost 33% of known countries in the world accessed this server in a period of 24 days.

Let’s take a look at how many times each country accessed the server, how many different IPs from the country, average, minimum, and maximum number of access.

How items were calculated:

• number of occurences: cat country_ip.log | awk '{ $1=""; print $0 }' | sort | uniq -c | sort -n | tail
• unique IPs: cat country_ip.log | grep 'PL' | sort | uniq | wc -l
• average : num occureneces / uniq IPs
• min: cat country_ip.log | grep '<country code>' | sort | uniq -c | sort -n | head -n 1
• max: cat country_ip.log | grep '<country code>' | sort | uniq -c | sort -n | tail -n 1

Poland really stood out, not only with the number of occurences, but basically a single individual showed up in the SSH log more than 17,000 times!

Port Scans per Country

The number of port scans performed by country can be obtained using this command:

cat megaauth.log | grep -oE 'Received disconnect.*' | awk '{ print $4 }' | sed -n -e 's/://p' |  awk '{ print $0 system("geoiplookup " $1) }' | tr '\n' '|' | xargs -n 2 -d '|' | sed -n -e 's/0$//p' | sed -n -e 's/GeoIP Country Edition: //p' | awk '{ printf("%s %s\n", $(NF), $0 ) } END { print }' | awk '{ $(NF)=""; print $0 }' | head -n -1 | awk '{ $1=""; print $0 }' | sort | uniq -c | sort -n | tail

Interesting, China & Poland based IP addresses seemed to have done the most port scans.

Attempted logins per Country

cat megaauth.log | grep -oE 'Invalid user.*' | awk '{ print $5 }' | awk '{ print $0 system("geoiplookup " $1) }' | tr '\n' '|' | xargs -n 2 -d '|' | sed -n -e 's/0$//p' | sed -n -e 's/GeoIP Country Edition: //p' | awk '{ printf("%s %s\n", $(NF), $0 ) } END { print }' | awk '{ $(NF)=""; print $0 }' | head -n -1 | awk '{ $1=""; print $0 }' | sort | uniq -c | sort -n | tail

The number of attempted logins was not really dominated by any one country like port scans.

ubnt???

From part 1 of this, I encountered many login attempts with the username: ubnt. How many countries tried to login with ubnt username?

cat megaauth.log | grep -oE 'Invalid user ubnt.*' | awk '{ print $5 }' | awk '{ print $0 system("geoiplookup " $1) }' | tr '\n' '|' | xargs -n 2 -d '|' | sed -n -e 's/0$//p' | sed -n -e 's/GeoIP Country Edition: //p' | awk '{ printf("%s %s\n", $(NF), $0 ) } END { print }' | awk '{ $(NF)=""; print $0 }' | head -n -1 | awk '{ $1=""; print $0 }' | sort | uniq | wc -l

A total of 19 different countries used ubnt to login.

How about which countries attempted the most?

cat megaauth.log | grep -oE 'Invalid user ubnt.*' | awk '{ print $5 }' | awk '{ print $0 system("geoiplookup " $1) }' | tr '\n' '|' | xargs -n 2 -d '|' | sed -n -e 's/0$//p' | sed -n -e 's/GeoIP Country Edition: //p' | awk '{ printf("%s %s\n", $(NF), $0 ) } END { print }' | awk '{ $(NF)=""; print $0 }' | head -n -1 | awk '{ $1=""; print $0 }' | sort | uniq -c | sort -n | tail

Most are non-English speaking countries, but even Canada and the United States are on this list.

City level

The interesting thing about geoiplookup, is that there is an option of using the city database to get finer

• most seen IP in log in general (city)

cat auth.log | grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" | awk '{ system("geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat " $1) }' | awk '{ $1="";$2="";$3="";$4="";$5=""; print $0 }' | sort | uniq -c | sort -n | tail

Conclusion

This time, exploring the SSH log with geoiplookup mapped IP addresses to physical locations and how many times SSH daemon rejected login attempts.

If there is a server openly accessible on the Internet, find a way to lock it down! If it can be accessed, someone, somewhere will try!

Next time, I will talk about using Fail2Ban to reduce continued failed access.