Red Green Repeat Adventures of a Spec Driven Junkie

Configuring fail2ban

This is part three of four in a series of articles about security.

  1. Text Processing Tools in UNIX
  2. Parsing SSH Logs
  3. Configuring fail2ban
  4. Configuring mail (and Slack) for SSH Notifications

In the first two parts of this series, I wrote aboit how much port scanning and login attempts happen on a server, I am not comfortable leaving a server openly accessible anymore.

Restricting Access Methods

If I do not want a server openly accessible, how van I access it?? How can one restrict access to a server that is one the Internet? Whitelist IP addresses and power on & off the server as needed.

  • Whitelist IP Addresses

    This option only allows specified IP addresses to access a server. To configure which addresses are allowed, there are security settings to specify which addresses to which port. AWS allows this to be configured through its Security Group settings. Another is on the server itself, through the firewall settings.

    AWS EC2 documentation

  • Power on & off the server

    A server off makes the system inaccessible and only powering on when you need it, just a computer that you have physical access. Even though the server is hosted in the cloud, powering on & off as needed is possible with AWS instances.

Securing Login

If whitelisting IP addresses and powering the server on & off is too much work, the next best thing is to secure the login. Some ways to secure:

  • SSH public key authentication

    This will protect from brute force username & password attempts as the ‘password’ is a VERY hard to guess. This will require private keys to be on every SSH client that accesses the server.

    details

  • SSH Two-factor authentication

    Configure SSH to use two factor authentication, so a password and a randomly generated token is required for login. This is great working on systems which you want to access temporarily (i.e. a friend’s computer) but do not want to copy the SSH key onto.

    my article on setting up TFA in SSH

These two methods step up system security by essentially defeating brute force login attempts. System is still accessible and make login very difficult without private keys or authentication tokens.

fail2ban

Even with secure SSH login, I want to step up my security, especially after exploring the SSH log of my server. There is definitely activity on a server that is accessible, even if the login method is secured.

fail2ban is a daemon (a UNIX system service) that automatically parses logs for malicious behavior and bans offenders through reconfiguration of the system firewall, iptables on Linux, that bans the offender from connecting in general. After a specified period, the ban is lifted.

fail2ban is a great way to add another level of security that deters repeat offenders while permitting authorized access.

Test Script

The following is a quick one-line script to just continuously attempt logging into a specified server every second:

$ while true; do ssh ubnt@feta.ddns.net 2>&1; sleep 1; done

This will be the ‘test’ for system reaction to multiple login attempts with and without fail2ban configured.

For purposes of this article, a server has been setup with address: feta.ddns.net that is accessible by ssh.

Test system without fail2ban:

On a fresh Linux install, running the test script against the server produces:

$ while true; do ssh ubnt@feta.ddns.net 2>&1; sleep 1; done
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
...

Note: feta.ddns.net will have fail2ban configured.

This is the result of a system that does not have fail2ban running.

The Permission denied (publickey). message will repeat continuously. The default configuration for SSH will not do very much against brute force attacks, even though it is noted in its log.

Install fail2ban

This is how to install fail2ban on a Debian based distribution such as Ubuntu:

$ sudo apt-get install fail2ban

For other UNIX systems, the fail2ban.org download page has more information.

Start fail2ban

At this point, fail2ban is installed and the service is already started. Running:

$ sudo service fail2ban status

Will display its current status.

$ sudo service fail2ban (start|stop|restart)

Will start, stop, or restart it. Required after any configuration changes to fail2ban.

Test with fail2ban results

Test multiple login attempts on the same server with the script and the results are:

$ while true; do ssh ubnt@feta.ddns.net 2>&1; sleep 1; done
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
Permission denied (publickey).
ssh: connect to host feta.ddns.net port 22: Connection refused
ssh: connect to host feta.ddns.net port 22: Connection refused
ssh: connect to host feta.ddns.net port 22: Connection refused
...

After exactly six login attempts, the Permission denied message changed to: ssh: connect to host feta.ddns.net port 22: Connection refused.

This is fail2ban in action. The current client cannot even get an SSH prompt because its IP address has been blocked completely.

Working with iptables

Some tips on working with the iptables firewall in Linux.

List Banned Offenders

To see the current list of SSH bans by fail2ban:

$ sudo iptables -L --line-numbers

Sample output

$ sudo iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    fail2ban-ssh  tcp  --  anywhere             anywhere             multiport dports ssh

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Chain fail2ban-ssh (1 references)
num  target     prot opt source               destination
1    REJECT     all  --  linux.example.com     anywhere             reject-with icmp-port-unreachable
2    RETURN     all  --  anywhere             anywhere

Remove a Banned Offender

To remove a ban set by fail2ban-ssh the command is:

$ sudo iptables -D fail2ban-ssh <chain num>

so, to remove the ban on: linux.example.com from the previous output:

$ sudo iptables -D fail2ban-ssh 1

Additional fail2ban Configurations

At this point, fail2ban is running and already protecting the system against more than six consecutive failed SSH login attempts within ten minutes.

Additional configuration to fail2ban can improve peace of mind and deter repeat offenders.

Additional Configuration File

fail2ban is configured through a file /etc/fail2ban/jail.conf for all of its global settings. Changing configuration in the jail.conf file work, but a jail.local file is in the same directory provides specific overrides.

I like to configure fail2ban with the jail.local, which provides a kind of backup for the global one. If something doesn’t work in the jail.local I configured, I can easily reset using the original.

To make a local config file, Copy the jail.conf to jail.local:

$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

A copy of my jail.local with the following changes can be found here

Note: although # is the comment delimiter on the jail.conf file, never put a comment on the same line as a setting. (i.e. bantime = 3600 # 60 * 60) This messes up fail2ban parsing.

Always Allow an IP address: ignoreip

Any IP addresses set with ignoreip are always allowed to access, especially after too many failed login attempts.

I put my most commonly used IP addresses but do not change frequently, like my home IP address.

ignoreip = 127.0.0.1/8
# ignoreip = 127.0.0.1/8 <common IP addresses used>

Note: space is the separator between addresses.

Ban Offenders Longer: bantime

bantime is the duration to ban offenders before allowing access again. This value is in seconds and the default is 600, which is equivalent to 10 minutes.

From my experience, 10 minutes (600) was too short, even one hour (3600) was not long enough to deter repeat offenders.

I recommend 24 hours (86400) as a good value for bantime. Make sure ignoreip is configured to an IP that is accessible to you within the bantime, just in case. ;-D

bantime = 600
# => bantime = 86400

Email fail2ban Activity

fail2ban can also send emails when a ban occurs and when service has been restarted. To have fail2ban email activity, install sendmail and configure fail2ban to mail.

Install sendmail

$ sudo apt-get install sendmail

No need to configure sendmail further. fail2ban can use sendmail with its default options.

Note: email clients may flag them as spam since the emails are coming directly from the server, not through a known SMTP host.

Email offenders: destemail

As a way to gauge effectiveness, I like having emails of every time fail2ban bans someone. So, configuring an email address:

destemail = root@localhost
# destemail = you@email.domain

Note: + markers in the address can be used, so you+label@email.domain can be used as is.

Email logs: action

If destemail is configured to send email to a real address, having additional information in the email is very nice.

To have fail2ban email when a ban occurs, action needs to be set to a value of either %(action_mw)s or %(action_mwl)s. Both emails will include WHOIS data. The latter option will also include relevant log entries.

action = %(action_)s
# => action = %(action_mw)s

This is what a sample email report will look like:

Hi,

The IP 35.164.29.53 has just been banned by Fail2Ban after
7 attempts against ssh.


Here are more information about 35.164.29.53:


#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#


#
# The following results may also be obtained via:
# https://whois.arin.net/rest/nets;q=35.164.29.53?showDetails=true&showARIN=false&showNonArinTopLevelNet=false&ext=netref2
#


# start

NetRange:       35.152.0.0 - 35.183.255.255
CIDR:           35.152.0.0/13, 35.176.0.0/13, 35.160.0.0/12
NetName:        AT-88-Z
NetHandle:      NET-35-152-0-0-1
Parent:         NET35 (NET-35-0-0-0-0)
NetType:        Direct Allocation
OriginAS:
Organization:   Amazon Technologies Inc. (AT-88-Z)
RegDate:        2016-08-09
Updated:        2016-08-09
Ref:            https://whois.arin.net/rest/net/NET-35-152-0-0-1



OrgName:        Amazon Technologies Inc.
OrgId:          AT-88-Z
Address:        410 Terry Ave N.
City:           Seattle
StateProv:      WA
PostalCode:     98109
Country:        US
RegDate:        2011-12-08
Updated:        2017-01-28
Comment:        All abuse reports MUST include:
Comment:        * src IP
Comment:        * dest IP (your IP)
Comment:        * dest port
Comment:        * Accurate date/timestamp and timezone of activity
Comment:        * Intensity/frequency (short log extracts)
Comment:        * Your contact details (phone and email) Without these we will be unable to identify the correct owner of the IP address at that point in time.
Ref:            https://whois.arin.net/rest/org/AT-88-Z


OrgTechHandle: ANO24-ARIN
OrgTechName:   Amazon EC2 Network Operations
OrgTechPhone:  +1-206-266-4064
OrgTechEmail:  amzn-noc-contact@amazon.com
OrgTechRef:    https://whois.arin.net/rest/poc/ANO24-ARIN

OrgAbuseHandle: AEA8-ARIN
OrgAbuseName:   Amazon EC2 Abuse
OrgAbusePhone:  +1-206-266-4064
OrgAbuseEmail:  abuse@amazonaws.com
OrgAbuseRef:    https://whois.arin.net/rest/poc/AEA8-ARIN

OrgNOCHandle: AANO1-ARIN
OrgNOCName:   Amazon AWS Network Operations
OrgNOCPhone:  +1-206-266-4064
OrgNOCEmail:  amzn-noc-contact@amazon.com
OrgNOCRef:    https://whois.arin.net/rest/poc/AANO1-ARIN

# end


# start

NetRange:       35.160.0.0 - 35.167.255.255
CIDR:           35.160.0.0/13
NetName:        AMAZO-ZPDX9
NetHandle:      NET-35-160-0-0-1
Parent:         AT-88-Z (NET-35-152-0-0-1)
NetType:        Reallocated
OriginAS:       AS16509
Organization:   Amazon.com, Inc. (AMAZO-47)
RegDate:        2016-08-29
Updated:        2016-08-29
Ref:            https://whois.arin.net/rest/net/NET-35-160-0-0-1


OrgName:        Amazon.com, Inc.
OrgId:          AMAZO-47
Address:        EC2, EC2 1200 12th Ave South
City:           Seattle
StateProv:      WA
PostalCode:     98144
Country:        US
RegDate:        2011-05-10
Updated:        2017-01-28
Ref:            https://whois.arin.net/rest/org/AMAZO-47


OrgAbuseHandle: AEA8-ARIN
OrgAbuseName:   Amazon EC2 Abuse
OrgAbusePhone:  +1-206-266-4064
OrgAbuseEmail:  abuse@amazonaws.com
OrgAbuseRef:    https://whois.arin.net/rest/poc/AEA8-ARIN

OrgNOCHandle: AANO1-ARIN
OrgNOCName:   Amazon AWS Network Operations
OrgNOCPhone:  +1-206-266-4064
OrgNOCEmail:  amzn-noc-contact@amazon.com
OrgNOCRef:    https://whois.arin.net/rest/poc/AANO1-ARIN

OrgTechHandle: ANO24-ARIN
OrgTechName:   Amazon EC2 Network Operations
OrgTechPhone:  +1-206-266-4064
OrgTechEmail:  amzn-noc-contact@amazon.com
OrgTechRef:    https://whois.arin.net/rest/poc/ANO24-ARIN

# end



#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
# If you see inaccuracies in the results, please report at
# https://www.arin.net/public/whoisinaccuracy/index.xhtml
#

Regards,

Fail2Ban

Conclusion

How to install fail2ban, test whether the service is running or not, work with Linux’s iptables firewall to see and remove bans, provide additional options for fail2ban for greater peace of mind.

Next time: setting up SSH to send an email on successful login.