Red Green Repeat Adventures of a Spec Driven Junkie

Configuring Mail (and Slack) for SSH Login Notifications

This is the final part of four in a series of articles about Linux system security.

  1. Text Processing Tools in UNIX
  2. Parsing SSH Logs
  3. Configuring fail2ban
  4. Configuring mail (and Slack) for SSH Login Notifications

This part will focus on configuring notifications on system login.

Why Login Notifications?

After configuring fail2ban with SSH key-based or two factor authentication login, a system is pretty secure. Random offenders are banned after too many failed login attempts, accounts require a complex key or token.

The paranoid part of me just wants a little more book keeping so I keep track of when I logged in, especially on a dedicated system that I am the only one using.

For example, if an intruder does gain access to the system, there is no way of knowing until I notice something is funny, if I look for it, or the intruder does something stupid.

An easy way to keep track if an intruder has gained access: have a notification sent on every successful login, even for myself.

If I get a login notification and I did not login, I immediately know something is wrong and take immediate action.

Approach for Mail Notifications

The way I will develop notifications on login: on successful PAM authorization, send an email using a script.

To accomplish this, these are the steps I took on a new Linux system:

  • Install mailutils
  • Test mail command to send an email from command prompt
  • Create script for mail command
  • Modify PAM to execute script
  • Add in info from PAM variables to the email

Install mailutils

mailutils includes a simple mail program, called mail, that is very easy to send an email from the command line. To install mailutils on a Debian based system:

$ sudo apt-get install mailutils

Other UNIX mail programs can work, like sendmail, the important part is to be able to email from a prompt or script.

Test: mail

Now, to send message from command prompt using mail, this is the command:

$ echo 'message body' | mail -s 'subject line' [email protected]

Check your inbox and the spam folder as well. mail will send using its built-in SMTP server that is unverified, so email clients will consider mail emails to be spam.

Mail Script

Now, putting this command into a script so any program can use it, send_email.sh:

#!/bin/bash

echo 'email from script' | mail -s 'script subject line' [email protected]

Make sure the script has executable permissions:

$ chmod +x send_email.sh

To run the script:

$ ./send_email.sh

The result is the same as the command prompt execution, emails from a command prompt!

Send email on login

With a script to send email, let’s configure PAM, the Pluggable Authentication Module, to call the script on any SSH activity:

In PAM’s SSH configuration file: /etc/pam.d/sshd, add this line at the bottom of the file:

# on any activity, execute send_email.sh
session   optional      pam_exec.so /absolute/path/to/send_email.sh

The optional keyword is very important here as PAM will still allow login, even if the email script fails.

The other value allowed is: required, which will always execute the script on login. If the script fails when the setting is required, login will fail, even with proper login authentication.

Logout of the system and login again to verify email notifications are being sent.

Enhanced PAM script

Now basic emails are being sent upon any PAM SSH activity. Let’s make the email script to only email upon successful login and use some of the environment variables set by PAM when it is executing the script.

Modify the send_email.sh script to become:

#!/bin/sh
if [ "$PAM_TYPE" = "open_session" ]
then
  {
	echo "User:        $PAM_USER"
	echo "Remote Host: $PAM_RHOST"
	echo "Service:     $PAM_SERVICE"
	echo "TTY:         $PAM_TTY"
	echo "Date:        `date`"
	echo "Server:      `uname -a`"
  } | mail -s "$PAM_SERVICE login on `hostname -s` for account $PAM_USER" [email protected]
fi
exit 0

I originally found this script from here

This script will email only upon successful logins and include some nice details:

Subject:

sshd login on ip-127-31-49-100 for account ubuntu

Message Body:

User:        ubuntu
Remote Host: mf52436d1.tmodns.net
Service:     sshd
TTY:         ssh
Date:        Tue Mar  7 07:49:15 EST 2017
Server:      Linux ip-127-31-49-100 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 GNU/Linux

Optional: Slack

Having login notifications sent to a messaging service such as Slack is also possible. Slack supports messages to be posted from the command prompt using curl with a properly configured Incoming Webhook.

Approach Slack Notifications

The approach to set up login notifications on Slack:

  • configure Slack to accept external messages
  • send Slack message from command prompt
  • put Slack message command into a script
  • have PAM also execute the script on SSH activity
  • update script to include PAM variables

Configure Slack for External Messages

Note: For this, one needs to be able to add an integration to a Slack.

Go to this page to add an Incoming Webhook to your Slack and send a message privately to yourself.

Send Slack Message from Command Prompt

There will be an example curl command in the Example section of the page, which will look like:

curl -X POST --data-urlencode 'payload={"channel": "#general", "username": "webhookbot", "text": "This is posted to #general and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}' http://slack.generated.webhook/services/xx...x/yy...y/zz....z

Run this to ensure messages are going from the command prompt to the #general room in your slack (magical, isn’t it?!)

Pare down the script to only send message to yourself:

curl -X POST --data-urlencode 'payload={ "username": "webhookbot", "text": "This is posted to #general and comes from a bot named webhookbot.", "icon_emoji": ":ghost:"}' http://slack.generated.webhook/services/xx...x/yy...y/zz....z

Message Slack from Script

Let’s put this into a script, message_slack.sh:

#!/bin/bash

curl -X POST --data-urlencode 'payload={ "username": "webhookbot", "text": "This message comes from a script: message_slack.sh.", "icon_emoji": ":ghost:"}' http://slack.generated.webhook/services/xx...x/yy...y/zz....z

Running at the command prompt:

$ ./message_slack.sh

Will put a message into your private slack. Great, isn’t it?

Configure PAM to Send Slack Notifications

Add this script to also be executable by PAM:

# on any activity, execute send_email.sh
session   optional      pam_exec.so /absolute/path/to/message_slack.sh

Logout and login again to check.

Now your private slack channel will receive a message on any PAM SSH activity. Yay!

Adjust Slack Script with PAM Variables

Let’s fine tune the script and make it better suited for SSH logins and include some PAM variables:

#!/bin/bash

function post_to_slack () {
	# format message as a code block ```${msg}```
	SLACK_MESSAGE="\`\`\`$1\`\`\`"
	SLACK_URL=http://slack.generated.webhook/services/xx...x/yy...y/zz....z

	case "$2" in
		INFO)
			SLACK_ICON=':slack:'
			;;
		WARNING)
			SLACK_ICON=':warning:'
			;;
		ERROR)
			SLACK_ICON=':bangbang:'
			;;
		*)
			SLACK_ICON=':slack:'
			;;
	esac

	curl -X POST --data "payload={\"text\": \"${SLACK_ICON} ${SLACK_MESSAGE}\", \"username\": \"login-bot\"}" ${SLACK_URL}
}

USER="User:        $PAM_USER"
REMOTE="Remote host: $PAM_RHOST"
SERVICE="Service:     $PAM_SERVICE"
TTY="TTY:         $PAM_TTY"
DATE="Date:        `date`"
SERVER="Server:      `uname -a`"
LOGINMESSAGE="$PAM_SERVICE login on `hostname -s` for account $PAM_USER"

if [ "$PAM_TYPE" = "open_session" ]
then
	post_to_slack "${LOGINMESSAGE}\n${USER}\n${REMOTE}\n${SERVICE}\n${TTY}\n${DATE}\n${SERVER}" "INFO"
fi
exit 0

_Note: post_to_slack function originally from pragbits.com.

Now, wherever there is Slack, there can be notifications:

slack ssh login notification

Pretty handy! :-)

Conclusion

Adding a simple email script upon successful login is a great way to keep track of server logins (and notice when oneself did not login!)

Using mailutils and sending an email on successful PAM authorization is easy to configure, hooking up scripts together.

With the ability to post login messages to slack ensures any system administrator is always on top of who is logging in or out of their system at any time.