While I was doing some regular maintaince on my minecraft server, I stumbled on a massive log of failed connections via ssh. I closed the port while I did some digging only to find that for the past month, my server was getting blasted ²⁴⁄₇ by a botnet. What follows in an analyis of the logs I collected and the lessons I learned about securing a homelab.

Attack Post Mortem

Thankfully, due to my already security focused outlook, all my homelabs vms where in a secure state. None of the login attempts came even close to breaching my security, but the sophistication and scale of the attack was certainly alarming.

IP Address Breakdown

One of the most scary parts of this attack was the size of the botnet attacking my lab. With an average of ~1.4 login attempts per IP, this attack avoided detection by fail2ban. Additionally each login attempt was spaced out by at least 10 minutes meaning that it was also able to avoid standard brute force detection.

I ran a number of these IP’s through a botnet registry and found none had yet be found. This is likely because these are freshly compromized machines and/or the attacker has been very carefully not to use the machines for any suspect behaviour. At the bottom of the article I have listed a collection of the uniq login attempts for anyone who also happens to get fuzzed by this botnet.

Username Breakdown

In total 1966 uniq names where used to fuzz ssh. This was the least sophisticated part of the attack and the list of usernames seems to be directly ripped from a common username list. A more complex attack would have noticed that my IP was a residential connection and would have excluded things like ec2, kubernetes, and some other common cloud default usernames.

To the attackers credit, the usernames did include the most common names like Tom, John, Andrew, ect.

Lessons Learned

Take Your Homelab Security Seriously

With the rise of automated attacks and botnets, most attackers could care less if you are school, government, or a kid hosting your personal project on a raspberry pi. All they see is an IP and a potentially venerable system. It can be very tempting to dismiss security as something only compaines and powerusers need to think about but this isn’t the case.

Obscurity != Security

Another often tempting axiom to belive is that obscurity is your security. This could not be further from the truth. When my lab was attacked, my ssh port was running on random port number. These days it is not enough to just put your server on a random port and hope for the best. Tools like zmap are able to scan the entire internet for open ports in less than an hour. With tools like these available for any user with even the most basic technical skills, all users need to take their security seriously.

Check Your Logs

One of the most important things you can do to keep your homelab secure is to frequently check your logs. This can done in a varity of ways and depends quite a bit on how you have your lab setup. Below I have included some simple scripts that can be useful for auditing SSHD logs. As with all things, you will need to take the time to come up with a solution that covers applications and vulnerabilities that exist in your stack.

# Scan your ssh logs for unique IP's
journalctl -u sshd | grep -E -o "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort | uniq

# Scan your ssh logs for failed login attempts
journalctl -u sshd | grep -i 'invalid\|error\|Unable to negotiate'

# Scan for users and IP's
journalctl -u sshd | grep 'Invalid user' | awk -F '[ ]' '{ print $1 "-" $2 "-" $3 " USER:" $8 " IP:" $10}' | column -t

Harden Your SSH Server Configs

This is by far the single most important config that you will secure. If an attacker gains a remote shell in any of your machines, you entire network is compromised. You mush harden your ssh configs even if you don’t have any of these ports open to the public. Below I’ve broken down my core rules for configs and an example config I use.

1. NO ROOT LOGINS!!!! - If you do this you are just asking trouble. Regardless of your application always disable root login.
2. Turn Password Authentication Off - This is the core of all a secure enviroment. If you are using an RSA key, make sure it is at least 4096 bits.
3. Disable both Host Based Authentication and Challenge Response Authentication - This adds a extra layer of security to your system an prevents more sophisticated attackers from exploiting weakeness many distros default configs.

Below is an example config I built for when I deploy new VM’s.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

### START HARDENED CONFIG ###
# version 1.2

# Allowed Configs
PubkeyAuthentication yes
AuthorizedKeysFile      .ssh/authorized_keys
UsePAM yes
Banner /etc/banner

# Authentication Hardening:
PermitRootLogin no
PasswordAuthentication no
PermitEmptyPasswords no
HostbasedAuthentication no
ChallengeResponseAuthentication no

# Protocol Hardening
StrictModes yes
IgnoreRhosts yes
GatewayPorts no
Protocol 2
Compression delayed

# Ubuntu and Debian
#Subsystem       sftp    /usr/lib/openssh/sftp-server

# Arch Linux and Alpine Linux
#Subsystem     sftp    /usr/lib/ssh/sftp-sever

# CentOS 8
#Subsystem     sftp    /usr/libexec/openssh/sftp-server

# FreeBSD 12
#Subsystem     sftp    /usr/libexec/sftp-server

#SyslogFacility AUTHPRIV
#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key


### END HARDENED CONFIG ###

Login Attempt Log