Do you fail2ban unauthorized server access?

Photo by Clint Patterson on Unsplash

When most of us think of a hacker, we imagine a sinister figure from the underworld who lives in perpetual darkness, whose only comprehension of light is of the panel from his screen. The keyboard — an extension of his limbs — does his bidding with each and every rapid, purposeful keystroke. Nothing stands in the way of his unparalleled wit, as he lays waste to the layers of security meant to keep him out. He grins and lets out a sigh of satisfaction, tapping the Enter key one final time, then leans back in his chair for a well-earned respite. He has gained unauthorized entry into a server, and with it, the troves of valuable data that it holds.

Hollywood has sensationalized our notion of the hacker, portraying their actions as highly manual yet effective, and directed towards entities that are of value. “ I’m not holding any information that is of value, no one would want to hack my systems… right?” Wrong! One thing we can be certain of is that the process of uncovering vulnerabilities to exploit requires tedious trial and error, far from the glorious imagery depicted by movies.

Most attacks today are executed by bots, methodically scouring IP addresses in the web, making attempts to identify the ports and systems that are in use, and establishing the attack surface. According to OWASP, the attack surface is:

1. the sum of all paths for data/commands into and out of the application
2. the code that protects these paths (including resource connection and authentication, authorization, activity logging, data validation and encoding)
3. all valuable data used in the application, including secrets and keys, intellectual property, critical business data, personal data and PII
4. the code that protects these data (including encryption and checksums, access auditing, and data integrity and operational security controls).

Each vector in the attack surface represents a potential doorway for a hacker to enter. All that a hacker needs to do, is persist in his search for a door that hasn’t been locked properly.

What is fail2ban?

In a nutshell, fail2ban prevents brute force attacks on your server. It can be configured to identify suspicious connections, and to jail (fail2ban lingo for ban) the originating IP address when transgressions exceed a defined limit. What this does is make it significantly harder for a hacker to establish the attack surface and consequently, perform targeted attacks.

I have configured my SSH server to jail IP addresses for 1 week if there are 3 failed or invalid attempts within 72 hours. To give you a sense of how prevalent these attacks are, take a look at my current SSH jail status:

SSH jails for 3 failed connections within 3 days

Every single day, I get at least 15 unique IP addresses prodding at my server. To get some sense about where these attacks are coming from, we can run the following code to read the fail2ban logs and map each offending IP addresses to a country:

Generate distribution of unauthorized SSH attempts

Now, are you convinced that fail2ban is worth your while?

Installing fail2ban

Note: Installing fail2ban is easy. Unfortunately it is equally easy to lock yourself out of your server while you are in the midst of configuring it properly. I strongly recommended that you proceed with the following steps when you have physical access to the server. If that is not possible, at least make sure that you have enough time to set it up properly with your trusted IP addresses whitelisted.

When you’re comfortable to begin, here’s how to install fail2ban:

Installation instructions for fail2ban

Understanding the configuration files

fail2ban installs configuration files to /etc/fail2ban. Let’s take a look at the important components of this folder:

Important components of the /etc/fail2ban configuration folder

• fail2ban.conf and fail2ban.local: These files determine the general settings of fail2ban — where log files are kept, the level of information to log. The convention is to leave fail2ban.conf as it is, but to copy it to fail2ban.local and modify the settings there.
• jail.conf and jail.local: These files contain the configurations for each jail — which logs to read; the number of failed attempts before an IP is jailed etc. Similarly, we typically make a copy of jail.conf to jail.local and make modifications there.
• filter.d: This folder contains the regex patterns that we would like each service to register as a failed attempt.
• action.d: This folder contains the list of actions that we can perform when an IP address gets jailed by a service. The most common one that you will use is iptables-allports.conf, as it allows us to block an IP address across all ports.
• jail.d: For services that do not have jail configurations setup in jail.local, you can define a new file in this folder like I did for Gitea.
• paths-common.conf: Contains environment variables used by the config files of various services, such as paths to system logs.

Whitelisting trusted IP addresses

Locking yourself out of the server is possible while you are tinkering with the regex patterns to use in filter.d. Unfortunately for myself, I’ve been there and done that… twice. To prevent this from happening, you can whitelist the trusted IP addresses which you connect from in /etc/fail2ban/jail.local. This would allow your IP address to violate any of the filter.d regex patterns without getting banned.

Input whitelisted IP addresses

IP ranges can be conveniently whitelisted using CIDR (Classless Inter-Domain Routing). You may use this tool to help convert between an IP range and CIDR. By default, the loopback IPv4 block 127.0.0.1/8 and IPv6 ::1 addresses are ignored. In addition to these, I have added 192.168.1.0/24 to whitelist all devices in my LAN. Do make sure that additions are space-delimited.

Enabling SSHD jail

We need to add in a block of code into /etc/fail2ban/jail.local to enable the SSHD jail with the desired configurations:

Configuring the SSHD jail

• enabled: true / false. Enough said.
• mode: normal, ddos, extra, aggressive. Take a look at /etc/fail2ban/filter.d/sshd.conf to see the additional regex matchers that get added as we opt for a stricter setting (cmnfailre, mdre-ddos, mdre-aggressive).
• port: keep it as ssh if you are using the default port 22 for your ssh server. Otherwise, change it accordingly.
• logpath: specifies log path for SSH authentication attempts. This variable is defined in paths-common.conf and points to /var/log/auth.log by default.
• backend: specifies how log files will be monitored; auto, pyinotify, gamin, polling…
• maxretry: failed attempts before a jail is activated.
• bantime: how long to keep an IP address in a jail.
• findtime: how far back to search for failed attempts.

Personally, i feel that even the aggressive mode of filters are too lenient on these bots. By occasionally browsing through the SSHD attempts in /var/log/auth.log, I have identified several more logs that i wanted to mark as a failed attempt. The corresponding regex patterns for the logs can be appended to /etc/fail2ban/filter.d/sshd.conf:

Append additional SSHD regex patterns to filter.d file

To update fail2ban with the new configurations, run:

$ sudo fail2ban-client reload

Next, leave your server running for a few hours, then come back to check the status of your SSHD jail!

$ sudo fail2ban-client status sshd

Testing regex patterns

Before adding new regex patterns, I would recommend temporarily increasing maxretry to prevent accidental lockouts. We can then make use of fail2ban-regex to validate if a regex pattern matches any of the log entries that we are trying to detect:

Testing patterns with fail2ban-regex

It takes time to build trust with a new tool, so if you are feeling a little skeptical about the output of fail2ban-regex, that’s completely understandable. To allay your worries, you can do a very rough test by grep’ing the output of the log file. In my case, the following returns the same IP addresses as fail2ban-regex:

Validating the output of fail2ban-regex against a simple grep

Once you’re happy with the results, remember to reload your configurations with sudo fail2ban-client reload. Once the new patterns have been applied, it is always prudent to perform a valid authentication, and check that it does not get categorized as a failed attempt via fail2ban-regex.

Creating jails for custom services

By default, fail2ban includes a handful of jail configurations in jail.local. But what if you wanted to create a jail that doesn’t ship with the defaults? There are only 3 things that you need to identify or take note of before creating a new jail:

1. Where are the logs stored? Depending on the service, this may be what you’d spend the most time figuring out. For example, NextCloud does not output logs until you install the plugin called “Auditing / Logging”.
2. Does the user have access to read the logs? In the case of my Gitea Docker container, the logs were output to a volume mounted to container. Fail2ban was unable to read the logs until chmod was run on it.
3. What regex patterns do you need in /etc/fail2ban/filter.d? What jail configurations do you want to setup in /etc/fail2ban/jail.d? Here’s an example that I use for Gitea:

filter.d and jail.d configurations for Gitea

Conclusion

As a whole, fail2ban provides a fairly straightforward way to protect your computer against brute force attacks, whether by bots or an actual human. Just by applying the configurations that it ships out of the box, you gain a significant security boost to your servers.

With a little more effort, configurations can be tailored to suit any service that you wish to protect. Starting today, don’t fail to ban (or should I say “do fail2ban”) unauthorized access to your server!