And you can always use net.ipv4.ip_local_reserved_ports to reserve the port for privileged use. (I have never had to do this in over a decade of using a high port for ssh access.)
What does it mean for an IP to fail “authentication” in that context? As I understand, they only check for failed connection attempts, not failed authentications?
The first thing it was integrated into was sshd, there's a small patch in sshd that sends a notification to the blacklist daemon when a connection has failed authentication.
The difference is blacklistd is just a api and a database for telling the firewall to block ip's. The decision to inform blacklistd is on the application side, and not a scanner that checks the log's produced by that application.
And yes, DoS is a very good example, if your application writes so much log's (during a attack) and fail2ban has to analyze them during that, then machine could very well DoS itself, the blacklistd approach is much simpler and straightforward BUT (and that's a big one) the applications has to support it.
I agree with this viewpoint. IMO, a lot of these types of threads fill up fast with gimmicky things that sound good at first but do nothing to increase security against realistic threats, plus are at least inconvenient to set up and maintain, if not prone to misfiring and blocking you.
The real advice for setting up new servers IMO is, SSH pubkey auth is perfectly adequate security. Set it up normally on the default port logging in as root, disable password login, and don't worry about the noisy failed bot logins. This is a lot more convenient for automated setup and maintenance and has no practical loss of security.
Yes. I don't think there's anything to be gained from monitoring login logs. The only real use is after you recognize some sort of breach to try and help determine exactly what did or didn't happen.
Fail2ban could be potentially useful in an era of passwords, especially when systems allowed anyone to login remotely, and you only had the one server because servers were very expensive.
However, the moment the industry went to SSH Keys en masse, and got away from passwords, fail2ban stop serving any real purpose other than to make people feel like they had done something to improve security. Which it didn’t really, especially if you enforced key usage only for logins.
Literally, the only argument you can make is that fail2ban might reduce the number of log entries.
> the moment the industry went to SSH Keys en masse [...] fail2ban stop serving any real purpose
That assumes people only use it for sshd, no? Which isn't the case [anecdatum] for me, at least - I use it for ssh, http (several servers), imap, smtp, mqtt, etc. Make a request for phpMyAdmin pages on my server? 10 day timeout for you, sir. IMAPS connection with incorrect username? 10 days in the bin. etc.
Yeah, it reduces log entries but it also reduces unwanted resource consumption and keeps my servers working instead of chugging under non-useful load.
Fail2ban can parse other logs and taken other actions. I have it on my NVR VM, reading the logs from the old Unifi NVR software. Three failed attempts and Fail2ban runs a script that does an API call to my OpnSense firewall to ban that IP from every port. Forever. Haproxy also monitors the HTTP response codes and drops the TCP connection, allowing the block to happen once the bad IP tries to reconnect.
While I agree with most of the points, I think the most important one is: your time is precious. Spending hours setting up things is are really high price to pay for something that might never be of use.
Setting up fail2ban is essentially just "apt install fail2ban", it already comes with sensible defaults. That's on Debian, I think it is the same on most distros.
For me, it is less of a hassle than looking at logs flooded with failed connection attempts.
Genuine question: Why not just tell the daemon to stop logging failed attempts? I personally have a hard time reconciling local username+password authentication and the need to audit failed attempts in the same use case but that may just be because I haven't run across the right use case.
The simplest reason is that installing fail2ban is just as easy.
Also, I sometimes want to know about failed attempts, mostly for legitimate attempts that failed and shouldn't have. As in: "why couldn't bob connect the the server?", and failed attempts would just show that he used the wrong password, or at least, that it is a password related problem.
If someone would fail to properly code something you can find your basic auth is slow to a crawl or even not responding but you would have no idea what is going on.
Setup in a containerized environment and with structured logging is a nightmare. Last time I checked structured logging was entirely unsupported. Fail2ban has fallen out of times.
I'm surprised no one has ported it to some container-friendly runtime.
In any case your threat model for container workflow is already 100x more complicated than some one-server blog with an open SSH port, I would give up on fail2ban immediately
Just two days ago I found out the sensible defaults doesn't include automagically enabling even the basic jails. The machine run for a year (or more) with enabled fail2ban which... did nothing. Not a Debian machine and it's entirely on me, but fail2ban setup is not "just apt install".
You need to be at least somewhat familiar with it's lingo and the minimals on working with it.
> at worst, fail2ban: causes you massive inconvinience or total lockout [...] it'll probably block you out.
I agree that fail2ban is useless once you've disabled password login, which you should have done.
But is anyone really getting locked out of their own systems by fail2ban unless they're doing something pretty weird? I thought it gave you about 5 password attempts, and then blocked you for 10 minutes - doesn't seem that draconian to me.
In one of my previous jobs, it blocked me out after 3 attempts indefinitely - that is, until I got a hold of the sysadmin and asked him to unblock me. It was a PITA, as you can imagine. And I don't understand why any sysadmin would want to put themselves in a position to get bothered regularly by people who tend to mistype their passwords like me.
The default configuration of fail2ban on Debian considers every offered key as a login attempt, so you can be immediately banned for merely connecting with too many keys in your ~/.ssh.
Never been locked out when I used to use it. But it got replaced with a default firewall rule to only allow SSH from my address in the end. Didn’t need it at all then!
I've stopped trying to convince people against using things like Fail2ban. As long as it's not a company or government where security is important, let them waste their time.
In general I agree with the recommendations here. Though it may be slightly over prescriptive in what security approaches universally make sense (e.g. privileged ports) it carries a very valid conclusion about the applicability of Fail2Ban.
The only time I almost used Fail2Ban was with a personal "guaranteed to work from wherever and whatever I'm on" jumphost which ran SSH on 22 and 443. 443 because, like the post says, outbound filter compatibility (Fun note: to listen on multiple ports simultaneously you can literally just include multiple "Port xyz" lines in your sshd_config). Ultimately I decided for this use case it made more sense to just let bad attempts to fail silently by changing the logging config. If I was ever in a situation I felt I needed the failed attempts logged I'm not sure I could simultaneously justify username+password login on a public host.
In fail2ban's defense, it can be used for a lot more than just blocking IP addresses based on SSH failures.
At one point in the past I adapted an existing fail2ban setup to ban IP addresses that were running malicious vulnerability scans against a large legacy PHP webapp, using some dumb heuristics that worked and were easy to put in place.
Would it have stopped sufficiently motivated expert attackers? Of course not.
Did it reduce the load on the legacy PHP app and improve the signal to noise ratio in our webserver logs, while reducing the chance a brute-force attacker would find some hole lurking in a long-forgotten corner of the app's unfortunately-large attack surface?
Yup.
So, I'd argue it can be useful, even if naive SSH blocking may not offer much improvement.
I think the log SnR point is a great one. Debugging aside, if you're able to filter out the "obviously malicious" traffic, any more sophisticated attacks will be easier to spot (or at the very least, analyse after-the-fact).
> At least one RCE, admittedly with a non-default fail2ban config and a somewhat unlikely attack chain:
you had to manipulate answers from a whois server.
Sure, it increases attack surface (like any additional piece of code) but argueing to better let an IP hammer your mailserver with infinite stuffed credentials just to avoid possible bugs, is questionable at least.
fail2ban's CVE track record is quite good in comparision, if you take that as metric. It might be even more secure than using multiple different rate-limiting implementations that were written in C.
They work wonders against lots and lots of traffic from Chinese bots which are too stupid to see that password logins are disabled but there's a fucking lot of them, they just won't stop knocking, and it does take cycles to fend them off in any way. You stop listening on port 22, and 99% of that shit traffic just goes away.
I know, it's not by the book, and that well ackchyually I should yak yak this and yada yada that. But it's so fucking cheap and simple to do, and it goes brrr, so I just don't care.
> changing your SSH port to non-root port (>1024) means that if the SSH daemon dies, any userland process can take its' place. this is bad.
Man, if you have sshd that likes to die a lot, you've got bigger problems already. If you have userland processes that can take your sshd port over maliciously, you're already fucked and should really reprioritize your worries.
> changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working.
~/.ssh/config goes brrrrrr
> you will very likely waste hours of your life.
Only if I need to dictate the contents of that file to my iPhone, otherwise it's seconds.
> you will have to google "rsync ssh non standard port" every time you want to use rsync
rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr. Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin?
> you will have to remember scp flags. this is also bad. probably worse.
scp from OpenSSH honours OpenSSH configs, so it goes brrrr.
> you will have to tell the port to anyone else who tries legitimately accessing your system.
"Here's a fragment that you copy and paste into your ~/.ssh/config. Remember to do a chmod 0600 on it if the file is not there already."
My god, is it me getting old, but when I was young it was customary to know the basics of your tools.
I always change my ssh ports from the default, not for any security reasons, but because I'm frequently on stupid networks that block traffic on port 22. In my experience, non-standard ports are more likely to work, in general.
> userland processes that can take your sshd port over maliciously,
It doesn't need to happen maliciously. The result is the same: You might be locked out permanently.
> if you have sshd that likes to die a lot, you've got bigger problems already
It doesn't have to die, it can also just happen during startup.
Let's say you initially configure port 2222, restart sshd, everything works well, so you remove port 22, restart one more time, and stop thinking about it.
Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad.
> rsync uses ssh process under the hood, and that will pick up my shiny config and go brrrrr.
And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed).
> Man, what if you need, dare I say, oh gosh, ProxyCommand!? Won't your head explode? How many times will you have to google it? Are you a goldfish who went sysadmin?
SSH is used by all kinds of people, not all of them condescending sysadmins.
> Let's say you initially configure port 2222, restart sshd, everything works well, so you remove port 22, restart one more time, and stop thinking about it.
> Then, on your next reboot, it turns out that port 2222 is actually the preferred port of some other automatically starting service that just happened to not be running during your experiment, but which is now racing sshd for port 2222: That's bad.
Isn't it sysadmin 101 "How not to get locked out"? You should know already what else might be running on the box, or learn the hard way.
> And just as many applications including an SSH client won't pick up the config from .ssh/config (for example because they can't read that directory because they're sandboxed).
Put the relevant config where the sandbox can see it.
> SSH is used by all kinds of people, not all of them condescending sysadmins.
SSH should be used by people who at least know what a file is and how to protect their private key, I think that putting in a few lines of configuration (or having a provisioning tool do it for them) is a non-issue. Except if you are willing to die on that hill and think up artificial obstacles.
> Man, if you have sshd that likes to die a lot, you've got bigger problems already.
This one.
One amusing thing what these apologists completely ignore (because they don't know shit?): just fucking DNAT a high port to localhost:22 and drop externalip:22. Bam, works.
it's fine until you run into some stupidly written tool or abstraction layer that relies on ssh being port 22.
yes, it sucks to make efforts to accommodate your worst software, but sometimes it's easier to throw the cycles away than to dick around with a custom iptables entry to fix someone elses poor software design choices.
> it's fine until you run into some stupidly written tool or abstraction layer that relies on ssh being port 22.
Good part of being me is not having to deal with such tools. If I see something like this, I consider finding an alternative immediately because the hardcoded port will be the least of its problems. If it assumes and insists on a port, there's a good chance it doesn't even piggyback on an ssh command or libssh and reimplements lots of ssh itself. What ever could go wrong with this approach?
If it's a tool to which source is available, it can be patched up. It can be used on the right side of the LAN where you can use well-known ports all you want. Embrace and use VPNs, they are good! I'm somehow convinced that if the ground rules are laid out up front in no ambiguous terms, the workaround will be found on the side of the badly-written software.
close the port open if you dont want ssh to be used.
i think i read once the rsync.net dude uses port-knocking and maybe f2b? what was the other note from rachaelbythebay, mac os updates always revert changes sshd settings or something like that; maybe they stopped?
i do hate the idea of ip whitelisting; IP's should be more dynamic and less able to fingerprint an individual - but that is opening another can of wurms.
Generally never trust ANY ip; dont use whitelists, and the blacklists should never be "forever". if your app is secured by a keypair like openssh then thats the route to take.
honestly though fail2ban is great in that it does silence some noise in the logs - there will still be an unbareable amount - but at least some less; but still regular log maint is required.
Also the groups Ive seen never recommend f2b for use to fight against a DoS attack; not even for "cleaning up logs" - i dont see where that point was created other than from this page
I wouldn’t call it a whitelist necessarily but there’s no reason why ssh has to be open to the world if it is possible to use a network overlay or vpn.
If ssh is only allowed on the management wireguard network, not much can beat that other than a host compromise which, additional checks to verify that a host can be trusted at that moment could potentially help stop unauthorized access.
I actually want to do this (one day i'll get to nerd-out again) but then tie it to a totp setup so the knocking-algorithm changes ever so slightly; could even utilize different ports
I'm not fond of this writing style without concrete substance; it feels immature. Yes, the author may have tried to be funny, but I don't see a compelling reason to stop using fail2ban.
Let's be clear here - fail2ban will not save you if you have a poorly configured SSH server or widely open Nginx/Postfix services, but it will offload traffic blocking to iptables (or whatever firewall you are using). This way, the mentioned services will not have to waste resources just to cancel some connections.
Will it save it against large DDoS? Surely not, but it will stop mild DDoS without problems, which it is designed for. We can argue that iptables/nftables can do a fail2ban job with rate limiting and other techniques many people have never heard of, but if you are at this level, you already know your craft, and I'm pretty sure you considered, tried or used fail2ban before.
* changing your SSH port means that you'll have to do a bunch of fuckery to get basic functionality working. you will very likely waste hours of your life. you will have to google "rsync ssh non standard port" every time you want to use rsync. you will have to remember scp flags. this is also bad. probably worse.
---
I dont need to remember non standard ports if i just use my .ssh/config.
Host myhost.tld
Hostname 127.0.0.1
Port 333
User MyUser
IdentityFile ~/.ssh/myhost.tld
I was brand new to managing an Ubuntu Hetzner server and the moment I saw how many port 22 scans the server received i decided to try changing the port number, followed by key-only passwordless logins. My logs immediately shrank in size. I have never once had an issue having moved to non standard ports and, moreso, feel almost naked logging into port 22.
I know security through obscurity is not an answer, but judging by the reduction in port scanning i've seen after moving as many standard ports as possible to new addresses above 20000 I have to believe its a reasonable first step. How many script kiddies are scanning all 65500 ports for each IP address?
I feel similarly. Switching ports is no real defense, but it at least means you are eliminating the drive-by attacks who are only interested in the trivially exploited. Such a simple thing to do and sharply reduces the log volume.
The next trick I think of implementing is port knocking. Should drop log noise to zero unless someone starts targeting me specifically. In which case, my goose is already cooked.
If it's not some sort of proxy/firewall remapping the port, you probably shouldn't use a port above 1000 for some services.
Consider this: an attacker (somehow) managed to get user access to your server. They can now dos the service until it crashes and then start their own service listening on that same port, maybe impersonating your service. Maybe they can use that to grab sensitive information or do something else.
Indeed, although because I heavily utilized Docker I also ended up using UFW-Docker. It was fairly straightforward to incorporate into my startup scripts.
Sure, if you use only the OpenSSH client and other software respecting its command line option paradigms or configuration files. But there's tons of other things connecting to SSH.
And even only in the OpenSSH universe, I find it quite annoying having to remember whether the `-p` can go after the hostname or has to appear before, whether SCP uses `-p` or `-P`, whether `ssh-copy-id` supports one or the other etc.
There are some protocols I wouldn't necessarily run on their default port and publicly accessible, but SSH is really not one of them, also given that sshd has been specifically hardened for that adversarial use case.
Normally I might put a title like this down to opinion but here, the author suffers not knowing enough about the subject to justify having an opinion, let alone publish one.
Fail2ban is a stupidly easy way to block lazy hacking scripts. It's easy to extend to handle simple honeypot services. It's no replacement for real security but it makes focussing easier.
THIS. I use non standard port for SSH by default and my single SSH config file manages this for me with no extra effort. Seems like a common sense behavior to me.
I use fail2ban. It is a mild security control which I know can fail. I install it, use its defaults and get on with my life.
At times, for bastion hosts, I use crowdsec instead.
I could not capture why the author was so bitter. I guess I would be the same if I was supporting some product, and its users were getting problems because of fail2ban - and I had to support it for them.
The approach my team takes is we put Wireguard gateways running on OpenBSD in front of network services. We go a step further and very carefully select what chips are used for the network interfaces and also attach a hardware random number generator to add to entropy.
Internal users' machines have these tunnels transparently loaded and running, configured by our internal GitOps-driven config management (SaltStack).
FreePBX/Asterisk comes with a default fail2ban setup that will lock out your SIP connections (from phones) while you're just fiddling to get it working, adding to the frustration.
It's way simpler. There is a tiny core to it, and a couple of modules specific to Exim, Apache, SSH. These just run as daemons on the machine.
I tried using fail2ban at that time, but it wasn't at all what I expected. It was way overcomplicated for what it achieves, and poorly documented. Just getting it running at all was fiddly, never mind the custom behaviors I had in mind.
I should review the Apache module; some of it is quite out of date, due to the changing landscape. Also, that webserver it is used with is now behind a reverse proxy, which only receives connections for specific applications.
A particularly fun combination is Fail2Ban on a mail server and Apple's Mail desktop email client when you need to change your password. We've got to change passwords every 90 days at work, and probably 90% of the times I try to do this I end up getting banned by Fail2Ban.
The problem apparently is that Apple Mail is pretty aggressive at retrying if a login fails, and so if it tries to connect before you have finished changing the password in the client it can try enough times in quick succession to trigger Fail2Ban.
There are at least 3 places you can change your password(s) for Apple Mail.
• Invoke "Mail/Settings..." in Mail to open the settings dialog. Select the "Accounts" tab. Then for the account you want to change go to the "Server Settings" pane. That has password fields for your incoming mail server and your outgoing mail server.
• Invoke "?/System Settings...", then go to the "Internet Accounts" settings. (You can also get there from "Mail/Account..." in Mail). Select you mail account, click the "Detail..." button, and you'll get a dialog that has a password field.
Note it only has one password field. I'm not sure what you are supposed to do if you have different passwords for you incoming and outgoing mail servers.
• Open "Keychain Access", and search for your incoming mail server. Look for an entry whose kind is "application password" named com.apple.account.IMAP.password. (I assume that IMAP would be POP3 if you use a POP3 server). Open it, verify that the Account field is you@that_server. Then you can click "Show password" to reveal the current password. Change that to the new password and save.
Do the same for your outgoing mail server, looking for com.apple.account.SMTP.password.
My current procedure is to first disable my work mail account in "Mail/Settings...", then change the password on the server, then update the client side passwords via Keychain access, and then via "Mail/Settings...", then enable my work mail account. That seems to usually work, although once it didn't until I also changed it in "Internet Accounts" in the system settings. I also had a time where I just could not get the new password to take until I rebooted.
Apple Calendar is also a pain with password changes, but we don't have Fail2Ban on our calendar server (it is not on a public facing network so there would be no point) so it is not as annoying as Mail.
For Calendar what I've found to mostly work is:
1. In "Calendar/Settings..." in the settings for my work account in the Accounts tab turn off all delegates. Then disable my work account. The quit Calendar.
On MacOS prior to 13 find and kill the CalendarAgent and CalNCService processes. On 13 they went away.
2. Change the password on the server.
3. Change the password in "Keychain Access". I find the entry by searching for name@calendar_server. The right entry is the "application password" named "com.apple.account.CalDAV.password".
4. Then change the password in "Internet Accounts". As with Mail, Calendar provides convenient access to this via "Calendar/Accounts...".
5. Open Calendar, reenable the account, and turn back on delegates.
All of the above comes from trial and error and I have no idea how much of it is actually necessary. I haven't had the patience to do the time consuming experimentation it would take to figure out the actual simplest way to change these passwords reliably and without hitting the server with the old password during the changing.
> what i'm saying is, if you're fucked, you're fucked. fail2ban will not help
I see this attitude frequently in online discussions around {security thing}, but it really doesn't match my understanding of reality. It's not about making any one part of the system perfectly secure, but instead about having all of the pieces of it contribute to a whole. Many real-life stories of caught/thwarted attacks include the threat actor successfully breaching/moving through multiple layers of a system but ultimately getting stuck on something relatively simple. Defense in depth is not just a catchphrase.
In evaluating whether something like this is genuinely useful, it can make sense to look at how difficult it is to operate, and how much it might reduce the probability of a successful attack. In my experience, fail2ban has been extremely easy to install and operate, and I think it brings at least some meaningful impact (particularly if applied to other things than SSH). At the very least it can cause someone making a focused attack to have work more slowly and take alternative approaches–"noisy" behavior that may itself trigger an alarm and get a human involved.
I agree completely. Security is not black and white, same goes for agents that are malicious, if you’re protected from script kiddies by changing your ssh port and adding fail2van then it’s a net positive any way you look at it. It’s important to be aware of course that you’ll need to go a step further if you want to protect from more competent attackers, but what percentage of us will be a target of one of those.
You're being far too polite. He's just plain wrong on this point:
> do not change your standard ports. do not use fail2ban.
That used to be my attitude too. But reality intruded.
Example: I ran a SIP Server for a business. SIP Servers are very attractive targets because if you get past the security, you can make then call paid numbers and make real money. Knowing that, I used huge security tokens. No one got past them. But - it didn't stop them from trying. 100's of times per second at times. The CPU load created by processing the login attempts was so great the SIP server started dropping calls. As it happens, asking the kernel to drop packets from miscreant IP's takes far, far less CPU. Fail2ban to the rescue.
Example: Move on a few years, and now spammers costs have dropped so much it is worth their while to target IMAP, POP and SMTP servers. The story is the same: they never got through, but despite that their attempts increased to the point the server became overloaded. I can't listen on a different port. Fail2ban to the rescue.
Example: Move on to the last couple of years. I move my personal stuff to a VPS. To save costs it's very memory constrained. A problem soon arises. After a few weeks of trouble free operation it keeps slowing down, and sometimes becoming unresponsive. Turns out it's running out of memory; the OOM killer is running amok. That's odd, as there should be almost no load on this thing. Well, apart from ssh login attempts it seems. Turns out there is one of them every few seconds. They have no hope of succeeding of course as I only allow pubic key auth, but the load it creates is a problem. Fail2ban to the rescue.
I don't see anything wrong with the principle, which is you block these attacks at the cheapest point possible. That happens to be the point of entry, before you spend CPU cycles processing the packet (even in TCP). Fail2ban leverages that by using the kernel's networking stack. It's forced to glean the information on who to ban from log files, which is messy, error prone and fragile. I don't like that either, but I'll take what I can get.
And the author insists you shouldn't change ports any way. But seems to have no trouble just saying "use the security in your applications". Well... what sort of security is there in an IMAP server beyond "validate credentials"?
My other thought, reading the author's post, and having seen enough of this type of attitude, is that this type of person would be the one to say "just use... <external mail service>" or similar. "You can't ever do better than them!" Well... I have to suspect that techniques like fail2ban - blocking network access after X suspicious access attempts - get employed at the big service providers too. Possibly not fail2ban specifically, but the same ideas.
Several years ago I was part of team managing one of the busiest recursive DNS resolvers in the country. Around the time, DNS reflection attacks were becoming all the rage (I think there might have been a vulnerability in PowerDNS to blame, I don't specifically recall the details).
The solution we employed was a "dumb" fail2ban like script to ban any abusive IP address. It worked wonders and practically completely neutralized the attacks. We received zero complaints from any legitimate users of the service.
If I recall correctly, even after the attacks subsided (CVEs was patched etc.), we still kept a toned down version of the script in place.
> at best, fail2ban: does nothing
This is just plain wrong. Even if you're only using key based auth and there's no way an attacker could have your keys, fail2ban will still block the abusive traffic.
I have no issues with blocking abusive traffic instead of letting the attackers beat their fists against my armored doors.
> at worst, fail2ban: causes you massive inconvinience or total lockout
The authors main criticism seem to be that you might lock yourself out of your own server one day. This hardly seems fair because the author is saying "just setup SSH correctly and you won't have any issues" while also saying "if you don't setup fail2ban correctly, you'll have a whole bunch of issues!" OK, so shouldn't we just setup fail2ban correctly as well?
Having said all that, my recommendation for people who are going to use an indiscriminate fail2ban setup on SSH access would be to have some out-of-band access in case you do happen to ban yourself, even if it's less convenient to use than SSH.
fwiw, I'm not trying to be anonymous. I've updated my profile to make that plain.
The examples where there to demonstrate the conclusion in the final paragraph matters in real life. The para could stand on it's own - it's just a statement of the obvious. But like all obvious things, it's only obvious once you see it. Like the author of the article, I didn't see it for years (decades?).
> fwiw, I'm not trying to be anonymous. I've updated my profile to make that plain.
Were you thinking that I was arguing that e.g. you shouldn't be able to use a pseudonym to post comments?
Sorry if that's how my comment reads. The thing I was wondering about was if/why we should allow spammers / crackers to retain their anonymity when attacking computers in the way you described. E.g., 100 attacks / second on some port, with no accountability for the persons who originated / propagated those packets.
I'm curious about the pros/cons of law-enforcement agencies having the ability (and mandate) to hunt down the sources of such traffic.
I realize that doing so would open a huge can of works, so I'm 10000% not advocating for it. I'd just like to understand the steel-man arguments for and against.
It's no use if you know your attackers are from SE Asia or Russia. It just creates another incentive for corporations to hoard use data and sell it. Block them and go on with your life. They have an IP address, so it's not 'anonymous'. Law enforcnent could probably get to them if it's worth it and they try hard enough.
Yeah, they also miss (IMO) one of the biggest advantages of using something like fail2ban which is the reduction in logging noise. I've seen multiple VMs effectively DoS-ed because their disk filled up with SSH log entries. I've been paged by security monitoring tools that registered all the failed login attempts as attacks, and I've had to filter out or just wade through all those logs when investigating other issues.
fail2ban also works with services other than SSH, and one of my favorite tricks is to point it at nginx logs, add a `robots.txt` with a honeypot entry and have fail2ban block any IP that hits that or any of a dozen common vulnerability endpoints that I know aren't on the server (like `wp-admin.php`). Keeps the web logs cleaner and more useful.
More seriously, I don’t see why anyone would want to disable fail2ban (outside of purposefully being against the grain) because it seems like “free” protection. I’m sure there’s some situations where it may be beneficial to, but those seem like extreme edge cases.
If you can't prevent some users from having passwords, it's a decent, low-effort way to discourage intrusion attempts. One place it DOESN'T necessarily work great is in k8s.
A few years ago I had to migrate an ancient VM running OpenSSH (for SFTP only) to our k8s cluster. 99% of the users had passwords instead of keys. Easy enough to run sshd in a container with a mounted volume, but I wanted something to recognize and block repeated bad login attempts (the old VM had no such protections).
sshd doesn't do it natively, fail2ban wasn't a good option since you generally don't want a pod messing with a k8 node's iptables. What I ended up with was a Ruby script with 3 threads: One to monitor the auth log for failed logins, a second to lock accounts with repeated failures during a given window, and a third to unlock them after some period (and in the darkness bind them). Definitely some caveats in there, but it was an improvement over the status quo.
Always use multiple layers of defence.
A sturdy 6ft fence and a locked wooden front door can be more secure than a 50m impenetrable stone wall, if it turns out that there is no such thing as an impenetrable stone wall.
Brute force protection is completely unnecessary for openssh when you've disabled password authentication because it's impossible[really expensive] to brute force.
Until you are impacted by an authentication vulnerability which still requires a valid user, then you are brute forcing a user away from being totally compromised. All of a sudden doing all the "unnecessary" things, like brute force protection, disabling root logins and not using default accounts make the difference between you being at the front or back of the line to get compromised in the race to get patched.
If you are one of those who believes a vulnerability like that is extremely unlikely rather than inevitable, then consider this. An ssh key gets leaked, the system it has access to is known / discoverable, the username is not. Brute force protection is now the main difference between being highly at risk and still reasonable safe.
It's a contrived scenario, but hardly an outrageous one.
I remember having a friend with an extemely sluggish server. Turns out the reason behind this sluggishness was that he set a permanent fail2ban bantime, which added thousands of records to iptabes; any packet, even not SSH, the server had to check the source IP against thousands of records.
After purging the rules and switching the bantime the server was usable again.
Fail2Ban can prevent password spraying from working, since an eventual good password will still be blocked if done within the fail2ban block period. And those block periods aren't publicized.
And given appropriate configuration, is another layer of security a potential threat has to deal with.
You could also temp-ban those IPs for a few hours and save yourself some CPU and bandwidth.
It may not seem like much, but I spent a year or two adminning a >1 million reqs / day cluster, and it does add up. Auto-banning this type of obvious offender (with fail2ban, as it happens) did result in a noticeable improvement for us.
They usually follow a predictable pattern, like trying to hit every (almost entirely invalid) URL starting /product/1 /product/2 /product/3 and on up through the millions. They also will rotate the User Agent but neglect realistic/authentic values for other request headers.
It adds rules to in-kernel netfilter tables. On Linux at least, there is really no such thing as a firewall app, there are just different ways to add and manage rules. The only 'approved' default app is nft (nftables), but any app can act on the in-kernel tables.
> you will have to remember scp flags. this is also bad. probably worse
Worse? How?
Hrm... If I don't use commands for a while, yes, I may have to remember them. If that's the tradeoff for one extra layer providing even a modicum of security... why is this a bad thing?
We have things like bash scripts... where you can just store the common flags you use for common things.
I wonder how often the author port tunnels over ssh? I usually have to go search for the syntax for ... 30 seconds. Then I have it. Or it might be in my history already.
"You might have to remember something" seems about the weakest argument against changing ssh ports I've ever read.
I have always changed my ssh port and I don't see any convincing argument in this rant not continuing to do so.
However, scanners have changed drastically during the last 5-10 years. I remember when you saw close to none login attempts when you used a port between 15000 and 20000. Nowdays there are 100s every day. I guess on 22 it's much worse, but I have not used 22 just for the sake of comparing.
For fail2ban it's the same: I know it's not a great solution (lightweight code base, better security gains and lower risk to affect legitimate users would be great), but there is no convincing argument and no better alternative in the submission. Good security for the whole installation is not an alternative, it's a precondition for connecting anything to the internet anyway.
I think ssh on a non-standard port is a no-brainer, have been doing this for years.
But at some point I got too many log entries of failed ssh access attempts. So I looked into fail2ban, but found the system too involved, and I don't like python for this kind of thing to start with. Then I started looking for alternatives, because I figured you only need something very simple, that checks certain log files and then instructs the kernel through netfilter to drop traffic from all infracting IPs. I like single-binary applications, especially for things installed outside the normal package manager. Found a skeleton of a golang app that claimed to work (it didn't), but I managed to rework it, and it serves my use cases: https://github.com/pepa65/fail2drop
> you will have to google "rsync ssh non standard port" every time you want to use rsync.
I have not the best memory but after using it several times I managed to remember "rsync -e 'ssh -p 223' ..." and "scp -P 223". It wasn't that hard really.
I've used fail2ban and a nonstandard ssh port before, for like 12 years. Never had any of the issues this article screams at you about, didn't take any time to setup. I spent a lot more time locking down the sshd_config itself and maintaining that over time. I do agree that is more important to do. Fuck me, I guess.
But that's not a reflection on the fail2ban maintainers, the engineering work that's gone into building it, its usability, bugs or indeed anything else connected to its implementation.
It sucks simply because it's solving the problem in the wrong way. There is no sensible security onion, or defence in depth model for open ports in front of private systems.
If the technical solution for keeping private, or limited access systems actually and effectively private on the public Internet requires open ports, it's the wrong solution.
Unless we're running services designed for public and anonymous access, exposing technologies like VPN servers, SSH or any other software for private consumption to the Internet is a mistake.
When we do this we're putting out invitations for abuse and pulling the burden of security responsibility onto ourselves. I'd argue however, that unless it's a full time concern for us, we're close to the project or maintainers, and able to contribute security fixes to source code, we're probably ill equipped to directly handle that responsibility and remain dependent on engaging with the surrounding security eco-system to remain secure.
I'm not knocking those security eco-systems, I'm just pointing out that there's a time lag inherent on depending on best-efforts from third parties which creates windows of vulnerability, and open ports on infrastructure ALWAYS puts us on the back foot. From getting patches applied in a timely fashion to the time and convenience cost of updating ACLs each time IPs change. Even if we get the maintenance tasks right 100% of the time, we're still open to the risks of something outside of our control happening, like zero day being sprayed across open ports.
The U.S. Government correctly issued an executive order mandating Federal Agencies move to adopt Zero Trust principles just six days after the Colonial Pipeline ransomware attack in 2021. Despite the marketing hype which has followed, if we take nothing else away from this motion, it should be that opening ports in our networks to access private systems using the public Internet is over.
There are at least 90 projects and businesses today dedicated to building modern private access technologies which allow secure remote connections and access to private networks without opening firewall ports. Many are commercial options serving businesses, but there are also lots of compelling open source offerings for non-commercial use too.
There's a directory of vendors and technologies here https://zerotrustnetworkaccess.info/ which attempts to dispense with some the Zero Trust marketing BS and instead focus on technical discourse, architecture and approach which some might find helpful.
Disclosure; founder of one of the businesses (enclave.io) with a commercial interest in this space.
reply