For three days I was very proud of myself.
I built sign-in for this website with modern password hashing, sessions stored so that a stolen database isn't a stolen login, and a login page that gives nothing away. The app runs as its own locked-down account that can barely see the rest of the machine. I wrote blog posts about security.
And the whole time, the server it all runs on had its doors open.
How you get here
Fresh Ubuntu install. Straight into the hosting control panel, because I wanted the site up. Straight into building the site, because that's the fun part. Then the blog, the accounts, the deploy pipeline.
"I'll harden the server later," said someone who was having a great time.
Later arrived on day three, with a question: what does this box actually look like to the internet?
The audit
No sign of a break-in. Every successful login that week was mine. Breathe.
About 1,400 failed password guesses in 24 hours. Roughly one a minute, because password login was still switched on.
fail2ban was banning them. Relief.
The firewall wasn't the one I thought. An earlier check had named a firewall that turned out not to be installed. The control panel's own firewall was the only one, doing its job, while I'd been working from a wrong answer about my own machine.
And IPv6 had no firewall at all. The server has a public IPv6 address, and SSH and the control panel were listening on it, unfiltered, where fail2ban didn't reach either. A locked front door, and a second one round the side with no lock and no bouncer.
I needed a minute. It kept going:
- phpMyAdmin was public on every domain, a login page for a database nothing uses.
- My admin account could run anything as root without a password.
- The control panel was open to the internet, with two-factor off.
- Nine old PHP versions were running. My sites use one.
- The web server, PHP and the database weren't getting automatic security updates, because they come from repositories the updater skipped.
The fix, done properly
The tempting move is to SSH in and start changing things by hand, fast and angry. That's how you lock yourself out of your own server at midnight.
So the fix became a script, with rules:
- Back up every file before touching it.
- Validate every change before applying it, and roll back if the check fails.
- Don't turn off password login until key login has actually worked. Its first step refuses to continue until it has.
- Safe to run again.
What it does: SSH becomes keys-only, with no root login and only the accounts that need it. sudo asks for a password again. The control panel closes to the internet; I tunnel in now. IPv6 gets a firewall that allows web traffic only. fail2ban gets stricter, kernel network settings get tightened, and every internet-facing package gets automatic updates. The unused API, public phpMyAdmin and the end-of-life PHP versions are removed.
The run
Eleven steps. ok. ok. ok.
ok: phpMyAdmin now returns 403 publicly
PHP 5.6 through 8.2, gone.
All steps succeeded.
The music swelled.
Never trust a green checkmark
Then I checked from the outside, the way an attacker would.
Password login: refused. Root: refused. Control panel: unreachable. IPv6: web only. sudo: asks. Deploys, services, sites: all fine. Old PHP: gone.
phpMyAdmin: still public. On every domain.
The script had said 403. The music stopped.
Here's why. This server runs two web servers in a chain: nginx at the front, passing some requests back to Apache. The phpMyAdmin request was one of those. The script changed the setting in nginx, but Apache had its own copy, and Apache's still said "come on in".
The change was valid, tested, and did nothing. And the success message came from the same script that made the mistake, so of course it agreed with itself.
A success message is just a claim.
The fix is an Apache rule that blocks phpMyAdmin on every site, survives control-panel updates, and has to pass Apache's config test before it loads. I ran the script again and checked from outside again.
403. Everywhere. Finally.
One more thing. Obviously.
I was about to close the laptop when one last question arrived: where are the backups?
The site's database backed itself up daily, and the control panel backed up the sites. Very responsible. And every one of those backups sat on the same disk as the data. If the server died, the backups died with it: a second copy standing next to the first in the same burning building.
The host's snapshots didn't make sense at this project's size, so backups now get pulled off the server every day, to a machine I control. The rule was that whatever does the pulling can do nothing else. It uses a new account that can only download, jailed to one folder of backup copies, with no shell, no tunnels, no uploads and no deletes. If its key ever leaked, the worst anyone could do is read backups. The pull fetches only what's changed, prunes old copies, and raises an alert if it fails or the copy goes stale.
And of course, the setup script's first run failed. Its SSH check came back red.
But this was exactly the situation it was built for. The check failed, so it removed its own change, and SSH was never touched. The real problem was a bug in the check itself. The safety net had a hole in it, and it still caught me. I fixed the check and ran it again. Green.
This time I didn't trust the green either. The download account refused everything it should refuse. The database copy passed an integrity check. The backup opened. If this server vanished tomorrow, I'd lose a day at most.
What I'd tell myself three days ago
- Harden the server before you build on it. "Later" is 1,400 password guesses a day while you pick fonts.
- IPv6 exists. A firewall that only speaks IPv4 is half a firewall.
- Every "ok" is a claim. Verify from the outside.
- Automate fixes with checks and rollbacks. They'll save you from your own bugs too.
- A backup on the same disk isn't a backup. Get it off the machine, and open what you backed up.
All of this is now in the deployment playbook, so the next app on this server starts on a hardened machine.
The apps were secure. Now the building is too, and there's a copy of everything that matters somewhere else, in case it burns down anyway.
I'm going to go lie down.
~ Jer