Xenial Xerxus and WordPress – upgrading your VPS

https://xpil.eu/7g1 📋

It is very important to keep your software updated. Forget to upgrade your apache to the latest version and you're screwed. Forget to install recent security patches and you're pwned. And so on.

So, when I logged on to my VPS this morning and saw a big message saying that "...blahblahblah new version of Ubuntu is now available, please upgrade blahblahblah...", I decided to give it a shot.

I had nothing to lose, really. This blog makes no money, its intellectual value to the world is close to zero and I have recent backups on Dropbox, OneDrive and GoogleDrive, so in the worst case scenario I would simply restore from backup and voila. Right?

Xenial Xerxus (or Ubuntu 16.04 LTS if you wish) was rolled out 5 days ago so it is still pretty fresh. But we've been there before, right? The blog survived worse things than simple do-release-upgrade. We know how to get out of a deep shite in case of any software issues, so.

So here we are. My blog was offline for a whopping half an hour today, and I know exactly why. If you are experiencing similar issues, chances are you will find some answers here. Or not.

I ran the do-release-upgrade script as told and waited patiently for all the 507MB of packages to download. That took some 6 minutes.

Then they started to install, occasionally pausing and asking me whether I wanted to keep my versions of certain config files or to install the ones from the repo. In most cases it was the latter but you have to be very careful here. Especially when you have https certificates configured.

In a separate window I was occasionally checking the blog site, just for curiosity. Would I notice anything? I wasn't expecting 100% uptime (at some point the server needs to be bounced to load the new kernel but that usually takes some 2-3 seconds so no biggie) but then I pressed F5 for the umptillionth time, and I felt some sinister, cold, very old feeling crawling up my spine: instead of the boring homepage I saw the php code behind it.

You may think this is nothing serious, right? Must have been the php engine being upgraded and it should come back to normal in no time.

Well... Nope.

First of all, the access details for the wordpress database are stored in open plaintext, in one of the php configuration files. I checked myself and to my horror the file was widely open to the public. Both user name and password for my mysql wordpress database were there, screaming out loud: STEAL ME, PLEASE, I'M ALL YOURS

The panic lasted for about 2 seconds, then I simply ssh-ed to a second terminal session and shut down apache.

It helped a bit. Server was now returning some "blahblahblah ...server is refusing blahblahblah..." error rathar than showing precious contents of PHP files.

OK.

I changed mysql password, and updated relevant PHP config file. I also checked apache logs to make sure nobody accessed the website in the last 5 minutes. Found a couple of bots and my own IP address so it wasn't too bad.

After some 5 or 10 minutes waiting, the OS upgrade was done and politely asking me for a reboot.

It rebooted.

And... nothing. The blog was simply dead.

I scratched my head and came up with a really brilliant idea. I was proud of myself, my razor-sharp mind forged for 20 years in various educational institutions. It was worth it, after all.

The webserver was stopped. Well, I stopped it myself, didn't I?

So I started it. Yay!

And... still nothing.

I examined error log and found this nice message:

Invalid Mutex directory in argument file:${APACHE_LOCK_DIR}

My knowledge on mutex directories in general, and invalid ones in particular, is much smaller than my expertise on Chinese liturgy so - just in case - I scratched my head again. With this pace, I was expecting to dig through the skull-bone and get to the grey goo in about three days. Not good.

Thankfully, other internauts had this issue before and I found a working solution which I did not understand:

source /etc/apache2/envvars

This, at least, allowed me to properly start the apache2 service.

But the blog was still dead. I started receiving emails from various automated services, telling me that my blog was down. Well thanks services, I know it, right? You better tell me what to do to bring it up...

Two and a half moments later I got enlightened again: I renamed that php file with wordpress database username and password, right? It would not be such a completely crazy idea to revert that change.

Yesssss. That was it. Before the change, the file name was completely different, and now, after I renamed it back, it was exactly as original. Hallelujah!

But the site was still down.

I scratched my head again, updating my medical records with the other hand to make sure I can get a neurosurgeon on time.

I went through a couple of on-line resources listing linux packages that are required for WordPress to run. And guess what: most of them were PHP5-related. But PHP5 was not there any more, replaced with PHP7. So, all the PHP5 packages became obsolete and were removed during the upgrade. I had to manually reinstall them (the trick is to use "php-" prefix instead of "php5-" for most of the packages). Things like php-cli, php-mysql, php-pfm and so on.

I installed all of them manually. In a separate terminal window I also started SCP-ing all files from the server onto my local machine - just in case I needed to re-install my VPS.

And I scratched my head again. It felt funny.

Of course installing missing PHP packages did not help.

I examined /var/log/apache2/error.log once again and I spotted something really interesting: the php-executioner.php file was giving plenty of errors every time I was trying to refresh the website.

This was actually a good sign. I knew the apache was working and the php was operative, too.

php-executioner.php is a part of a WordPress plugin that allows you to execute SQL code directly from your WordPress admin area. Very handy. But apparently not compatible with Xenial Xerxus so I manually removed it.

And...

Believe me or not, this last step actually worked and resolved the issue. To my shock, the blog opened as if there was no issue at all. I received a couple of soothing emails from various services, that my blog was now up. Thanks again, services...

So, to sum up:

1. Before you upgrade to Xenial Xerxus, shut down your apache and - once upgraded - make sure all previously existing php5-* packages are now replaced with their php-* counterparts
2. If you're getting the "Invalid Mutex directory..." apache error, execute "source /etc/apache2/envvars" command as root, then restart the apache service
3. Don't use the SQL Executioner WordPress plug-in.

And now please carpe diem before it's too late 😉

https://xpil.eu/7g1 📋