The recent Heartbleed vulnerability sent a pronounced scare throughout the tech community. Fortunately Linux distributions were quick to deploy a patch, with companies and System Administrators following suit.
However, we found ourselves in a bind when we realized 2 of our non-public facing servers were still running Ubuntu Server 13.04. Canonical hadn’t released a Heartbleed patch for 13.04 due to it reaching end of life back in January. Yikes!
The more we researched, the more we found others in the same situation. Unfortunately, or perhaps fortunately, the only correct path is to upgrade to 13.10. With 14.04 so close to its release date, we’d rather of waited and updated to it, but security issues are critical. Prompt action is always better than no action.
So the choice is clear: for those running Ubuntu Server 13.04, an upgrade to 13.10 is required if you want a supported Heartbleed fix. Though perhaps more importantly, it’s bad practice to run an unsupported OS version, so upgrade to 13.10 today regardless. Even better, upgrade to 14.04 LTS when it’s released.
Beware Of Breaking Changes
Fortunately research turned up breaking changes in Apache configuration files that took place in 13.10. We also encountered breaking changes with PHP, and one provider-specific change that prevented our server’s ability to boot! So in addition to the upgrade process, I’ll outline those below, and how we worked around them.
Before proceeding I want to make a recommendation: perform the upgrade on a test server first, perhaps by cloning your target server environment to a new VM or cloud server. Once everything checks out, proceed with updating production servers.
On with the upgrade then.
Upgrading From Ubuntu Server 13.04 to 13.10
Upgrading to 13.10 will affect PHP, Apache, and maybe a cloud or VM server’s ability to boot (heads up to Dediserve customers!) if your provider uses a custom menu.lst file. Ours did, which we’ll mention below.
First, it’s a good idea to get all 13.04 updates installed, so run:
sudo apt-get upgrade
Second, proceed with the update to 13.10 by issuing the following commands:
That will kick off the upgrade process.
It’s important to note that during that process you’ll be asked if you want to keep any changed system files, or have them overwritten by the new release’s version. Since no one can make those decisions for you, it’s best to diff each file (which you can do during the upgrade process) and make your own decisions.
Here are the changes that were important to us, including what changed and how we worked around any breaking changes.
Please beware of changes to this file, as it usually specifies disk or partition paths. Changes in this file can affect a server’s ability to boot. We host our servers with Dediserve, and prior experience taught us to keep their custom menu.lst file in place, else our server failed to boot.
So when asked by the upgrade process if we wanted to keep our own version or install the new version, we decided to keep our own.
13.10 broke our PHP installation, which included changes to our php.ini file. After diff’ing the current vs. new version, the new php.ini’s changes were relatively simple. The new version’s php.ini:
- turned short tags off
- set error_reporting back to a default value
- reverted our session.cookie_lifetime and session.gc_maxlifetime settings
- set default_charset back to an empty default
We accepted the new version, just in case it contained other important updates, and then re-instated the settings above in the new php.ini file:
- short tags were turned back on
- error_reporting was set back to our preferred value
- session.cookie_lifetime and session.gc_maxlifetime were set back to preferred values
- default_charset was set back to UTF-8
There were 2 additional errors we experienced.
The first was an error stating that json_decode()/json_encode() functions were undefined. I’m not sure why 13.10 changed that, but to resolve we simply re-installed the json package:
The second was due to no timezone setting. To resolve that we specified a date.timezone setting in php.ini:
After that we tested image generation, pdf generation, mail delivery, ftp, etc. Fortunately all that still worked in our PHP apps.
13.10 introduced some important changes to Apache, mostly with configuration files. They will break your 13.04 configuration, so please do your own research in addition to noting the changes below.
There were 2 major changes we were affected by.
The first is that all config files in /etc/apache2/conf.d should be moved to /etc/apache2/conf-available.
This is because 13.10 now treats those config files the same as sites-enabled/available and mods-enabled/available. We use a custom.conf file in /etc/apache2/conf.d that includes the ServerName and AddDefaultCharset directives; we needed to move that to /etc/apache2/conf-available, then enable with:
Second, vhost files in /etc/apache2/sites-available previously had no file extensions. That’s changed in 13.10; they now must have a .conf extension. Otherwise, apache will report an error like this upon start:
ERROR: Site site-name does not exist!
Fortunately this is pretty easy to fix. Just append .conf to each of your vhost files in /etc/apache2/sites-available.
Once that’s done, you’ll need new symlinks between sites-enabled and sites-available. You can re establish those by first removing your existing sym links:
Then re-enable your sites with a2ensite:
And that should take care of things. I had additional PHP packages installed (curl, gd, etc.), along with sites behind SSL. Fortunately all that continued to work after the upgrade.
Verify Heartbleed fix
Finally, with the upgrade complete, you’ll also want to verify that OpenSSL is the version with the Heartbleed patch. You can do so by running:
… and verifying that your openssl version is 1.0.1e-3ubuntu1.2.
In addition to updating to 13.10 and verifying the Heartbleed patch, you’ll also want to change any passwords used to access the server, or for apps hosted on it, since those would have been vulnerable. You’d also need to reissue any SSL certificates used to secure sites hosted on affected servers. And it’s important to note that you’d want to do those after the Heartbleed patch is installed.