The recent Heartbleed vulnerability sent a scare throughout the tech community.  Fortunately Linux distributions were quick to deploy a patch, allowing companies to quickly follow suit.

However, we found ourselves in a bind after realizing 2 of our non-public facing servers were still running Ubuntu Server 13.04. Canonical hadn’t released a Heartbleed patch for 13.04 due to it reaching end of life back in January. Yikes!

The more we researched, the more we found others in the same situation.  Unfortunately, or perhaps fortunately, the only correct path is to upgrade to 13.10. With 14.04 so close to its release date, we’d rather of waited and updated to it, but security issues are critical.  Prompt action is always better than no action.

So the choice is clear:  for those running Ubuntu Server 13.04, an upgrade to 13.10 is required if you want a supported Heartbleed fix. Although you’ll also want to consider upgrading to 14.04 when it’s released since it’s a LTS version.

Beware Of Breaking Changes

Fortunately research turned up breaking changes in Apache configuration files that took place in 13.10.  We also encountered breaking changes with PHP, and one provider-specific change that prevented our server’s ability to boot! So in addition to the upgrade process, I’ll outline those below, and how we worked around them.

Before proceeding I want to make a recommendation: perform the upgrade on a test server first, perhaps by cloning your target server environment to a new VM or cloud server.  Once everything checks out, proceed with updating production servers.

On with the upgrade then.

Upgrading From Ubuntu Server 13.04 to 13.10

Upgrading to 13.10 will affect PHP, Apache, and maybe a cloud or VM server’s ability to boot (heads up to Dediserve customers!) if your provider uses a custom menu.lst file. Ours did, which we’ll mention below.

First, it’s a good idea to get all 13.04 updates installed, so run:

sudo apt-get update
sudo apt-get upgrade

Second, proceed with the update to 13.10 by issuing the following commands:

sudo apt-get install update-manager-core
sudo do-release-upgrade

That will kick off the upgrade process.

It’s important to note that during that process you’ll be asked if you want to keep any changed system files, or have them overwritten by the new release’s version. Since no one can make those decisions for you, it’s best to diff each file (which you can do during the upgrade process) and make your own decisions.

Here are the changes that were important to us, including what changed and how we worked around any breaking changes.

/boot/grub/menu.lst

Please beware of changes to this file, as it usually specifies disk or partition paths. Changes in this file can affect a server’s ability to boot. We host our servers with Dediserve, and prior experience taught us to keep their custom menu.lst file in place, else our server failed to boot.

So when asked by the upgrade process if we wanted to keep our own version or install the new version, we decided to keep our own.

PHP

13.10 broke our PHP installation, which included changes to our php.ini file.  After diff’ing the current vs. new version, the new php.ini’s changes were relatively simple.  The new version’s php.ini:

• turned short tags off
• set error_reporting back to a default value
• reverted our session.cookie_lifetime and session.gc_maxlifetime settings
• set default_charset back to an empty default

We accepted the new version, just in case it contained other important updates, and then re-instated the settings above in the new php.ini file:

• short tags were turned back on
• error_reporting was set back to our preferred value
• session.cookie_lifetime and session.gc_maxlifetime were set back to preferred values
• default_charset was set back to UTF-8

There were 2 additional errors we experienced.

The first was an error stating that json_decode()/json_encode() functions were undefined.  I’m not sure why 13.10 changed that, but to resolve we simply re-installed the json package:

sudo apt-get install php5-json

The second was due to no timezone setting.  To resolve that we specified a date.timezone setting in php.ini:

date.timezone = "America/Chicago"

After that we tested image generation, pdf generation, mail delivery, ftp, etc.  Fortunately all that still worked in our PHP apps.

Apache

13.10 introduced some important changes to Apache, mostly with configuration files.  They will break your 13.04 configuration, so please do your own research in addition to noting the changes below.

There were 2 major changes we were affected by.

The first is that all config files in /etc/apache2/conf.d should be moved to /etc/apache2/conf-available.

This is because 13.10 now treats those config files the same as sites-enabled/available and mods-enabled/available.  We use a custom.conf file in /etc/apache2/conf.d that includes the ServerName and AddDefaultCharset directives; we needed to move that to /etc/apache2/conf-available, then enable with:

sudo a2enconf custom

Second, vhost files in /etc/apache2/sites-available previously had no file extensions. That’s changed in 13.10; they now must have a .conf extension. Otherwise, apache will report an error like this upon start:

ERROR: Site site-name does not exist!

Fortunately this is pretty easy to fix. Just append .conf to each of your vhost files in /etc/apache2/sites-available.

Once that’s done, you’ll need new symlinks between sites-enabled and sites-available. You can re establish those by first removing your existing sym links:

sudo rm /etc/apache2/sites-enabled/*

Then re-enable your sites with a2ensite:

sudo a2ensite site-name

And that should take care of things. I had additional PHP packages installed (curl, gd, etc.), along with sites behind SSL.  Fortunately all that continued to work after the upgrade.

Verify Heartbleed fix

Finally, with the upgrade complete, you’ll also want to verify that OpenSSL is the version with the Heartbleed patch.  You can do so by running:

dpkg -l | grep "openssl"

… and verifying that your openssl version is 1.0.1e-3ubuntu1.2.

Other Precautions

In addition to updating to 13.10 and verifying the Heartbleed patch, you’ll also want to change any passwords used to access the server, or for apps hosted on it, since those would have been vulnerable.  You’d also need to reissue any SSL certificates used to secure sites hosted on affected servers.  And it’s important to note that you’d want to do those after the Heartbleed patch is installed.