In a recent post I discussed some of the options for patch management in the RightScale platform, this time I will talk about what happens when a patch is not available through traditional patch channels from the vendor. This typically happens in one of two cases:

1. A “workaround” or configuration “fix” is made available from the vendor of a package
2. The vendor of a package applies a security patch to their distribution, but the patch has not been applied to the packages distributed by the operating system vendor

In both of these scenarios, updating from the Security Repositories from the vendors will not provide a fix, it is necessary to do some custom “configur-ating” to get the patch or workaround applied to instances. In both situations, a patch/fix is deployed with custom RightScripts to running instances, as well as those that will be launched until the vendor package patch is released and ServerTemplates are updated.

I’ll walk through one possible way to accomplish this for the recent Apache HTTPd Denial of Service vulnerability. To refresh everyones memory, a vulnerability was found in various versions of Apache that allowed a remote attacker to consume all the CPU on the system the Apache server was running on. Workarounds were issued shortly after the vulernability disclosure, then about a week later, Apache released an official patch in the form of an updated version. We were running some HTTPd 2.2.x servers and the specific version we needed was HTTPd 2.2.20. At the time of the patch release, the linux distros had not yet updated their packages, so we needed to implement an out of band patch (i.e., work around our normal process).

Here are the steps we took to update HTTPd running on a CentOS based image. We did the initial building on a test server as you will see there was some hefty debugging needed to get it right. Here are the steps we followed:

1. Start by reviewing the applicable CVE and find information about the vulnerability.
2. Pull down the sources into the test instance. The instance should be launched with the same ServerTemplate as current running instances that will need to be updated
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.gz http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.gz
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.bz2 http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.bz2
• wget http://mirrors.servercentral.net/fedora/releases/test/16-Alpha/Fedora/source/SRPMS/httpd-2.2.19-4.fc16.src.rpm
3. Run an rpmbuild to get a list of dependencies for the update
• rpmbuild –rebuild httpd-2.2.19-4.fc16.src.rpm
4. Install the dependencies
• yum install xmlto libselinux-devel apr-devel apr-util-devel pcre-devel openssl-devel -y
5. Update to the latest package available for CentOS
• rpm -Uvh httpd-2.2.19-4.fc16.src.rpm –force –nomd5
6. Start configuring to create our own rpm (this is where the hard part begins)
• cd /usr/src/redhat/SPECS/
• edit httpd.spec to add
• Version: 2.2.19 => 2.2.20
• Release: 10%{?dist}.1 => 1%{?dist}.0
7. Build the rpm, expect a boatload of errors to walk through:
• rpmbuild -ba httpd.spec -> Fix error -> repeat
8. Once successful, the newly built package is in /usr/src/redhat/RPMS/
9. Install the update and restart the server
• rpm -Uvh httpd httpd-tools –nodeps
• service httpd restart
10. Take the newly created rpm and upload it as an attachment to be used by the RightScript
11. Create a RightScript that performs the update and restarts the server
• rpm -Uvh $RS_ATTACH_DIR/httpd*.rpm
• service httpd restart
12. Run the RightScript as an “Any” or “Operational” script to update servers in the deployment.

While this process is for CentOS, Ubuntu requires similar heavy lifting to get things functioning. This process took one of our Professional Services engineers about 4 hours to complete (obviously the most time was spent on step #7 ). This type of process takes a lot of hackery to back port a version into an srpm. It is not trivial, but can be done.

So basically the answer to “How?” is “RightScript”. Even though it is non trivial to get that custom rpm or package debugged, once it is, then the deployment to systems is very quick and painless using the RightScale platform.

A final note, once the Linux distribution actually issues the patch, you should transition from “fix” mode your standard “patch” mode for overall consistency. Remember that very little if any testing is given to “fixes” that are released, whereas, a certain level of regression testing is typical for vendor released patches (i.e., distro packages or Windows Updates).