Introduction to RightLink

RightLink (link) is the instance side agent that supports RightScale’s RightNet protocol. The agent provides an improved and secure ability to leverage RightScale to manage large numbers of instances in the cloud. In the RightScale architecture, we leverage a light-weight RightLink agent on every instance to support our latest automation features. Prior to RightLink, which was released a bit over year ago, RightScale leveraged the command execution features of SSH to perform tasks on remote instances. With the introduction of RightNet and the RightLink agent, we are no longer reliant on SSH access for instance management.

The RightLink agent communicates with the core RightScale systems using the Advanced Message Queuing Protocol (AMQP). RightNet leverages AMQP’s Simple Authentication and Security Layer (SASL) support to perform a basic session authentication to ensure that the RightLink agent is talking to a legitimate RightScale core component (a broker in our lingo). This session authentication uses a shared key to authenticate the ends. After the session is authenticated RightNet uses payload encryption (openssl with X509 certificates, PKCS7 envelopes and AES 256 CBC cipher for encryption) to protect that data while in transit, and to provide a much stronger authentication mechanism (public-private key versus only the shared key of the session). Both of these security features are to ensure that packets are properly segmented and protected in the highly multi-tenant aspect of the cloud.

All our version 5 (v5) RightImages (and Multi Cloud Images, MCIs) include the RightLink agent by default. We started releasing v5 images over a year ago, and have seen a large, but not complete, adoption. For those of you still on v4 images, I am going to try to give you a couple more security motivations that may encourage you down the upgrade path.

1. Ability to restrict SSH access on the instance: Because RightLink does not use SSH you can restrict access to the ssh service on Linux systems. With non RightLink enabled images (i.e., v4 and earlier by default), the RightScale platform ran scripts on the instance by ssh-ing into that instance directly, thus the need for ssh port to be accessible on the instance from the RightScale platform usually meant that it was accessible from any IP address. This created some exposure with potential brute force attacks. I will say that by default, RightImages configured SSHD to support public-key authentication only, so the risk of brute force password guessing was not an issue. What was an issue was that any vulnerability found in the SSHD server would then be potentially exploitable by anyone on the Internet.  With RightLink, this exposure can be mitigated.
2. Managed SSH: In addition, v5 RightImages introduce a “Managed SSH Login” feature. This allows you to use a different SSH key for each user logging into a server. It can either use an SSH key uploaded by each user or the dashboard can generate a key for each user.  When using EC2 you may still select an EC2 SSH Key when launching the instance, however, it’s only really necessary if you need to log-in before RightLink starts to troubleshoot something in the bootstrap process. Note that the SSH connection is from your desktop system (wherever you are running the dashboard UI from, not RightScale) to your instance, thus working seamlessly with any SSH access restrictions you put in place.

SPOILER-ALERT: one of the items we are working on for RightLink v5.8 (next version coming out) is a Managed SSH Login that will bind each RightScale authentication principal to a distinct, non-root Unix user whenever they login via the dashboard. This is intended to improve the login auditing as well a enable each user to load a customized shell profile. We’d be very interested in your feedback as to the usefulness and desire of this specific feature.

Upgrade options

The cleanest and best way to move to v5 images is to find a v5 ServerTemplate, clone it and make the modifications needed to effectively duplicate the functionality you currently have. This will work like a charm if you if you did your scripts right and took a modular approach to deployment.

Next option is to change the RightImage (i.e. Multi Cloud Image, MCI) you’re using to a v5 one and relaunch. The V5 execution of RightScripts is almost fully compatible with v4 so, in theory, that’s all you need to do. The catch typically is that this brings updated versions of the OS and packages with it and may cause some incompatibilities. You will probably spend a bit more time troubleshooting this avenue.

Lastly, you can get RightNet support by RightLink enabling your v4 instance (see http://support.rightscale.com/12-Guides/RightLink/04-Creating_RightScale-enabled_Images_with_RightLink), and many might be motivated to go that route. I would encourage you to move to v5. While you’ll get the “not using ssh for command and control” benefit, you will miss many other benefits of the v5 image update.

Why Again?

Because there are some really cool features in v5:

• Managed SSH
• Bug fixes
• Faster Execution of Operational Scripts
• Added Chef Support in addition to RightScritps

More details can be found http://support.rightscale.com/06-FAQs/FAQ_0180_-_What_are_the_differences_between_v4_and_v5_RightImages%3F

It will take a bit of effort, but I guarantee the improvements you gain will be worth it! My one-liner of advice to those RightScale customers with older versions ”if you’re one of those hanging onto v4 or earlier you really should upgrade.”

Building your own Salesforce.com? Yup, sounds like a lot of work. Yet here at RightScale, I see many companies trying to build their own cloud management solutions. Perhaps it is the DevOps mindset that has made cloud computing so popular: “If I can’t get approval, I’ll just do it myself on the side.” Or perhaps it is because we are still in the early stages of cloud, and people are experimenting and discovering what is possible internally versus what is available in the market.

I did an informal poll of our sales team, and here’s what they said were the top reasons companies try to make their own solutions rather than use a cloud management product:

1. They want control, or the ability to highly customize their environment.
2. PaaS and IaaS, as concepts, seem simple, easy to jump on. A “cloud computing management platform” seems like a complex paradigm to adopt.
3. Because they can, and they want the challenge of exploring a new frontier.
4. The cost of a cloud management solution is too high.

OK, so these appear to be valid reasons at first glance. But these statements are typically founded in misconceptions about cloud management solutions in general or RightScale in particular, which I’ll address here:

Control: RightScale is not a PaaS service. We let you get into everything – perhaps more so than we should. Change the images if you must, run custom scripts against our API, and export usage data to include in your own data warehouse. Fifty-two percent of the servers running on RightScale are controlled by completely custom ServerTemplates, not ones we provide. Our product philosophy is to let you “get under the hood” if you need to – so please do.

Complexity: Cloud management is complex, and I don’t argue that. What RightScale aims to do is provide a layer of abstraction that makes the difficult and mundane tasks, like auto-scaling, much easier. It is unfortunate that the term seems complex, because if anything, a cloud management solution can make managing your entire cloud infrastructure and applications so much easier.

Conquering the new frontier: You’re being told by your boss to “Learn cloud now – just figure it out.” You want to truly understand what’s possible, how to build it, and deliver on expectations. As you start down this path, you cobble together some tools to accomplish your first foray into the cloud. Unfortunately, technologists have a tendency to “reinvent the wheel” as they continue along their path to the cloud. We’re many steps ahead, and we’re happy to share what we’ve already learned.

Cost: Netflix is a poster-child for DIY cloud, and has been forthcoming about its experience, which has helped grow this new paradigm. Netflix “designed its cloud architecture so that it has the option to move to an Amazon Web Services competitor” if needed, according to this NetworkWorld article. At a recent conference, Adrian Cockcroft, Cloud Architect for Netflix, mentioned that Netflix has 50+ engineers working on this cloud-independent solution. Doing some quick math, that’s about $8.3 MM per year Netflix spends building and maintaining this platform. That could buy a lot of RightScale Enterprise Editions!

At the end of the day, we see many customers who come to us after they outgrow their own internal solutions. They eventually discover that there are just too many things to stitch together: configuration management, systems automation, monitoring, application automation, provisioning, user permissions, reporting…it goes on.

We have hundreds of employees and have spent many millions creating the most comprehensive cloud management platform in the world. And we designed our product to drive the same way no matter which cloud you choose. So while cloud management may seem like a fun weekend project to tackle, it’s not – please don’t try it at home.

Yes, Amazon is still the dominant cloud, but a tornado of new clouds is swirling. The next thing your boss will likely ask is, “So what if we wanted to use this other cloud instead?”

These clouds have been in the works for a little while, and I’m pleased they are now available in the RightScale platform for our customers.  When we integrate with a given cloud, we work hard to ensure a seamless experience across all the clouds we support.  We provide a generic interface to each of the clouds integrated within RightScale.  This is not to limit functionality from the clouds themselves; but rather to ensure all that cool functionality is usable.  If I’m using SoftLayer and Datapipe, I don’t want to deal with different storage solutions like volumes or instance based storage (or at least not until I’m ready to optimize the storage).  Likewise, keep networking off my plate…I don’t care whether it’s security groups or ip tables.  Just make that infrastructure stuff work so that my app can run.

As a user, I want to  easily port what I have in one resource pool to another resource pool.  For this purpose, RightScale has generic constructs for things like instances, instance types, images, volumes, volume snapshots, etc, that are exposed in our dashboard.  Then, in our ServerTemplates (stay tuned by the way, a release is imminent), we use chef to abstract features for individual ServerTemplates that work, albeit very differently, across different resource pools.  Using the above example, someone launching servers in SoftLayer’s Amsterdam cloud and Datapipe’s Hong Kong cloud doesn’t have to worry about the differences between network configuration and storage management.  You can launch an entire 3-tier PHP architecture on both environments using ServerTemplates from the MultiCloud Marketplace.  We’ll take care of dealing with instance based storage in Amsterdam and set up the proper security groups for you in Hong Kong through the platform.

Why does RightScale spend so much time touting ‘MultiCloud’ and why should anyone care?

It’s a good question to ask actually.  I spend a lot of my time working with service providers and various companies looking to deliver infrastructure as a service for public consumption.   A number of people, our existing customers included, come to us and say “hey, I know I will have multiple clouds (if I don’t already)…help me make that happen.”  Analysts also agree – Forrester’s Holger Kisker touts “multi cloud becomes the norm” as his number 1 cloud computing prediction for 2012.

It’s real.  And it’s great validation for being the leader in ‘MultiCloud Management”.

Perhaps even more interesting (and contradictory if you think about it) is that the service providers say the same thing!  We describe how RightScale offers clouds to consumers and the choice consumers have to use what works best for their business needs.  And, IaaS providers are more than happy (okay, some take it as a challenge to deliver an even better service for their users. ).  In truth though, they recognize that cloud is a heterogeneous environment.  A single customer will use more than one cloud offering in a single environment.  Cost is one factor, and another I hear often is performance.  In some cases, geographic location is important and they “can’t get there with their current IaaS provider.”  It’s an opportunity for some to seize, and we’re partnering with the best to deliver the multi-cloud solutions our customers want.

Within RightScale, you can use any or all of the following clouds – all the Amazon regions, SoftLayer, Rackspace Cloud across US and UK, Datapipe, Logicworks as well as private cloud management with CloudStack and Eucalyptus.

I encourage you to click and try the new clouds on RightScale.  Use a new app or an existing one that’s already in cloud and as always, let us know what you think.

1. A “workaround” or configuration “fix” is made available from the vendor of a package
2. The vendor of a package applies a security patch to their distribution, but the patch has not been applied to the packages distributed by the operating system vendor

In both of these scenarios, updating from the Security Repositories from the vendors will not provide a fix, it is necessary to do some custom “configur-ating” to get the patch or workaround applied to instances. In both situations, a patch/fix is deployed with custom RightScripts to running instances, as well as those that will be launched until the vendor package patch is released and ServerTemplates are updated.

I’ll walk through one possible way to accomplish this for the recent Apache HTTPd Denial of Service vulnerability. To refresh everyones memory, a vulnerability was found in various versions of Apache that allowed a remote attacker to consume all the CPU on the system the Apache server was running on. Workarounds were issued shortly after the vulernability disclosure, then about a week later, Apache released an official patch in the form of an updated version. We were running some HTTPd 2.2.x servers and the specific version we needed was HTTPd 2.2.20. At the time of the patch release, the linux distros had not yet updated their packages, so we needed to implement an out of band patch (i.e., work around our normal process).

Here are the steps we took to update HTTPd running on a CentOS based image. We did the initial building on a test server as you will see there was some hefty debugging needed to get it right. Here are the steps we followed:

1. Start by reviewing the applicable CVE and find information about the vulnerability.
2. Pull down the sources into the test instance. The instance should be launched with the same ServerTemplate as current running instances that will need to be updated
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.gz http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.gz
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.bz2 http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.bz2
• wget http://mirrors.servercentral.net/fedora/releases/test/16-Alpha/Fedora/source/SRPMS/httpd-2.2.19-4.fc16.src.rpm
3. Run an rpmbuild to get a list of dependencies for the update
• rpmbuild –rebuild httpd-2.2.19-4.fc16.src.rpm
4. Install the dependencies
• yum install xmlto libselinux-devel apr-devel apr-util-devel pcre-devel openssl-devel -y
5. Update to the latest package available for CentOS
• rpm -Uvh httpd-2.2.19-4.fc16.src.rpm –force –nomd5
6. Start configuring to create our own rpm (this is where the hard part begins)
• cd /usr/src/redhat/SPECS/
• edit httpd.spec to add
• Version: 2.2.19 => 2.2.20
• Release: 10%{?dist}.1 => 1%{?dist}.0
7. Build the rpm, expect a boatload of errors to walk through:
• rpmbuild -ba httpd.spec -> Fix error -> repeat
8. Once successful, the newly built package is in /usr/src/redhat/RPMS/
9. Install the update and restart the server
• rpm -Uvh httpd httpd-tools –nodeps
• service httpd restart
10. Take the newly created rpm and upload it as an attachment to be used by the RightScript
11. Create a RightScript that performs the update and restarts the server
• rpm -Uvh $RS_ATTACH_DIR/httpd*.rpm
• service httpd restart
12. Run the RightScript as an “Any” or “Operational” script to update servers in the deployment.

While this process is for CentOS, Ubuntu requires similar heavy lifting to get things functioning. This process took one of our Professional Services engineers about 4 hours to complete (obviously the most time was spent on step #7 ). This type of process takes a lot of hackery to back port a version into an srpm. It is not trivial, but can be done.

So basically the answer to “How?” is “RightScript”. Even though it is non trivial to get that custom rpm or package debugged, once it is, then the deployment to systems is very quick and painless using the RightScale platform.

A final note, once the Linux distribution actually issues the patch, you should transition from “fix” mode your standard “patch” mode for overall consistency. Remember that very little if any testing is given to “fixes” that are released, whereas, a certain level of regression testing is typical for vendor released patches (i.e., distro packages or Windows Updates).

Within the RightScale platform, there are 3 primary options that can be used to automate the patching of instances:

1. Unfreeze Security Repositories and enable automatic updates on systems, hope that the updates don’t break anything.
2. Manually unfreeze Security Repositories for test systems and update. Perform regression testing, then update & refreeze Security Repositories for production systems and apply updates. Do this regularly (say monthly or weekly).
3. Update each ServerTemplate with the latest Security Repository. Regression test each updated ServerTemplate. During a schedule maintenance period, force all servers to be relaunched with updated ServerTemplates.

Of course, there’s also always the option to hide underneath a pile of coats and hope it all works out for the best. It goes without saying that while many people de-facto implement this last option, it is not a viable long-term strategy!

Let’s dive into each of the options a bit more and look at some pros and cons, so you are in a better position to pick the one (or combination) that works best for you.

• Unfreeze with automatic updates: Since many (most?) of the core Linux distributions have functionality to allow selecting of security updates only, you freeze all channels, and then set the security repo to /latest via a RightScript. You then configure the system to install those updates on an interval you desire (daily seems to be a good choice). For example, on Debian based systems, such as Ubuntu, security patches are broken out into a separate repository. For a given release it is possible to only automatically install updates from http://security.ubuntu.com/ubuntu/ instead of http://us.archive.ubuntu.com/ubuntu/, making this very easy to implement. Just unfreeze that repository and updates will apply as they are released.
  With CentOS you can run “yum update –security” and only install security related patches. Using this method allows rapid access to the latest security updates, with almost no work required to enable this behavior as the unattended upgrade packages do all the work for you.
  The downside is that if a broken package is released, say into Ubuntu-security, it could affect production. A side note is that as it relates to security patches, the industry at large has pretty much come to the acceptance that the risk of problems with automatically patching security vulnerabilities outweighs the potential risks with doing it. For example, Debian, Ubuntu and Windows 2003-2008 all ship this way. For those who determine that the risk of automatic patching is too great, there is …
• Unfreeze in test, test it, update production: This option is to apply the security patches to instances in a test deployment, then after regression testing, deploy them to production using a RightScript to update repositories and perform an update on production servers. This has the advantage of some level of regression testing prior to deploying security patches in production. The downside is that there is a high manpower cost to perform the functional testing on a regular basis. There is also the fact that you should test the specific items that the security fix supposedly touched which involves a bunch of research. This is a non-trivial effort. It would likely require a special test environment dedicated to security testing. From a purely dogmatic standpoint, this is the way it should be done, but the pragmatist in me knows that for many organizations, the additional cost associated with this is not justified by the increase in risk posed by just installing security updates. I’d rather have patched systems, than people not doing it because it was not the absolute best way to go about it.
• Update ServerTemplates and relaunch: This may be the cleanest and seemingly easiest approach. There is relatively little change in current operations, as many of you use this method currently. This also ensures that all packages are tested before being deployed in production. The upside is that systems are cleanly built, and ServerTemplates are updated more often. The downside to this is that your patch level is only as good as your latest ServerTemplate update, and while it works for servers that can be frequently updated (app servers, web servers, etc.), it really doesn’t work well for services that are infrequently updated, or difficult to relaunch (databases, load balancers, etc.). Further, it forces you to relaunch servers you wouldn’t otherwise relaunch during maintenance windows.

So, you may be asking “You use RightScale to manage RightScale, so how do you do it?” Well, at RightScale, we have chosen a hybrid approach of #1 & #2. Our default patching policy is “Unfreeze with automatic updates”. As stated earlier, there is some inherent risk in this stance, but we feel that getting critical fixes in outweighs the incremental risk of taking too long to get the patch deployed. In instances where the risk of any patch (security or not) breaking a system, we use the “Unfreeze in Test, test it, update production” patch policy. Further, we design our platform with mitigating controls to restrict access to systems and services that may not get the latest patches on a daily basis. This policy/stance works for us, and we think it is a reasonable one for others to start with (if you didn’t already have a stance).

I would be remiss if I did not point out that there are likely a myriad of other ways that you can perform security patching, but that these are ones you get “out of the box” with the RightScale platform. The specific approach you choose will be driven by your business requirements. Remember that you have options, so use them to develop a process the works for you and your organization. My next blog will be on deploying workarounds and non packaged fixes. Until then, Happy security “patching!”

First, 3 million is impressive in the data center business.  Many well-known hosting companies house between 50,000 and 100,000 servers, and estimates for the world’s largest computer companies with large data centers range up to 1 million.  (See the DataCenterKnowledge report here.)  It’s difficult to compare our statistic with these installations, since many may be running largely under a pre-cloud operational model.  Nevertheless, launching 3 million is quite a number by any comparative metric, and there’s no question that it was achieved only with new levels of automation and dynamic configuration that are core to RightScale.

The second reason 3M is worth noting has to do with how fast we got there.   After our founding in 2007, it took us about 27 months to reach 1M, another 12 months to reach 2M, and then just 6 months to reach 3M.  That’s more than twice as fast for each subsequent 1M servers.  Likewise, one year ago in Sept. 2010, we had launched 1.5M servers – and we doubled in the last 12 months.

The third reason this milestone matters is that the servers our users launch have increased in power, and persist for a longer duration, as each month passes.  In fact, since January this year server runtime has increased on average 30%. So the trend is clear: companies are running “bigger iron” in the cloud — and keeping it running longer — than ever before.  Here is a graph of the size distribution we recorded this summer:

Certainly, the growth rate we’re tracking for the quantity, power and longevity of servers launched on RightScale remains quite healthy and mirrors the broad adoption of cloud services industry-wide. But equally important is the range of customers driving this growth, representing a wide variety of industries, use cases and services powered by RightScale on the cloud. For example, during the last year:

• media giant Pearson converted a traditional educational software offering to a SaaS based model that allowed faster onboarding of new customers;
• consumer goods company American Girl (a division of Mattel) launched their virtual world with a major advertising push behind it and sailed smoothly through the holiday season;
• online game company Zynga launched new games that consistently broke records;
• and companies like Amdocs and Trader Media spoke at our User Conference last June about new enterprise services launched on both public and hybrid clouds.

All of these RightScale customers contributed toward the 3M milestone, and we continue to be dazzled by the solutions they achieve using cloud infrastructure. We’re looking forward to the next million servers launched by our customers, and the amazing services they’ll power with them.

Drum roll please…Introducing the auto-scaling, high availability .NET Stack on Amazon!

Some of you might think, ‘well Finally!’.  As a product manager, I empathize with that sentiment. It was certainly tricky getting this to hum on the cloud. I’m very proud of our team’s work, especially when reviewing the challenges they overcame to get this out the door.  Let’s take a deeper look into some of them.

Database Manager for SQL Server

When I first proposed the SQL Server Database Manager last year to our development team with what ultimately would become our first SQL Server ServerTemplate, I met with mixed reactions.  EVERYONE thought it was a great idea and would be useful to users.  However, a lot of reservations too…doing something like what we have for MySQL with master/slave replication is no easy feat.  Adding in Microsoft complexity with Powershell as well as unexpected Windows behavior in the cloud, the solution seemed out of grasp. Some of the notable questions we asked ourselves:

• How does MS Licensing work as existing orgs transition to the cloud?  Is SQL Server Standard good enough or will users demand Enterprise?
• How will we backup data on SQL Server?  Native backups guarantee “sane” backups without service interruptions but take a long time.
• What are the setup best practices and how do we implement them in the cloud?  Multiple Data / Log Volumes, default monitoring and alerts, backup scheduling, etc…all need to be considered.
• How do we set up replication with SQL Server?  There are so many supported options, what’s best for cloud?

We started at the beginning, pushing licensing off to the likes of the service providers (Amazon) and focusing on prototyping our implementation.  One of our developers found that backups using Volume Shadow Services was a better option than SQL Server native backups.  The following is an excerpt from his report:

Now that we have a clear understanding of how to proceed with backups, we tried to figure out best practice configuration — focusing mainly on the volume configuration and management.  We encountered issues attaching EBS volumes before Windows was ‘ready’ which resulted in out of order drive letter assignment.  We solved that, and moved on only to find that Powershell was running as a 32-bit process on the x64 environments…great.  Fixed that too.  Those two issues actually got us pretty far and enabled our first release of a Beta ServerTemplate that supported Standalone backup/restore functionality.

Going through the Beta of our backup/restore ServerTemplate, we learned enough to facilitate building a High Availability SQL Server Solution (that’s what we published recently!).  We focused on a few things:

• SQL Server Mirroring (Set up, Monitoring and Alerting)
• Authentication and Data transfer Encryption
• Failover to the mirror

One key challenge in setting up the mirroring session was waiting.  We had to have not only the mirror server in operational state but also a set of database full and differential backups from the principal to initialize the database on the mirror.  We utilized the RightScale ability to tag servers and locate servers by tag to help automate this whole process.

I recommend you try out our ServerTemplate to see just how much we managed to take off your shoulders for the database management.

IIS Application Server

Phew.  When you think about the complexity of the Database Manager, IIS seems like a walk in the park!  But a lot of work went into this template too.  I went over and chatted with the dev lead whose team built this template to get the inside scoop of the challenges they had to overcome.  Here’s his list:

• Built with the use case of the scalable app tier in mind
• Integrate a front-end load balancing solution
• Figure out how to get the app on the server
• Figure out how to tell app servers where the db is when they are ready
• Oh, and of course, best practice configuration of IIS App Servers (this is Microsoft after all)

Doesn’t sound too complicated does it?  Luckily, with standardized images and use of RightScale tags to discover the Database server, it worked pretty well.  Also, based in large part on the design, we were able to get this ServerTemplate to work equally well on both Rackspace and Amazon.  That actually was a cool product win, even though it created more work for our testing team (details of which I went into in my last post).

I encourage you to take a look at this ServerTemplate too.

Together, these two ServerTemplates with either the HAProxy Load Balancer ServerTemplate or Elastic Load Balancing from Amazon make up our .NET Stack.  Take them for a spin, and as always, please send us your feedback.  Enjoy!

It’s been a R.A.C.E. to the finish line this sprint, but we have some exciting news!  RightScale’s current ServerTemplate release showcases the entire PHP 3-tier stack across the Rackspace Cloud, Amazon’s Elastic Compute Cloud, Cloud.com’s CloudStack and Eucalyptus Systems (hence R.A.C.E).  These new HAProxy Load Balancer, PHP App Server and Database Manager for MySQL 5.1 ServerTemplates are available now in the MultiCloud MarketPlace.

Underneath all these templates, we’re releasing a new CentOS 5.6 MultiCloud Image with RightLink 5.7 for all EC2 regions, Rackspace Cloud Servers, Cloud.com’s CloudStack and Eucalyptus Systems.  For a complete list of ServerTemplates and MultiCloud Images we released, check out our latest release notes.

People often talk about developing for the cloud and associated challenges.  But what about building platforms on which other people develop their apps in the cloud?  It is very challenging building for the “general use case”, but it is even more important when you build for generality that it works well.  A lot of time and effort goes into designing and building our ServerTemplates.  During development, and especially before release, we conduct extensive manual and automated testing cycles where we put our templates through the wringer.  Below is a sample test matrix that represents our checklist for one ServerTemplate.

Notice that we test this one ServerTemplate across 2 separate images in 7 clouds each.  Considering that we released 9 ServerTemplates last week, this makes for quite a few permutations.  Of course, we also find bugs and have to retest rapidly.

Luckily we have help with a home-grown automation tool that we call “Virtual Monkey.” The monkey uses the RightScale API to create deployments with servers to test, launches them, runs tests against them, collects the results, shuts everything down, and cleans up. In most cases the tests include entire 3-tier application deployments so we can test the interactions between the servers and ensure things work end-to-end. All in all we launch hundreds of servers a day in various clouds to test these ServerTemplates and make sure they work before we let them loose in the wild.

Not all clouds are created equal…

If you’ve done anything on multiple clouds, you’ll appreciate the behind-the-scenes presented above.  When launching hundreds of servers and developing infrastructure-as-a-service agnostic solutions, small nuances can be big blockers towards expected end-user functionality on the solution.  From security groups on Amazon versus iptables management on Rackspace to volume snapshot API response differences in Eucalyptus and CloudStack, the ServerTemplate needs to be aware and abstract away differences.  Do our customers really care about these differences?  Of course not.  They just expect one solution to work on one cloud type just as well as it works on another cloud type.  You may have noticed that we utilized Chef for many of our recently released ServerTemplates.  Chef allows us to abstract the business logic away from the details of the cloud,  making it easier to propagate the same solution to as many clouds as we can get our hands on.

Wait, each cloud has different profiles for instance types!

But Chef isn’t the only innovation that we use.  We also have to support many ‘by design’ cloud architecture differences. For example, pre-defined instance types.  Instance types in Amazon, while being the same across all regions within Amazon, do not align well to flavors in Rackspace.  Plus, in private clouds, you can either use the default instance types or custom configure to what your application demands.  How then does a specific ServerTemplate correctly configure a new instance for optimal performance?  Seems like that’s (yet another) real prerequisite for proper application setup.

In the specific case of MySQL, our ServerTemplate will auto-tune configuration parameters including innodb_additional_mem_pool_size and table_cache.  The tuning is based off of an instance’s available memory.  Of course this can be overridden on a per-server basis.  This mechanism extends well to our PHP App and Load Balancer ServerTemplates where you override parameter defaults specified in apache2.conf and haproxy_http.

All of this is just the tip of the iceberg.  Check out the ServerTemplates for yourself and let us know what you think.

Add Server and Add Server Array Assistants

First, we’ve created new assistants to simplify the process of creating a server or server array. It’s a big change, requested by our customers, designed to make your life easier in the day-to-day usage of the Dashboard. We worked with a few customers over the last few months to refine this flow, so we know it will be a welcome change. The previous process was getting a little disjointed after a few years of rapid cloud innovation! Learn more about the new assistants shown below.

Community Translations

Next, in May of this year, we launched our first language translation for the RightScale Dashboard: Japanese. Back then, we mentioned that we accomplished this through a platform we planned to make available to the community. That time is here. You will now notice a “Help Us Translate” link in the footer:

So how can you help us translate (and why would you)? First, the how. When you click on this link, you’ll be taken to a tool that will allow you to see all the phrases that need to be translated, translate these phrases, and vote on translations that others might have submitted. You can then link back to the Dashboard to see the translations in real-time! To get started, click on the link, read the instructions, and choose your language:

Now, why would you help us? Well, many of you have already offered out of the goodness of your heart, and we appreciate that. If you are someone that needs an incentive, we appreciate that too. That’s why we’re going to give the top translator for each of the following languages an Amazon Kindle: German, Chinese Simplified, Japanese, French, Spanish, and Korean. For how to get started, and for more information on this “Translation Showdown,” read the RightScale Dashboard Translation Guide.

New MultiCloud API

We’ve been incubating a new API with a few of our largest customers for over half a year now. This API is a complete redesign, and takes into account everything we have learned over the years on how to manage multiple clouds behind a single “pane of glass.” Or in this case, a single set of XML/JSON instructions.

We are making it available today as a public beta, supporting Cloud.com, Eucalyptus, and Rackspace. Not everything that is in API 1.0 is available in this new API yet, but it is burning a hole in our pocket, and will be extremely useful to our customers who want to begin automating their multi-cloud deployments. As we equalize the feature set between this new API and the 1.0 EC2 API, we will move AWS EC2 support into this new API and retire API 1.0.

One new feature here (available on all clouds) is the ability to provision and manage users via the API. You can now list all users in an account, add users, and set their permissions. Coming up in the October release, Enterprise plan customers will also be able to provision new accounts.

Learn more about the new API and how to get started with it.

Release Notes

As always, please read the Release Notes for a detailed list of changes made to the Dashboard and API.  Look out for the ServerTemplate & MultiCloud Image release next week – we have some great solutions coming up for both Linux and Windows cloud administrators.

Enjoy!

Security testing is one aspect of a security program that is often overlooked. Organizations who take security seriously understand that testing systems and applications is just smart business. We felt that one way we could help our customers is to describe the process, and nuances, that we go through during our testing. Since RightScale runs in the cloud, the information should help any RightScale customer accomplish the same tasks on their environment.

Our process is basically broken down into the following steps:

1. Identify instances and applications that will be tested
2. Select tools and systems that will be used to perform the testing
3. Coordinate with the cloud service provider to get authorization for testing
4. Execute the test
5. Communicate the results

Below I have outlined some of the practical details of each of these steps.

Identify Targets

Before we start testing, we identify what we want to test. For this particular test, we decided that we would include all of the systems that make up our platform, as well as the main dashboard application. Since we use RightScale to manage RightScale, and one of the main functions of our service is using ServerTemplates™ and RightScripts™ to ensure that systems are deployed consistently, there was a temptation to select a representative sample.

Since this was my first time testing RightScale since becoming the Director of Security and Compliance, we decided to test them all. We figured it is good practice, and provided a “validation” of sorts that we were following the practices we champion. We did however decide to limit the testing to publicly addressable AWS IP addresses. (Note: Anyone trying to be PCI compliant in AWS will likely need to test private IPs as well.)

As for the application, we decided on the entire dashboard, and not just a portion (mostly because I wanted a good overview to have as a baseline).

Select Testing Tools

Along with determining which systems/instances and applications we were testing, we selected tools that would help us automate the testing. We had agreed that a primarily automated vulnerability test (with manual validation) was acceptable, but that the application scanning would require a more manual approach given the complexity of our application. To that end, we had the following basic selection criteria:

• Vulnerability scanner: Number one criterion was its ability to appropriately identify vulnerabilities. We did not want a lot of false positives, but felt that false negatives would be much worse. A second criterion for the vulnerability scanner, was the flexibility of its reporting mechanism.
• Application testing: Number one criterion was our ability to use it, not what others think of it. A second criterion for the application testing tool was its ability to test against the framework of our application.

Given those “requirements” we chose three vulnerability scanners that we wanted to evaluate, in hopes of selecting one as the foundation for our ongoing testing program. Those were SAINT, NeXpose, and OpenVAS. Many will point out that there are other tools out there, and I agree, but these were tools I personally have history with, and one is free. We had to start somewhere.

As far as the application testing, I have used Burp Pro for a number of years and am a fan of it, and selected that as an application testing tool of choice. It should be noted that a number of other tools have recently come out that may rival Burp Pro in its functionality, but familiarity of use was important. We wanted to test the application, not the tool.

Where to Run Them?

Once we determined the tools that we wanted to use, we had to figure out where we wanted to run them:

• SaaS
• Instance in the same cloud
• Instance in a different cloud
• Traditional hosting environment
• Physical system on our network

We chose the “Instance in the same cloud” for a couple of reasons:

• Flexibility: We were able to install multiple tools to evaluate and test
• Eating our own dog food: RightScale is all about configuring and managing systems, so what better way for us to help our customers be able to deploy scanning systems than to do it ourselves
• Bandwidth cost: By using an instance within the same availability zones on AWS, bandwidth was not an issue
• Access to internal IPs: By running in the same cloud (AWS region) we can test internal IP addresses

Once we decided to build our own, we downloaded a trial version of SAINT, the community version of NeXpose, and followed the Ubuntu installation directions for OpenVAS. Then we wrote some RightScripts to automate the majority of the install and we were “cooking with gas” so to speak.

Get Authorization from Cloud Provider

Once we identified all our instances we were going to test, and had our testing sources (one in our case), per the AWS usage agreement, we needed to get authorization from AWS to perform the testing.
AWS provides a form that we filled out to request penetration testing of instances. We had to supply the AWS instance IDs and IPs that we obtained earlier, as well as the source of the testing. AWS uses this to create a ticket that AWS security team will get, and subsequently white list the account so the IDS systems are not triggering alerts during the testing. This prevents getting nasty emails about policy violation as well as port blocking, which would affect the test results.

AWS security responded back within a couple of days with approval for the scanning. It is interesting to note that it appears it is the vulnerability scanning that this applies to, for all intents and purposes you should make this request for application-based scanning as well, but it’s been my experience that testing the application does not cause abuse reports to be generated within AWS. During the testing, launching and relaunching of the scanner we did accidentally perform a number of scans from an IP address other than the one we provided to AWS and we did receive two abuse notices.

Probably the biggest point to note with respect to testing instances running in AWS is that instance size must be medium or greater. AWS policy does not allow pen testing, including port/service scanning, of smalls or below, presumably because they want to avoid that the testing degrades the other VMs on the same host. It should be noted, that we were just testing in AWS, depending on your cloud service provider, what you need to provide as far as what you are testing will vary. For AWS, we provided the instance ID as well as the public IP that will be tested, and the source of the testing.

For AWS, the quickest way to get the list of all AWS instance IDs and associated IPs is to use the rest_connection API. It can be used to programmatically generate a list of the instances and associated IP addresses that will be the targets of testing. We ignored the security groups in this test and hit all the “well known ports” that the tools scan. An alternative would be to only test the accessible ports.

Execute the Test

Once we obtained the authorization for the testing, we coordinated with the ops team to make sure they were ready for any potential problems. Once we got their “we are a go” signal, we commenced the testing. The general methodology looked something like this:

1. A sequential vulnerability scan, using each of the scanners. For both SAINT and NeXpose, we utilized the “exploit” portion of the tools (when it existed) on any noted vulnerability. (Note that we performed multiple scans with each scanner over the course of our 3 weeks of testing.)
2. General walk through and Burp Pro “passive” testing of the entire dashboard. Attempting to get an overall feel for the testing tool with the dashboard, and basically doing a full manual spider of the site.
3. Next we specifically performed testing of our session state mechanism, looking for entropy, manipulation, and injection flaws.
4. We then stepped through each of the dashboard’s main function areas, “Reports,” “Manage,” “Design,” “Clouds” and “Settings,” looking for well-known attack vectors. In particular focusing on identifying Cross Site Scripting and Request Forgers (XSS and CSRF), Injection, parameter manipulation, and other common web app exposures. See the OWASP testing guide for a good discussion of things that should be tested for in web applications.

Note that all testing we performed was done in both an authenticated state as well as an unauthenticated state.

As stated earlier, we made the decision that the vulnerability scanning portion of our testing would be mostly automated, and the application testing mostly manual. It took us approximately 3 weeks to identify the systems, get the authorization, and perform the testing. About 2 weeks of that was dedicated to the manual app testing.

A Bit More on the Application

It could be argued, that the bulk of “cloud” security testing should revolve around the application. This is not to say that making sure supporting services like Apache and MySQL versions are patched is not important (it is, just ask Sony), but meaning that much of the exposure to your data will come through the application. Taking the time to assess the mechanisms protecting the application is critical. For example:

• Are the security groups appropriate?
• Do you have appropriate controls on who can access API calls or make security related changes via the UI?
• Does your authorization mechanism enforce appropriate controls via all interfaces?

Items like these are things that will be critical for long-term protection of information. Make sure that you include them in your testing regiment.

Communicate Results

We are an Agile shop, so frequent communication is part of our culture, and we leveraged that to provide feedback from the testing to the appropriate engineering or ops teams as we uncovered potential threats. This allowed us to create records of our testing results, as well as provided timely information to be fed into our sprint process. At the completion of the testing, we wriote a summary report and included details of the vulnerabilities from each of the tools as appendices. Even though the information is already fed into the appropriate groups, including details along with the final report allowed stakeholders the ability to review the overall testing methodology and findings, as well as dig down into the details of any vulnerabilities found.

Your process may vary, and you may have a much more formal reporting requirement. The most important part is to get the appropriate information to the people who can get the system services or applications fixed in a timely manner.

Summary

The process of identifying targets, maintaining testing tools, coordinating with cloud service providers, and communicating those results should be formalized within your organization. Security testing should become an integral part of the IT culture. There will always be issues, as nothing is absolutely secure, but trying to stay ahead of the curve is a worthy cause. With a formal process, you can make it a regular occurrence, thus enhancing your security program and likely meeting many practical as well as compliance requirements.

One side note about the testing is that for all practical purposes, it was exactly the same methodology and tools that I have used previously in non-cloud environments. So I encourage you to roll up your sleeves and implement a testing program for your infrastructure and applications.

In it, you will find concrete examples of how to maintain advanced database architectures in the cloud, how to auto-scale Windows .NET applications, and even how to move database information between clouds. With your feedback, these examples will become production solutions that you can extend and modify. Before you know it, you’re a sysadmin rock star for your organization – people will wonder how you accomplish such magic.

Don’t hold back your secrets.

Rackspace ServerTemplate Support for Linux

On the heels of our Windows ServerTemplate Support for Rackspace, we’re elated to also bring you Linux ServerTemplate support – starting with CentOS. We have a few templates available for you to start with: a Base ServerTemplate that works across many clouds, a LAMP ServerTemplate with backups to CloudFiles, and a Database Manager for MySQL that takes snapshot backups to CloudFiles. These templates are all new and part of the public beta – please provide feedback and let your sales rep know if you’re experimenting.

With this release, you now have most of the Management (monitoring, user controls, auto-scaling, etc.) and Configuration (RightImages, ServerTemplates) aspects of our product available for Rackspace. We’re close to wrapping up our API support as well, and will be adding more RightImages, OSes, and ServerTemplates from here on out.

MultiCloud Magic

Let’s point out something important about the new Database Manager for MySQL that I mentioned in the Rackspace section above: it also works on Amazon.

This is one of our newer templates specifically designed to adapt to different clouds.  It is based off of another new MultiCloud ServerTemplate, the Storage Toolbox, which allows you to set up an LVM filesystem on instance drives or attachable volumes.  It also helps you take snapshot backups of your filesystem and upload it to the clouds object storage (in the MySQL case, to AWS S3 or Rackspace CloudFiles).

For the RightScale User Conference this week in New York, I demonstrated a database server running on the Rackspace Cloud, taking backups to Amazon S3, and restoring to a warm EC2 server.  I could have also gone the other way using Rackspace CloudFiles, or moved between Amazon regions using S3.

Magic? Nope. Here’s a tutorial so you can try it yourself.

CloudStack CentOS RightImages

Let’s move from MultiCloud to multi-hypervisor. With this release, CentOS RightImages are now available on all popular hypervisors for Cloud.com CloudStack clouds. We have RightImages for KVM, Xen Server, and VMWare ESX. Contact your sales rep for access to these private cloud images.

Database Manager for PostgreSQL

Many of you asked for a Database Manager for PostgreSQL 9 since replication issues from previous Postgres versions have been resolved. Well, the team took the structure of our MySQL Manager on Amazon and managed to replace MySQL with PostgreSQL – so you’re in luck! Full master-slave support, use of EBS volumes, assisted DNS failover, etc. Read the setup guide to get started.

Windows

Earlier this year, we released our first ever Database Manager for Microsoft SQL Server. We received great feedback, adding smart volume configuration with best practice disk configuration for system and user databases.  Now, not only are master, msdb and temp on EBS Volumes, but we create default locations so your database data and log files are directed to separate EBS volumes.  We coordinate simultaneous volume snapshots so we have sane and consistent backups that can be used for disaster recovery purposes with built-in restore RightScripts.  We also optimally configure SQL Server for you, enabling mixed authentication mode and creating an equal number of tempdb data files to the number of CPUs on the server. Check out the beta for our Manager for Microsoft SQL Server.

We’re also pleased to announce the Microsoft IIS Application Server, which can be used in an Array to serve as an auto-scaling .NET application tier. The ServerTemplate has built in Powershell-based RightScripts to register to either the AWS Elastic Load Balancer or to HAProxy.  In a similar vein to our other Application Servers in the MultiCloud Marketplace, this ServerTemplate will automatically download and deploy your application code and connect to a local or remote database server. Together with the Database Manager for Microsoft SQL Server, you can quickly get a multi-tier .NET app up and running in the cloud – get started with this setup guide.

Of course, both of these ServerTemplates are powered by RightScale RightImages.  We’ve enhanced our RightImages as part of this release to include support for Windows Server 2008 R2, bringing our total number of RightImages on Windows to 70!

There’s More!

We’ve added Nginx-based PHP and Rails Application servers too. To see the rest, please read the May and  June release notes for details and starting points.

Enjoy!

Citrix’s OpenStack announcement is especially notable given that it comes from the company that provides the hypervisor that has the longest history and powers more virtual servers than any other in the cloud today: Xen. So it will be interesting to see what it means to have a “cloud optimized version of XenServer” under the covers of Citrix’s OpenStack. That also brings another question to the forefront: what will it mean to have several flavors of OpenStack? Citrix uses the phrase “version of OpenStack”, Jim Curry (RackSpace) uses the phrase “OpenStack distribution”, and Barton George (Dell) also uses “OpenStack Distro.” It is clear they’re not just talking about a little packaging since, for example, Citrix states “Project Olympus will come pre-integrated with the Citrix Cloud Networking fabric.” In other words, it will have functionality different from ‘stock’ OpenStack.

From a selfish point of view I’m wondering how many versions or distros of OpenStack we will have to support and how compatible they will be with one-another? Of course, to be fair, other private cloud offerings also contain variants such as having multiple networking modes that differ substantially from one-another and that we support. This form of flexibility is a clear need. But for the larger community it will be intereting to see how things play out. At this point, my expectation is that this represents healthy differentiation and innovation in the OpenStack community, and we’ll continue to work with the various vendors to ensure we can support the architectures they’re implementing.

With the advent of these commercial OpenStack offerings, we’re witnessing a new emergence of reference architectures and an ecosystem of major players who can deliver complete IaaS solutions to enterprises and service providers who want to stand up and deliver clouds that can be managed by RightScale. Ultimately, that means more choice for our customers.

The best thing about working closely with our customers over the past four years is that we’ve been able to learn from your experience and use it to guide us in building an even better cloud management platform. So we thought it made perfect sense to focus this year’s conference around the theme Real Cloud Experience. Shared.   Our agenda is packed with sessions from the RightScale team, presentations from our customers, partners and from Forrester Research, Inc.  Come spend the day with us to find out what new stuff RightScale has cooking, gather insight from our customers and Forrester and attend in-depth breakout sessions from RightScale and our partners.  We’ll then cap the day with a RightScale hosted cocktail at the hip Gansevoort Hotel.

The final agenda is up and registration is open – check out all the sessions and get your free pass today!  As an added benefit, if you register for our conference you also get a free pass to the Cloud Expo happening that same week in NYC. Hope to see you there!

We’ve just put out a new release of the Dashboard, and with it, we’re expanding our ability to meet the needs of this global community.

First, the initial Japanese translation for the Dashboard is now available. With it, we built a foundation for many more translations to be provided to and by the community in the future. For now, you can switch to Japanese by toggling the language in the lower left hand corner of the Dashboard (thanks to our team in Japan for your hard work!):

Next, we’ve release a couple features that customers have been asking for. You can now associate AWS ELB and RDS services with RightScale Deployments so you can view your whole AWS system on one page. We’ve also made it possible to add Cluster Monitoring heat maps and stacked graphs to your Dashboard:

On that note, we’re making it even easier for our community to partner with us to create the best Cloud Management Platform in the world. Today, we launched feedback.rightscale.com, a place where you can submit and vote on ideas to improve the RightScale products. In addition, for any feature you show interest in, we’ll tell you the instant it becomes available.

We’ve seeded it with a few recent requests already, so start voting…

Finally, let’s not forget you can already submit your ServerTemplate ideas… as ServerTemplates! We’ve launched our second ServerTemplate Showdown, where you can win prizes by simply publishing the ServerTemplates you use everyday. This spring, the grand prize is a 4-day trip to Santa Barbara… bring your surfboard, not your laptop. Read more details about the ServerTemplate Showdown.

What’s not obvious in the release? We’ve made a number of improvements in the back-end to enable better scaling of our service and to allow us to reduce the impact of releases. This work will be ongoing and we hope to be able to show the benefits soon.

Read the full Dashboard Release Notes for a complete list of new features and changes in this release.

The outage was triggered by an operator error during a router upgrade which funneled very high-volume network traffic into a low-bandwidth control network used by EBS (Elastic Block Store). The resulting flooding of the control network caused a large number of EBS servers to be effectively isolated from one another, which broke the volume replication, and caused these servers to start re-replicating the data to fresh servers. This large-scale re-replication storm in turn had two effects: it failed in many cases causing the volumes to go offline for manual intervention, and it flooded the EBS control plane with re-replication events that affected its operation across the entire us-east region.

The steps taken by AWS to regain control started by stopping the re-replication attempts to quiesce the system and prevent new volumes from being drawn into the outage. AWS then isolated the affected availability zone from the EBS control plane to restore normal operation in other zones. Finally, AWS started to recover volumes by adding storage capacity to allow the re-replication to succeed where possible, by restoring data from snapshots on S3, and finally by manually restoring data. Ultimately 0.07% of the volumes could not be restored to a consistent state.

The Relational Database Service RDS was also affected by the outage. 45% of single-availability-zone databases in the affected availability zone went down because each database server stripes data across multiple EBS volumes with the result that one stuck volume halts the entire database. A number of multi-AZ  RDS databases whose  master server was in the affected zone failed to fail-over because of a bug in the fail-over process.

The post mortem lists a number of system improvements that AWS is working on. These primarily target improving the resiliency of EBS when replication fails as well as improving the tools created and used during the outage to recover from the situation. Customer communication improvements, especially regarding the frequency of updates, are also listed and AWS is crediting affected users a significant fraction of this month’s charges, this way beyond anything covered in its SLAs.

It is interesting to see how a network configuration error caused such a chain reaction within the EBS system. The outage trigger really is pretty incidental, a similar set of events could have probably been triggered by something else as well. The measures taken by AWS to contain and repair the outage highlight the deep technical expertise and full mastery of the entire software and hardware stack at AWS. Clearly deep code changes were made and sophisticated recovery tools were written 24×7 under the pressure of the outage, without which the situation most likely would have spun completely out of control.

The impact of the outage, the public reaction, and the measures necessary to control it show the scale at which AWS operates. It is pretty clear that this type of outage is part of growing the service to unprecedented scale. I find it amazing that this type of outage, where the sophisticated systems necessary to provide cloud computing at scale fail massively hasn’t happened years ago. This is a testament to AWS’s sophistication.

The outage summary exposes interesting technical details about the architecture of the services that AWS has kept confidential until now, however, more than providing information to competitors I believe that it provides education to cloud customers. All cloud providers who are planning world-wide cloud roll-outs absolutely must understand the power of and the need for availability zones in a region and isolation between regions (or equivalent constructs to “differentiate” from AWS). Without that redundancy and isolation, it has now become crystal clear: “how can we sell that to customers?”

An aspect of EBS durability which is not often mentioned is the role of snapshots during recovery. The EBS product description states “the durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot.” Here’s what this means. Suppose there are two copies of the volume (i.e. mirroring) and one fails, then a fresh mirror can fetch data contained in snapshots from S3 (which is itself replicated) but must retrieve other data from the single remaining copy, which may itself fail or become unreachable. Sadly the performance impact of taking a snapshot is such that most of our customers with high volume database cannot snapshot the master DB volume. Please fix that AWS!

An item missing from the remedies list in my opinion is EBS performance improvement. Better performance would have helped in the outage. Specifically I’d like AWS to reduce the impact of snapshots on volume performance so customers can actually snapshot high-volume servers and improve the performance of volumes so customers don’t have to stripe across multiple volumes which reduces availability (as it did with RDS).

I also am not satisfied with the communication improvements AWS proposes. I was fine with the frequency of status updates because it was clear that the EBS team was on top of it and didn’t have much new to report. I would like to see improved responsiveness so we don’t have to open a ticket before something shows up on the status page. But foremost I would like better content in the status updates. I’d like to be constructive, so I’ll make it concrete. Here is some of what I would have liked to see (I naturally have to make some assumptions about what was concluded when within AWS):

• explicit mention that the initial network event was contained, status updates kept talking about “increased latencies”, which made it unclear whether there was a general ongoing network issue
• clear statement that the outage revolved around EBS and noting the impact on launching servers from EBS images, but also stating that there was no impact on servers not using EBS
• clear statement that certain API calls were disabled instead of vaguely referring to “increased error rates affecting EBS CreateVolume API calls”
• timely reporting, e.g., the post mortem states “by 5:30 AM PDT, error rates and latencies again increased for EBS API calls across the Region” while the status updates only mentioned this at 7am
• the fact that the outage was due to failed EBS volumes as opposed to just connectivity or latency issues accessing the volumes was only reported at 8:54am, yet this is crucial piece of information
• the status updates never made it clear that EBS volumes continued to fail after the initial event, nor did they mention when this infection was halted
• the isolation of the other availability zones from the “affected one” was reported several hours after it was put in place
• it would have been useful to see some relative numbers, such as % of volumes deemed operational, % being recovered automatically soon, % slated for later manual recovery; best would have been emails to users with specific volume IDs

I’m sure that some of the items above weren’t quite as obvious at the time and in the heat of the moment it’s always difficult to determine what to say. But there is no question that the status updates were filled with vague terms, such as “increased latencies”, “moderate increase in error rates”, “affected availability zone”, “a network event”, etc. Perhaps foremost it’s not until 8 hours after the onset of the outage that AWS made it clear that volumes in the affected zone weren’t going to return to normal for hours to come. Up to that point it seemed that everything could return to normal any minute. This lack of clarity made it much harder for users to take the right decisions promptly.

On the public reaction front, while I understand it, I’m still baffled by reporters stating that the loss of 0.07% of volumes as not recoverable is a fundamental problem. This is equivalent to complaining about users losing data because their RAID array failed (happens all the time from operator error to 6ft drop due to earthquake). Users that lost data and were not aware of the risk they were taking need to seriously reflect on what they’re doing (and get help as appropriate).

This episode provides a key lesson to all cloud companies regarding architecting to withstand failure, and communicating with customers when failures do occur. While RightScale got through the outage relatively unscathed, we are working to improve on both those fronts ourselves. And we intend to continue to work with customers to enable AWS as well as other providers with independent, best-practice solutions that are resilient and highly available.

We’ve been working with our partners at Canonical to make it possible to use Ubuntu AMIs out of the box with RightScale. This means you can start playing with Natty Narwhal 11.04 in RightScale today!

How does it work?

The method was pioneered by Eric Hammond, who helps maintain Ubuntu and Debian AMIs for EC2 along with Scott Moser. An AMI is setup to fetch the user-data when an EC2 instance launches, and execute any scripts delineated by a special format. Canonical has incorporated this into the Ubuntu releases as their cloud-init software.

In the latest Natty Narwhal 11.04 release, RightScale’s RightLink software is now compatible with the Canonical cloud-init. Furthermore, Canonical’s officially supported AMIs for Natty Narwhal 11.04 are capable of configuring RightLink automatically.

We’ve created the following MultiCloud Images supporting all AWS regions:

• RightImage_OSS_Ubuntu_11.04_i386_v5.6.28
• RightImage_OSS_Ubuntu_11.04_x64_v5.6.28
• RightImage_OSS_Ubuntu_11.04_i386_v5.6.28_EBS
• RightImage_OSS_Ubuntu_11.04_x64_v5.6.28_EBS
• RightImage_OSS_Ubuntu_11.04_x64_v5.6.28_HVM

We’re sure this will benefit Ubuntu customers and partners, and we hope this method becomes more widely adopted by other virtual machine image creators.

Our hats off to the Ubuntu community for your continued advances in the cloud.

I will try to summarize what happened, what worked and didn’t work, and what to learn from it. I’ll do my best to add signal to all the noise out there, in that respect I liked a tweet by Beaker (Christofer Hoff): “Happy with my decision NOT to have written a blog about the misfortune of AWS, stating nothing but the obvious & sounding like a muppet”.

Executive summary

• The Amazon cloud proved itself in that sufficient resources were available world-wide such that many well-prepared users could continue operating with relatively little downtime. But because Amazon’s reliability has been incredible, many users were not well-prepared leading to widespread outages. Additionally, some users got caught by unforseen failure modes rendering their failure plans ineffective.
• Some ripple effects within EC2 and in particular EBS caused by the initial failure should not have happened. There’s important work Amazon needs to do to prevent such occurrences.
• Amazon’s communication, while better than during previous outages still earns an F. This is probably the #1 threat to AWS’s business.
• The cloud architecture provides ample opportunities to design systems to withstand failures. The material cost of such designs is a fraction of what comparative measures would cost using traditional hosting means. However, designing, building, and testing everything is not cheap. Many of our customers who used our best practices fared well (I’m not claiming we’re perfect or that everything is automatic!) and we got numerous calls from other companies that were wholly unprepared.
• Overall this is just one of many bumps in the cloud computing road. It reminds us that this is still “day one” of the cloud and that we all have much to learn about building and operating robust systems on a large scale. We are receiving a stream of calls from EC2 users that realize they need help in setting up a more robust architecture for their systems.

Outage analysis

At the time of writing Amazon has not yet posted a root cause analysis. I will update this section when they do. Until then, I have to make some educated guesses.

We got the first alerts at 1:01am on Thursday, the proverbial Christmas lights lit up indicating I/O issues on a large number of our servers. We started failing servers over and opened a ticket with Amazon. They finally posted a status message at 1:41am containing no useful details, sadly this is a typical sequence of events.

It appears that a major network failure was the initial cause of problems but that the real damage happened when EBS (Elastic Block Store) volume replication was disrupted. We did some extrapolations and concluded that there must have been on the order of 500k EBS storage volumes in the affected availability zone. It appears that a significant fraction of the volumes concluded that the replication mirroring was out-of-sync and started re-replicating causing further havoc, including an overload of the EBS control plane. It is also possible that the EBS replication problem was the root cause and that the network issues were a consequence, hopefully Amazon’s root cause analysis will shed light on this.

The biggest problem, from my point of view, was that more than one availability zone was affected. We didn’t see servers or volumes fail in other zones but we were unable to create fresh volumes elsewhere, which of course makes it difficult to move services. This is “not supposed to happen” and is an indication that the EBS control plane has dependencies across zones. Amazon did manage to contain the problem to one zone approx 3 hours after the onset.

After Amazon managed to contain the problems to one zone, it took a very long time to get the EBS machinery under control and to recover all the volumes. Given the extrapolated number of volumes it would not be surprising that an event of this scale exceeded the design parameters and was never tested (or able to be tested). I’m not sure there is any system of comparable scale in operation anywhere.

I do want to state that while “something large” clearly failed, namely the EBS system as a whole, the real big failure is that multiple availability zones were affected for ~3 hours. I also want to mention two important things that didn’t fail: we didn’t see capacity constraints in relaunching servers in other zones after the initial cross-zone issues and we didn’t see other regions affected at all. This is clearly good news!

Amazon communication failure

In my opinion the biggest failure in this event was Amazon’s communication, or rather lack thereof. The status updates were far too vague to be of much use and there was no background information whatsoever. Neither the official AWS blog nor Werner Vogels’ blog had any post whatsoever 4 days after the outage! Here is a list of improvements for Amazon:

• Do not wait 40 minutes to post the first status message!
• Do not talk about “a small percentage of instances/volumes/…”, give actual percentages! Those of us with many servers/volumes care whether it’s 1% or 25%, we will take different actions.
• Do not talk about “the impacted availability zone” or “multiple availability zones”, give each zone a name and refer to them by name (I know that zone 1a in each account refers to a different physical zone, so give each zone a second name so I can look it up).
• Provide individualized status information: use email (or other means) to tell us what the status of our instances and volumes is. I don’t mean things I can get myself like cpu load or such, but information like “the following volumes (by id) are currently recovering and should be available within the next hour, the following volumes will require manual intervention at a later time, …”. That allows users to plan and choose where to put their efforts.
• Make predictions! We saw volumes in the “impacted availability zone” getting taken out many hours after the initial event. I’m sure you knew that the problem was still spreading and could have warned everyone. Something like: “we recommend you move all servers and volumes that are still operating in the impacted availability zone [sic] to a different zone or region as the problem is still spreading.”
• Provide an overview! Each status update should list which functions are still affected and which have been repaired, don’t make everyone scan back through the messages and try to infer what the status of each function is.
• Is it so hard to write a blog post with an apology and some background information, even if it’s preliminary? AWS tweeters that usually send multiple tweets per day remained silent. I’m sure there’s something to talk about 24 hours after the event! Don’t you want to tell everyone what they should be thinking instead of having them make it up???

Coverage from around the web

Since Amazon did not communicate much of substance beyond the rather sparse and obscure status updates everyone else was left to speculate. Most of the blog posts or news articles contained little information. Here’s a list of blog posts that I found interesting:

• Amazon outage and the auto-immune vulnerabilities of resiliency by Lydia Leong at Gartner. An early post during the outage that has a good overall analysis.
• Amazon.com’s real problem isn’t the outage, it’s the communication by Keith Smith from BigDoor.
• AWS is down: Why the sky is falling by Justin Santa Barbara. One of the early posts with technical information.
• Standing on the shoulders of giants and stumbling with them – the Amazon AWS outage’s “pain” statistics by PagerDuty (a service we happily use) with nice stats about alerts going out during the outage.
• Cloud crash has a silver lining by Leon Katsnelson from IBM about DB2 and replication.
• nostromo comment on news.ycombinator.com (look for nostromo’s comment if it’s not at the top anymore), a brief description of the pain with RDS.
• Today’s EC2 / EBS Outage: Lessons learned by Stephen Nelson-Smith. One of the first good lessons learned posts I saw.
• Why Twilio Wasn’t Affected by Today’s AWS Issues has some interesting recommendations on how to architect for failure.
• The AWS Outage: The Cloud’s Shining Moment by George Reese of Enstratus has a very nice analysis of what designing for failure means and how it contrasts with more traditional approaches.
• Amazon’s Trouble Raises Cloud Computing Doubts on the front-page of the New York Times business section.
• Working around the EC2 outage by Jérôme Petazzoni of dotcloud with an interesting account of the issues they faced.

Lessons learned

Our services team handled 4x the incident volume last Thursday compared to a normal Thursday. A large number of callers needed help in assessing the situation or in bringing their servers back up. A typical request was: “It looks like my db server is down due to the outage, can you help confirm and assist with a migration?” Unfortunately we also heard from a good number of users who were using a single availability zone or didn’t set up redundancy properly. Hindsight is always 20-20.

A clear lesson for everyone is obviously that backup and replication have to be taken seriously (duh). In EC2 this means live replication across multiple availability zones and backups to S3 (and ideally elsewhere also). It has also become clear that a minimum of replicas must be running and a certain degree of over-provisioning is necessary to handle the load spike after a massive failure. Adrian Cockroft from Netflix summarized their strategy in a tweet a while ago: “Deploy in three AZ with no extra instances – target autoscale 30-60% util. You have 50% headroom for load spikes. Lose an AZ -> 90% util.” (Also see the discussion around the tweet.) Users that relied on launching fresh servers or on creating fresh volumes from snapshots were not able to do so for several hours. The only previous event that I remember where multiple availability zones were affected was the July 20th 2008 S3 outage that took down S3 in the US and EU (multiple regions!).

A number of blogs mention NoSQL databases as a solution to the replication and failure difficulties with traditional relational databases. While we’ve started to use Cassandra ourselves it has become pretty clear to me that this is not a silver bullet by a long shot. When a single node fails the built-in replication and recovery functions well, although the extra load on remaining nodes is high when the failing node is repaired and resynchronizes. But when large numbers of nodes in the cluster lock-up one-by-one over the course of an hour, I’d be hesitant to make a prediction about the outcome both in terms of the cluster’s availability and its consistency. We have two applications that make very different use of Cassandra and the behavior of the database is very different in both cases. My conclusion from what I have observed thus far is that clusters of replicated eventually-consistent NoSQL stores have pretty complex dynamics that can easily lead to unpleasant surprises. Sometimes it’s nice to have a comparatively simple MySQL master-slave set-up that experiences some downtime during the fail-over but acts very predictably.

I can’t help but feel uncomfortable about the performance of Amazon’s RDS “database-as-a-service” in that some databases that were replicated across multiple availability zones did not fail-over properly. It evidently took more than 12 hours to recover a number of the multi-az databases. The obvious failure here is compounded by the fact that Amazon has made it difficult for users to backup their databases outside of RDS, leaving them no choice but to wait for someone at Amazon to work on their database. This lock-in is one reason many of our customers prefer to use our MySQL master-slave setup or to architect their own.

The biggest lesson we learned abut operating RightScale itself is that we have to continue pushing hard on reducing the load on our central MySQL database and distributing our service. The database has grown too big and failover consequently takes too long because it takes forever to load the working set (over 30GB) into memory. We have some short-term measures we will be implementing to reduce the failover time, but more is needed. We also need to provide our users a choice of RightScale systems located in different regions and clouds: users operating primarily out of one region need to be able to use RightScale in an independent region or cloud. Ironically the first thing every public cloud operator and every company with a private cloud asks us is whether we can run RightScale inside their cloud: that seems pretty misguided to me!

We also were confused by Amazon’s status messages. In hindsight we should have intentionally failed-over our master database which was operating in the “impacted availability zone” early on at a time where we could minimize downtime. We were lucky that it didn’t get affected until about 12 hours after the start of the outage but we didn’t connect one and one. A clear message from Amazon that more and more volumes were continuing to fail in the zone would have been really helpful.

What’s next?

With Amazon’s overall stellar operating reliability it is easy to become complacent. This outage was a wake-up call for many of us. What remains to be seen is whether Amazon decides to take a lead and provide more granular descriptions of failure modes and recommended actions or whether they will leave it to everyone else to guess and figure it out. I see this as being one of the main long-term problems of cloud computing, namely that it is extremely difficult for users to list the possible failure modes and even more difficult to actually test any of them.

In the big picture I find Lew Moorman‘s analogy in the NYT article very appropriate: “The Amazon interruption was the computing equivalent of an airplane crash. It is a major episode with widespread damage. But airline travel is still safer than traveling in a car — analogous to cloud computing being safer than data centers run by individual companies. Every day, inside companies all over the world, there are technology outages, each episode is smaller, but they add up to far more lost time, money and business.” Most of the articles that predict a run away from cloud computing fail to explain where to run to. Unless you can hire superman to run your private datacenters my experience tells me that you’ll be worse off.

We’ve been working with Zend for a long time and it’s gratifying to see the Zend PHP Solution Pack offered on RightScale graduate to a full Zend PHP PaaS offering. Zend PHP is important to us because 37% of our customers use PHP, which is to be expected given that apparently roughly a third of the web runs on PHP and there are 4 million PHP developers! We’re also seeing our customers combine IaaS and PaaS, basically some portion of their overall system runs within a PaaS framework and then “punches out” to or combines with other services that don’t fit the PaaS mold. We believe this is where RightScale can really shine: provide the flexibility to deploy and operate a wide variety of services in the cloud.

The Zend PHP Solution Pack is a PaaS offering that consists of a multi-server cluster that includes a number of Zend application servers, a Zend cluster manager, load balancers and a MySQL master/slave pair.  This configuration – runnable today on Amazon Web Services and in the future on other clouds, including private clouds – provides a production-ready high availability environment that will auto-scale up and down as required. We worked with Zend to make it easy for you to stand up a standardized PaaS environment on whatever cloud you choose then deploy PHP applications to your heart’s content! The solution pack includes RightScale’s premium onboarding service – a step-by-step coaching program to deploying on the cloud using best practices backed by RightScale and Zend. Find out more on our website or attend the joint webinar next week thursday, April 28th, @ 11am PT: register here. Also, Andi Gutmans, Zend’s CEO, wrote a nice blog post on our joint offering.

CloudFoundry Architecture

From a technical point of view I see two main innovations in Cloud Foundry. The first is that the software is released as an open source project with an Apache license, which gives users and third-parties access to make customizations and to operate Cloud Foundry on their own. The second is that Cloud Foundry is very modular and separates the data path from the control plane, i.e. the components that make user applications run from the ones that control Cloud Foundry itself and the deployment and scaling of user applications. The reason the latter innovation is significant is that it really opens up the door to innovate on the management of the PaaS as well as integrate it into existing frameworks such as RightScale’s Dashboard.

Enough prelude, the pieces that make up Cloud Foundry are:

• At the core the app execution engine is the piece that runs your application. It’s what launches and manages the Rails, Java, and other language app servers. As your app is scaled up more app execution engines will launch an app server with your code. The way the app execution engine is architected is nice in that it is fairly stand-alone. It can be launched on any suitably configured server, then it connects to the other servers in the PaaS and starts running user applications (the app execution engines can be configured to run a single app per server or multiple). This means that to scale up the PaaS infrastructure itself the primary method is to launch more suitably configured app execution engines, something that is easy to do in a RightScale server array!
• The request router is the front door to the PaaS: it accepts all the HTTP requests for all the applications running in the PaaS and routes them to the best app execution engine that runs the appropriate application code. In essence the request router is a load balancer that knows which app is running where. The request router needs to be told about the hostname used by each application and it keeps track of the available app execution engines for each app. Request routers are generally not scaled frequently, in part because DNS entries point to them and it’s good practice to keep DNS as stable as possible, and also because a small number of request routers go a long way compared to app execution engines. It is possible, however to place regular load balancers in front of the request routers to make it easy to scale them without DNS changes.
• The cloud controller implements the external API used by tools to load/unload apps and control their environment, including the number of app execution engines that should run each application. As part of taking in new applications it creates the bundles that app execution engines load to run an application. A nice aspect of the cloud controller is that it is relatively policy-free, meaning that it relies on external input to perform operations such as scaling how many app execution engines each application uses. This allows different management policies to be plugged-in.
• A set of services provide data storage and other functions that can be leveraged by applications. In analogy with operating systems these are the device drivers. Each service tends to consist of two parts: the application implementing the service itself, much as MySQL, MongoDB, redis, etc. and a Cloud Foundry management layer that establishes the connections between applications and the service itself. For example, in the MySQL case this layer creates a separate logical database for each application and manages the credentials such that each application has access to its database.
• A health manager responsible for keeping applications alive and ensuring that if an app execution engine crashes the applications it ran are restarted elsewhere.

All these parts are tied together using a simple message bus, which, among other things allows all the servers to find each other.

Auto-scaling Cloud Foundry

“So, does it auto-scale”? seems to be the question everyone asks. (I wonder who started this auto-scaling business? ) The answer is “no, but trivially so”. There are actually two levels at which Cloud Foundry scales, whether automatically or not. The first is at the Cloud Foundry infrastructure level, e.g. how many app execution engines, how many request routers, how many cloud controllers, and how many services there are. The second level is at the individual application level and is primarily expressed in how many app execution engines are “running” the application (really, how many have the application loaded and are accepting requests from the request router).

The first level of scaling the Cloud Foundry infrastructure is the responsibility of the PaaS operator. The operator needs to monitor the load on the various servers and launch additional or terminate idle ones as appropriate. In particular, there should always be a number of idle app execution engines that can accept the next application or that can be brought to bear on an application that needs more resources. This level of scaling can be performed relatively easily manually or automatically in RightScale. The app execution engines can be placed in a server array and scaled based on their load.

The second level of scaling is the responsibility of each application’s owner. The nice thing about the modularity of Cloud Foundry is that it exposes the necessary hooks to adding external application monitoring and scaling decisions. It is also interesting that Cloud Foundry in effect exposes the resource costs and lets the application owner decide how much to consume–and pay for. This is in contrast to other systems that make it difficult to limit the resources other than by setting quotas at which point an application is suspended as opposed to simply running slower.

What we envision in working with Cloud Foundry is simple: RightScale will be able to monitor the various servers in the Cloud Foundry cluster, and determine for example when it’s “slack pool” of warm, ready-to-go app execution engines has dropped below a given threshold (or exceeded an idle threshold), and either boot new servers to add to the “slack pool” or de-commission unnecessary ones to save on cost, as appropriate.

PaaS and IaaS Synergy

The benefits of PaaS come from defining a constrained application deployment environment. That makes it necessary for many applications to “punch out” and leverage services outside of the PaaS framework. In some cases this may be a simple service, like a messaging server or a special form of data storage. In other cases it will end up being almost a reversed situation where a large portion of the application runs outside of the PaaS and the portions in the PaaS are really just complements or front-ends for the main system. Cloud Foundry makes it relatively easy to make outside services available to applications in the PaaS, but these outside applications still need to be managed. This is where an IaaS management framework like RightScale is great because it can bring the whole infrastructure under one roof.

Some examples for this punching out:

• Databases from the SQL variety to NoSQL and other models. Accessing legacy databases as well as leveraging popular DB setups like our MySQL Manager, which provides master slave replication.
• Different load balancers in front of the request routers, perhaps with extensive caching features, global load balancing, or other goodies. Examples would be Zeus, Squid and many others.
• Legacy or licensed software, for example video encoding software or PDF generators.
• Special back-end services, such as a telephony server.

If there’s one thing I’ve learned about customers at RightScale it’s the incredible variety of needs, architectures, and software packages that are in use. For this reason alone I see PaaS as another very nice tool in the RightScale toolbox.

Can you run Cloud Foundry without RightScale? Of course. It certainly runs on raw servers. They can PXE boot a base image and join the PaaS in one of the above server roles. However in a mixed environment it is much more beneficial to run the Cloud Foundry roles within a managed infrastructure cloud.

It seems obvious from the traditional SaaS/PaaS/IaaS cloud diagrams that these different layers were made to interoperate. And that’s what we’ve already seen our customers doing: combining PaaS and IaaS in ways that meet their needs. There are a number of PaaS solutions in the market with more on the horizon. We will continue to support as many as we can and to the extent that their architectures allow it, because cloud is a heterogeneous world and customers want choice. In the case of Cloud Foundry, we have a particularly open architecture that provides a compelling fit – and we’re excited to see where our joint customers take us together.

Advent of private PaaS

Until now the notion of PaaS has lumped together the author of the PaaS software and its operator. For example, Heroku developed its PaaS software and also offers it as a service. If you want to run your application on Heroku your only choice is to sign-up to their service and have them run your app. Google AppEngine has the same properties. All this is very nice and has many benefits, but it doesn’t fit all use-cases by a long shot. What if you need to run your app in Brazil but Heroku and your PaaS service doesn’t operate there? Or if you need to run your app within the corporate firewall? Or if you want to add some custom hooks to the PaaS software so you can punch out to custom services that are co-located with your app? All these options become a reality with Cloud Foundry because the PaaS software is developed as an open-source project. You can customize it and you can run it where you want to and how you want.

Of course you can also go to a hosted Cloud Foundry service whenever you don’t want to be bothered running servers. This could be a public Cloud Foundry service that is in effect competing with Heroku, AppEngine and others, but it could also be a private service offered by IT or your friendly devops team mate. This opens the possibilities for departmental PaaS services that may have a relatively small scale and can be tailored for the specific needs of their users.

Benefits of PaaS

PaaS is really about two things: simplicity of deployment and resource sharing. The way a PaaS makes deployment simpler is by defining a standard deployment methodology and software environment. Developers must conform to a number of restrictions on how their software can operate and how it needs to be packaged for deployment. Restrictions is perhaps the wrong word here, a set of standards is a better way to phrase it because just as some flexibility is lost a lot of benefits are gained out of the box. It’s similar to no longer writing applications that tweak device interfaces directly and instead have to go through a modern operating system device driver interface. In the PaaS context, instead of having custom deployment and scaling methodologies for each application there is a standard contract. This makes for much simpler and cheaper deployment and reduces the amount of interaction necessary between the teams that produce applications and those that run it.

Resource sharing is a second benefit of PaaS in that many applications can time-share a set of servers. This is similar to virtualization but at a different level. Where this resource sharing becomes interesting is when there are many applications that receive an incredibly low average number of requests per second. For example, a corporate app that is used once a quarter for a few days is likely to receive just a trickle of requests at other times. If virtualization were used then at least some virtual machines would have to be consuming sufficient resources to keep the operating system ticking, the monitoring system happy, log files rotating and a number of other things that are just difficult to turn off completely without shutting the VMs down, which may not be desirable for a number of reasons. In a PaaS the cost of keeping such applications alive drops down significantly.

PaaS running in IaaS — Cloud Foundry with RightScale

PaaS is sometimes believed to be at odds with IaaS, as if you have to choose one or the other. We believe in both models and CloudFoundry starts to fulfill that vision. RightScale enables Cloud Foundry to be deployed in a number of different configurations that vary in size, in location, in underlying cloud provider, in geographic location, or in who controls the deployment or pays for it

With RightScale it becomes easy to set-up a number of Cloud Foundry configurations for different use-cases. It is possible to set up a large deployment for many applications and really leverage the resource sharing benefits. But as some applications mature and have more stable resource needs and perhaps need to be separated from other to improve monitoring, resource metering, or allow for customization this can be easily accomplished by launching appropriate deployments. Finally, some applications may outgrow the capabilities of a PaaS environment and require a more custom deployment architecture.

Try it out!

We’ve created an All-In-One ServerTemplate in RightScale that launches Cloud Foundry in one server on Amazon EC2. If you do not have a RightScale account you can sign up for one free (you will have to pay for the EC2 instance time though). The ServerTemplate is called “Cloud Foundry All-In-One“. When you launch it, take a coffee, and come back, and you’ll be able to load your apps up! (Note that currently a lot of components are compiled at boot from the source repositories, so the server takes ~10-15 minutes to boot, we will be optimizing that as soon as the code base settles down a bit.)

I must say that this is one of the more exciting cloud developments in a while. I’ve been wanting to add good PaaS support to RightScale for a long time and Cloud Foundry is now making it possible. We’ve been talking to Mark Lucovsky about his secret project for many moons and it’s really refreshing to see the nice clean simple architecture he and his team (hi Ezra!)  have developed see the light of day. We’re now planning RightScale features around PaaS support so please let us know what you’d like to see from us!

NB: I had wanted to write about the architecture of Cloud Foundry and how it fits with RightScale ServerTemplates, but the timing was too tight. Stay tuned for a follow-on blog post in the next couple of days… Update: I did write the follow-up post Cloud Foundry Architecture and Auto-Scaling