These clouds have been in the works for a little while, and I’m pleased they are now available in the RightScale platform for our customers.  When we integrate with a given cloud, we work hard to ensure a seamless experience across all the clouds we support.  We provide a generic interface to each of the clouds integrated within RightScale.  This is not to limit functionality from the clouds themselves; but rather to ensure all that cool functionality is usable.  If I’m using SoftLayer and Datapipe, I don’t want to deal with different storage solutions like volumes or instance based storage (or at least not until I’m ready to optimize the storage).  Likewise, keep networking off my plate…I don’t care whether it’s security groups or ip tables.  Just make that infrastructure stuff work so that my app can run.

As a user, I want to  easily port what I have in one resource pool to another resource pool.  For this purpose, RightScale has generic constructs for things like instances, instance types, images, volumes, volume snapshots, etc, that are exposed in our dashboard.  Then, in our ServerTemplates (stay tuned by the way, a release is imminent), we use chef to abstract features for individual ServerTemplates that work, albeit very differently, across different resource pools.  Using the above example, someone launching servers in SoftLayer’s Amsterdam cloud and Datapipe’s Hong Kong cloud doesn’t have to worry about the differences between network configuration and storage management.  You can launch an entire 3-tier PHP architecture on both environments using ServerTemplates from the MultiCloud Marketplace.  We’ll take care of dealing with instance based storage in Amsterdam and set up the proper security groups for you in Hong Kong through the platform.

Why does RightScale spend so much time touting ‘MultiCloud’ and why should anyone care?

It’s a good question to ask actually.  I spend a lot of my time working with service providers and various companies looking to deliver infrastructure as a service for public consumption.   A number of people, our existing customers included, come to us and say “hey, I know I will have multiple clouds (if I don’t already)…help me make that happen.”  Analysts also agree – Forrester’s Holger Kisker touts “multi cloud becomes the norm” as his number 1 cloud computing prediction for 2012.

It’s real.  And it’s great validation for being the leader in ‘MultiCloud Management”.

Perhaps even more interesting (and contradictory if you think about it) is that the service providers say the same thing!  We describe how RightScale offers clouds to consumers and the choice consumers have to use what works best for their business needs.  And, IaaS providers are more than happy (okay, some take it as a challenge to deliver an even better service for their users. ).  In truth though, they recognize that cloud is a heterogeneous environment.  A single customer will use more than one cloud offering in a single environment.  Cost is one factor, and another I hear often is performance.  In some cases, geographic location is important and they “can’t get there with their current IaaS provider.”  It’s an opportunity for some to seize, and we’re partnering with the best to deliver the multi-cloud solutions our customers want.

Within RightScale, you can use any or all of the following clouds – all the Amazon regions, SoftLayer, Rackspace Cloud across US and UK, Datapipe, Logicworks as well as private cloud management with CloudStack and Eucalyptus.

I encourage you to click and try the new clouds on RightScale.  Use a new app or an existing one that’s already in cloud and as always, let us know what you think.

First, 3 million is impressive in the data center business.  Many well-known hosting companies house between 50,000 and 100,000 servers, and estimates for the world’s largest computer companies with large data centers range up to 1 million.  (See the DataCenterKnowledge report here.)  It’s difficult to compare our statistic with these installations, since many may be running largely under a pre-cloud operational model.  Nevertheless, launching 3 million is quite a number by any comparative metric, and there’s no question that it was achieved only with new levels of automation and dynamic configuration that are core to RightScale.

The second reason 3M is worth noting has to do with how fast we got there.   After our founding in 2007, it took us about 27 months to reach 1M, another 12 months to reach 2M, and then just 6 months to reach 3M.  That’s more than twice as fast for each subsequent 1M servers.  Likewise, one year ago in Sept. 2010, we had launched 1.5M servers – and we doubled in the last 12 months.

The third reason this milestone matters is that the servers our users launch have increased in power, and persist for a longer duration, as each month passes.  In fact, since January this year server runtime has increased on average 30%. So the trend is clear: companies are running “bigger iron” in the cloud — and keeping it running longer — than ever before.  Here is a graph of the size distribution we recorded this summer:

Certainly, the growth rate we’re tracking for the quantity, power and longevity of servers launched on RightScale remains quite healthy and mirrors the broad adoption of cloud services industry-wide. But equally important is the range of customers driving this growth, representing a wide variety of industries, use cases and services powered by RightScale on the cloud. For example, during the last year:

• media giant Pearson converted a traditional educational software offering to a SaaS based model that allowed faster onboarding of new customers;
• consumer goods company American Girl (a division of Mattel) launched their virtual world with a major advertising push behind it and sailed smoothly through the holiday season;
• online game company Zynga launched new games that consistently broke records;
• and companies like Amdocs and Trader Media spoke at our User Conference last June about new enterprise services launched on both public and hybrid clouds.

All of these RightScale customers contributed toward the 3M milestone, and we continue to be dazzled by the solutions they achieve using cloud infrastructure. We’re looking forward to the next million servers launched by our customers, and the amazing services they’ll power with them.

Security testing is one aspect of a security program that is often overlooked. Organizations who take security seriously understand that testing systems and applications is just smart business. We felt that one way we could help our customers is to describe the process, and nuances, that we go through during our testing. Since RightScale runs in the cloud, the information should help any RightScale customer accomplish the same tasks on their environment.

Our process is basically broken down into the following steps:

1. Identify instances and applications that will be tested
2. Select tools and systems that will be used to perform the testing
3. Coordinate with the cloud service provider to get authorization for testing
4. Execute the test
5. Communicate the results

Below I have outlined some of the practical details of each of these steps.

Identify Targets

Before we start testing, we identify what we want to test. For this particular test, we decided that we would include all of the systems that make up our platform, as well as the main dashboard application. Since we use RightScale to manage RightScale, and one of the main functions of our service is using ServerTemplates™ and RightScripts™ to ensure that systems are deployed consistently, there was a temptation to select a representative sample.

Since this was my first time testing RightScale since becoming the Director of Security and Compliance, we decided to test them all. We figured it is good practice, and provided a “validation” of sorts that we were following the practices we champion. We did however decide to limit the testing to publicly addressable AWS IP addresses. (Note: Anyone trying to be PCI compliant in AWS will likely need to test private IPs as well.)

As for the application, we decided on the entire dashboard, and not just a portion (mostly because I wanted a good overview to have as a baseline).

Select Testing Tools

Along with determining which systems/instances and applications we were testing, we selected tools that would help us automate the testing. We had agreed that a primarily automated vulnerability test (with manual validation) was acceptable, but that the application scanning would require a more manual approach given the complexity of our application. To that end, we had the following basic selection criteria:

• Vulnerability scanner: Number one criterion was its ability to appropriately identify vulnerabilities. We did not want a lot of false positives, but felt that false negatives would be much worse. A second criterion for the vulnerability scanner, was the flexibility of its reporting mechanism.
• Application testing: Number one criterion was our ability to use it, not what others think of it. A second criterion for the application testing tool was its ability to test against the framework of our application.

Given those “requirements” we chose three vulnerability scanners that we wanted to evaluate, in hopes of selecting one as the foundation for our ongoing testing program. Those were SAINT, NeXpose, and OpenVAS. Many will point out that there are other tools out there, and I agree, but these were tools I personally have history with, and one is free. We had to start somewhere.

As far as the application testing, I have used Burp Pro for a number of years and am a fan of it, and selected that as an application testing tool of choice. It should be noted that a number of other tools have recently come out that may rival Burp Pro in its functionality, but familiarity of use was important. We wanted to test the application, not the tool.

Where to Run Them?

Once we determined the tools that we wanted to use, we had to figure out where we wanted to run them:

• SaaS
• Instance in the same cloud
• Instance in a different cloud
• Traditional hosting environment
• Physical system on our network

We chose the “Instance in the same cloud” for a couple of reasons:

• Flexibility: We were able to install multiple tools to evaluate and test
• Eating our own dog food: RightScale is all about configuring and managing systems, so what better way for us to help our customers be able to deploy scanning systems than to do it ourselves
• Bandwidth cost: By using an instance within the same availability zones on AWS, bandwidth was not an issue
• Access to internal IPs: By running in the same cloud (AWS region) we can test internal IP addresses

Once we decided to build our own, we downloaded a trial version of SAINT, the community version of NeXpose, and followed the Ubuntu installation directions for OpenVAS. Then we wrote some RightScripts to automate the majority of the install and we were “cooking with gas” so to speak.

Get Authorization from Cloud Provider

Once we identified all our instances we were going to test, and had our testing sources (one in our case), per the AWS usage agreement, we needed to get authorization from AWS to perform the testing.
AWS provides a form that we filled out to request penetration testing of instances. We had to supply the AWS instance IDs and IPs that we obtained earlier, as well as the source of the testing. AWS uses this to create a ticket that AWS security team will get, and subsequently white list the account so the IDS systems are not triggering alerts during the testing. This prevents getting nasty emails about policy violation as well as port blocking, which would affect the test results.

AWS security responded back within a couple of days with approval for the scanning. It is interesting to note that it appears it is the vulnerability scanning that this applies to, for all intents and purposes you should make this request for application-based scanning as well, but it’s been my experience that testing the application does not cause abuse reports to be generated within AWS. During the testing, launching and relaunching of the scanner we did accidentally perform a number of scans from an IP address other than the one we provided to AWS and we did receive two abuse notices.

Probably the biggest point to note with respect to testing instances running in AWS is that instance size must be medium or greater. AWS policy does not allow pen testing, including port/service scanning, of smalls or below, presumably because they want to avoid that the testing degrades the other VMs on the same host. It should be noted, that we were just testing in AWS, depending on your cloud service provider, what you need to provide as far as what you are testing will vary. For AWS, we provided the instance ID as well as the public IP that will be tested, and the source of the testing.

For AWS, the quickest way to get the list of all AWS instance IDs and associated IPs is to use the rest_connection API. It can be used to programmatically generate a list of the instances and associated IP addresses that will be the targets of testing. We ignored the security groups in this test and hit all the “well known ports” that the tools scan. An alternative would be to only test the accessible ports.

Execute the Test

Once we obtained the authorization for the testing, we coordinated with the ops team to make sure they were ready for any potential problems. Once we got their “we are a go” signal, we commenced the testing. The general methodology looked something like this:

1. A sequential vulnerability scan, using each of the scanners. For both SAINT and NeXpose, we utilized the “exploit” portion of the tools (when it existed) on any noted vulnerability. (Note that we performed multiple scans with each scanner over the course of our 3 weeks of testing.)
2. General walk through and Burp Pro “passive” testing of the entire dashboard. Attempting to get an overall feel for the testing tool with the dashboard, and basically doing a full manual spider of the site.
3. Next we specifically performed testing of our session state mechanism, looking for entropy, manipulation, and injection flaws.
4. We then stepped through each of the dashboard’s main function areas, “Reports,” “Manage,” “Design,” “Clouds” and “Settings,” looking for well-known attack vectors. In particular focusing on identifying Cross Site Scripting and Request Forgers (XSS and CSRF), Injection, parameter manipulation, and other common web app exposures. See the OWASP testing guide for a good discussion of things that should be tested for in web applications.

Note that all testing we performed was done in both an authenticated state as well as an unauthenticated state.

As stated earlier, we made the decision that the vulnerability scanning portion of our testing would be mostly automated, and the application testing mostly manual. It took us approximately 3 weeks to identify the systems, get the authorization, and perform the testing. About 2 weeks of that was dedicated to the manual app testing.

A Bit More on the Application

It could be argued, that the bulk of “cloud” security testing should revolve around the application. This is not to say that making sure supporting services like Apache and MySQL versions are patched is not important (it is, just ask Sony), but meaning that much of the exposure to your data will come through the application. Taking the time to assess the mechanisms protecting the application is critical. For example:

• Are the security groups appropriate?
• Do you have appropriate controls on who can access API calls or make security related changes via the UI?
• Does your authorization mechanism enforce appropriate controls via all interfaces?

Items like these are things that will be critical for long-term protection of information. Make sure that you include them in your testing regiment.

Communicate Results

We are an Agile shop, so frequent communication is part of our culture, and we leveraged that to provide feedback from the testing to the appropriate engineering or ops teams as we uncovered potential threats. This allowed us to create records of our testing results, as well as provided timely information to be fed into our sprint process. At the completion of the testing, we wriote a summary report and included details of the vulnerabilities from each of the tools as appendices. Even though the information is already fed into the appropriate groups, including details along with the final report allowed stakeholders the ability to review the overall testing methodology and findings, as well as dig down into the details of any vulnerabilities found.

Your process may vary, and you may have a much more formal reporting requirement. The most important part is to get the appropriate information to the people who can get the system services or applications fixed in a timely manner.

Summary

The process of identifying targets, maintaining testing tools, coordinating with cloud service providers, and communicating those results should be formalized within your organization. Security testing should become an integral part of the IT culture. There will always be issues, as nothing is absolutely secure, but trying to stay ahead of the curve is a worthy cause. With a formal process, you can make it a regular occurrence, thus enhancing your security program and likely meeting many practical as well as compliance requirements.

One side note about the testing is that for all practical purposes, it was exactly the same methodology and tools that I have used previously in non-cloud environments. So I encourage you to roll up your sleeves and implement a testing program for your infrastructure and applications.

The outage was triggered by an operator error during a router upgrade which funneled very high-volume network traffic into a low-bandwidth control network used by EBS (Elastic Block Store). The resulting flooding of the control network caused a large number of EBS servers to be effectively isolated from one another, which broke the volume replication, and caused these servers to start re-replicating the data to fresh servers. This large-scale re-replication storm in turn had two effects: it failed in many cases causing the volumes to go offline for manual intervention, and it flooded the EBS control plane with re-replication events that affected its operation across the entire us-east region.

The steps taken by AWS to regain control started by stopping the re-replication attempts to quiesce the system and prevent new volumes from being drawn into the outage. AWS then isolated the affected availability zone from the EBS control plane to restore normal operation in other zones. Finally, AWS started to recover volumes by adding storage capacity to allow the re-replication to succeed where possible, by restoring data from snapshots on S3, and finally by manually restoring data. Ultimately 0.07% of the volumes could not be restored to a consistent state.

The Relational Database Service RDS was also affected by the outage. 45% of single-availability-zone databases in the affected availability zone went down because each database server stripes data across multiple EBS volumes with the result that one stuck volume halts the entire database. A number of multi-AZ  RDS databases whose  master server was in the affected zone failed to fail-over because of a bug in the fail-over process.

The post mortem lists a number of system improvements that AWS is working on. These primarily target improving the resiliency of EBS when replication fails as well as improving the tools created and used during the outage to recover from the situation. Customer communication improvements, especially regarding the frequency of updates, are also listed and AWS is crediting affected users a significant fraction of this month’s charges, this way beyond anything covered in its SLAs.

It is interesting to see how a network configuration error caused such a chain reaction within the EBS system. The outage trigger really is pretty incidental, a similar set of events could have probably been triggered by something else as well. The measures taken by AWS to contain and repair the outage highlight the deep technical expertise and full mastery of the entire software and hardware stack at AWS. Clearly deep code changes were made and sophisticated recovery tools were written 24×7 under the pressure of the outage, without which the situation most likely would have spun completely out of control.

The impact of the outage, the public reaction, and the measures necessary to control it show the scale at which AWS operates. It is pretty clear that this type of outage is part of growing the service to unprecedented scale. I find it amazing that this type of outage, where the sophisticated systems necessary to provide cloud computing at scale fail massively hasn’t happened years ago. This is a testament to AWS’s sophistication.

The outage summary exposes interesting technical details about the architecture of the services that AWS has kept confidential until now, however, more than providing information to competitors I believe that it provides education to cloud customers. All cloud providers who are planning world-wide cloud roll-outs absolutely must understand the power of and the need for availability zones in a region and isolation between regions (or equivalent constructs to “differentiate” from AWS). Without that redundancy and isolation, it has now become crystal clear: “how can we sell that to customers?”

An aspect of EBS durability which is not often mentioned is the role of snapshots during recovery. The EBS product description states “the durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot.” Here’s what this means. Suppose there are two copies of the volume (i.e. mirroring) and one fails, then a fresh mirror can fetch data contained in snapshots from S3 (which is itself replicated) but must retrieve other data from the single remaining copy, which may itself fail or become unreachable. Sadly the performance impact of taking a snapshot is such that most of our customers with high volume database cannot snapshot the master DB volume. Please fix that AWS!

An item missing from the remedies list in my opinion is EBS performance improvement. Better performance would have helped in the outage. Specifically I’d like AWS to reduce the impact of snapshots on volume performance so customers can actually snapshot high-volume servers and improve the performance of volumes so customers don’t have to stripe across multiple volumes which reduces availability (as it did with RDS).

I also am not satisfied with the communication improvements AWS proposes. I was fine with the frequency of status updates because it was clear that the EBS team was on top of it and didn’t have much new to report. I would like to see improved responsiveness so we don’t have to open a ticket before something shows up on the status page. But foremost I would like better content in the status updates. I’d like to be constructive, so I’ll make it concrete. Here is some of what I would have liked to see (I naturally have to make some assumptions about what was concluded when within AWS):

• explicit mention that the initial network event was contained, status updates kept talking about “increased latencies”, which made it unclear whether there was a general ongoing network issue
• clear statement that the outage revolved around EBS and noting the impact on launching servers from EBS images, but also stating that there was no impact on servers not using EBS
• clear statement that certain API calls were disabled instead of vaguely referring to “increased error rates affecting EBS CreateVolume API calls”
• timely reporting, e.g., the post mortem states “by 5:30 AM PDT, error rates and latencies again increased for EBS API calls across the Region” while the status updates only mentioned this at 7am
• the fact that the outage was due to failed EBS volumes as opposed to just connectivity or latency issues accessing the volumes was only reported at 8:54am, yet this is crucial piece of information
• the status updates never made it clear that EBS volumes continued to fail after the initial event, nor did they mention when this infection was halted
• the isolation of the other availability zones from the “affected one” was reported several hours after it was put in place
• it would have been useful to see some relative numbers, such as % of volumes deemed operational, % being recovered automatically soon, % slated for later manual recovery; best would have been emails to users with specific volume IDs

I’m sure that some of the items above weren’t quite as obvious at the time and in the heat of the moment it’s always difficult to determine what to say. But there is no question that the status updates were filled with vague terms, such as “increased latencies”, “moderate increase in error rates”, “affected availability zone”, “a network event”, etc. Perhaps foremost it’s not until 8 hours after the onset of the outage that AWS made it clear that volumes in the affected zone weren’t going to return to normal for hours to come. Up to that point it seemed that everything could return to normal any minute. This lack of clarity made it much harder for users to take the right decisions promptly.

On the public reaction front, while I understand it, I’m still baffled by reporters stating that the loss of 0.07% of volumes as not recoverable is a fundamental problem. This is equivalent to complaining about users losing data because their RAID array failed (happens all the time from operator error to 6ft drop due to earthquake). Users that lost data and were not aware of the risk they were taking need to seriously reflect on what they’re doing (and get help as appropriate).

This episode provides a key lesson to all cloud companies regarding architecting to withstand failure, and communicating with customers when failures do occur. While RightScale got through the outage relatively unscathed, we are working to improve on both those fronts ourselves. And we intend to continue to work with customers to enable AWS as well as other providers with independent, best-practice solutions that are resilient and highly available.

I will try to summarize what happened, what worked and didn’t work, and what to learn from it. I’ll do my best to add signal to all the noise out there, in that respect I liked a tweet by Beaker (Christofer Hoff): “Happy with my decision NOT to have written a blog about the misfortune of AWS, stating nothing but the obvious & sounding like a muppet”.

Executive summary

• The Amazon cloud proved itself in that sufficient resources were available world-wide such that many well-prepared users could continue operating with relatively little downtime. But because Amazon’s reliability has been incredible, many users were not well-prepared leading to widespread outages. Additionally, some users got caught by unforseen failure modes rendering their failure plans ineffective.
• Some ripple effects within EC2 and in particular EBS caused by the initial failure should not have happened. There’s important work Amazon needs to do to prevent such occurrences.
• Amazon’s communication, while better than during previous outages still earns an F. This is probably the #1 threat to AWS’s business.
• The cloud architecture provides ample opportunities to design systems to withstand failures. The material cost of such designs is a fraction of what comparative measures would cost using traditional hosting means. However, designing, building, and testing everything is not cheap. Many of our customers who used our best practices fared well (I’m not claiming we’re perfect or that everything is automatic!) and we got numerous calls from other companies that were wholly unprepared.
• Overall this is just one of many bumps in the cloud computing road. It reminds us that this is still “day one” of the cloud and that we all have much to learn about building and operating robust systems on a large scale. We are receiving a stream of calls from EC2 users that realize they need help in setting up a more robust architecture for their systems.

Outage analysis

At the time of writing Amazon has not yet posted a root cause analysis. I will update this section when they do. Until then, I have to make some educated guesses.

We got the first alerts at 1:01am on Thursday, the proverbial Christmas lights lit up indicating I/O issues on a large number of our servers. We started failing servers over and opened a ticket with Amazon. They finally posted a status message at 1:41am containing no useful details, sadly this is a typical sequence of events.

It appears that a major network failure was the initial cause of problems but that the real damage happened when EBS (Elastic Block Store) volume replication was disrupted. We did some extrapolations and concluded that there must have been on the order of 500k EBS storage volumes in the affected availability zone. It appears that a significant fraction of the volumes concluded that the replication mirroring was out-of-sync and started re-replicating causing further havoc, including an overload of the EBS control plane. It is also possible that the EBS replication problem was the root cause and that the network issues were a consequence, hopefully Amazon’s root cause analysis will shed light on this.

The biggest problem, from my point of view, was that more than one availability zone was affected. We didn’t see servers or volumes fail in other zones but we were unable to create fresh volumes elsewhere, which of course makes it difficult to move services. This is “not supposed to happen” and is an indication that the EBS control plane has dependencies across zones. Amazon did manage to contain the problem to one zone approx 3 hours after the onset.

After Amazon managed to contain the problems to one zone, it took a very long time to get the EBS machinery under control and to recover all the volumes. Given the extrapolated number of volumes it would not be surprising that an event of this scale exceeded the design parameters and was never tested (or able to be tested). I’m not sure there is any system of comparable scale in operation anywhere.

I do want to state that while “something large” clearly failed, namely the EBS system as a whole, the real big failure is that multiple availability zones were affected for ~3 hours. I also want to mention two important things that didn’t fail: we didn’t see capacity constraints in relaunching servers in other zones after the initial cross-zone issues and we didn’t see other regions affected at all. This is clearly good news!

Amazon communication failure

In my opinion the biggest failure in this event was Amazon’s communication, or rather lack thereof. The status updates were far too vague to be of much use and there was no background information whatsoever. Neither the official AWS blog nor Werner Vogels’ blog had any post whatsoever 4 days after the outage! Here is a list of improvements for Amazon:

• Do not wait 40 minutes to post the first status message!
• Do not talk about “a small percentage of instances/volumes/…”, give actual percentages! Those of us with many servers/volumes care whether it’s 1% or 25%, we will take different actions.
• Do not talk about “the impacted availability zone” or “multiple availability zones”, give each zone a name and refer to them by name (I know that zone 1a in each account refers to a different physical zone, so give each zone a second name so I can look it up).
• Provide individualized status information: use email (or other means) to tell us what the status of our instances and volumes is. I don’t mean things I can get myself like cpu load or such, but information like “the following volumes (by id) are currently recovering and should be available within the next hour, the following volumes will require manual intervention at a later time, …”. That allows users to plan and choose where to put their efforts.
• Make predictions! We saw volumes in the “impacted availability zone” getting taken out many hours after the initial event. I’m sure you knew that the problem was still spreading and could have warned everyone. Something like: “we recommend you move all servers and volumes that are still operating in the impacted availability zone [sic] to a different zone or region as the problem is still spreading.”
• Provide an overview! Each status update should list which functions are still affected and which have been repaired, don’t make everyone scan back through the messages and try to infer what the status of each function is.
• Is it so hard to write a blog post with an apology and some background information, even if it’s preliminary? AWS tweeters that usually send multiple tweets per day remained silent. I’m sure there’s something to talk about 24 hours after the event! Don’t you want to tell everyone what they should be thinking instead of having them make it up???

Coverage from around the web

Since Amazon did not communicate much of substance beyond the rather sparse and obscure status updates everyone else was left to speculate. Most of the blog posts or news articles contained little information. Here’s a list of blog posts that I found interesting:

• Amazon outage and the auto-immune vulnerabilities of resiliency by Lydia Leong at Gartner. An early post during the outage that has a good overall analysis.
• Amazon.com’s real problem isn’t the outage, it’s the communication by Keith Smith from BigDoor.
• AWS is down: Why the sky is falling by Justin Santa Barbara. One of the early posts with technical information.
• Standing on the shoulders of giants and stumbling with them – the Amazon AWS outage’s “pain” statistics by PagerDuty (a service we happily use) with nice stats about alerts going out during the outage.
• Cloud crash has a silver lining by Leon Katsnelson from IBM about DB2 and replication.
• nostromo comment on news.ycombinator.com (look for nostromo’s comment if it’s not at the top anymore), a brief description of the pain with RDS.
• Today’s EC2 / EBS Outage: Lessons learned by Stephen Nelson-Smith. One of the first good lessons learned posts I saw.
• Why Twilio Wasn’t Affected by Today’s AWS Issues has some interesting recommendations on how to architect for failure.
• The AWS Outage: The Cloud’s Shining Moment by George Reese of Enstratus has a very nice analysis of what designing for failure means and how it contrasts with more traditional approaches.
• Amazon’s Trouble Raises Cloud Computing Doubts on the front-page of the New York Times business section.
• Working around the EC2 outage by Jérôme Petazzoni of dotcloud with an interesting account of the issues they faced.

Lessons learned

Our services team handled 4x the incident volume last Thursday compared to a normal Thursday. A large number of callers needed help in assessing the situation or in bringing their servers back up. A typical request was: “It looks like my db server is down due to the outage, can you help confirm and assist with a migration?” Unfortunately we also heard from a good number of users who were using a single availability zone or didn’t set up redundancy properly. Hindsight is always 20-20.

A clear lesson for everyone is obviously that backup and replication have to be taken seriously (duh). In EC2 this means live replication across multiple availability zones and backups to S3 (and ideally elsewhere also). It has also become clear that a minimum of replicas must be running and a certain degree of over-provisioning is necessary to handle the load spike after a massive failure. Adrian Cockroft from Netflix summarized their strategy in a tweet a while ago: “Deploy in three AZ with no extra instances – target autoscale 30-60% util. You have 50% headroom for load spikes. Lose an AZ -> 90% util.” (Also see the discussion around the tweet.) Users that relied on launching fresh servers or on creating fresh volumes from snapshots were not able to do so for several hours. The only previous event that I remember where multiple availability zones were affected was the July 20th 2008 S3 outage that took down S3 in the US and EU (multiple regions!).

A number of blogs mention NoSQL databases as a solution to the replication and failure difficulties with traditional relational databases. While we’ve started to use Cassandra ourselves it has become pretty clear to me that this is not a silver bullet by a long shot. When a single node fails the built-in replication and recovery functions well, although the extra load on remaining nodes is high when the failing node is repaired and resynchronizes. But when large numbers of nodes in the cluster lock-up one-by-one over the course of an hour, I’d be hesitant to make a prediction about the outcome both in terms of the cluster’s availability and its consistency. We have two applications that make very different use of Cassandra and the behavior of the database is very different in both cases. My conclusion from what I have observed thus far is that clusters of replicated eventually-consistent NoSQL stores have pretty complex dynamics that can easily lead to unpleasant surprises. Sometimes it’s nice to have a comparatively simple MySQL master-slave set-up that experiences some downtime during the fail-over but acts very predictably.

I can’t help but feel uncomfortable about the performance of Amazon’s RDS “database-as-a-service” in that some databases that were replicated across multiple availability zones did not fail-over properly. It evidently took more than 12 hours to recover a number of the multi-az databases. The obvious failure here is compounded by the fact that Amazon has made it difficult for users to backup their databases outside of RDS, leaving them no choice but to wait for someone at Amazon to work on their database. This lock-in is one reason many of our customers prefer to use our MySQL master-slave setup or to architect their own.

The biggest lesson we learned abut operating RightScale itself is that we have to continue pushing hard on reducing the load on our central MySQL database and distributing our service. The database has grown too big and failover consequently takes too long because it takes forever to load the working set (over 30GB) into memory. We have some short-term measures we will be implementing to reduce the failover time, but more is needed. We also need to provide our users a choice of RightScale systems located in different regions and clouds: users operating primarily out of one region need to be able to use RightScale in an independent region or cloud. Ironically the first thing every public cloud operator and every company with a private cloud asks us is whether we can run RightScale inside their cloud: that seems pretty misguided to me!

We also were confused by Amazon’s status messages. In hindsight we should have intentionally failed-over our master database which was operating in the “impacted availability zone” early on at a time where we could minimize downtime. We were lucky that it didn’t get affected until about 12 hours after the start of the outage but we didn’t connect one and one. A clear message from Amazon that more and more volumes were continuing to fail in the zone would have been really helpful.

What’s next?

With Amazon’s overall stellar operating reliability it is easy to become complacent. This outage was a wake-up call for many of us. What remains to be seen is whether Amazon decides to take a lead and provide more granular descriptions of failure modes and recommended actions or whether they will leave it to everyone else to guess and figure it out. I see this as being one of the main long-term problems of cloud computing, namely that it is extremely difficult for users to list the possible failure modes and even more difficult to actually test any of them.

In the big picture I find Lew Moorman‘s analogy in the NYT article very appropriate: “The Amazon interruption was the computing equivalent of an airplane crash. It is a major episode with widespread damage. But airline travel is still safer than traveling in a car — analogous to cloud computing being safer than data centers run by individual companies. Every day, inside companies all over the world, there are technology outages, each episode is smaller, but they add up to far more lost time, money and business.” Most of the articles that predict a run away from cloud computing fail to explain where to run to. Unless you can hire superman to run your private datacenters my experience tells me that you’ll be worse off.

Let’s see what’s under the tree…

New Database Managers

Our Database Manager for MySQL is one of our most popular ServerTemplates, serving as the rock-solid foundation of many of our customer’s deployments. With this release, we’ve expanded our MySQL 5.1 manager to support CentOS as well as Ubuntu.

We’re also happy to bring this same foundation to our Microsoft Windows customers, with our new Database Manager for MS SQL Server. This new ServerTemplate automates the provisioning and operation of SQL Server, and includes scheduled backups, assisted restoration, and database monitoring. The template supports both SQL Server 2005 on Windows 2003 and SQL Server 2008 on Windows 2008.

Windows ServerTemplates on Rackspace

We have been working closely with Rackspace to expand our support to include ServerTemplates. We’re making Windows ServerTemplates available first. Our Base ServerTemplate for Windows includes RightImages for both AWS and Rackspace and can be launched on either cloud. This is a public beta, and we are very interested in your feedback – please let your account representative know if you plan to experiment with Windows on Rackspace.

Base and LAMP ServerTemplates for Cloud.com CloudStack

New CentOS 5.4 RightImages have been released for Cloud.com private cloud customers. We have also released a LAMP All-In-One ServerTemplate based on these images. The LAMP template uploads a backup of the database to either AWS S3 or Rackspace CloudFiles. Enterprise customers can contact their account representative for access to these images.

RightImages and ServerTemplates for EC2 AP-Tokyo

Following our Dashboard support for the new region, we have released a full set of RightImages (Ubuntu & CentOS) and ServerTemplates (Database Manager for MySQL, HAProxy Load Balancer, App Servers for PHP/Rails/Tomcat, and more) for EC2 AP-Tokyo.

New Free All-In-One Developer ServerTemplates

Developer templates are designed for you to quickly plug in your code & database and get up and running on the cloud with a single all-in-one server.

• LAMP All-In-One with MySQL 5.0 or 5.1
• WordPress All-In-One example
• Rails All-In-One with Nginx and Passenger
• Mephisto All-In-One example

Our First Compatibility Release

Our first “Compatibility Release” is now generally available (GA). What is a compatibility release, you might ask? Let’s break it down to two words: compatible and release. We test and release all of our most popular ServerTemplates and RightImages at the same time with the same software repository date. This helps ensure that RightScripts developed for one template and operating system should be compatible with other templates and operating systems that are also part of the release.

Why did we do this? We found that customers mixed and matched pieces from our ServerTemplates to create their own. Great! But without guaranteeing the same software repository or operating system version, their mileage varied. Now customers and partners can develop within a compatibility release, and take full advantage of all of the pre-built configurations we offer. You can see all of the ServerTemplates in the 11H1 Compatibility Release in the Library. (Hint to publishers: If you build a template within the Compatibility Release, be sure to add the “11H1 Compatible” category.)

New Base ServerTemplates

It’s actually easy for you to get most of the benefits of the compatibility release without even thinking about it. The trick is to start with one of our Base ServerTemplates for any custom ServerTemplate development. You’ll automatically get monitoring, the latest production RightImages, and the ability to pull compatible scripts from any related Compatibility Release templates.

and finally, while it was hard to wrap…

RightScale Experience and Expertise in the Cloud

About a month ago, a few of our customers started reporting sudden crashes of Ubuntu servers. RightScale worked closely with Canonical to identify the obscure kernel bug and brainstorm potential solutions. We then ran heavy tests on the Ubuntu 10.04 release with a variety of other kernels. Heavy testing in RightScale equates to multiple automated runs of hundreds of servers performing a variety of different workloads (database, load balancer, application server, etc.). We finally found a kernel that worked, and it is included with the latest Ubuntu RightImages.

With our current scale, we have the visibility and partnerships to identify and solve major cloud issues. We’re happy to package this expertise up for all of our customers as we continue to invest in the future of cloud computing.

Enjoy the new toys!

For a complete description of what was released, visit the Current Release Notes for ServerTemplates and MultiCloud Images.

We recently took a look at how our customers and ISV partners use ServerTemplates and just published some of our findings in a press release. Among the interesting facts that surfaced, our customers have developed a total of 42,500 RightScale ServerTemplates for Cloud Deployments as of 2011, doubling the number from the previous year. Of those, 42% of the customer-developed ServerTemplates were built from RightScale ServerTemplates and 58% were created from scratch or from partner templates. I was myself intrigued by the sheer number of ServerTemplates and looked a bit into what our users are doing. Of course a good number of the derived ServerTemplates come from customers making improvements to the ones we’ve published, and many of them are good fuel for our roadmap. But the vast majority of customer-created ServerTemplates are variations that are specific to the customer’s environment or have very customer-specific software. Looking at these numbers is really a good reminder that while most organizations use standard software, and most sysadmins try to adhere to standard ways for installing and operating this software, the reality for the leading companies is that customization is a vital necessity and RightScale supports just that.

The ServerTemplate methodology is also uniquely suited to supporting multiple cloud implementations. This ability to leverage our and our partners’ ServerTemplate investment has led to the just announced partnership with Tata Communications where we’ll support their cloud offering with our multi-cloud ServerTemplates. The Tata clouds are based on Cloud.com’s system, which means that customers will be able to deploy servers based on the same RightScale ServerTemplate in Amazon EC2, Tata’s InstaCompute cloud, as well as the customer’s in-house private Cloud.com cloud (if they have one). What really excites us in enabling this level of portability is that it leads to more business for everyone! Most of our larger EC2 customers have plans for other clouds, whether public clouds by other providers or private clouds, and in every single case the result will be more EC2 consumption because the overall goal isn’t to move EC2 servers back in-house but to enable more parts of the business to leverage cloud computing!

Dynamic Configuration

The dynamic configuration of servers through ServerTemplates is key to all this. The way we see a server coming together within RightScale is:

• The cloud provides a virtualization container, the “hardware,” with requested resources for processing, memory, disk, and network
• The cloud then boots a machine image which sets up the operating system and the RightLink agent
• RightScale runs the set of boot scripts identified in the server’s ServerTemplate, these perform several types of tasks:
  • install the full complement of software that the server needs to operate,
  • add additional hardware resources to the server, for example, disk volumes restored from snapshot, assignable ip addresses, etc, and
  • configure all the software and bring the server into an operational state
• RightScale starts collecting monitoring data and generates alerts according to the alert specifications of the ServerTemplate

Where we’ve innovated in the face of the new opportunities offered by cloud computing is in moving the boundary between what’s cast into the machine image vs. what’s done dynamically at boot time. Traditionally in the virtualization world there is little difference between an image and a server. An image is just a snapshot of a server, or perhaps its embodiment when the server is not running and has no hardware resources allocated to it. While this methodology works in the cloud as well, it doesn’t allow the benefits of the cloud to be leveraged.

How to Fully Leverage the Cloud

To fully leverage cloud architecture the role of images has to change. An image has to be the basis for launching and relaunching many servers. This is how auto-scaling happens, this is how failing servers get replaced quickly, this is how IT becomes agile and can spin up or rev servers. The key insight needed is that fewer things should be cast in the image and more needs to be customized on demand to each server’s role and configuration. The truth is that to leverage the cloud we have to recover the operating system as an abstraction layer!

What has happened is that the operating system, the application, and often also the application data are mushed together in machine images. Remember that one of the main roles of an operating system is to abstract the specifics of the hardware and present a portable machine abstraction to applications. We need to preserve this in the cloud so we can replicate and move servers and server configurations!

For example, as a server configuration moves from development to production the hardware (really the virtualization container) can change drastically. It may move from an internal dev cloud to an external production cloud or simultaneously to multiple internal and hosted production clouds around the world. Each time the operating system needs to be able to change while the stack on top of it remains the same. Some of the typical OS changes include: 32-bit vs. 64-bit platforms, uniprocessor vs. multiprocessor, slight changes in OS revision/configuration, changes in disk layout, hypervisor changes, etc.

The best way to recover the abstraction layer is to dedicate the machine image to contain only a cloud-optimized version of the operating system. This is exactly what we do with RightImages. These are base OS installs that we’ve been producing for years that provide effective abstraction across multiple clouds and virtualization containers. They consist of an optimized operating system install, the RightLink agent, and a few other very common supporting software packages. RightImages serve as the foundation upon which ServerTemplates dynamically assemble the rest of the software stack at boot time. This methodology lets us reliably stand up the same server configuration across a large number of clouds and machine types.

Often users get hung up about what happens at boot time versus what is already present on the image from a performance perspective. The performance of installing software at boot is a non-issue in most cases, but not always, and that doesn’t really break the model. Many of our customers create their own images that simply have more software pre-installed. Some good examples are pre-installing the version of Java a customer uses across many servers, or pre-installing SharePoint on Windows as it takes a long time to install. The advantage is that booting is faster, but the disadvantage is that a larger portion of the stack becomes more difficult to manage. It helps to remember that you’re effectively adding this software to the operating system abstraction, so to speak.

Keeping the application software separate from the operating system adds great runtime and maintenance flexibility. For example, you can test something out on a running server by adding a new script, making sure it works, committing the ServerTemplate, then rolling forward all running servers to that revision of the ServerTemplate to have it applied immediately. With bundled images, you would have to push the script to all machines manually and re-snapshot an image to use for future instances. Or, If you’re an ISV publisher of a solution on RightScale, and you make a simple bug fix to a script on a ServerTemplate, anyone using that solution can pull in the bug fix onto their existing servers where they may have added additional scripts and applications. When ISVs deliver their software through pre-bundled images, customers have to get the new image with any fixes, then reinstall and re-configure any applications and take a new snapshot – very tedious! In short, ServerTemplates are flexible like playlists on an iPod, where virtual machine images are like burning and shipping a CD.

RightScale Learnings

A related trend that we’ve found ourselves pursuing is to be more procedural and less declarative. At first blush a declarative template that states “this server needs X Y and Z” may seem better than one that cobbles things together in a script, but we’ve had relatively little success with that beyond simple demo examples. Disk volumes are a case in point. It may seem natural to specify in the ServerTemplate that it needs a 1TB volume, or perhaps even a volume restored from a specific snapshot (this is what the EC2 API supports). But in reality a server will need to restore data from a snapshot backup, generally the most recent consistent backup of the role the server is about to take on. This ends up involving listing recent snapshots, filtering for those pertaining to the role and that are tagged as being consistent (in many cases several volumes have to be backed up and are then tagged as consistent if all the snapshots complete correctly within the allotted timeout), and then selecting the most recent one. After attaching the volume, it needs to be mounted and appropriate recovery procedures often have to be run. All this ends up being procedural, and the best methodologies we’ve seen are to use configuration languages that don’t repeat work that has already been performed and are idempotent, such as Puppet and Chef.

Thoughts on AWS’s CloudFormation

In this context, it’s interesting that Amazon decided to use a declarative approach in the new CloudFormation (CF) templates. The CF templates are mostly at a different level of abstraction from ServerTemplates in that they focus on putting together multiple servers and their associated resources. We’ve been using our macro feature for that purpose for some time and have published a number of “getting started” macros to set up demo deployments as well as partner macros that setup clusters with software from multiple vendors. We’ve again found that past the simple examples, we really want to have procedural control over the setup of a server cluster, if only to be able to let the user input some parameters, like the number of app servers, and adjust how everything unfolds accordingly, or handle the cases where some resources already exist and should be used as-is.

Another area where we’ve taken a broader approach: CloudFormation really only targets setup (of course that’s the most juicy part where $$ start flowing if you’re the underlying cloud provider ). Servers also need actions performed on them during operation and often their roles and complexity are no different from initial setup. For example, when you run a RightScale operational script to re-mount the latest production snapshot on a staging database, that’s really no different at the core from what happened when the server originally launched. Or, if you choose to relaunch the staging server to get the latest snapshot, then the operational script to reconnect all the app servers to the new database is the same as the app server to database connection when all the servers were originally launched. A dynamic runtime configuration system is needed really at all three stages of a server’s lifespan: launch, runtime, and decommissioning.

What I hope becomes clear through these examples is the value of the integrated approach that RightScale offers. We see tremendous leverage in integrating all the pieces that developers and sysadmins need to architect, deploy, manage, track, audit, and recycle servers in one platform. We are leading the cloud industry in that respect, and we also still have a lot of cool new features under development in the pipeline!

First the upshot: across the board we now support Windows 2003 and 2008 on par with Linux. We provide RightImages that work well with RightScale out of the box, they can be managed in the dashboard, we support extensible monitoring, associated alerts and automation, and we support ServerTemplates with Windows and even provide a few sample templates. Plus, if you have existing images, we provide a RightLink installer with documentation and a starter ServerTemplate that gives you monitoring out of the box. This will give you a good dose of the RightScale love (monitoring, scaling arrays, user access control, etc.).

The mountain of work that we’ve been chipping away at for the past two weeks is being able to release 40 Windows RightImages. This work wasn’t what we had imagined, it crept up on us little by little. When we release RightImages, whether it is for Windows or Linux, we strive to provide a consistent software environment across the RightImages of a generation and to ensure they work smoothly with RightScale. Both the Linux and Windows RightImages work great without RightScale as well and many EC2 users have been using Linux RightImages for a long time. Here is what we had to do for the Windows RightImages:

• install RightLink for integration with RightScale
• ensure the images are automatable, meaning psexec can be used, which also involved getting consistent settings for the windows firewall, file sharing and a few other details
• install PowerShell 2.0 on all images, including the Windows 2003 ones, because according to everyone we’ve talked to it’s a must
• ensure all the right DLLs are installed to manage SQL server from PowerShell
• increase the size of the root disk for the 2008 images because the std 40GB are insufficient for a lot of interesting software installs (SharePoint being one example), and increase the pagefile to 1.5x ram size (up to 8GB)
• ensure the server’s clock is successfully synchronized to NTP at boot and periodically thereafter
• install logic to determine when the server is fully ready after boot before starting any automation, which is not as simple as it may sound due to sysprep, double boot, and an ec2-specific service that runs at boot and sets a few things up
• ensure complete install of asp.net on Windows 2003

This list kept growing as we discovered issues. In addition, we are supporting all four EC2 regions (us-east, us-west, eu-west, ap-southeast) and it turns out we have to build each image from scratch in each region because it’s not possible to copy a Windows image from one region to another (licensing restrictions). We are supporting 10 images in each region:

• Windows 2003 /2008 across i386/x64
• Windows w/SQLServer Express 2003 /2008 across i386/x64
• Windows w/SQLServer 2003 /2008 on x64

Getting to the final images involved a fair number of iterations (we don’ know how many images we built total, and frankly, we’d rather not be reminded) so we built automation to crank out the images. This makes them consistent and reproducible. Unfortunately, the AWS images we had to start from are not all automatable, so we also had to build some “intermediate” images by hand with just a few settings tweaked so we could target them with our automation. Some of our developers ended up with nightmares about clicking through endless circular install dialog boxes!

While we were launching lots of Windows instances for testing we noticed that a fair number had their clock off by a large amount, like days. Digging into the issue we discovered that, unlike with stock AWS Linux images, the Windows wall clock is not synchronized to the virtualization host’s time and that the initial NTP synchronization doesn’t always succeed, partially because the Windows NTP service is “challenged” but also because it is pointed at a public pool of servers. Since performing automation on a server that thinks it’s yesterday or tomorrow is a non-starter we concluded that we had to beef up the time synchronization by ensuring that we get an NTP sync before proceeding with any automation and also run our own set of NTP servers so we can ensure our customers always have in-cloud NTP servers available to synchronize with. We think this really is part of the famous “muck / undifferentiated heavy lifting” that Amazon prides itself to take care of, but they politely declined, which is really a shame. We also noticed that the new HVM Linux images don’t lock their clocks to the host, so we will probably switch the way the clock sync works across all RightImages.

Another interesting issue we discovered during testing was that automation at boot time would frequently fail. The root cause ended up being that our service, which was configured to launch at boot, was starting too early and that the server just wasn’t ready yet. The way Windows boots in the cloud is quite different from Linux or from the “normal” world. With Windows, each OS install generates a server key that is embedded in the registry and uniquely identifies the server, so when an image is booted many times a fresh key needs to be generated for each instance and a number of things need to be updated (“sysprep”) and the server rebooted once. Towards the end of the second boot a special Ec2Config service finalizes the config, including admin password and hostname. This means that any automation has to wait for the reboot plus the ec2 service to complete its changes, which is not trivial due to an oversight by AWS. It’s interesting how the security details of Windows (i.e. the server key) ripple down into the whole boot process, making it take twice as long as it should. The net is that while Linux instance boot times on EC2 have come down from a typical 6-8 minutes back in 2006 to under a minute now when using EBS images the Windows boot times are starting out around 10-15 minutes. Hopefully Microsoft can be sensitized to the notion that fast boot times are an important asset in the cloud because they enable a lot of automation that is very painful if one has to wait so long for additional capacity or replacement servers to come online.

The final and biggest piece is implementing all the automation support we offer in Linux for Windows as well. As part of this we ported Chef to Windows (we’re working with Opscode to feed the changes back into the mainline) and we built out support for PowerShell. This means that software can be installed and configured on Windows servers similar to the way it’s done in Linux. We find that larger software packages often need to be installed manually, but even in that case it’s nice to be able to automate as much as possible and choose which portions to run “ahead of time” and bake into a custom image and which portions to leave off for the actual server launches. To round out the automation we wrote a nice little monitoring plugin that speaks the collectd protocol and that offers a simple meta-language that can be used to query many of the WMI statistics available on Windows servers. And all this is available out of the box to all our customers.

One of the open question we have concerns Windows updates. We will most certainly republish fresh RightImages when AWS updates its base set. What is not clear to us is whether we should be publishing “fully updated” RightImages on a regular schedule. Most of the customers we asked told us that they really want to carefully manage the exact set of updates on their servers. It’s going to be interesting to see how the whole update management with Windows servers plays out and how it will affect the amount of image rebuilding that everyone will have to do. We will definitely build our ServerTemplates directly on RightImages or RightImages augmented by just base installs of larger software packages and do all the config using Chef/PowerShell. This way we minimize the work we have to do when we swap out the underlying Windows install or update level.

I must say that overall I’m very happy with where we’re ending up, which is that we’re getting Windows to a point where it definitely is usable in cloud-style. What I mean by that is not just migrating a set of traditional servers into a equivalent set of servers in the cloud, but rather automating Windows servers for the cloud and leveraging the flexibility of the cloud to enable the business. The friction along the way certainly is higher than with Linux, whether it’s from license questions that crop up everywhere to the mechanics that currently require double-booting, but it is totally possible and Microsoft can, if it focuses on it, make it a lot better yet!

Last week we released RightLink install packages for Linux and Windows enabling our users to take almost any image and RightScale-enable it. Most of you are probably well aware that I’m not a fan of creating images: it’s a slow process that results in hard to maintain monsters. This hasn’t changed with the RightLink installer, but what has changed is that we wanted to make it easier for our partners to build RightScale support for their flavor of Linux and we wanted to make it easier for users that already have servers up in the cloud to benefit from RightScale. But we didn’t just put the RightLink packages out in the wild, we also published the ServerTemplates we use to build our RightImages. We’ve been publishing these images since 2007, the idea being that we provide a clean base OS install for CentOS and Ubuntu. In RightScale these serve as the base boot image on top of which custom software gets installed and configured dynamically at boot time. What has set our images apart is that we built them automatically and have published the scripts we used for that purpose. This means they’re clean and reproducible. More recently we’ve used ServerTemplates to build the images, basically we boot a server that then builds and saves an image as it comes up. The benefit of using ServerTemplates to build images is that they are easy to maintain and 100% automated, they also make it easy to understand what went into an image later on. This is an excellent framework for anyone else to customize and build their own special image. But I won’t let an opportunity pass by to recommend that you save yourself a lot of time and grief and simply use the RightImages we already built and maintain!

Last but not least we’ve introduced public beta support for Windows in RightScale, and that’s not just launching AMIs, it’s the full ServerTemplate machinery and associated automation, including initial monitoring support. Our sales guys all asked which of our standard features we don’t support for windows… the response is “uhhh, syslog consolidation?”. I’m really interested in seeing how the usage pattern of Windows deployments will differ from Linux. I assume that Windows folks will have to be much more in the image bundling business to create base installs of software packages that are too slow to install at boot time. But I also wonder about things like auto-scaling because it still takes a Windows server a lot longer to boot up. We may have to play more games with stopping and starting servers instead of booting them cold. I’m sure our users will tell us!

Jeff Barr wrote an article describing SNS, the key points are:

• you can create topics and publish messages to these topics
• others can subscribe to the topics and they will get messages pushed to them
• messages can be pushed over http or smtp (email)
• using access control policies one can control who is allowed to subscribe to a topic
• the SNS system is redundant and retries message delivery if necessary
• the cost at volume is $0.06 per 100,000 messages published and $0.06 per 100,000 HTTP message pushes. Email pushes cost 33x more. Good news is the first 100,000 message publish and pushes are free.

It really is great that AWS provides such a service. It’s relatively easy to fire up a messaging server, like RabbitMQ, but it’s a different story to set up a redundant scalable messaging system. While this can be done with RabbitMQ, for many users having this provided as a service is the right way to go.

Unfortunately SNS does not use a standard messaging API, it’s all proprietary. This is a major weakness of SQS, SNS, and SDB: once you use their interface you’re locked in to using AWS. Granted, the SNS interface isn’t particularly big, but then why did they have to roll their own?

My biggest beef with SNS is what is being said, or more precisely, not being said about reliability. I have no reason to believe that SNS doesn’t do all the right things, but AWS isn’t very forthcoming with specifics. Here is what the SNS docs state:

• “Reliable – Amazon SNS runs within Amazon’s proven network infrastructure and datacenters, so topics will be available whenever applications need them. To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple servers and data centers.” source
• “Although most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end.” source

So there is talk about redundant storage, at least once delivery, and delivery retries. But what I’d really want to know is not all this fuzzy feel-good stuff. The question is not that difficult:

• If SNS returns a HTTP “200 OK” to my publish request, what is the probability that each subscriber will receive at least one delivery attempt?

ok, I guess I really need to factor in time, which would also give an indication of performance:

• If SNS returns a HTTP “200 OK” to my publish request, what is the probability distribution over time that each subscriber has received at least one delivery attempt?

This would let me reason about what I can use SNS for and what not, or whether I need a back-up synchronization mechanism or not, etc. If the story stays at the warm&fuzzy level, AWS could at least specify when messages are stored redundantly, e.g. is a redundant copy stored by the time I get an HTTP “200 OK” response? Also specifics about how long and how often retries are made. (I’m focusing on the HTTP delivery, I don’t think it makes much sense talking about email delivery reliability.)

I hope that many others will also ask AWS to be more specific about what in the end is really the SLA offered by SNS (same goes for some of the other AWS services). I’m not asking for damages if the SLA isn’t met, I just want to know what AWS is publicly holding itself accountable for and thus what I can design for and apply my “trust in AWS” judgement to. While the service is in beta, the SLA might be a target.

The way it works is that from the account you use for billing you can send invites to others so they can add their account to your billing method. They can still see their bill, it just doesn’t hit their credit card, it hits yours. You get to see what they used too, since you’re paying for it. I created a new account that I’m not using for any service at all and consolidated all my “real” accounts onto it. This way I can pretty freely hand out the credentials in finance & admin so anyone there who wants to see the numbers can log in without having any power over any instances, buckets, or whatnot.

One of the nice benefits of consolidated billing is that you can share some of the savings of reserved instances across accounts, but the details are pretty weird. The part that makes sense is that if account A doesn’t use a reserved instance “slot” then another account B can make use of it and get the discounted hourly rates. Where it gets weird is that if account A purchased a reserved instances in zone us-east-1a then account B gets the benefit also in zone us-east-1a. While this may sound logical it makes no sense because the zone names are permuted between accounts, so A’s 1a is typically different from B’s! Amazon: so why do we need to buy reserved instances in an availability zone as opposed to a region since it clearly doesn’t really matter???

I’ve actually always been unhappy with reserved instances. They seem to combine two completely different notions that I believe don’t go together because the use-cases are disjoint. One notion is that of a “revenue commit” or “buying into a discount tier”: you pay some money up-front to get a lower rate. Makes perfect sense, but why does it have to be tied to an availability zone? Or even to an instance type for that matter? The second notion is that of guaranteed availability, i.e., your reserved instance slot is always guaranteed to be available, you won’t get an “insufficient capacity” error. I understand why that has to be for a specific zone and type.

The reason the two notions don’t mix for me is that in order to get the discount benefits you have to run your instance virtually all the time, it really only is advisable for instances that always run. Well, in that case the whole notion of reservation is moot since you’re running all the time anyway! If you’re looking for the availability guarantee it’s generally because the instance is *not* running all the time and you want to make sure that the day you need it you can get it. Think disaster recovery scenario. Well, in that case the whole “discount” is moot, since you’re like to actually pay more due to the up-front reservation cost! I wish I could just buy the discount without the availability guarantee and without the tie to a specific zone!

One thing I noticed is that most of our larger customers are not using reserved instances. I wonder why…

How it works is simple yet complex. You can read the official product page,  Jeff Barr’s blog, and Werner’s blog. Here’s my attempt at explaining it. AWS publishes a spot price for each instance size in each region. The spot price is the per-hour cost of a server and if you launch a spot price server now that’s what you pay for the next hour. So instead of $0.10/hr for a small server you might only pay $0.03/hr if that’s the current spot price. AWS adjusts the spot price periodically based on the idle capacity available, so the price might be low at night or week-ends when many sites auto-scale down and it might be high during the day when everything is busy.

Now comes the complex part. You don’t just launch a spot instance and forget about it, you actually specify a maximum price you are willing to pay and for each hour you have your server running you pay the spot price current at the start of the hour. As the spot price continues to vary while your instance is running this maximum becomes very important because should the spot price exceed your maximum then your instance will be terminated by AWS! It’s also possible to work the maximum price in reverse: specify a price lower than the current spot price in the evening and your request stays queued until the spot price drops below what you specified and AWS then automatically launches your instances. You can revise your maximum at any time, so if at 4am the spot price has not dropped enough you can raise your max so your instances get to run before sunrise.

It should be clear from the way the spot pricing functions that this is intended for transient compute capacity. For your database instances you should carefully stay with the on-demand or reserved instances, but for late night batch jobs where it doesn’t matter whether they run a bit earlier or later the spot pricing can save quite some money.

One thing that is not obvious at the outset is what would motivate Amazon to keep the price down. Part of the answer lies in the fact that instances whose max bid drops below the current spot price get terminated, thus if the price goes up too much, too many instances get terminated which results in less revenue. So there is a balance between more instances at a lower price and fewer instances at a higher price. But I’m sure it’s a lot more complex than that.

We will be supporting spot pricing in the RightScale platform over the coming months and we’re curious about the functionality our customers would like to see in that respect. There are a lot of opportunities for automation here!

Amazon introduces US west coast cloud

Almost exactly a year after the first geographical expansion of EC2 to Europe today is the second big step to the west coast. What is notable about the EC2 architecture is that each one of these expansions constitutes a new cloud or “region” in EC2 speak. This means that now in addition to the US-EAST-1 and EU-WEST-1 regions we have a new US-WEST-1 region. Each region operates autonomously from the others in order to provide failure isolation, which has benefits as well as downsides. A major benefit is obviously the redundancy one can get by operating in more than one region or placing DR in a region other than the one used for one’s primary service. The downside is that sharing across regions is not as easy as one might imagine. For example, machine images (AMIs) are not shared, so for each image you’re using in one region you have to copy and re-register the image in the other, and then it has a different id you need to keep track of and reference. We didn’t plan it this way, but our multi-cloud support turns out to be very helpful in managing operations in multiple EC2 regions. For example, in RightScale you can define ServerTemplates that use different images in different clouds, this means that as you update your ServerTemplate it automatically works across clouds and thus EC2 regions.

For redundant operations the comparison between the cloud and DIY datacenters is becoming ever more lopsided. Who can really afford to lose the man-hours, the cap-ex, the time-to-market, and incur the headaches it takes to set-up a datacenter from scratch, even if it’s in a traditional colo? And who can afford to go through all that again to set-up a second or DR site? The ease with which it is now possible to set-up a DR site in the cloud that is a faithful replica of the primary site is really remarkable. And the best is that the second site can be extremely low cost because very little needs to be running there: most of it can be fired up on-demand in the case something happens. If you already have your own datacenter/colo set-up then all hope is not lost. Setting up DR in the cloud is one of the common use-cases we see.

Amazon Instances Boot from EBS

The real sea change about to occur in EC2 is booting from EBS. Tonight’s release includes a ton of new features which build on the recently introduced ability to publish EBS snapshots. Here’s a quick summary:

• instances can boot from an EBS snapshot instead of a traditional AMI, EC2 creates an EBS volume from the snapshot and makes it the root partition
• instances can also boot from an EBS volume, which means that a “boot from EBS” instance can effectively be stopped and restarted later by keeping the volume around and launching a fresh instance from the same volume
• instances can now be stopped and restarted later, which works almost exactly as described in the bullet above except for the fact that the instance id (the i-12345678 number) remains the same
• almost all attributes of an instance can change while stopped, including the instance size (naturally the availability zone is one thing that can’t change)
• EBS snapshots can be registered and published as images, so now we have “traditional images” as well as “EBS images” (I wonder what AWS will call these)
• images can specify snapshots and volumes to be automatically mounted at boot, and they can specify EIPs to be attached at boot, the run-instances API call can add/override these “image defaults”
• instances can be “locked”, which prevents their accidental termination
• instances can be bundled into images using an API call (with shutdown or optionally without)

That’s a long list of features to digest! What’s going on here is that AWS is responding to the needs of enterprise customers who have many ‘legacy’ applications that are not designed to scale out or to play nice with the operations agility enabled by the cloud. It’s for the apps that sysadmins spend weeks setting up and then do their utmost not to touch again. Now they can be installed on an EBS root volume and servers can be launched and relaunched as needed without having to touch the config. Basically this enables the old-school way of managing servers to be applied to EC2.

But these new features are also of great benefit to those operating scalable arrays of servers or web 2.0 web sites. It is now much easier to make changes to a clean server image: mount the image as a volume onto an extra server, edit the software/config on the image (e.g. using chroot and the native packaging system), when happy create an image from the volume and boot a server. Test it out and fix any problems in the original volume. Repeat until happy. If done correctly, this results in clean images that are not polluted by repeated boots and other operations, which is one goal we’ve always pursued with the RightImages we publish.

The stopping and starting of servers can also make development more cost effective. Developers that use dev & test servers can stop them at the end of the day and start them back up when they next need them. In fact, many servers could be set-up to stop by themselves if there has been no activity for a while. (This reminds me that I saw that the three longest running instances visible by RightScale have been running for over 1000 days and that the account they run in has seen no activity since then, except for credit card charges I assume, impressive and scary at the same time!)

Stopping and starting servers can also be abused. For example, it can be used to implement “dumb auto-scaling”: simply stop some servers when the load drops and start them back up later. The good thing is that you don’t end up with fresh servers on start, so they don’t have to self-configure, the bad thing is, well, that you don’t end up with fresh servers, servers come up believing the world hasn’t changed since they were last stopped. I think of this as abuse because it’s easy to forget to update one of the stopped servers when making changes to the system, whether these are changes to the software installed on each server or changes to the rest of the system each server needs to communicate with. In other words, the danger of having a zombie come back to life and create mayhem is high. Better keep a basic amount of hygiene and start with fresh servers.

The Cloud Marches On…

It will be interesting to see how EC2 and its user base continue to evolve. With each release Amazon offers more options. That’s more ways to do interesting stuff, but also more ways to shoot oneself in the foot and more stuff to ‘grok’ to get started. Maybe the most important, though, is that the Boot from EBS features rank very high on the “remove sales objections” scale: not every application is ready for the former EC2 cloud, not every sysadmin is ready for it either, by far not. I have to admit that all this leaves me with mixed feelings. EC2 used to have a simple & clean model, it required some rethinking but that was for the better. It was clear how to deploy highly scalable, highly redundant applications with a high degree of automation. Now that there are 10 ways to skin the proverbial cat it’s much harder to stay on track and to leverage automation. Where early customers needed help figuring out how to operate in the world of EC2′s disposable servers today’s customers need help just navigating through all the options available in EC2 and which to apply to each application or use-case.

Support for the new features and the new US-WEST region in RightScale will become available with our next release, currently scheduled to go live just before xmas. Full support for booting from EBS will take a little longer as it has far-reaching implications. I’m sure that many of our customers will be operating in the new west coast region and that  it may even have some appeal to those in the far east and south pacific as “one step closer” to a local presence.  As always, we’d love to hear your thoughts on the new features, how you’re planning to use them, and how you’d like to see us support them.

Updates:

• AWS now gives each region a little local character: US-WEST-1 is listed as “N. California”, US-EAST-1 as “N. Virginia”, and “EU-WEST-1″ as “Ireland”.
• Nice blog post on some of the mechanics of using Boot from EBS by Shlomo Swidler (but see comment below)
• Some things you can’t do with traditional AMIs: start & stop instance, create image (new way of bundling)
• Some things you can’t do with EBS-based AMIs: dev pay, protect the content of public AMIs (someone can mount the content as a data volume and pull files off it)
• If you plan to create a public EBS-based AMI beware of deleted files: don’t just “delete” files with sensitive data on the volume because they can be “undeleted”, you have to erase the blocks, or better, not put anything sensitive there in the first place

We’ve had rather sophisticated sharing of ServerTemplates in RightScale for over a year now allowing certain users to share ServerTemplates, RightScripts and other design artifacts with other RightScale users. This enables us to publish free ServerTemplates to all our users, premium ones to our customers and it also lets ISVs on our platform publish ServerTemplates for free or for pay to their users and customers. In addition, each of the design artifacts is versioned such that users who have launched servers with a ServerTemplate last year can still launch new servers with exactly the same version of that ServerTemplate.

A result of all this publishing, sharing and versioning is that there’s a lot to choose from. So much that drop-down menus have become really unwieldy and this is where the new library comes into play. In the past, when adding a server to a deployment one had to find the correct ServerTemplate from the list of all available templates in the RightScale system. Now this has become a two-step process where you first import the ServerTemplates of interest from the library into your account and then only the imported templates are shown in all the drop-down selection menus. Separating the library import/export step will also allow us to significantly upgrade the experience browsing all the design artifacts in the library over the coming releases, stay tuned…

We introduced Flickr style machine tags recently and we’re expanding their use with this release. One of the really exciting new features is that servers now have tags and we’ve integrated the tags with the routing of messages between servers, with Chef (via the RightLink agents) and with the UI. All this is still in alpha but it’s starting to take shape. Our first real use-case is the registration of application servers with load balancers. The way it works is that when a load balancer comes up and is ready for operation it adds a “loadbalancer:lb=www” tag to say “I’m a load balancer for the www vhost”. When an app server starts up, it requests all servers in the deployment with a “loadbalancer:lb=www” tag to run a Chef recipe that adds the app server to the load balancer rotation. This way, the app server doesn’t need to know which or how many load balancers there are. The tag matching, communication, and running of the Chef recipe are all done by the RightLink agents.

In order to let new load balancers come up when app servers are already running we can do the same tag-location in reverse: app servers announce “loadbalancer:app=www” to say “I’m an app server serving vhost www” and load balancers on start-up can add all app servers to their config by querying for all servers with that tag. For overall resiliency it’s a good idea for load balancers to re-query the set of app servers and to update their config accordingly. This catches race conditions as well as issues where portions of the app servers may be temporarily invisible due to network partitions. The theme here is “eventual consistency” and we’re still evaluating what the best primitives are to support high availability.

You may wonder why the examples above use such long tags and that’s really where machine tags come in. The “loadbalancer:” prefix helps isolate the tags to coordinate the load balancer registration from other tags. Think of “loadbalancer” as being the name of the application or feature that uses these tags, e.g. the load balancer registration. The “lb=www” and “app=www” tag predicate and value can be used to support multiple vhosts. So a load balancer could announce “loadbalancer:lb=www” and “loadbalancer:lb=api” to indicate that it’s load balancing the www and api vhosts. And an api app server then would only query for the “lb=api” tag and it would only announce the “app=api” counterpart.

While all this is happening amongst the servers, the RightScale UI provides access to all the tags, so one can see the servers announce the various tags and one can even intervene and manually modify these tags. We might provide a “don’t touch” notion for some tags, but right now it’s much more important to us to be able to expose all this machinery. As an ops guy there are few things I loathe more than hidden automation that I can’t inspect and override when I need to.

Of course there’s more in the new release than just these two features: more support for RackSpace (monitoring in particular), improved support for Chef, support for new AWS features, and more…

The basics

What does this persistent storage look like? We’ve been testing it for awhile and are thoroughly impressed. The Amazon folks are clearly still fine-tuning a lot of the details, but basically you can create storage volumes in the cloud next to the server instances you launch in the cloud. Think of having a really big SAN in the cloud in which you can create volumes of up to 1TB each with a single API call, or with a simple click in the RightScale UI (yes, of course we’ll have nice support for the storage volumes on our site coupled with some neat automation and an array of pre-packaged solutions). You can mount one or multiple volumes on an instance and they appear just like the other local drives, so you can format them as you like, set-up striping and do other useful things.

The feature that really makes the storage volumes sizzle is the ability to snapshot them to S3 and then create new volumes from the snapshots. The snapshots are great for durability: once a snapshot is taken it is stored in S3 with all the reliability attributes of S3, namely redundant storage in multiple availability zones. This essentially solves the whole backup issue with one simple API call or click in the RightScale UI. You can also easily restore a snapshot by creating a fresh volume from it. This feature is useful beyond just restoring a backup: you may restore to another instance where you now have a clone of the data and can do whatever you want to it. Wow!

The cool stuff

There are so many great uses for the storage volumes that it’s impossible to write them all up in a single blog post, and we obviously haven’t thought of them all (or even close). The first usage scenario we looked into is running a database. Up to today the only setup for a mission critical database we recommend is using two instances with real-time database replication and frequent backups to S3. We’ve now installed our Manager for MySQL replicated set-up for many, many customers and it works very well. In short, we use MySQL replication for redundancy and frequent (like every 10 minutes) backups to S3 on the slave to guard against the unlikely event of simultaneous failure of both instances located in different availability zones.

With the storage volumes the Manager for MySQL set-up works even better. Instead of having to tar-up the database files and upload them to S3 we can just take a snapshot. And in order to initialize a slave we simply create a volume for it from the last snapshot of the master and launch the replication: no more rsync of the data is necessary. It’s really nice to see how all the automation we’ve built stays in place with the new Amazon capabilities and saves just as many headaches as before, it just gets turbocharged by the storage volumes!

In addition, the storage volumes enable slightly lower-end database offerings. Since the storage volumes are more durable than local instance storage a lot of the risk of losing it all if the instance dies goes away. It is now possible to run a single instance with the database data living on a storage volume and to take frequent snapshots to get backups onto S3. Should the instance die, it is very simple to launch a fresh one using the same storage volume. Typically it would take only a few minutes for the new instance to come up and take off where the old one stopped! Of course this set-up has more downtime when compared to the redundant database set-up, and one has to be really careful in setting everything up to minimize the time it takes to mount the volume and to ensure a successful database recovery.

Just as the storage volumes enable the reliable use of single-instance databases they also enable single-tenant appliances in EC2. It is now possible to host the data for a single-tenant virtual appliance on a storage volume and mount it on an instance. What’s really cool is the decoupling of the data from the instance. It means that you can start a customer on a small instance and if they outgrow it, you can migrate them almost seamlessly to a large and later an x-large instance, all using the same storage volume. Beyond an x-large a couple of interesting options are possible to increase performance further, such as striping multiple storage volumes. EC2 really brings virtual appliances to the next level!

The S3 snapshots enable some completely different and very intriguing usage scenarios. Suppose you’re doing some DNA matching against a Genome data set on 1000 instances. In addition to firing-up 1000 instances on a whim you can, also on a whim, clone a nicely prepared snapshot of the data set 1000-times to create 1000 volumes, one for each instance. BANG! This way they can all independently crawl over the data set. This type of massive (essentially read-only) cloning really opens-up new possibilities in running such large computations in a cost effective manner.

Summing it up

I’ll stop here, but clearly the cloud has just squared in size! Two years ago, when I started on EC2 there were only small instances available and the sentiment was that in order to get the horizontal scalability and pricing of the cloud you had to accept inferior features. In the meantime we’ve gotten multiple instance sizes plus recently the remappable IP addresses and availability zones. That already indicated that computing in the cloud would soon surpass computing in traditional colos or in your own datacenter not just in scale and price, but also in feature set. With the addition of the storage volumes with all the cool snapshot features it’s now a fait accomplit: the cloud adopters will have much more computing horsepower and flexibility at their fingertips than those who are still racking their own machines. It’s going to be like agile software development: if you want to survive as an internet/web service you will have to compute in the cloud or your competitors will leave you in the dust by being able to deploy faster, better, and cheaper.

Update: Werner Vogels, Amazon’s CTO also blogs about the storage volumes in all-things-distributed with a little more background perspective. The Amazon folks are getting pretty coordinated with news appearing at the same time on their blogs and the forums. Maybe I missed it, but I don’t think they even press release this stuff…

The most confusing thing about availability zones is the name: In the cloud, what exactly is an “availability zone”? The easiest way to think about it is that a zone equals a datacenter. If power goes out in one datacenter and the generators fail to start (naah, that never happens…) then it doesn’t affect the other datacenter. Or if there’s a fire, one datacenter may burn out or be otherwise incapacitated, but others are unaffected. In reality zones don’t necessarily correspond to datacenters. Given careful engineering, it’s possible to have multiple “rooms” in a datacenter that are highly failure isolated while technically still being part of the same datacenter (imagine football-sized fields here).

The point of availability zones is the following: if I launch a server in zone A and a second server in zone B, then the probability that both go down at the same time due to an external event is extremely small. This simple property allows us to construct highly reliable web services by placing servers into multiple zones such that the failure of one zone doesn’t disrupt the service or at the very least, allows us to rapidly reconstruct the service in the second zone.

The one caveat to consider when using multiple zones is that there is no free lunch (you knew there was a catch, didn’t you?). First of all there’s the speed of light. The zones Amazon is exposing are all on the East coast (indicated by the names, such as “us-east-1a”. I don’t have inside information about the location of their facilities, but I imagine some may be in New York and others may be in Virginia, so the distance between zones may be considerable, thus translating into some network latency. And even if the actual facilities used by EC2 today are not that far apart, they may be someday in the future.

The second “gotcha” is that bandwidth across zone boundaries is not free.  Amazon is charging $0.01/GB for what they call “regional” traffic. This is less than 1/10th the cost of Internet traffic, which seems perfectly reasonable to me. In the days where I was managing multiple datacenters the cost of traffic between them was essentially the same as the cost of random Internet traffic. Actually, I take that back, it cost twice as much: once to exit one datacenter and once to enter the other. (Granted, at high volume one can do interesting things to save some money, but it doesn’t become free by a long shot.)

An example

Enough talk, let’s show a diagram of how a simple redundant web site looks like with Availability Zones and Elastic IPs. At the core we’ll have two web servers (e.g. with Apache and PHP) running the web application and accessing the master database. All this occurs in one zone. We’ll allocate two Elastic IP addresses that we assign to the two web servers and then we create a round-robin DNS entry for our web site that maps the domain name to the two IP addresses (this is commonly called “round-robin DNS”).

In order to ensure the survival of the data in the case of a massive failure, we start a slave database in a second availability zone and replicate the data in real-time. This is how we’ve set-up all our customers to date, except that up until now we haven’t been able to specify the placement of the slave with respect to the master. In the RightScale Dashboard the zone of each server is shown and at server launch time one can select the desired zone.

Now suppose the zone with the web servers and database fails due to a fire! After receiving an alert, we first promote the slave in the second zone to master using the RightScale Manager for MySQL automation. We then launch fresh web/app servers in the same zone as the slave database. Once the promotion completes and the two new servers are up, it is a simple matter of reassigning the Elastic IPs to the two new servers to redirect all the users to the new servers and we’re up and running again.

The next step is to recreate the redundancy and for this the third availability zone that each account has access to comes into play. We start a fresh database slave in the third zone again using the automation in the Manager for MySQL. Once that comes up and starts replicating we are back to having a redundant setup!

If you have never tried to set something like this up yourself starting from renting colo space, purchasing bandwidth to buying and installing servers, you really can’t appreciate the amount of capital expense, time, headache, and ongoing expense saved by EC2′s features! And best of all, using RightScale it’s just a couple of clicks away .

Beyond the simple redundant setup

As an astute reader you probably noticed that the site described above would go down if there was a failure in the primary zone, which would require a manual restarting of new servers in order to bring it back up. Some of this can be easily remedied by placing one or multiple web servers into the secondary zone and having them talk to the master DB across the zone boundary. The performance of these servers may be slightly lower due to the inter-zone latency and there is some cost to the database access traffic. It’s somewhat application-dependent how these play out.

A more sophisticated setup uses load balancers to reduce the impact of the cross-site traffic. The idea is to place one load balancer instance in each zone and route the requests primarily to a set of redundant web/app servers in the primary zone, as shown in the figure below. A third app server can be running in the secondary zone and perhaps get a trickle of traffic from the load balancers just to keep it “warm.” Keeping it warm makes it easy to monitor and ensure that it’s operating properly.

The good thing about this setup is that the traffic shipped across the zone boundary is exactly the same as comes into the second load balancer. This means that for half the total Internet traffic there is a $0.01/GB surcharge, which results in less than 5% extra cost overall. (This is not counting the DB replication traffic.) Also, the extra latency from one zone to the other is negligible when compared to the already incurred Internet latency.

In the case of a primary zone failure, browsers will fail over to the load balancer in the remaining zone (this is a feature built into web browsers related to the round-robin DNS set-up). The load balancer will direct all traffic to the third web/app server. At that point the secondary database needs to be promoted to master and the third app server repointed to that database and everything will be back up and running. With automation the DB promotion could be done automatically, but it’s better to be conservative: a promotion due to a false alert could cause a lot of harm.

This second set-up is a bit more complicated than the previous one, but it requires less machinery and no server launches in the case of a failure. It also requires one extra machine if one assumes that each load balancer can run on the same instance as a web/app server (typically not a problem). Many more variants on this basic setup are clearly possible and should be considered on a case-by-case basis.

Wow, it’s mind-boggling how much power Amazon is giving us in designing sophisticated distributed redundant Internet services! In combination, the availability zones, the elastic IPs and the overall programmatic control over all the resources make the cloud a superior environment for deploying sophisticated Internet services. At RightScale we’re extremely excited and are hard at work to incorporate the new features into our standard deployment templates such that all our customers can easily take advantage of the new features in their deployments. We’re also automating a number of the failure scenarios so that you don’t need to have an alert wake you up if there a fire at Amazon in the middle of the night!

Following up with more information about this morning’s event. At 1:31am PST, a network engineer made a change to a pair of redundant aggregation routers fronting a portion of EC2. This change caused both to no longer route traffic to a subset of EC2 instances. The change should have been non-intrusive. We are taking steps to prevent this type of failure in the future. The failure affected a portion of EC2 instance connectivity for 2 hours and 5 minutes. In response to the question about RSS feeds, we are planning to provide this functionality in the future.

Sincerely,
The Amazon Web Services Team

Thanks Kathrin for posting this. We all know that this stuff happens and we all want to use a provider who is committed to getting to root causes of incidents and eradicating them as much as possible!

The fact that Amazon says up-front that computers fail seems to be the number one concern and criticism of EC2, specially from people who have not used it extensively. I don’t actually spend much time thinking about that because in our experience it’s not something to worry about. It’s essential to take into account when designing a system: whenever we set something up on a machine we immediately think “and what do we do when it fails?” That’s good thing, not a bad thing as anyone with production datacenter experience can attest.

Since it’s such a hot topic, I’ve been keeping a close eye on all the “my instance disappeared” threads on the EC2 forum, and it’s not easy to sort them out. I have no doubt that the vast majority has to do with operator error:

• trying SSH and forgetting to open port 22 in the security group (or similarly with other ports)
• having difficulties with the SSH keys, or forgetting to set-up a key to begin with
• using/constructing an AMI that does not have SSH properly set-up
• using/constructing an AMI that does not boot properly (network and/or sshd issues) and failing to look into console output
• instance reboot failing, for example disk mounts failing due to mount point changes that were not reflected into init scripts
• sshd killed by kernel out-of-memory reaper, failing to look into console output for diagnosis
• … and many more

Some of these are beginners failing to read the getting started guide, some are more subtle and can happen even to veteran EC2 users. Then there are emails from Amazon saying “we have noticed that one or more of your instances are running on a host degraded due to hardware failure” and I wonder how many users don’t get these emails because their AWS account’s email address points into a bit bucket.

No doubt there are real failures as well where a host dies and takes the instances with it, or one of the disks used by an instance gives up which is the end of that instance. The question here is how frequent this is relative to the total number of instances running, and since Amazon is so secretive with their numbers it’s really difficult to make even an educated guess. I tried to go back into our year of logs to see whether I could estimate the failure rate, but I don’t have enough data to distinguish failure from shutdown, sigh.

The failures that concern me the most are actually not instance failures but network failures. Anyone having set-up a large datacenter will know that network issues are the most difficult to get under control. The damn network just keeps changing, and as soon as you try to hold still your service providers change stuff. Some of the instance disappearances are really network issues that cause an instance to be unreachable, or unreachable from certain other instances. These are hard to troubleshoot and on more than one occasion I’ve had to run tcpdump on both ends to see packets departing and never arriving. If I can get to the target instance at all to run tcpdump, that is… I hope Amazon gets a better handle onto this type of failure and provides us with better troubleshooting tools. In the meantime, it’s important to flag issues to them so they can troubleshoot and eliminate the root causes.

The really good news is that the Amazon folks are very dedicated to figuring out what’s going wrong and fixing it. So if you have an issue, be sure to do the troubleshooting you can, then set the instance aside, launch a new one to take its place, and post all the details on the forum. Shut the instance down only after the issue is looked-at. Looking back, there were two big issues causing instance termination, one was the day where some EC2 front-end explicitly terminated a bunch of instances by error. Not good, but from what we saw it wasn’t a massive failure either. They clearly have done their best to ensure this doesn’t recur. The other was an instance reboot bug which caused many instances to die in the reboot process. We learned not to reboot ailing instances but instead to relaunch and rescue any data. This issue also seems to be fixed at this point.

• To summarize, if you can’t reach an instance, here is what you should do:try to SSH, check the security group
• distinguish SSH timeout from key issues (timeout vs. permission denied type of errors)
• use ping to test connectivity (enable ICMP if you have the bad habit of disabling it)
• check the console output (use the convenient button on the RightScale dashboard), note that it can take a few minutes for stuff to appear
• look at the RightScale monitoring to see whether the instance is still sending monitoring data
• hop onto an instance in the same security group and try connecting from there (launch an instance if you don’t have any)
• post details (instance ID, what you’ve tried, symptoms observed) on the forum and set instance aside

All in all, the number one lesson is “relaunch!” There are thousands of instances waiting to be utilized so use a fresh one if you see trouble with an existing one. If you master this step you can use it in so many situations: to scale up, to scale down, to handle instance failure, to handle software failure, to enable test set-ups, etc. If you use RightScale you will notice that that’s also what we focus on: making it easier to launch fresh instances.

The overall architecture concept we’re using for this customer is to build a number of app+database clusters and to load multiple sites onto each one. The number of sites per cluster can be adjusted such that the database portion of each cluster is loaded up optimally, and it’s designed such that sites can be moved around easily, for example to offload a cluster that may have become too heavily loaded as some of the sites on it have grown.

In the end, the architecture boils down to having two instances running a mysql master/slave set-up managed by our Manager for MySQL plus two instances running load balancers and Rails/Mongrel as redundant app servers. This makes for a fully redundant cluster on which a number of sites can be hosted. It is also easy to add a few more EC2 instances running Rails depending on the Rails vs. MySQL workload balance.

Each site on a cluster has its own logical database (i.e. what MySQL calls a “database”), this makes it easy to backup and restore a site individually, and most importantly, to move a site to another cluster in order to free up resources on the original one. The sites on a cluster can also share the app servers as long as there is no HTTPS involved. The reason for this caveat is that each Amazon EC2 instance has only a single IP address and it is not possible to do “virtual hosting” with HTTPS sites. With HTTP all the www.site1.com, www.site2.com, etc. DNS entries point to the same two load balancing instances (using what’s called “round-robin DNS” for fail-over purposes) and the load balancer (or front-end Apache, if used) figures out which site the user is visiting based on the “host” header included in every HTTP 1.1 request.

What’s really nice about these 4-6 machine clusters is that they’re very powerful yet so simple. There’s no “infinitely scalable” magic under the hood that breaks at the worst moments. No, it’s a plain set-up that anyone with a bit of experience can fully understand. The magic is that it’s so easy to set these clusters up with Amazon EC2 plus RightScale so you can really take advantage of the same “horizontal scaling” as the big guys (Google, Yahoo!, etc.).

One of the interesting design decisions with all this is how to set-up DNS. For example, the app for site1 needs to locate the IP address of the database it’s supposed to talk to. We use DNS as follows:

• the app connects to db-master.site1.com
• db-master.site1.com resolves to a CNAME for db-master-cluster3.company.com
• db-master-cluster3.company.com resolves to the IP address of the instance that currently hosts the master
• DNS for db-master-cluster3.company.com is set-up with a low TTL (we use 75 secs) and supports dynamic updates
• if the DB master crashes or is otherwise replaced the db-master-cluster3.company.com DNS entry is automatically updated by the RightScale MySQL manager, which switches all the sites hosted by that cluster over with one stroke
• if site1 is moved to different cluster, then the CNAME has to be updated to point to the correct cluster DB

For the web sites themselves its also nice to use CNAMEs:

• www.site1.com points to www-cluster3.company.com
• www-cluster3.company.com resolves to the IP addresses of all the load balancer instances
• if a load balancer instance is restarted, the www-cluster3.company.com entry is dynamically updated
• if the site is moved to different cluster, the CNAME needs to be updated

Wow, it’s amazing all this is actually possible and not just a dream! Amazon EC2 enables it and RightScale makes it possible to manage without an army of sysadmins running around and tweaking servers all the time.

Archived Comments

Dan Croak
Do you have one of your pretty graphics to go along with this setup? For all us visual learners out there…

Sushi
A script/tutorial/template like “rails-on-ec2-standard” post will be of great help.

Thorsten
Dan – good suggestion, I’ll pull out my brushes and paints…

Sushi – I fear this is all still a little too new to condense into a tutorial. It’s one thing to set everything up for one site, but another story altogether to figure out a solution that covers many different sites. What is needed is a design pattern that can be used in many slightly different situations and so it needs to be adaptable while also keeping the core that makes it tick constant. We’re not quite there yet.