Windows in the Cloud with RightScale

August 17, 2010 at 3:00 am · Filed under AWS, EC2, RightImage, RightLink, Thorsten

As many of you know we’ve been working on supporting Windows on par with Linux within RightScale for close to a year. Well, the big moment has finally arrived where we can announce the GA (General Availability) of our Windows support! By “support” we mean much more than being able to launch a Windows instance on EC2, which has been possible for a long time, so I thought it worthwhile to expand a little on what it took to make Windows behave nicely in the cloud and what the outcome is.  This is also a good opportunity to examine whether there is a “Windows handicap” in the cloud, or whether it works just as smoothly as Linux.

First the upshot: across the board we now support Windows 2003 and 2008 on par with Linux. We provide RightImages that work well with RightScale out of the box, they can be managed in the dashboard, we support extensible monitoring, associated alerts and automation, and we support ServerTemplates with Windows and even provide a few sample templates. Plus, if you have existing images, we provide a RightLink installer with documentation and a starter ServerTemplate that gives you monitoring out of the box. This will give you a good dose of the RightScale love (monitoring, scaling arrays, user access control, etc.).

The mountain of work that we’ve been chipping away at for the past two weeks is being able to release 40 Windows RightImages. This work wasn’t what we had imagined, it crept up on us little by little. When we release RightImages, whether it is for Windows or Linux, we strive to provide a consistent software environment across the RightImages of a generation and to ensure they work smoothly with RightScale. Both the Linux and Windows RightImages work great without RightScale as well and many EC2 users have been using Linux RightImages for a long time. Here is what we had to do for the Windows RightImages:

  • install RightLink for integration with RightScale
  • ensure the images are automatable, meaning psexec can be used, which also involved getting consistent settings for the windows firewall, file sharing and a few other details
  • install PowerShell 2.0 on all images, including the Windows 2003 ones, because according to everyone we’ve talked to it’s a must
  • ensure all the right DLLs are installed to manage SQL server from PowerShell
  • increase the size of the root disk for the 2008 images because the std 40GB are insufficient for a lot of interesting software installs (SharePoint being one example), and increase the pagefile to 1.5x ram size (up to 8GB)
  • ensure the server’s clock is successfully synchronized to NTP at boot and periodically thereafter
  • install logic to determine when the server is fully ready after boot before starting any automation, which is not as simple as it may sound due to sysprep, double boot, and an ec2-specific service that runs at boot and sets a few things up
  • ensure complete install of asp.net on Windows 2003

This list kept growing as we discovered issues. In addition, we are supporting all four EC2 regions (us-east, us-west, eu-west, ap-southeast) and it turns out we have to build each image from scratch in each region because it’s not possible to copy a Windows image from one region to another (licensing restrictions). We are supporting 10 images in each region:

  • Windows 2003 /2008 across i386/x64
  • Windows w/SQLServer Express 2003 /2008 across i386/x64
  • Windows w/SQLServer 2003 /2008 on x64

Getting to the final images involved a fair number of iterations (we don’ know how many images we built total, and frankly, we’d rather not be reminded) so we built automation to crank out the images. This makes them consistent and reproducible. Unfortunately, the AWS images we had to start from are not all automatable, so we also had to build some “intermediate” images by hand with just a few settings tweaked so we could target them with our automation. Some of our developers ended up with nightmares about clicking through endless circular install dialog boxes!

While we were launching lots of Windows instances for testing we noticed that a fair number had their clock off by a large amount, like days. Digging into the issue we discovered that, unlike with stock AWS Linux images, the Windows wall clock is not synchronized to the virtualization host’s time and that the initial NTP synchronization doesn’t always succeed, partially because the Windows NTP service is “challenged” but also because it is pointed at a public pool of servers. Since performing automation on a server that thinks it’s yesterday or tomorrow is a non-starter we concluded that we had to beef up the time synchronization by ensuring that we get an NTP sync before proceeding with any automation and also run our own set of NTP servers so we can ensure our customers always have in-cloud NTP servers available to synchronize with. We think this really is part of the famous “muck / undifferentiated heavy lifting” that Amazon prides itself to take care of, but they politely declined, which is really a shame. We also noticed that the new HVM Linux images don’t lock their clocks to the host, so we will probably switch the way the clock sync works across all RightImages.

Another interesting issue we discovered during testing was that automation at boot time would frequently fail. The root cause ended up being that our service, which was configured to launch at boot, was starting too early and that the server just wasn’t ready yet. The way Windows boots in the cloud is quite different from Linux or from the “normal” world. With Windows, each OS install generates a server key that is embedded in the registry and uniquely identifies the server, so when an image is booted many times a fresh key needs to be generated for each instance and a number of things need to be updated (“sysprep”) and the server rebooted once. Towards the end of the second boot a special Ec2Config service finalizes the config, including admin password and hostname. This means that any automation has to wait for the reboot plus the ec2 service to complete its changes, which is not trivial due to an oversight by AWS. It’s interesting how the security details of Windows (i.e. the server key) ripple down into the whole boot process, making it take twice as long as it should. The net is that while Linux instance boot times on EC2 have come down from a typical 6-8 minutes back in 2006 to under a minute now when using EBS images the Windows boot times are starting out around 10-15 minutes. Hopefully Microsoft can be sensitized to the notion that fast boot times are an important asset in the cloud because they enable a lot of automation that is very painful if one has to wait so long for additional capacity or replacement servers to come online.

The final and biggest piece is implementing all the automation support we offer in Linux for Windows as well. As part of this we ported Chef to Windows (we’re working with Opscode to feed the changes back into the mainline) and we built out support for PowerShell. This means that software can be installed and configured on Windows servers similar to the way it’s done in Linux. We find that larger software packages often need to be installed manually, but even in that case it’s nice to be able to automate as much as possible and choose which portions to run “ahead of time” and bake into a custom image and which portions to leave off for the actual server launches. To round out the automation we wrote a nice little monitoring plugin that speaks the collectd protocol and that offers a simple meta-language that can be used to query many of the WMI statistics available on Windows servers. And all this is available out of the box to all our customers.

One of the open question we have concerns Windows updates. We will most certainly republish fresh RightImages when AWS updates its base set. What is not clear to us is whether we should be publishing “fully updated” RightImages on a regular schedule. Most of the customers we asked told us that they really want to carefully manage the exact set of updates on their servers. It’s going to be interesting to see how the whole update management with Windows servers plays out and how it will affect the amount of image rebuilding that everyone will have to do. We will definitely build our ServerTemplates directly on RightImages or RightImages augmented by just base installs of larger software packages and do all the config using Chef/PowerShell. This way we minimize the work we have to do when we swap out the underlying Windows install or update level.

I must say that overall I’m very happy with where we’re ending up, which is that we’re getting Windows to a point where it definitely is usable in cloud-style. What I mean by that is not just migrating a set of traditional servers into a equivalent set of servers in the cloud, but rather automating Windows servers for the cloud and leveraging the flexibility of the cloud to enable the business. The friction along the way certainly is higher than with Linux, whether it’s from license questions that crop up everywhere to the mechanics that currently require double-booting, but it is totally possible and Microsoft can, if it focuses on it, make it a lot better yet!

Comments (5)

More servers, bigger servers, longer servers, and 10x of that!

August 4, 2010 at 10:00 am · Filed under EC2, Thorsten, cloud computing

As a rapid growing start-up we obviously keep a close eye on where and how the growth is occurring and year after year it keeps amazing us. We just published a press release that showcases some of the trends we’re seeing, but given how press releases are I thought it would be nice to go into a bit more detail and background behind the numbers.

Maybeserver size distribution the most surprising growth to some people is the size of the servers. EC2 now has 8 instance sizes so we decided to look at them as xlarge vs. smaller to make it easy, i.e., m1.xl, c1.xl, m2.xl, m2.2xl, m2.4xl vs. m1.small, c1.medium, and m1.large. For normal people, that’s machines with 4+ cores or 16+GB vs. smaller ones. In june 2009 just 12% of the fleet was on xlarge servers but by june 2010 this grew to 56%, yikes! Some people’s reaction to this is “I thought the cloud was about horizontal scalability” and while horizontal scalability may have been a requirement in 2006-2007 it no longer is. It’s a wonderful option, but not a requirement. There are a number of simple observations:

  • If you’re starting out building a web app on one server, it’s often smart to postpone the hassle of scaling it horizontally for as long as you can stand it by growing to a single larger server first. This simple principle applies pretty broadly and in particular to the database tier. The catch is that you need to build things right so you don’t have to redo everything to scale horizontally, but that’s not rocket science anymore.
  • If you need 200-300GB of memcache I’m sure you would rather operate 10 32GB servers over 200 1.7GB servers (the only server available in  ’06-’07). And these are not the largest such clusters that we see. (One catch with this calculation is that the large servers can’t handle 10x the request rate due to an approx 100k pkt/sec limit, see our load balancing whitepaper that details some of this.)
  • More and more types of apps are being moved to the cloud, it’s not just scalable web apps and many of the “legacy apps” don’t scale horizontally, so the larger servers are very needed.

At the same time we’ve also seen more and more servers in the cloud. More customers, each launching more servers. At the end of June 2010 almost 3x more servers were running than at the end of June 2009. That’s more companies moving more computing to the cloud and others “simply” growing their cloud footprint. Many companies start with proof-of-concepts or peripheral projects to test the waters, and that’s definitely what we always recommend. Once they get comfortable and experience the benefits the projects grow and multiply.

Lastly servers run longer. We looked at the proportion of servers running full time, as defined by running for more than 30 days and as opposed to more transient servers. The upshot is that of the servers launched in June 2009 3.3% still ran 30 days later and for June 2010 this grew to 6.3%. This growth is significant (don’t be fooled by the small percentages: they’re to be expected given what we measure) because production servers tend to run full-time while tests and prototypes tend to run for a few hours to a few days. Of course this is not an exact science because some production server arrays grow and shrink on a daily basis and some test servers are left running all the time. But nevertheless there is a clear trend that shows a continued move of business critical computing to the cloud.

The upshot of all this is that in dollar terms the spending we’ve seen on cloud computing servers has more than ten-folded between June 2009 and June 2010. We’re living in interesting times where the way computing is consumed is changing dramatically, whether at the personal level with smartphones and iPads or at the server level in the cloud!

Comments (14)

OpenStack: game changing open source cloud platform

July 18, 2010 at 11:19 pm · Filed under OpenStack, RackSpace, Thorsten, cloud computing

OpenStack logoToday RackSpace announced that it is open-sourcing it’s cloud platform, i.e. the code that powers CloudServers and CloudFiles, at openstack.org. We’ve been working with RackSpace for quite a while and have also participated in this new initiative at a workshop held by RackSpace last week. The bottom line is that we believe this to be a potentially game changing event. The reason is that RackSpace has committed itself to a true open source project, meaning that it’s not just source thrown over the wall into the open, but also an open design process, an open development process, and an open community. Of course RackSpace and NASA, whose Nebula team is co-leading the effort, will need to show that they can live up to those promises, but last week’s workshop definitely showed that their hearts are in it. Despite the fact that there were over 40 technical folks discussing architectural details, Rick Clark managed the requirements gathering very openly using a “scaled-up” version of the Ubuntu design conference process he is very familiar with, having been the lead of the Ubuntu Server project. The process felt very inviting yet still focused on firming up a first release later this year, which doesn’t leave much room for crazy new stuff. I expect we’ll see a good number of companies contributing code to this project.

From the RightScale perspective we will of course continue to support a variety of public and private clouds. We already have basic support for RackSpace’s API, which OpenStack will start out with, and we have a number of implementations under way with Eucalyptus and Cloud.com which we’re looking forward to multiply. At the same time, having many fragmented cloud efforts doesn’t really help build a compelling alternative to Amazon which keep adding incredible new features at a blazing pace. And the industry needs an alternative to Amazon, not because of some problem with AWS, but because in the long run cloud computing cannot fulfill its promise to revolutionizing the way computing is consumed if there aren’t a multitude of vendors with offerings targeting different use-cases, different needs, different budgets, different customer segments, etc. OpenStack promises to build enough momentum to create an exciting cloud offering that is as feature rich as AWS, that is implemented by a number of service providers, like RackSpace, and that enterprises can also run internally, like NASA. Of course, at this point it’s just the beginning, we’ll have to see how the first complete release shapes up at the end of the year.

Comments (2)

VMops rebrands to Cloud.com and open sources

May 4, 2010 at 11:15 am · Filed under EC2, Eucalyptus, Thorsten, cloud computing, cloud.com

Pretty bold move by VMops! They evidently bought the cloud.com domain and are rebranding to Cloud.com on the occasion of their new release, now called CloudStack 2.0, which they’re also open-sourcing at open.cloud.com. With this they’re joining the open source private cloud landscape which Eucalyptus pioneered. We’ve been seeing a lot of interest in the private cloud space and have been working with Eucalyptus for a long time and with VMops/Cloud.com more recently. Eucalyptus support has been available in the RightScale dashboard for over a year, but stay tuned for more in-depth support of private clouds by us.

It’s interesting to see that Cloud.com is planning to support the EC2 API as well as the vCloud API, plus their current (native?) API. Eucalyptus is similarly able to support the vCloud API in addition to the EC2 API. This brings home the point that the API itself is more a programming exercise than a fundamental issue, it’s the semantics of the resources behind the API that really matter. Seemingly obscure stuff like: can you reattach a disk volume across datacenters, or can you move an IP address across datacenters, things that deeply affect how one builds redundant services in a particular cloud.

The move to support the EC2 API as well as the vCloud API is a clear indication of the market direction, which is what we call hybrid cloud use-cases where organizations use private/internal clouds for some use-cases and external/public clouds for others. Many of the larger organizations we talk to want to enable these hybrid clouds, and to manage them from one unified platform such as RightScale.

Comments (1)

Accelerating Cloud Adoption

April 28, 2010 at 6:00 pm · Filed under AWS, EC2, Thorsten, cloud computing

Over the past weeks we’ve been releasing a number of new features in response to numerous requests from our user base to broaden the reach of the RightScale platform.  Just today we launched support for Amazon’s new Singapore region, see Werner’s blog post as well as Jeff Barr’s. We’ve worked closely with the EC2 team so we can support the new region right out of the gate in our dashboard as well as with RightImages and the whole ServerTemplate, monitoring and automation stack. We’ll soon re-release our popular ServerTemplates in our library such that Singapore shows up in the list of supported clouds in each template. We’ve gotten a fair number of questions around who exactly has been asking for RightScale in south-east Asia and interestingly there isn’t a single answer.  Given that we support a large number of social gaming companies we’ve certainly heard requests from there. Some even were ready to set-up a Eucalyptus cloud managed with RightScale. Obviously latency, specially with mobile devices, has been an issue so being able to spin up servers in the region is a major enabler. Additionally, enterprises have been asking for ways to provide better cloud services to their development and research locations in Asia. How soon until it’s easier to list world regions where there isn’t an EC2 cloud than to list where they are?

Last week we released RightLink install packages for Linux and Windows enabling our users to take almost any image and RightScale-enable it. Most of you are probably well aware that I’m not a fan of creating images: it’s a slow process that results in hard to maintain monsters. This hasn’t changed with the RightLink installer, but what has changed is that we wanted to make it easier for our partners to build RightScale support for their flavor of Linux and we wanted to make it easier for users that already have servers up in the cloud to benefit from RightScale. But we didn’t just put the RightLink packages out in the wild, we also published the ServerTemplates we use to build our RightImages. We’ve been publishing these images since 2007, the idea being that we provide a clean base OS install for CentOS and Ubuntu. In RightScale these serve as the base boot image on top of which custom software gets installed and configured dynamically at boot time. What has set our images apart is that we built them automatically and have published the scripts we used for that purpose. This means they’re clean and reproducible. More recently we’ve used ServerTemplates to build the images, basically we boot a server that then builds and saves an image as it comes up. The benefit of using ServerTemplates to build images is that they are easy to maintain and 100% automated, they also make it easy to understand what went into an image later on. This is an excellent framework for anyone else to customize and build their own special image. But I won’t let an opportunity pass by to recommend that you save yourself a lot of time and grief and simply use the RightImages we already built and maintain!

Last but not least we’ve introduced public beta support for Windows in RightScale, and that’s not just launching AMIs, it’s the full ServerTemplate machinery and associated automation, including initial monitoring support. Our sales guys all asked which of our standard features we don’t support for windows… the response is “uhhh, syslog consolidation?”. I’m really interested in seeing how the usage pattern of Windows deployments will differ from Linux. I assume that Windows folks will have to be much more in the image bundling business to create base installs of software packages that are too slow to install at boot time. But I also wonder about things like auto-scaling because it still takes a Windows server a lot longer to boot up. We may have to play more games with stopping and starting servers instead of booting them cold. I’m sure our users will tell us!

Comments (2)

RightScale Meetup next week and iPad contest

April 13, 2010 at 11:30 pm · Filed under cloud computing

We’re holding our second RightScale Meetup next week: April 21st in New York, in conjunction with Cloud-Expo. The first one was really exciting with a lot of information for the attendees and well as feedback for us, so I’m looking forward to this NY edition! I hope you’re curious about the latest RightScale developments, want to hear how to RightScale-enable arbitrary images, are wondering about Windows, just want to talk to some of us in person, or perhaps you have some feedback for us you’d like to deliver in person with some candy :-)   Please check out the meetup page and sign-up (space is limited)!

While you’re at it, you might as well add your guess into our iPad drawing: guess how many servers will have been launched in the cloud by RightScale as of April 21. The contest page has some hints and you have less than 24 hours left… (April 14th 23:59pm PDT to be precise, I’m told!)

Comments (1)

Thoughts about the Amazon Simple Notification Service

April 12, 2010 at 9:40 pm · Filed under EC2

Amazon just released a new service, the Simple Notification Service (SNS), which is a distributed message delivery service roughly similar to AMQP or JMS message services. It uses a publish-subscribe paradigm and supports push delivery of notifications using HTTP and email. It seems mostly targeted at back-end applications, e.g. servers sending notifications to one another, but given that is has an email delivery mechanism it can also be used to deliver notifications to users.

Jeff Barr wrote an article describing SNS, the key points are:

  • you can create topics and publish messages to these topics
  • others can subscribe to the topics and they will get messages pushed to them
  • messages can be pushed over http or smtp (email)
  • using access control policies one can control who is allowed to subscribe to a topic
  • the SNS system is redundant and retries message delivery if necessary
  • the cost at volume is $0.06 per 100,000 messages published and $0.06 per 100,000 HTTP message pushes. Email pushes cost 33x more. Good news is the first 100,000 message publish and pushes are free.

It really is great that AWS provides such a service. It’s relatively easy to fire up a messaging server, like RabbitMQ, but it’s a different story to set up a redundant scalable messaging system. While this can be done with RabbitMQ, for many users having this provided as a service is the right way to go.

Unfortunately SNS does not use a standard messaging API, it’s all proprietary. This is a major weakness of SQS, SNS, and SDB: once you use their interface you’re locked in to using AWS. Granted, the SNS interface isn’t particularly big, but then why did they have to roll their own?

My biggest beef with SNS is what is being said, or more precisely, not being said about reliability. I have no reason to believe that SNS doesn’t do all the right things, but AWS isn’t very forthcoming with specifics. Here is what the SNS docs state:

  • “Reliable – Amazon SNS runs within Amazon’s proven network infrastructure and datacenters, so topics will be available whenever applications need them. To prevent messages from being lost, all messages published to Amazon SNS are stored redundantly across multiple servers and data centers.” source
  • “Although most of the time each message will be delivered to your application exactly once, the distributed nature of Amazon SNS and transient network conditions could result in occasional, duplicate messages at the subscriber end.” source

So there is talk about redundant storage, at least once delivery, and delivery retries. But what I’d really want to know is not all this fuzzy feel-good stuff. The question is not that difficult:

  • If SNS returns a HTTP “200 OK” to my publish request, what is the probability that each subscriber will receive at least one delivery attempt?

ok, I guess I really need to factor in time, which would also give an indication of performance:

  • If SNS returns a HTTP “200 OK” to my publish request, what is the probability distribution over time that each subscriber has received at least one delivery attempt?

This would let me reason about what I can use SNS for and what not, or whether I need a back-up synchronization mechanism or not, etc. If the story stays at the warm&fuzzy level, AWS could at least specify when messages are stored redundantly, e.g. is a redundant copy stored by the time I get an HTTP “200 OK” response? Also specifics about how long and how often retries are made. (I’m focusing on the HTTP delivery, I don’t think it makes much sense talking about email delivery reliability.)

I hope that many others will also ask AWS to be more specific about what in the end is really the SLA offered by SNS (same goes for some of the other AWS services). I’m not asking for damages if the SLA isn’t met, I just want to know what AWS is publicly holding itself accountable for and thus what I can design for and apply my “trust in AWS” judgement to. While the service is in beta, the SLA might be a target.

Comments (5)

Benchmarking Load Balancers in the Cloud

April 1, 2010 at 3:48 pm · Filed under EC2, Thorsten, cloud computing

Load balancing is one of the technologies that virtually all our customers are using within EC2, and there is an increasing set of options available for doing it. We’ve been giving advice to our customers for years on what we’ve seen work but we finally decided to spend some time and do a real A-B benchmark comparison of a number of solutions. The result is a white paper Load Balancing in the Cloud: Tools, Tips, and Techniques we just published.

The test we ran compared the following solutions:

We focused purely on request rate scalability, i.e., how many requests per second the load balancer can sustain. We didn’t focus on feature set, bandwidth, or other metrics. So we ended up requesting a tiny web page over and over as fast as the system under test would serve up and we measure requests/second, or rather, responses/second. We also didn’t take advantage of more advanced features, such as caching in the load balancers, so we ran aiCache in a pure LB mode, for example.

Cutting to the chase, we ran the HAproxy, Zeus, and aiCache tests on an m1.large EC2 instance. After chasing down all kinds of options, trying to tune the kernel, trying other instance types, and finally conferring with AWS– the result is 100,000 packets per second in+out! I know, that’s not requests/sec or responses/sec so let me explain. Basically, with the current virtualization technology implemented in EC2 the network speed of light on an instance is getting about 100,000 packets per second through the two networking stacks of the host OS and the guest OS. Your load balancing solution and the tests you run may use these 100K pps to requests and responses in various ways which give you slightly different performance as measured in resp/sec. On average you can get about 5000 requests/sec through a load balancer. If you use HTTP1.1 persistent connections you get a few more resp/sec because there are a couple fewer packets per request, but the difference is not all that dramatic. If you turn some form of caching on you can roughly double the resp/sec because you’re eliminating the packets to the back-end servers. Tuning kernel params has very little effect, but pinning the load balancer process to a specific core does help quite a bit and makes the performance a lot more even. But in the end it’s all about pps (packets per second).

We then turned to ELB, the Amazon Load Balancing service. It operates differently from the other solutions in that it is a scalable service provided by Amazon. Everything ELB does for you can be replicated using the above solutions running on EC2 instances, but of course that requires extra work. Unfortunately benchmarking ELB is really tricky. One has to use many clients applying load, ensure that they requery DNS frequently, and run tests for a long time so ELB scales up. In the end we produced some pretty graphs like this, showing the requests/sec handled over time:

This shows how our tests ramped up to around 20K requests/sec over the course of about three hours (note that we ramped up the load slowly to see the progress, so this is not all time taken by ELB to ramp up). We could have continued higher but lost interest :-) . I would prefer it if ELB were more transparent and easier to test, but it certainly delivers real world performance!

The whole benchmarking project was interesting in that it once again showed that until you really understand what is going on your benchmark is not done. We chased down more supposed performance bottlenecks than we care to remember and we drove the helpful folks at aiCache batty because they expected to see better performance given their results on non-virtualized machines. But in the end the results make a lot of sense and 100K pps is easy to remember.

I hope all this has whetted your appetite for the full white paper Load Balancing in the Cloud: Tools, Tips, and Techniques.

Load Balancing in the Cloud:
Tools, Tips, and Techniques

Comments (7)

RightScale ServerTemplates Explained

March 22, 2010 at 6:00 am · Filed under EC2, Thorsten, cloud computing

One of the distinguishing features of RightScale is that from day one we’ve focused on automating the configuration management of servers. The reason is simple: you can’t get the benefits of the cloud if you have to spend a lot of manual time configuring each launched server. The cloud itself only solves part of the problem: calling EC2′s runInstances API is just the beginning of a rather complex process that gets the server into full operation. After all, most launched servers are intended to go into operation/production and not just sit around idle waiting for someone to log in.

After doing a webinar on ServerTemplates, I thought it would be good to go back and write-up why the core of RightScale does what it does. It’s on the long side so you may want to skim over the sections that you’re already familiar with.

Why machine images don’t work

The standard method of operation in the virtualization world is to work with machine images. This translates to: launch a server with an image that is “close” to what you want, then log in and install the software you need plus make any config changes, then create an image from that server (“bundling” in EC2 parlance). Later on, when you need an instance of that server, you launch the image and hopefully it comes up ready to go. While this process may sound simple, it’s actually anything but that and takes quite some time. I went through the bundling treadmill back in 2006 and concluded this wasn’t productive. The reasons are simple:

Images are too monolithic. Everything on a server is bundled up in one image file which makes it difficult to manage a collection of images. Change the version of one software package you commonly use and you have to recreate all images that happen to use that package. This quickly gets out of hand.

Images are opaque. From the outside it’s hard to tell what’s in an image. Even if you fire it up it’s not convenient to poke around to figure out what’s installed and how it’s configured. Try determining the difference between two images: not a pleasant task.

Images are too big. They are unwieldy to work with. Take two images of different versions of your app server. More than 90% of the bits are typically identical, (often more than 99%). But finding the interesting ones that differ is like finding a needle in a haystack. This is ridiculous and contributes to making images hard to work with.

Images are too static. You can’t fully configure each server. When you launch the 10th app server it needs to know it’s number 10 and not 9. When you launch a test app server it needs to know it’s in a test deployment (e.g., don’t send alert emails to the ops team), yet you want the same image to be used in test as in production, otherwise, what are you really testing? So you need some dynamic configuration mechanism to “personalize” each server at boot time.

The bottom line is that my experience with images has been very frustrating. I know that tools exist to help manage them, but I’m not convinced this is a productive avenue. This why late in 2006 I set out to build what we now call ServerTemplates – and I can’t imagine going back.

The ServerTemplate concept

The idea behind ServerTemplates is to boot any server from a small set of very generic images and configure the server dynamically at boot time. We noticed that the Linux package managers are very fast and running a “yum install apache” or “apt-get install apache” takes just a few seconds, so there is little value in baking such software into an image. There are special cases where every second of boot time counts, but those are very limited and at that point it’s still possible to create a more specialized image for that purpose.

Simplifying a bit, a ServerTemplate is composed of the following pieces:

  • a group of settings to define the type of server, e.g., i386 vs. x64, etc.
  • a reference to a base image that is to be booted
  • a list of scripts and Chef recipes that are to be run at boot time to install and configure all the software

The illustration on the right shows the layers that are typically found in a ServerTemplate, starting from the bare virtual machine at the bottom, the OS, and various layers of software packages, and finally the application at the very top. On Linux we prefer to boot bare bone images that contain just the OS, while on Windows it is often required to pre-install some of the larger software packages on the image and use the dynamic ServerTemplate functionality primarily to configure these apps.

What happens at boot time is the following:

  • when launching a server, RightScale passes the server its identity in the launch call using a crypto token that uniquely identifies the server to RightScale. This is important because at any one time many servers may be booting from the very same image
  • when it comes up the server contacts RightScale with its token to obtain instructions on what to download and run
  • RightScale also sends the server a set of variables that can be configured on the web site, this is the way dynamic information is fed into the config to specify things like the server’s name, test vs. production, names/IPs of other servers it needs to contact, etc.
  • the server then typically downloads packages from a distribution mirror and runs a set of scripts to install and configure everything
  • throughout the process, the server sends audit entries to RightScale so that it’s possible to monitor the progress on the web site and also for the persistent audit record

Using Server Templates

ServerTemplates provide a very modular building block approach to managing server configurations. In practice many different constituencies contribute to a ServerTemplate. Vendors, OS distributions, and RightScale provide the lower layers in the form of standard software packages. The sysadmin or operations team often provides standard configurations for fleet-wide software, such as logging, intrusion detection, user account management, network config, etc. Developers provide higher layers, such as app server install and the application code itself. The modular approach makes it easy to integrate all the pieces and especially to manage the update process.

The modularity of ServerTemplates also enables flexible software development and test practices. In our case we use the same building blocks to create a large variety of ServerTemplates that are appropriate for the different stages from development to production:

  • In production we use many servers that are rather specialized, so we end up with many ServerTemplates, e.g. for load balancers, app servers, api servers, etc.
  • For staging we start to aggregate functions so we are running fewer servers, this saves money and also simplifies updates a bit. To achieve this, we combine scripts that are in different ServerTemplates in production onto fewer ServerTemplates.
  • For test setups we combine again so we have a number of test systems without having to launch and manage too many servers.
  • Finally, developers often use an “all-in-one” ServerTemplate for they dev&test. This ServerTemplates combines all the building blocks in a single ServerTemplate.

The beauty here is that we can reuse the exact same RightScript and Chef cookbook building blocks that we use in production for the other stages of development. This reduces set-up time and issues where developers test configurations that have little resemblance to what will go into production.

Making ServerTemplates reliable

In IT management there is often a tension between flexibility and reliability: if everything can change at any moment it’s hard to lock down a reliable and reproducible configuration. We discovered this early on and spent a lot of engineering resources to provide a good reliability harness around ServerTemplates to solve the problem. Our solution has a number of aspects:

  • ServerTemplates are version controlled, so you can commit a version and come back to it at any later point. If you want to relaunch a server with last year’s version of a ServerTemplate you can. Or perhaps you just want to see a diff of what has changed since.
  • We mirror the Linux distribution mirrors such that a booting server retrieves the packages from a local redundant set of fast servers.
  • We also keep a daily snapshot of the Linux distribution mirrors such that when you relaunch a server with last year’s version of a ServerTemplate it can retrieve the software packages as they were at that point in time. This is under user control: you can “freeze” the repos to any day of your choice.

RightScale reliably launches thousands of servers using ServerTemplates each day. Keeping this machinery reliable as both RightScale and the underlying clouds we manage evolve at breakneck pace is a top priority for us and involves a significant amount of engineering. One of the tricks we use to stay ahead of the curve is that we use ServerTemplates ourselves: we leverage RightScale to manage RightScale. The benefits in terms of automation, control and reliability have been incredible, and at this point we cannot imagine going back to a pure machine image model of operation.

Summary

ServerTemplates are a leap forward over images, they enable servers to be launched and dynamically configured at run time – no tweaking machine images. This is the underpinning of truly automated infrastructure. They also support launching reliable infrastructure – you know what you’re launching at all times, it is completely repeatable. Finally, they are built on re-usable components, which saves untold time in creating new or similar configurations, or expanding from test and dev to production.

Comments (8)

Top Cloud API Sins

February 13, 2010 at 9:51 pm · Filed under Thorsten, cloud computing

… while I’m at it, I might as well… Here is a short list of the top poor design decisions that I’ve seen in cloud APIs. Let me rephrase that… Here’s a short list of the top API features that got in our way or simply didn’t work for us. There may well be other use-cases where these features make sense.

  • Listing of resources without the details, e.g., a list-servers call that doesn’t return all the details for each server. This makes it very expensive to poll for server state changes because the listing doesn’t have enough information and so one has to do a show-server for each individual server. Imagine polling for an account that has several thousand servers – ouch. It’s fine to have a “with details” flag in the request so one can get the bare list, but we’d always set that flag.
  • Not returning a resource id on creation. Some APIs don’t give you a server id when you request a server to be launched, the response is just “ok, we’ll launch a server”. This means you end up guessing “is that new server that just appeared in the list the one I just launched?”
  • Providing a task queue. Several APIs I’ve seen have a task queue that is supposed to provide updates on tasks that are in progress. E.g., you launch a server and you get a handle onto a task descriptor. For us that’s just overhead. Just include a state field in the resource itself and we’ll just keep track of the state changes on the resource. So if mounting a volume takes a while, create the volume resource and set its state to “attaching” (or whatever is appropriate). Having a separate resource to say “that volume you created is attaching” is just overhead and means that the state of a resource is now in several places.
  • Lacking publishable images or the equivalent of EC2′s user_data (small amount of data that is passed to the launching server via the launch API call). I touched on these in my previous blog post.

UPDATE: reviewing YACA (yet another cloud API) reminded me of another two sins:

  • Not returning deleted resources in a “list resource” call. In particular, terminated servers must be returned in a list servers call for a certain duration, probably at least for an hour. The reason is that otherwise the client has to infer that the server self-terminated or failed when it no longer finds it in the result of list servers calls. Well, we have seen multiple completely different clouds fail to list running servers. In the case of EC2, which lists terminated instances for a good amount of time this resulted in error emails alerting us of the situation. In another cloud this resulted in servers marked as terminated, which is an irreversible operation and often triggers alerts and automation. And then the servers “resurrected”. Ouch! Now combine this with the next sin:
  • Pagination that goes page-wise instead of using a marker, e.g. where you get page 1 or the first 100 resources and then issue a query for “page 2″ or “from 100 on”. Explain to me how a client can get a consistent resource listing when resources can be added and removed concurrently. This is particularly fun if the client has to infer deletion from the absence of a resource in the listing: was it deleted or did it fall through the crack between pages due to a different resource being deleted concurrently with the listing? The proper way to do pagination is using markers like Amazon does it, but for a cloud API I actually don’t see the value in pagination. We always retrieve the whole list.

If you’re working on a cloud API, please think twice if you’re doing one of the above. Again, I don’t know all the use cases, just ours.

Now here’s what I’d really like to see. This is what we’re working on for internal purposes and it’s not easy, which is an event based interface instead of a request-reply based interface. Request-reply is fine if you have a system that sends commands to the cloud. It’s a problem when you build a system that reacts to changes in the cloud because you have to keep polling all these resources. We run a good number of machines that do nothing but chew up 100% cpu polling EC2 to detect changes. Fortunately cpu cycles are cheap :-) .

Comments (12)

Older Posts »