To be successful with cloud, IT decision makers must first ask themselves:

1. What do I want to achieve with cloud computing? Agility? Scalability? Cost savings?
2. How do I select the best clouds for my use case?
3. How do I correctly architect my systems for the cloud?
4. What tools and solutions do I need to make it happen?

Over the past six years, RightScale has developed a deep expertise in cloud. RightScale engineers have worked with forward-thinking IT leaders from companies like Pearson Education, Fox Media, and CareCloud to help them develop and implement their cloud management strategy.

We’ve built a cloud management solution that will continue to serve companies as cloud adoption accelerates and as they build out their multi-cloud portfolio to include public, private, and hybrid cloud technologies.

RightScale now gives users the ability and choice to manage their cloud applications on 7 public clouds (Amazon Web Services, Rackspace Cloud, and SoftLayer among them) as well as with 3 private cloud providers, including Citrix CloudStack, Eucalyptus Systems, and OpenStack.

At RightScale, we’ve developed the expertise to solve the problems that plague corporate data centers, from managing VM sprawl and underutilized hardware to enabling self-service provisioning and metered chargeback, and we’ve built solutions for managing them into RightScale.

When the sixth RightScale Conference kicks off in New York City on June 13, some of the most progressive minds in cloud computing will be there, among them our engineers, our customers, and our solutions partners such as Zend, 10gen, and Riverbed who provide the expertise and the enabling technologies to help you design and develop applications for the cloud. You’ll have the chance to talk with these experts and take advantage of their collective knowledge to jumpstart your cloud project or fine tune your existing systems architecture or deployments.

Following a keynote from RightScale CEO Michael Crandell, we’ll feature presentations from RightScale customers including Pearson Education, Fox Media, and CareCloud. The afternoon will be devoted to breakout sessions on a variety of topics, including a track each on Private and Hybrid Clouds, HA/DR, and Big Data.

Private and Hybrid Clouds

We’ll be holding several breakout sessions on multi-cloud strategy, including one by RightScale Professional Services Architect Brian Adler who helps many of our enterprise customers custom design their architectures. He’ll share key considerations for building a private or hybrid cloud as well as selecting hardware, cloud infrastructure software, hosting vendors, systems integrators, and reference architectures.

Peder Ulander with the Cloud Platforms Group at Citrix will demonstrate how CloudStack, CloudBridge, and RightScale can enable Enterprise IT to extend resource pools beyond physical datacenter boundaries and leverage additional private clouds or public clouds to meet peak usage requirements and smoothly manage planned or unplanned capacity spikes.

Jeff Dickey, Chief Cloud Architect at Redapt, will provide specific use cases and examples for creating a hybrid cloud that can accommodate legacy applications while providing flexibility for future architectures and applications. He will also cover securing and segregating data, ensuring elastic demand, and accommodating mixed workloads on bare metal and hypervisors, as well reducing data center waste while increasing application to hardware efficiency.

HA/DR

Some view the cloud as a silver bullet to solve performance issues. If only it were that simple. Although the cloud does provide a superior way to scale hardware on demand, performance must be optimized at the application level to realize maximum gains. Apica COO Craig Irwin will present key strategic elements employed by innovative companies and share actionable insights on how they are leveraging technology to proactively identify bottlenecks, improve performance, and optimize their environments. Learn from high-profile crashes and common mistakes that enterprises make — and how not to become another headline.

Once you’ve optimized your application performance, you need to make sure your database can keep up. Senior Systems Architect Josep Blanquer will demonstrate how RightScale uses SQL and NoSQL databases such as mySQL and Cassandra to provide a scalable, distributed, and highly available service around the world that is designed to recover from failures of an entire cloud region.

Big Data

Because it exceeds the processing capacity of conventional database systems, big data is proving to be a popular use case for deriving huge benefits from on-demand, unlimited compute resources on a pay-as-you-go basis. In a Big Data 101 session led by RightScale Engineer Vijay Tolani, you’ll find out what others are doing with big data in the cloud and how to get started. He’ll cover solutions from RightScale technology partners IBM BigInsights, Couchbase, and MongoDB in the context of storing large data sets with NoSQL solutions and performing data analytics using Hadoop. Vijay will demonstrate the IBM BigInsights ServerTemplateTM and how it can be used to deploy analytics clusters for companies of all sizes. IBM offers free database and Hadoop courses through BigDataUniversity.com, which has more than 20,000 registered participants.

RightScale Zero-to-Cloud™ Training

If you’re looking for training on either RightScale fundamentals or advanced concepts, such as using RightScale with Chef or using RightScale to set up a hybrid cloud, we’ll be offering instructor-led hands-on courses onsite at the Javits Convention Center the two days prior to the conference. Our training classes sold out last conference, so get more details and sign up soon for RightScale training.

Get Your Free RightScale Conference Pass

So if you’re looking to move to the cloud — or you’re already there — we think you’ll find the RightScale Conference to be the place for expert advice on cloud computing and managing your cloud strategy. We’ll see you in NYC.

At the upcoming RightScale Conference in June, high availability (HA) and disaster recovery (DR) strategies will be one focus of discussion as this is a challenge that many companies are attempting to manage. When we advise our customers on designing their architectures for HA in the cloud, there are four tried-and-true steps we typically recommend:

1. Build for server failure

Instances in the cloud — just as in a typical data center — are ephemeral. You need to be prepared for server failure. Building for server failure begins with designing stateless applications that are resilient through a server or service reboot or relaunch.

• Set up auto-scaling so that your application can respond to dynamic traffic patterns based on a set of performance metrics.
• Set up database mirroring, master/slave configurations, and/or priming to ensure data integrity and minimum downtime.
• Use dynamic DNS and static IPs so that components of your application’s infrastructure always have the right context.

2. Build for zone failure

Sometimes more than just single servers fail — there are power failures, network outages, and lightning strikes. You need to make sure that your applications are prepared for zone failures. Zones (Amazon Web Services refers to them as “availability zones”) are distinct locations that are engineered to be insulated from failures in other zones.

• Spread the servers in each of your application tiers across at least two zones.
• Replicate data across zones. Note that this is usually cheap, though not free.

3. Build for cloud failure

On rare occasions multiple zones in a region can experience outages due to system-wide issues — the April 2011 Amazon Web Services (AWS) outage is a notable example. Each region is an independent system of resources with its own API endpoint — which is how we define cloud here.

To achieve multiple 9s of availability, you will need to have a process in place for cloud failures. Building across clouds can be difficult: APIs, services, and configurations differ. You will need to design your architecture using generic concepts (durable storage) yet deploy using cloud specifics (EBS volumes).

Cloud management systems abstract away these differences and make it easier for you to implement a fault-tolerant strategy by providing reusable building blocks. These building blocks can be used to not only withstand moving across different regions of the same provider, but also across different infrastructure providers.

• Back up or replicate data across regions or providers. Make sure you secure and authenticate your communications across these regions as the traffic will traverse the open Internet.
• Maintain sufficient capacity to absorb zone or cloud failures, using reserved instances if necessary.
• Remember to crawl, then walk: Build high availability across zones and then expand to multiple clouds.

4. Automate and test everything

As you set up your infrastructure to handle server, zone, and cloud failure, you should be automating your processes in the event such a failure occurs. Cloud management systems allow you to execute pre-planned failover processes across servers, zones, and clouds. In an emergency, time is precious, so automate everything.

• Automate backups so that your data is ready whenever disaster strikes.
• Set up monitoring and alerts to identify and pinpoint problems as they occur because you may not receive timely information from your cloud providers.
• Your disaster recovery plan is only good if you test it to make sure it works. By directing high loads to your production servers and disabling your various servers, services, and zones, you can test the ability of your infrastructure to withstand failure.

Cloud infrastructure has made DR and HA remarkably affordable compared to other options. Despite recent highly publicized failures, many organizations successfully run critical services on the cloud when they architect correctly and use the right management tools. For a reference architecture and additional best practices, check out this white paper on HA and DR scenarios.

Our customers run the gamut from the classic “two guys in a garage” success story all the way up to large enterprises, and many have already started to look at the private cloud for a lot of different and interesting use cases. What was once seen as an interesting science project has now been proven as a viable (and scalable) IT delivery model. Some companies are building “IT vending machines” that allow their internal users to quickly spin up pre-configured software stacks for use in siloed test and development environments.

Others are using their private cloud in combination with one or more of the public cloud offerings to create a hybrid cloud  in which workloads are assigned to one side of the fence or the other depending on the characteristics of the applications they are looking to deploy.

And as the cloud outages over the last year or so have sent companies searching for viable disaster recovery solutions, the hybrid cloud has proven time and again to be up to this task for many of our customers.

In my role within the Professional Services group, I have the opportunity to talk to lots of our customers, and the trend continues to point more and more down the private/hybrid path. And as each of these customers have their own distinctive application workloads, they also bring to the private cloud their own unique set of requirements, desires, hopes and dreams. And while we in Professional Services do our best to say yes (as opposed to our Sales guys, who *always* say yes ) there are technical considerations that need to be considered, along with the normal practical ones like budget and schedule.

The companies I see that are the most successful at building private clouds are doing it for a few typical reasons, the first of which is workload/infrastructure interaction. In this case, the private cloud is being built for a very specific workload, and the hardware being procured is appropriate for the task(s) at hand. For example, one of our customers has an application with very high CPU demands. To most efficiently handle this workload in their private cloud, they are purchasing high-end compute hardware that has the horsepower to service the high demand. While there are public cloud resources out there that might be able to handle the load (such as cluster compute, high bandwidth instances), the quantity of the data involved is also quite substantial, and the cost (and latency) of pushing this data around, along with the security complications of shipping this data across the public Internet, has shown that the public cloud is a more costly solution over the long run.

Other customers have applications that demand (well, “very strongly desire”) high IOPS rates from their storage devices, and anyone who has run benchmark tests on the current selection of cloud-based storage systems knows that “your mileage may vary” doesn’t just apply to your choice of automobile and driving style. In order to get a higher (and much more consistent) IOPS rate, some customers are building out their private cloud with solid-state drives (SSD) for the nodes that require super-fast reads and writes. For the other parts of their application with more pedestrian needs in regard to storage, they have more traditional disk subsystems in place, and launch nodes in the appropriate pool of resources (segregating by zone or cluster).

Still other customers are using the private cloud to meet their security compliance needs. Since all the data stays within their own infrastructure, and never even sniffs the public Internet, they are able to run cloud-based workloads that still meet all of their stringent regulations. Basically, if you can keep your packets local and eliminate the need for them to travel out into the Wild, Wild West of the public Internet, you are going to have happier packets and unhappier bad guys who are out looking for those packets.

All of these use cases, the considerations surrounding them, and lots of other related and relevant bits can be found in our recently published white paper on architectural best practices for designing private and hybrid clouds.

Much like sex in high school, the prospect of having a private or hybrid cloud can be exciting, but if you’d never done it before, it can be reassuring to have that mentor who can tell you what to expect. We’re here when you need us.

Lack of excitement is not something the cloud market can be accused of! Citrix just announced a bold roadmap for its CloudStack platform coming right on the heels of a Eucalyptus and Amazon announcement to extend API compatibility and just two weeks before the OpenStack summit. My prediction for 2012: we will exit the year with as many cloud APIs as we had when we entered, and it doesn’t really matter!

Railroad gauges & compatibility...
(Source Wikipedia)

Citrix’s announcement is interesting because it is the first bold move around CloudStack coming after the acquisition of Cloud.com by Citrix. Over the past year the CloudStack team and Citrix have worked with the OpenStack project, contributing code and adopting the Swift storage service. I know that the CloudStack team was always quite open around adopting OpenStack technology but at the same time it always seemed like they would take a big step backwards if they adopted the compute portion. After all, CloudStack has powered some of the most successful large scale private cloud deployments in production and is powering a good number of large public cloud service providers too. From that point of view it only makes sense for Citrix to stick to its proven technology and continue to develop the CloudStack codebase, pulling-in parts of OpenStack where appropriate, such as for the storage service, and using its own technology for other parts. With this move to make CloudStack an Apache Foundation Project, Citrix puts a stake in the ground around its commitment to CloudStack as an independent technology, and its roadmap to remain competitive on the API front.

Last week Amazon Web Services and Eucalyptus jointly announced a partnership to enhance the compatibility of Eucalyptus with AWS APIs. This announcement is a great validation of Eucalyptus’ strategy which has always been to be API compatible with AWS instead of trying to establish yet another API. It’s also interesting from the point of view of how AWS’ strategy around its API has played out over time. When Eucalyptus first gained momentum and RackSpace first worked on their cloud offering there was a lot of speculation whether AWS would try to prevent API clones, even possibly with legal means.  It was clear at the time that the issue had been discussed internally at Amazon but AWS never said anything publicly one way or the other. This wait-and-see approach let them observe the market’s evolution and start playing their cards when it suited them (which they have now done). At this point we can only speculate whether an earlier play would have been beneficial — for example, to support Mark Shuttleworth’s appeal half a year ago to the OpenStack community to adopt the EC2 API. In any event, now that Citrix has announced that CloudStack will be AWS compatible (they do mean API compatible even if their press release doesn’t state it verbatim), it’s clear that AWS API compatibility will become increasingly common in the IaaS market.

But what does all this mean? How important is Cloud API compatibility? What should Cloud API compatibility even mean?

The pitfalls of Cloud API compatibility

I’ve said it many times and I’ll repeat it again: it’s the semantics of the resources in the cloud that matter, not the syntax of the API. This means that “API compatibility” has to reach very, very deep to be meaningful. Let me give you a couple of examples around EC2.

#1: EC2 has a number of discrete instance types, from a fractional core to 8, from 512MB to 64GB of memory, from no local disk to 4 spindles, from shared network bandwidth to dedicated 10Gps ports, etc. While all these instance type can be replicated in a local cloud, it’s not an easy task. This means that moving workloads to local clouds where instance types differ could well require some non-trivial architectural adaptations. And even if it is possible to exactly replicate the AWS configurations, it turns out that doing so may actually not be desirable! Some of our large hybrid (public + private) cloud customers report achieving significant cost savings precisely because they are using customized instance configurations specifically tuned for their applications.

#2: EC2 EBS block storage devices have quite peculiar performance characteristics (that are not universally liked…) both for regular I/O as well as for snapshots. It would seem rather crazy to try to duplicate those characteristics and not benefit from improvements that are possible in a smaller purpose-built private cloud. But by doing so the operating procedures for deployments may change rapidly making the notion of “compatibility” questionable. Put differently, should one retain compatibility if compatibility is worse?

#3: The AWS clouds are divided into regions that are independent of one another and all resources are bound to a region. This means that if you create an image in the us-east region and want to have a DR set-up in the us-west region you need to copy the image explicitly to us-west and keep maintaining any updates you make. Some of the more interesting innovations being pioneered by other cloud providers allow the ability to publish images and even volume snapshots worldwide across regions. Does something like that break the EC2 API? Not necessarily, but it certainly changes tools and the way one would maintain DR set-ups. So, API compatibility doesn’t help much here either.

#4: Along those lines it’s interesting to see how Softlayer leverages their worldwide network infrastructure to offer each customer a worldwide private network: the 10.* addresses of all instances worldwide are interconnected (on a customer-by-customer basis). That makes operating servers across multiple regions a lot easier than in EC2. If Softlayer were to use the EC2 API they could offer these benefits without being “API incompatible”, yet global deployment architectures and tools to manage them would be radically different between Softlayer and AWS.

The point of all this is that API compatibility is not a panacea for cloud portability or interoperability. For a simple tool that launches a few servers, API compatibility is helpful. But at the same time the amount of work to port that tool to a different API is minimal as well. When it comes to more important aspects of cloud appeal — real elasticity, high performance, cost efficiency, better governance and control, failure isolation & resiliency — then API compatibility is simply not a ticket to portability across clouds.

The bigger compatibility picture

The bigger picture around Cloud API compatibility starts with the observation that there are really many AWS APIs, and the EC2 API is really only a relatively small portion of the pie. And even there, I would claim that there are actually two EC2 APIs: the “EC2 classic” API and the “EC2 VPC” API for the virtual private cloud. They are very similar to one another, but have significant differences beyond what you might imagine the VPC differences to be. This ranges from VPC security groups having egress rules which “classic” security groups don’t all the way down to little trip-me-ups like being able to move IP addresses from one instance to another using one EC2 classic call but requiring two EC2 VPC calls.

But to get back to the bigger picture: to me “Amazon compatible” really brings all 20 services offered in the AWS cloud into view (+/- a couple depending on how you count). Many AWS deployments use features that are close to EC2, such as Elastic Load Balancing, features that are further removed, such as the Relational Database Service or the Simple Queue Service, and features that are not very private cloud amenable, such as the CloudFront content distribution service. For many organizations, if the services they use aren’t included in an “Amazon compatible” cloud, then it’s not really compatible, and loses attractiveness — hence the long-running claim of “lock-in” around this proliferation of services.

My prediction is that increasingly cloud users will look for equivalent portfolios of services in clouds and less for strict API compatibility. For example, RackSpace added load balancing as a service support to their offering not too long ago. SoftLayer offers a content distribution service. And if one looks at the services around Google AppEngine there are Task Queues, Email services, various SQL and NoSQL storage services, and more. It’s clear that other cloud providers also see the portfolio of cloud services as being key. Currently these various services are somewhat equivalent to what AWS offers and for many of them the semantics beyond the API veneer are even more important than for EC2. It will be really interesting to see how the cloud compatibility — or should we say cloud “equivalency” — landscape evolves around these cloud service portfolios.

Summing it up

I have to say that Citrix’s three-fold statement that ‘our technology is the best’, ‘we’ll open source it under the apache foundation’ and ‘we’ll make it AWS compatible’ is bold. Since I’ve commented mostly on the AWS compatibility aspect, I should add that their increased ‘open sourcing’ of CloudStack does open the door for others to complement anything Citrix does with more AWS compatible services. It’s still a wide open game at this level of the cloud market, with, as I said at the beginning, no lack of excitement!

My recommendation to cloud users, however, is twofold: 1) do continue to dive in with these technologies and expand your use of different cloud offerings for more & more sophisticated projects, 2) don’t get too hung up on the compatibility debate and fail to see the cloud automation forest for the API trees.  The issues of portability and interoperability are being addressed well at higher levels of the stack – that’s what we do at RightScale, after all .

A few weeks ago, this brilliant idea crossed my mind. Why don’t I build a private cloud at home, in my garage, so that I can get the experience of setting up the technology?

Vijay told me I was crazy, so to prove just how easy it would be I did it anyway, and as a result I’ve had a private cloud running in my garage since early March.

First, Some Background

Before I get started there are a few housekeeping items I’d like to quickly address.

TL;DR – Too Long, Didn’t Read

I think it’s only fair to tell you that this post is a bit lengthy since I wanted to tell the full story. I also wanted to cover as much of the details of the process as possible. Even so, I wasn’t able to deep dive into everything so if you have questions please fire them off in the comments and I’ll be happy to keep the conversation going there. With that said, grab a cup of coffee and enjoy the story!

If you’re looking for the tl;dr version the diagram under So Meta… should tell you everything you need to know.

The Road Ahead

It may also be helpful to have a quick outline of the topics I’m going to discuss.

First, I’ll walk you through some of the decision making process. How did I arrive at the decision to build a private cloud? How did I choose the technologies and hardware?

Then I’ll tell you a bit about the mechanics. How did I put the pieces together? What does my infrastructure look like?

And lastly I’ll show you some of the fruits of my labor (my favorite part).

Cloud-in-My-Garage Epoch

I had been contemplating an upgrade to my local home file server for quite a while. My research had lead me through all of the NAS devices and other consumer options but at the end of the day I really wanted to build my own system.

That realization led me to looking at some rack mountable servers with hot swappable SATA on eBay for hours on end. When I finally found the right server, and it was enroute to me via the logistics loving courier, I realized that I wasn’t far off from being able to set up a small private cloud with just a little more hardware.

Technology Shuffle

In order to proceed with some confidence I needed to make a few decisions about what technologies I would use. I wanted to be able to attach additional storage to my compute VMs. I also wanted to have the option for my cloud enabling technology to handle network security.

Another requirement was technologies that could be acquired and run for free, but had enterprise support available. Of course, whatever I chose would need to be compatible with RightScale so that I could easily provision and automate anything I might choose to run on my new garage cloud.

For the cloud enabling technology I settled on Citrix CloudStack 2.2.14, because it met my above criteria and has been proven to power some of the largest public and private clouds in the world reliably. Of course I could also have chose Eucalyptus or OpenStack which are technologies supported by RightScale. In fact, I’m seriously considering doing it all over with each of them for the experience.

From there, the decision to use the Citrix XenServer 5.6 hypervisor was a clear choice since I knew it would work seamlessly with CloudStack, plus it supported security groups.

So Meta..

I realized pretty quickly that the loss of my CloudStack Management Server could be pretty catastrophic if I intended to run anything of consequence on my garage cloud. In order to have some resiliency I would have to do continuous backups of the management database at a minimum. Having a runbook or automation scripts to easily reconfigure the management server from my backup would be even better.

If you’re thinking that sounds a lot like a RightScale ServerTemplate, I’m glad you’re enjoying the koolaid you’re right on the money. In fact the Database Manager for MySQL 5.1 handles safely and continuously backing up your database content, as well as providing tools for restoring that data in case of a disaster.

Given that I needed most of the functionality of the database manager for my management server, I decided I would simply make a clone of that template and add the CloudStack specific installation to that new template. The result is my CloudStack Management Server Single-Node ServerTemplate which you can use to set up your own small scale private cloud without having to worry about the CloudStack installation and disaster recovery planning.

The AWS Node’s Connected to the Garage Cloud

By now you may be asking yourself how I was able to use a RightScale ServerTemplate to setup my CloudStack Management Server on my bare metal hardware inside my network. If you took a closer look at the diagram above you’ve probably already figured it out, I’m not. Instead I’m running my CloudStack Management Server on an AWS instance and connecting it to my network via an OpenVPN IPSec tunnel.

Let’s just get this out of the way, it’s a little crazy to run the cloud management server for your private cloud; on a public cloud. I did it so that I could take full advantage of the configuration and orchestration tools built into the RightScale Database Manager ServerTemplate. Also that way I didn’t have to “waste” a physical system that I could be utilizing for more important tasks, like running VMs!

This approach has it’s own issues of course. Such as, How will my management server communicate with my hypervisor hosts? Will the images be stored on the management node, requiring my hypervisors to essentially download their OS from the internet each time?

As it turns out, setting up connectivity became the most complex part of my implementation, and I can’t say that I would recommend it to anyone. I wound up using OpenVPN, setting up my All-in-One CloudStack Management server as the VPN server, and connecting my home network to it with an OpenVPN client.

*NOTE: I did do a fair amount of hacking to get OpenVPN to work in my environment, so that my management server could talk to all the servers on my local network. That’s built into the ServerTemplate linked above, but chances are you’ll need to do some tweaking to get your network working smoothly. Your mileage may vary, caveat emptor, etc.

Needless to say, networking is hard, when you get stuck it’s probably a routing problem, and I’ll stick with software engineering.

On the topic of having to traverse the VPN for images and disk I/O, since my file server was acting as the primary and secondary storage for CloudStack, my hypervisor hosts would be using a resource local to my network for it’s root drive as well as any additional volumes I would attach. So, no weird situation where VMs are traversing the internet or my VPN.

Cloudifying the Bare Metal

With my storage, CloudStack Management Server, hypervisors, and networking figured out it was time to bolt everything together which meant spending some quality time with the CloudStack Install and Administrator guides.

Since I didn’t have a very large cloud, nor did I have any serious networking gear to setup VLANs etc. I chose to use the basic networking mode of CloudStack. I also chose not to create a separate storage network to handle the burden of disk I/O traffic.

With such a simple configuration and flat networking my configuration came together pretty quickly. The one thing which was fairly confusing is that once I’d added all of the appropriate components (a Zone, Cluster, Hosts, IP ranges etc) I expected to be presented with a big red “GO” button to initialize my environment. Instead, CloudStack was periodically checking the configuration of my environment and when it determined that it had everything it needed the initialization started automatically. That struck me as not terribly intuitive, but it became clear if I was watching the log file.

Orchestrating the Garage Cloud

Once CloudStack was happy with my environment and configuration I could take a step back and start interacting with my new cloud the way that I’ve become familiar - using RightScale.

Cloud Registration Two-Step

The steps were easy here. I went to Settings -> Administered Clouds in the RightScale dashboard and added a new CloudStack based cloud. I was then prompted to provide the name of my cloud (I know, I get no style points for ‘rjghome’, it’s boring) and a set of CloudStack API credentials with Admin permissions.

Now, this is where the process gets a tiny bit confusing. Once I’ve completed those steps, I’m immediately asked for another set of credentials. This time I’m not asked for detail like the API endpoint URL or the cloud name, but just a set of API credentials with User permissions. What’s happening here is that the first step is simply registering my private cloud with the RightScale dashboard. That first step requires Administrator permissions because it has to be done by someone who has the authority to allow the cloud to be consumed by RightScale users.

Imagine that I were doing this as an IT project for a corporation, and I had built out a much larger infrastructure in my corporate datacenter. The purpose of that private cloud would be to allow people in the organization to use RightScale to deploy their own workloads on that private cloud without having to depend upon IT. However, IT would have to “register” that cloud with RightScale, making it available to other consumers within the organization. That step of initial registration requires Administrator permissions, then the IT department can distribute API credentials with User permissions to each group that will be consuming. You can read more about registering a CloudStack cloud in the support docs.

So, it’s a producer/consumer model. Make sense? Cool!

RightImages for CloudStack

Okay just one more step before I can start launching servers through RightScale on my Garage Cloud! Each of the ServerTemplates that RightScale publishes now includes a placeholder for a RightImage for each of the supported Hypervisors on CloudStack (KVM, XenServer, ESXi/VMWare) so that when you import them from the MultiCloud MarketPlace they will “just work.”

However, in order for them to “just work” you need to upload the appropriate RightImage into your CloudStack environment, specifying the MD5 fingerprint of the image. RightScale will then match up the image you’ve uploaded to your CloudStack environment with the placeholder that’s in the published ServerTemplates allowing you to launch them on your cloud.

So…Why again?

After completing all of those steps I was finally able to launch a simple Base ServerTemplate for Linux which brings us full circle to the beginning of this post. It was super exciting to see a server become operational (something I’ve witnessed thousands of times) on my own private cloud, knowing that it was running on hardware I setup and configured myself.

So, really, why did I do it? Really the short answer is “because I could”, but it was also an excellent learning experience. This exercise gave me the opportunity to really see the moving parts of a private cloud implementation, and let me get my hands dirty with the actual technologies.

Plus, how cool is it to be running a full 3-Tier PHP stack in about 30 minutes on your own private cloud?

I also now have a great test bed for building ServerTemplates without running up any public cloud usage fees. I guess that means I’ll have more motivation to make my WordPress All-in-One ServerTemplate multi-cloud, eh?

IDC Frontier Cloud is one of two clouds RightScale supports in Japan.    Over the past several years, IDC Frontier has been delivering managed public cloud solutions to the Japanese market.  Recently, they added self-service cloud products to their portfolio which is the focus of our integration.

IDC Frontier Cloud, as integrated with RightScale, is built on top of Citrix technology.  In particular, they have deployed CloudStack and CloudPortal.  CloudStack is the technology layer powering compute, network and storage that exposes the cloud API that RightScale uses to control, manage and orchestrate resources within the cloud.  CloudPortal provides a web based user interface that enables users to interact with IDC Frontier for things like customer support (ticketing, help desk, forums), payment and account management.  It is integrated with CloudStack and also gives users abilities around virtual machine provisioning.  The combination of RightScale on top of the Citrix technology gives users the ability to control more than their virtual machines and raw cloud resources.  They can provision dynamic application workloads using the power of RightScale’s cloud management solution.  With our pre-configured application stacks and our automation engine complete with application monitoring, alerting and scaling, users focus their efforts on deploying and maintaining their applications in the cloud.

RightScale adds ‘Scale’ to IDC Frontier

With the popularity of social gaming and web applications in APAC and growing need for resource pools in e-commerce, e-publishing and advertising, IDC Frontier and RightScale provide a complete solution for businesses looking to leverage cloud.  RightScale’s ServerTemplates provide the basis for server configuration that enable customers to run production web-scale architectures in the cloud.  Day 1, IDC Frontier has ported many of our ServerTemplates on their cloud and customized them to leverage web services that are available within the IDC Frontier cloud.  Those services include their own file storage solution, called Contents Store, and an internal dynamic DNS Service called NSUpdate.  Both of these services, along with advanced networking settings such as Firewall rules and LoadBalancer policies are available for configuration with the combination of RightScale and IDC Frontier Cloud.  You can read more about this offering on our support site, including how to set up a three tier application stack using RightScale ServerTemplates here.

RightScale continues to partner with the best clouds around the world, and continues to focus on helping our customers access the optimal resource pool for their cloud workload.  As some of us here at RightScale like to say…it’s a cloudy world

Give IDC Frontier and RightScale a go, and as always, let us know what you think.

A recent O’Reilly Radar article on big data in the cloud included a mention of RightScale and its ability to orchestrate server management across multiple clouds. It prompted me to think about the role of RightScale in supporting the implementation of big data solutions for our users. But first, what exactly is big data? Edd Dumbill, a contributor to O’Reilly Radar, offers this explanation:

“Big data is data that exceeds the processing capacity of conventional database systems[because it] is too big, moves too fast, or doesn’t fit the structures of your database architectures.”

A commonly accepted way to identify big data is to determine if it meets any of these three key criteria:

Volume – The quantity of data is too large to fit into conventional database structures. There is a tremendous amount of information that can be gained by analyzing data stored on social networks, for example, but you have to analyze all of the data to extract the portion for your need at hand. Here are a few examples of the volume of data stored in social networks:

• Facebook is expected to have more than 1 billion users by August 2012, handles 40 billion photos, and generates 10 TB of log data per day.
• Twitter has more than 100 million users and generates some 7 TB of tweet data per day.

Velocity – Data is being generated at a meteoric pace. In many cases, data needs to be processed in real time or its value is diminished tremendously. Example use cases include:

• Sensor Networks – A single airplane engine generates more than 10 TB of data every 30 minutes. In the past, this data was deleted at the end of each flight. How would you feel if you knew that your airline was instead using this data to proactively monitor the health of its engines and replacing them before they failed?
• Stock Market – For every trading session, the NYSE captures 1 TB of trade information. In this case, processing stale data is worse than processing no data at all.

Variety – Did you know that 80 percent of the world’s data is unstructured? This data does not easily fit into the structures of a relational database, so storing and analyzing it requires a new approach. Two examples of unstructured data mining are:

• Transaction Records – A recent Forbes article highlights how Target tailors marketing campaigns to pregnant women. The company accomplishes this by analyzing customer transaction records and assigning each shopper a “pregnancy prediction.”
• Social Media Accounts – There is invaluable data contained in Facebook and LinkedIn users’ posts, emails, and text messages. But conventional data analytics platforms do not easily allow for storage or analysis of free-form text.

What is the connection between big data and cloud computing?

Analyzing big data can potentially lead to big profits, but is it worth the time and money? I recently attended a seminar where cloud computing was described as “the democratization of software.” I think this is particularly apt to any discussion of big data because, historically, analyzing a large data set required significant investments of both time and money. By using servers on a pay-as-you-go basis, you not only can reduce your costs but also benefit from increased compute resources to faster analyze your data. For example, let’s say you want to run a monthly report with Hadoop that takes 100 compute hours to complete.

Option 1: Build a 10-node Hadoop cluster

• Your job will take 10 hours to complete.
• Your cluster will be utilized for 10 hours out of each month and will remain idle for the remaining 720 hours.

Option 2: Build a 100-node Hadoop cluster in the cloud

• Your job will take 1 hour to complete.
• Your cluster will be terminated after the job is complete, and you will only be charged for the compute resources used.

With Option 2, you can gain access to the same tools everyone else has at a fraction of the cost. There is no need for a multi-million dollar data warehousing solution. RightScale and our partner, IBM, offer a solution for performing intensive business analytics, batch processing, and grid computing in the cloud of your choice. IBM has developed a set of RightScale ServerTemplates^TM for its Hadoop-based BigInsights product. The best part is that the BigInsights Basic Edition ServerTemplates are available at no charge for data sets up to 10 TB. IBM also offers BigInsights Enterprise Edition ServerTemplates, which include many additional tools and features. These ServerTemplates are available in the RightScale MultiCloud Marketplace.

Making the System Adjust Itself

First, let’s look at one that’s been out there for a little while on version 5.7 of our RightLink agent – the ability for a server to call a script on a different set of servers. Let’s say I’m running this command from a boot script on an application server:

% rs_run_right_script -n "Register App Server" -p "IP_ADDRESS=text:10.100.250.200" -r "provides:type=proxy"

In layman’s terms, this means “find all the proxy servers out there, and run the ‘Register App Server’ script passing my IP address.”  The script will run on the proxy server and register that new app server’s IP address to the available app server pool. On the flipside, the application server can call a “Deregister App Server” script when it is terminated to remove itself from the pool.

So if you need an army of new app servers, you can simply click a few “launch” buttons (or adjust the size of a RightScale array), and voila, the system adjusts itself as the servers become operational. In other words, we don’t just spin up new instances, we help you code how the overall system should behave. (Oh, and this works on Windows too!)

Pushing a Single Change to the Whole System

I know you sysadmins out there write bug-free scripts.  But let’s just say you found a bug in a script one day (we’re talking hypothetical here), and realized that script was being used by 10 images, and these images were behind 100 servers. Now you have to create all new images and cycle all those instances at some point. There is a high possibility for mistakes when you have to manually hunt down and update all these images and instances.

With last week’s release, we completed a request that was at the top of our feedback forum to help with just this (see requests here and here). Now, with a click of a few buttons in our UI, you can queue up this change for all your servers, and have a complete record of what each ServerTemplate and Server is now using.

Here’s a glimpse into this “bulk update” capability. Just click “Update Selected” on both screens below:

…and then you’re done.

Configuring Systems

Finally, and also with last week’s release, we made it even easier to configure your entire system at once. As you specify the variables you need to define for your system, you can also specify the categories these variables belong to (see original feedback here). Furthermore, you can even specify that some variables are advanced – hiding them from view for most users who may never need to manipulate them. This allows you to focus on just the area you wish to change within your system when editing the configuration, and not on some giant list or JSON file.

More to Come

Since RightScale is a SaaS platform, we can continue to evolve our ability to manage cloud computing at scale and roll out new features to all of our customers with each release.  Please keep the feedback coming, and we’ll keep making your cloud life easier.

RightScale has focused on automation from day one. We provided auto-scaling of server arrays early on: automatically launching and terminating servers based on monitoring metrics, such as cpu load. Something most newcomers don’t appreciate is that making the call to launch the next server when the cpu load goes up on the running ones is not the difficult part. The difficult part is bringing the new servers into full operation. That involves loading all required software, configuring everything, and connecting the server with other services, such as load balancing and the databases. This is why a big piece of the RightScale functionality concerns itself with configuration management and automating the entire boot process all the way to the point where the application is in production.

With coming releases we are continuing to build on top of this platform and introduce server orchestration. Server orchestration uses a workflow language that lets you automate at the level of RightScale resources, such as servers, deployments, etc. The first functionality we implemented is to let you customize the three key pieces of auto-scaling: (1) deciding when to scale up or down and by how many servers, (2) launching new servers, (3) terminating existing servers.

The way this works is that the RightScale system calls a user-defined decision function every minute to find out whether the server array should be scaled up or down and by how much. The decision function simply returns an integer that indicates how many servers to launch (value >0) or to terminate (value <0), or a value of zero to keep the server count the same. The decision function can retrieve monitoring data using our API and do a calculation similar to the built-in one or it could do something completely different. An interesting example would be to use knowledge of application specific state and metrics to better predict requirements. You may be able to tell early that a flash event is coming and that you need to launch a large number of servers all at once. That’s just one example, the sky is really the limit and I know some of our customers have pretty cool ideas in this area!

When the decision function asks for more servers, RightScale runs a scale-up workflow to actually launch the servers. This puts you in control of how the servers are to be launched and creates an interesting opportunity to carefully manage where the servers are launched. For example, you may want to ensure your servers are equally spread across a number of datacenters for availability reasons. Or you may want to launch where it’s the cheapest. Very similarly, the scale-down workflow can be picky about which servers are being terminated. In the built-in auto-scaling we terminate the oldest servers to ensure a continuous refresh of the running stock. But for some applications it’s preferable to terminate the youngest servers. In addition, the scale-down workflow can gracefully shut down the application, take a last backup, save away log files, and then terminate the server.

As we designed the orchestration functionality we kept coming back to two key requirements: concurrency and fault tolerance. We need to express concurrent activities with ease because, when one operates on many servers, it’s the only way tasks complete in a reasonable amount of time. For example, to perform a rolling upgrade on a number of servers the orchestration ought to grab a set of servers, run them through the upgrade process in parallel, and then move on to the next set.

Where orchestration becomes really exciting is when it is used to recover from failures and automatically relaunches failed resources, possibly in a different datacenter or cloud. That immediately raises the question about the resiliency of the orchestration process itself: what if it is affected by the same broader failure and can’t perform the recovery? Similar concerns arise when an orchestration process runs for a long time. The array auto-scaling example above could be implemented using a “parent” workflow that runs forever and invokes the decision function and scaling sub-workflows periodically. And again, this execution must be resilient to failures.

In order to provide good support for concurrency and to offer a fault-tolerant execution environment we decided to base our orchestration system on a workflow language that is built around the open source Ruote workflow system. Ruote offers a multitude of very nice structured concurrency constructs. For example, you can express strategies such as “run concurrently and wait for all” or “run concurrently and wait for the first, then cancel the rest”. The latter may sound unusual but it’s useful when you need a resource and you want to try multiple avenues and pick the first one that succeeds.

Amazon SWF came in very handy to ensure fault-tolerant execution of the Ruote workflows. We retargeted Ruote to leverage Amazon SWF as an execution back-end with the result that workflows are executed by many servers distributed across multiple availability zones. SWF takes care of scheduling the execution of workflow actions, collecting the results, and then atomically handing the results back to Ruote so it can schedule the next wave of actions. The result is a highly resilient orchestration system that can continue the execution of workflows in the face of major failures.

We’re obviously very excited about the upcoming features and can’t wait to make them available to our customers. Now that Amazon SWF is live we’re on the home stretch and hope to be ready for a private beta shortly after the upcoming release. If you’re interested in early access, please send me an email.

RightScale cloud management provides a ‘single pane of glass’ to private and public cloud resource pools where administrators can provision and launch servers in a matter of minutes. In June of 2011, we launched our hybrid cloud product called RightScale myCloud to further deepen our integration with CloudStack’s private cloud infrastructure solution. With today’s announcement, myCloud will incorporate the CloudStack 3 enhancements and make the process of implementing and building a private cloud easier while offering more choice.

Let’s examine a few key features of the CloudStack 3 release and what they mean for RightScale customers using Citrix CloudStack technology.

Storage

Before, most customers were either using off cloud storage mechanisms in their CloudStack deployments, or using a public cloud storage engine, such as Simple Storage Service from Amazon or Cloud Files from Rackspace. Now, with Swift integration, customers will have the option to use their own storage with RightScale.

Networking

We often see our customers require complex and custom networking configurations within their deployments and clouds. Having more flexible networking configurations, specifically the catalog of networking services along with NetScaler integration, when constructing their deployments will enable additional use cases and capabilities that can be automated with RightScale.

Our integration with private cloud infrastructure partners gives RightScale the ability to focus solely on multicloud management. With RightScale myCloud you can manage multiple private clouds in your data centers and link these with public clouds. You can scale across your clouds and back up or failover among them. You can configure, monitor, and operate all of your cloud resource pools from one Dashboard and API — choosing from global public clouds such as Amazon, Rackspace, SoftLayer and many others.

To find out more, view our webinar with Forrester Research about how to get started managing a private and hybrid cloud. We also offer a free trial of myCloud with CloudStack, to get started you can contact us here or visit us at the Cloud Connect show this week in Santa Clara – we’re at booth #604.

Introduction to RightLink

RightLink (link) is the instance side agent that supports RightScale’s RightNet protocol. The agent provides an improved and secure ability to leverage RightScale to manage large numbers of instances in the cloud. In the RightScale architecture, we leverage a light-weight RightLink agent on every instance to support our latest automation features. Prior to RightLink, which was released a bit over year ago, RightScale leveraged the command execution features of SSH to perform tasks on remote instances. With the introduction of RightNet and the RightLink agent, we are no longer reliant on SSH access for instance management.

The RightLink agent communicates with the core RightScale systems using the Advanced Message Queuing Protocol (AMQP). RightNet leverages AMQP’s Simple Authentication and Security Layer (SASL) support to perform a basic session authentication to ensure that the RightLink agent is talking to a legitimate RightScale core component (a broker in our lingo). This session authentication uses a shared key to authenticate the ends. After the session is authenticated RightNet uses payload encryption (openssl with X509 certificates, PKCS7 envelopes and AES 256 CBC cipher for encryption) to protect that data while in transit, and to provide a much stronger authentication mechanism (public-private key versus only the shared key of the session). Both of these security features are to ensure that packets are properly segmented and protected in the highly multi-tenant aspect of the cloud.

All our version 5 (v5) RightImages (and Multi Cloud Images, MCIs) include the RightLink agent by default. We started releasing v5 images over a year ago, and have seen a large, but not complete, adoption. For those of you still on v4 images, I am going to try to give you a couple more security motivations that may encourage you down the upgrade path.

1. Ability to restrict SSH access on the instance: Because RightLink does not use SSH you can restrict access to the ssh service on Linux systems. With non RightLink enabled images (i.e., v4 and earlier by default), the RightScale platform ran scripts on the instance by ssh-ing into that instance directly, thus the need for ssh port to be accessible on the instance from the RightScale platform usually meant that it was accessible from any IP address. This created some exposure with potential brute force attacks. I will say that by default, RightImages configured SSHD to support public-key authentication only, so the risk of brute force password guessing was not an issue. What was an issue was that any vulnerability found in the SSHD server would then be potentially exploitable by anyone on the Internet.  With RightLink, this exposure can be mitigated.
2. Managed SSH: In addition, v5 RightImages introduce a “Managed SSH Login” feature. This allows you to use a different SSH key for each user logging into a server. It can either use an SSH key uploaded by each user or the dashboard can generate a key for each user.  When using EC2 you may still select an EC2 SSH Key when launching the instance, however, it’s only really necessary if you need to log-in before RightLink starts to troubleshoot something in the bootstrap process. Note that the SSH connection is from your desktop system (wherever you are running the dashboard UI from, not RightScale) to your instance, thus working seamlessly with any SSH access restrictions you put in place.

SPOILER-ALERT: one of the items we are working on for RightLink v5.8 (next version coming out) is a Managed SSH Login that will bind each RightScale authentication principal to a distinct, non-root Unix user whenever they login via the dashboard. This is intended to improve the login auditing as well a enable each user to load a customized shell profile. We’d be very interested in your feedback as to the usefulness and desire of this specific feature.

Upgrade options

The cleanest and best way to move to v5 images is to find a v5 ServerTemplate, clone it and make the modifications needed to effectively duplicate the functionality you currently have. This will work like a charm if you if you did your scripts right and took a modular approach to deployment.

Next option is to change the RightImage (i.e. Multi Cloud Image, MCI) you’re using to a v5 one and relaunch. The V5 execution of RightScripts is almost fully compatible with v4 so, in theory, that’s all you need to do. The catch typically is that this brings updated versions of the OS and packages with it and may cause some incompatibilities. You will probably spend a bit more time troubleshooting this avenue.

Lastly, you can get RightNet support by RightLink enabling your v4 instance (see http://support.rightscale.com/12-Guides/RightLink/04-Creating_RightScale-enabled_Images_with_RightLink), and many might be motivated to go that route. I would encourage you to move to v5. While you’ll get the “not using ssh for command and control” benefit, you will miss many other benefits of the v5 image update.

Why Again?

Because there are some really cool features in v5:

• Managed SSH
• Bug fixes
• Faster Execution of Operational Scripts
• Added Chef Support in addition to RightScritps

More details can be found http://support.rightscale.com/06-FAQs/FAQ_0180_-_What_are_the_differences_between_v4_and_v5_RightImages%3F

It will take a bit of effort, but I guarantee the improvements you gain will be worth it! My one-liner of advice to those RightScale customers with older versions ”if you’re one of those hanging onto v4 or earlier you really should upgrade.”

Building your own Salesforce.com? Yup, sounds like a lot of work. Yet here at RightScale, I see many companies trying to build their own cloud management solutions. Perhaps it is the DevOps mindset that has made cloud computing so popular: “If I can’t get approval, I’ll just do it myself on the side.” Or perhaps it is because we are still in the early stages of cloud, and people are experimenting and discovering what is possible internally versus what is available in the market.

I did an informal poll of our sales team, and here’s what they said were the top reasons companies try to make their own solutions rather than use a cloud management product:

1. They want control, or the ability to highly customize their environment.
2. PaaS and IaaS, as concepts, seem simple, easy to jump on. A “cloud computing management platform” seems like a complex paradigm to adopt.
3. Because they can, and they want the challenge of exploring a new frontier.
4. The cost of a cloud management solution is too high.

OK, so these appear to be valid reasons at first glance. But these statements are typically founded in misconceptions about cloud management solutions in general or RightScale in particular, which I’ll address here:

Control: RightScale is not a PaaS service. We let you get into everything – perhaps more so than we should. Change the images if you must, run custom scripts against our API, and export usage data to include in your own data warehouse. Fifty-two percent of the servers running on RightScale are controlled by completely custom ServerTemplates, not ones we provide. Our product philosophy is to let you “get under the hood” if you need to – so please do.

Complexity: Cloud management is complex, and I don’t argue that. What RightScale aims to do is provide a layer of abstraction that makes the difficult and mundane tasks, like auto-scaling, much easier. It is unfortunate that the term seems complex, because if anything, a cloud management solution can make managing your entire cloud infrastructure and applications so much easier.

Conquering the new frontier: You’re being told by your boss to “Learn cloud now – just figure it out.” You want to truly understand what’s possible, how to build it, and deliver on expectations. As you start down this path, you cobble together some tools to accomplish your first foray into the cloud. Unfortunately, technologists have a tendency to “reinvent the wheel” as they continue along their path to the cloud. We’re many steps ahead, and we’re happy to share what we’ve already learned.

Cost: Netflix is a poster-child for DIY cloud, and has been forthcoming about its experience, which has helped grow this new paradigm. Netflix “designed its cloud architecture so that it has the option to move to an Amazon Web Services competitor” if needed, according to this NetworkWorld article. At a recent conference, Adrian Cockcroft, Cloud Architect for Netflix, mentioned that Netflix has 50+ engineers working on this cloud-independent solution. Doing some quick math, that’s about $8.3 MM per year Netflix spends building and maintaining this platform. That could buy a lot of RightScale Enterprise Editions!

At the end of the day, we see many customers who come to us after they outgrow their own internal solutions. They eventually discover that there are just too many things to stitch together: configuration management, systems automation, monitoring, application automation, provisioning, user permissions, reporting…it goes on.

We have hundreds of employees and have spent many millions creating the most comprehensive cloud management platform in the world. And we designed our product to drive the same way no matter which cloud you choose. So while cloud management may seem like a fun weekend project to tackle, it’s not – please don’t try it at home.

Yes, Amazon is still the dominant cloud, but a tornado of new clouds is swirling. The next thing your boss will likely ask is, “So what if we wanted to use this other cloud instead?”

Update Feb 3, 2012: Since I published this post, I’ve received a lot of feedback regarding DIY in the cloud computing space.

A few of our customer developers pointed out that they actually appreciated learning the cloud through RightScale – it gave them both an understanding of the underlying IaaS cloud as well insight into ideal cloud management frameworks. Forbes ran articles on how this extensive cloud computing knowledge is in high demand in IT and is a ticket to the corner office, and we’re starting to see RightScale listed as a required skill on some of these cloud job postings.

Next, I’ve heard from a few more larger companies who have built their own internal cloud management solution. They also cited approximately 50 engineers in their cloud computing groups, so it seems this is the sweet spot for development and maintenance of a robust internal solution. Let’s not forget about the PaaS-like solutions we offer with our ServerTemplates in this regard – it is not just automated provisioning that these larger companies ultimately need to build.

I’m not saying you can’t “do it yourself” in cloud computing (or in anything for that matter), I just want to encourage developers to avoid the trappings of #3 above – namely ignoring off-the-shelf solutions in the interest of personal discovery. It may work in the short term… until you hit one of the many walls that we’ve already had to plow through. At that point, you’ll either have to scale the solution and team, or re-architect for a product that offers the necessary solutions already.

Update March 15, 2012: This post stirred a lot of pots, so I expanded it into an article on SYS-CON called “Do-It-Yourself Cloud Computing Management – Is It Worth It?” However you come down on DIY cloud management, I think you’ll find more material worth mulling over.

These clouds have been in the works for a little while, and I’m pleased they are now available in the RightScale platform for our customers.  When we integrate with a given cloud, we work hard to ensure a seamless experience across all the clouds we support.  We provide a generic interface to each of the clouds integrated within RightScale.  This is not to limit functionality from the clouds themselves; but rather to ensure all that cool functionality is usable.  If I’m using SoftLayer and Datapipe, I don’t want to deal with different storage solutions like volumes or instance based storage (or at least not until I’m ready to optimize the storage).  Likewise, keep networking off my plate…I don’t care whether it’s security groups or ip tables.  Just make that infrastructure stuff work so that my app can run.

As a user, I want to  easily port what I have in one resource pool to another resource pool.  For this purpose, RightScale has generic constructs for things like instances, instance types, images, volumes, volume snapshots, etc, that are exposed in our dashboard.  Then, in our ServerTemplates (stay tuned by the way, a release is imminent), we use chef to abstract features for individual ServerTemplates that work, albeit very differently, across different resource pools.  Using the above example, someone launching servers in SoftLayer’s Amsterdam cloud and Datapipe’s Hong Kong cloud doesn’t have to worry about the differences between network configuration and storage management.  You can launch an entire 3-tier PHP architecture on both environments using ServerTemplates from the MultiCloud Marketplace.  We’ll take care of dealing with instance based storage in Amsterdam and set up the proper security groups for you in Hong Kong through the platform.

Why does RightScale spend so much time touting ‘MultiCloud’ and why should anyone care?

It’s a good question to ask actually.  I spend a lot of my time working with service providers and various companies looking to deliver infrastructure as a service for public consumption.   A number of people, our existing customers included, come to us and say “hey, I know I will have multiple clouds (if I don’t already)…help me make that happen.”  Analysts also agree – Forrester’s Holger Kisker touts “multi cloud becomes the norm” as his number 1 cloud computing prediction for 2012.

It’s real.  And it’s great validation for being the leader in ‘MultiCloud Management”.

Perhaps even more interesting (and contradictory if you think about it) is that the service providers say the same thing!  We describe how RightScale offers clouds to consumers and the choice consumers have to use what works best for their business needs.  And, IaaS providers are more than happy (okay, some take it as a challenge to deliver an even better service for their users. ).  In truth though, they recognize that cloud is a heterogeneous environment.  A single customer will use more than one cloud offering in a single environment.  Cost is one factor, and another I hear often is performance.  In some cases, geographic location is important and they “can’t get there with their current IaaS provider.”  It’s an opportunity for some to seize, and we’re partnering with the best to deliver the multi-cloud solutions our customers want.

Within RightScale, you can use any or all of the following clouds – all the Amazon regions, SoftLayer, Rackspace Cloud across US and UK, Datapipe, Logicworks as well as private cloud management with CloudStack and Eucalyptus.

I encourage you to click and try the new clouds on RightScale.  Use a new app or an existing one that’s already in cloud and as always, let us know what you think.

1. A “workaround” or configuration “fix” is made available from the vendor of a package
2. The vendor of a package applies a security patch to their distribution, but the patch has not been applied to the packages distributed by the operating system vendor

In both of these scenarios, updating from the Security Repositories from the vendors will not provide a fix, it is necessary to do some custom “configur-ating” to get the patch or workaround applied to instances. In both situations, a patch/fix is deployed with custom RightScripts to running instances, as well as those that will be launched until the vendor package patch is released and ServerTemplates are updated.

I’ll walk through one possible way to accomplish this for the recent Apache HTTPd Denial of Service vulnerability. To refresh everyones memory, a vulnerability was found in various versions of Apache that allowed a remote attacker to consume all the CPU on the system the Apache server was running on. Workarounds were issued shortly after the vulernability disclosure, then about a week later, Apache released an official patch in the form of an updated version. We were running some HTTPd 2.2.x servers and the specific version we needed was HTTPd 2.2.20. At the time of the patch release, the linux distros had not yet updated their packages, so we needed to implement an out of band patch (i.e., work around our normal process).

Here are the steps we took to update HTTPd running on a CentOS based image. We did the initial building on a test server as you will see there was some hefty debugging needed to get it right. Here are the steps we followed:

1. Start by reviewing the applicable CVE and find information about the vulnerability.
2. Pull down the sources into the test instance. The instance should be launched with the same ServerTemplate as current running instances that will need to be updated
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.gz http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.gz
• curl -o /usr/src/redhat/SOURCES/httpd-2.2.20.tar.bz2 http://apache.cyberuse.com//httpd/httpd-2.2.20.tar.bz2
• wget http://mirrors.servercentral.net/fedora/releases/test/16-Alpha/Fedora/source/SRPMS/httpd-2.2.19-4.fc16.src.rpm
3. Run an rpmbuild to get a list of dependencies for the update
• rpmbuild –rebuild httpd-2.2.19-4.fc16.src.rpm
4. Install the dependencies
• yum install xmlto libselinux-devel apr-devel apr-util-devel pcre-devel openssl-devel -y
5. Update to the latest package available for CentOS
• rpm -Uvh httpd-2.2.19-4.fc16.src.rpm –force –nomd5
6. Start configuring to create our own rpm (this is where the hard part begins)
• cd /usr/src/redhat/SPECS/
• edit httpd.spec to add
• Version: 2.2.19 => 2.2.20
• Release: 10%{?dist}.1 => 1%{?dist}.0
7. Build the rpm, expect a boatload of errors to walk through:
• rpmbuild -ba httpd.spec -> Fix error -> repeat
8. Once successful, the newly built package is in /usr/src/redhat/RPMS/
9. Install the update and restart the server
• rpm -Uvh httpd httpd-tools –nodeps
• service httpd restart
10. Take the newly created rpm and upload it as an attachment to be used by the RightScript
11. Create a RightScript that performs the update and restarts the server
• rpm -Uvh $RS_ATTACH_DIR/httpd*.rpm
• service httpd restart
12. Run the RightScript as an “Any” or “Operational” script to update servers in the deployment.

While this process is for CentOS, Ubuntu requires similar heavy lifting to get things functioning. This process took one of our Professional Services engineers about 4 hours to complete (obviously the most time was spent on step #7 ). This type of process takes a lot of hackery to back port a version into an srpm. It is not trivial, but can be done.

So basically the answer to “How?” is “RightScript”. Even though it is non trivial to get that custom rpm or package debugged, once it is, then the deployment to systems is very quick and painless using the RightScale platform.

A final note, once the Linux distribution actually issues the patch, you should transition from “fix” mode your standard “patch” mode for overall consistency. Remember that very little if any testing is given to “fixes” that are released, whereas, a certain level of regression testing is typical for vendor released patches (i.e., distro packages or Windows Updates).

Within the RightScale platform, there are 3 primary options that can be used to automate the patching of instances:

1. Unfreeze Security Repositories and enable automatic updates on systems, hope that the updates don’t break anything.
2. Manually unfreeze Security Repositories for test systems and update. Perform regression testing, then update & refreeze Security Repositories for production systems and apply updates. Do this regularly (say monthly or weekly).
3. Update each ServerTemplate with the latest Security Repository. Regression test each updated ServerTemplate. During a schedule maintenance period, force all servers to be relaunched with updated ServerTemplates.

Of course, there’s also always the option to hide underneath a pile of coats and hope it all works out for the best. It goes without saying that while many people de-facto implement this last option, it is not a viable long-term strategy!

Let’s dive into each of the options a bit more and look at some pros and cons, so you are in a better position to pick the one (or combination) that works best for you.

• Unfreeze with automatic updates: Since many (most?) of the core Linux distributions have functionality to allow selecting of security updates only, you freeze all channels, and then set the security repo to /latest via a RightScript. You then configure the system to install those updates on an interval you desire (daily seems to be a good choice). For example, on Debian based systems, such as Ubuntu, security patches are broken out into a separate repository. For a given release it is possible to only automatically install updates from http://security.ubuntu.com/ubuntu/ instead of http://us.archive.ubuntu.com/ubuntu/, making this very easy to implement. Just unfreeze that repository and updates will apply as they are released.
  With CentOS you can run “yum update –security” and only install security related patches. Using this method allows rapid access to the latest security updates, with almost no work required to enable this behavior as the unattended upgrade packages do all the work for you.
  The downside is that if a broken package is released, say into Ubuntu-security, it could affect production. A side note is that as it relates to security patches, the industry at large has pretty much come to the acceptance that the risk of problems with automatically patching security vulnerabilities outweighs the potential risks with doing it. For example, Debian, Ubuntu and Windows 2003-2008 all ship this way. For those who determine that the risk of automatic patching is too great, there is …
• Unfreeze in test, test it, update production: This option is to apply the security patches to instances in a test deployment, then after regression testing, deploy them to production using a RightScript to update repositories and perform an update on production servers. This has the advantage of some level of regression testing prior to deploying security patches in production. The downside is that there is a high manpower cost to perform the functional testing on a regular basis. There is also the fact that you should test the specific items that the security fix supposedly touched which involves a bunch of research. This is a non-trivial effort. It would likely require a special test environment dedicated to security testing. From a purely dogmatic standpoint, this is the way it should be done, but the pragmatist in me knows that for many organizations, the additional cost associated with this is not justified by the increase in risk posed by just installing security updates. I’d rather have patched systems, than people not doing it because it was not the absolute best way to go about it.
• Update ServerTemplates and relaunch: This may be the cleanest and seemingly easiest approach. There is relatively little change in current operations, as many of you use this method currently. This also ensures that all packages are tested before being deployed in production. The upside is that systems are cleanly built, and ServerTemplates are updated more often. The downside to this is that your patch level is only as good as your latest ServerTemplate update, and while it works for servers that can be frequently updated (app servers, web servers, etc.), it really doesn’t work well for services that are infrequently updated, or difficult to relaunch (databases, load balancers, etc.). Further, it forces you to relaunch servers you wouldn’t otherwise relaunch during maintenance windows.

So, you may be asking “You use RightScale to manage RightScale, so how do you do it?” Well, at RightScale, we have chosen a hybrid approach of #1 & #2. Our default patching policy is “Unfreeze with automatic updates”. As stated earlier, there is some inherent risk in this stance, but we feel that getting critical fixes in outweighs the incremental risk of taking too long to get the patch deployed. In instances where the risk of any patch (security or not) breaking a system, we use the “Unfreeze in Test, test it, update production” patch policy. Further, we design our platform with mitigating controls to restrict access to systems and services that may not get the latest patches on a daily basis. This policy/stance works for us, and we think it is a reasonable one for others to start with (if you didn’t already have a stance).

I would be remiss if I did not point out that there are likely a myriad of other ways that you can perform security patching, but that these are ones you get “out of the box” with the RightScale platform. The specific approach you choose will be driven by your business requirements. Remember that you have options, so use them to develop a process the works for you and your organization. My next blog will be on deploying workarounds and non packaged fixes. Until then, Happy security “patching!”

First, 3 million is impressive in the data center business.  Many well-known hosting companies house between 50,000 and 100,000 servers, and estimates for the world’s largest computer companies with large data centers range up to 1 million.  (See the DataCenterKnowledge report here.)  It’s difficult to compare our statistic with these installations, since many may be running largely under a pre-cloud operational model.  Nevertheless, launching 3 million is quite a number by any comparative metric, and there’s no question that it was achieved only with new levels of automation and dynamic configuration that are core to RightScale.

The second reason 3M is worth noting has to do with how fast we got there.   After our founding in 2007, it took us about 27 months to reach 1M, another 12 months to reach 2M, and then just 6 months to reach 3M.  That’s more than twice as fast for each subsequent 1M servers.  Likewise, one year ago in Sept. 2010, we had launched 1.5M servers – and we doubled in the last 12 months.

The third reason this milestone matters is that the servers our users launch have increased in power, and persist for a longer duration, as each month passes.  In fact, since January this year server runtime has increased on average 30%. So the trend is clear: companies are running “bigger iron” in the cloud — and keeping it running longer — than ever before.  Here is a graph of the size distribution we recorded this summer:

Certainly, the growth rate we’re tracking for the quantity, power and longevity of servers launched on RightScale remains quite healthy and mirrors the broad adoption of cloud services industry-wide. But equally important is the range of customers driving this growth, representing a wide variety of industries, use cases and services powered by RightScale on the cloud. For example, during the last year:

• media giant Pearson converted a traditional educational software offering to a SaaS based model that allowed faster onboarding of new customers;
• consumer goods company American Girl (a division of Mattel) launched their virtual world with a major advertising push behind it and sailed smoothly through the holiday season;
• online game company Zynga launched new games that consistently broke records;
• and companies like Amdocs and Trader Media spoke at our User Conference last June about new enterprise services launched on both public and hybrid clouds.

All of these RightScale customers contributed toward the 3M milestone, and we continue to be dazzled by the solutions they achieve using cloud infrastructure. We’re looking forward to the next million servers launched by our customers, and the amazing services they’ll power with them.

Drum roll please…Introducing the auto-scaling, high availability .NET Stack on Amazon!

Some of you might think, ‘well Finally!’.  As a product manager, I empathize with that sentiment. It was certainly tricky getting this to hum on the cloud. I’m very proud of our team’s work, especially when reviewing the challenges they overcame to get this out the door.  Let’s take a deeper look into some of them.

Database Manager for SQL Server

When I first proposed the SQL Server Database Manager last year to our development team with what ultimately would become our first SQL Server ServerTemplate, I met with mixed reactions.  EVERYONE thought it was a great idea and would be useful to users.  However, a lot of reservations too…doing something like what we have for MySQL with master/slave replication is no easy feat.  Adding in Microsoft complexity with Powershell as well as unexpected Windows behavior in the cloud, the solution seemed out of grasp. Some of the notable questions we asked ourselves:

• How does MS Licensing work as existing orgs transition to the cloud?  Is SQL Server Standard good enough or will users demand Enterprise?
• How will we backup data on SQL Server?  Native backups guarantee “sane” backups without service interruptions but take a long time.
• What are the setup best practices and how do we implement them in the cloud?  Multiple Data / Log Volumes, default monitoring and alerts, backup scheduling, etc…all need to be considered.
• How do we set up replication with SQL Server?  There are so many supported options, what’s best for cloud?

We started at the beginning, pushing licensing off to the likes of the service providers (Amazon) and focusing on prototyping our implementation.  One of our developers found that backups using Volume Shadow Services was a better option than SQL Server native backups.  The following is an excerpt from his report:

Now that we have a clear understanding of how to proceed with backups, we tried to figure out best practice configuration — focusing mainly on the volume configuration and management.  We encountered issues attaching EBS volumes before Windows was ‘ready’ which resulted in out of order drive letter assignment.  We solved that, and moved on only to find that Powershell was running as a 32-bit process on the x64 environments…great.  Fixed that too.  Those two issues actually got us pretty far and enabled our first release of a Beta ServerTemplate that supported Standalone backup/restore functionality.

Going through the Beta of our backup/restore ServerTemplate, we learned enough to facilitate building a High Availability SQL Server Solution (that’s what we published recently!).  We focused on a few things:

• SQL Server Mirroring (Set up, Monitoring and Alerting)
• Authentication and Data transfer Encryption
• Failover to the mirror

One key challenge in setting up the mirroring session was waiting.  We had to have not only the mirror server in operational state but also a set of database full and differential backups from the principal to initialize the database on the mirror.  We utilized the RightScale ability to tag servers and locate servers by tag to help automate this whole process.

I recommend you try out our ServerTemplate to see just how much we managed to take off your shoulders for the database management.

IIS Application Server

Phew.  When you think about the complexity of the Database Manager, IIS seems like a walk in the park!  But a lot of work went into this template too.  I went over and chatted with the dev lead whose team built this template to get the inside scoop of the challenges they had to overcome.  Here’s his list:

• Built with the use case of the scalable app tier in mind
• Integrate a front-end load balancing solution
• Figure out how to get the app on the server
• Figure out how to tell app servers where the db is when they are ready
• Oh, and of course, best practice configuration of IIS App Servers (this is Microsoft after all)

Doesn’t sound too complicated does it?  Luckily, with standardized images and use of RightScale tags to discover the Database server, it worked pretty well.  Also, based in large part on the design, we were able to get this ServerTemplate to work equally well on both Rackspace and Amazon.  That actually was a cool product win, even though it created more work for our testing team (details of which I went into in my last post).

I encourage you to take a look at this ServerTemplate too.

Together, these two ServerTemplates with either the HAProxy Load Balancer ServerTemplate or Elastic Load Balancing from Amazon make up our .NET Stack.  Take them for a spin, and as always, please send us your feedback.  Enjoy!

It’s been a R.A.C.E. to the finish line this sprint, but we have some exciting news!  RightScale’s current ServerTemplate release showcases the entire PHP 3-tier stack across the Rackspace Cloud, Amazon’s Elastic Compute Cloud, Cloud.com’s CloudStack and Eucalyptus Systems (hence R.A.C.E).  These new HAProxy Load Balancer, PHP App Server and Database Manager for MySQL 5.1 ServerTemplates are available now in the MultiCloud MarketPlace.

Underneath all these templates, we’re releasing a new CentOS 5.6 MultiCloud Image with RightLink 5.7 for all EC2 regions, Rackspace Cloud Servers, Cloud.com’s CloudStack and Eucalyptus Systems.  For a complete list of ServerTemplates and MultiCloud Images we released, check out our latest release notes.

People often talk about developing for the cloud and associated challenges.  But what about building platforms on which other people develop their apps in the cloud?  It is very challenging building for the “general use case”, but it is even more important when you build for generality that it works well.  A lot of time and effort goes into designing and building our ServerTemplates.  During development, and especially before release, we conduct extensive manual and automated testing cycles where we put our templates through the wringer.  Below is a sample test matrix that represents our checklist for one ServerTemplate.

Notice that we test this one ServerTemplate across 2 separate images in 7 clouds each.  Considering that we released 9 ServerTemplates last week, this makes for quite a few permutations.  Of course, we also find bugs and have to retest rapidly.

Luckily we have help with a home-grown automation tool that we call “Virtual Monkey.” The monkey uses the RightScale API to create deployments with servers to test, launches them, runs tests against them, collects the results, shuts everything down, and cleans up. In most cases the tests include entire 3-tier application deployments so we can test the interactions between the servers and ensure things work end-to-end. All in all we launch hundreds of servers a day in various clouds to test these ServerTemplates and make sure they work before we let them loose in the wild.

Not all clouds are created equal…

If you’ve done anything on multiple clouds, you’ll appreciate the behind-the-scenes presented above.  When launching hundreds of servers and developing infrastructure-as-a-service agnostic solutions, small nuances can be big blockers towards expected end-user functionality on the solution.  From security groups on Amazon versus iptables management on Rackspace to volume snapshot API response differences in Eucalyptus and CloudStack, the ServerTemplate needs to be aware and abstract away differences.  Do our customers really care about these differences?  Of course not.  They just expect one solution to work on one cloud type just as well as it works on another cloud type.  You may have noticed that we utilized Chef for many of our recently released ServerTemplates.  Chef allows us to abstract the business logic away from the details of the cloud,  making it easier to propagate the same solution to as many clouds as we can get our hands on.

Wait, each cloud has different profiles for instance types!

But Chef isn’t the only innovation that we use.  We also have to support many ‘by design’ cloud architecture differences. For example, pre-defined instance types.  Instance types in Amazon, while being the same across all regions within Amazon, do not align well to flavors in Rackspace.  Plus, in private clouds, you can either use the default instance types or custom configure to what your application demands.  How then does a specific ServerTemplate correctly configure a new instance for optimal performance?  Seems like that’s (yet another) real prerequisite for proper application setup.

In the specific case of MySQL, our ServerTemplate will auto-tune configuration parameters including innodb_additional_mem_pool_size and table_cache.  The tuning is based off of an instance’s available memory.  Of course this can be overridden on a per-server basis.  This mechanism extends well to our PHP App and Load Balancer ServerTemplates where you override parameter defaults specified in apache2.conf and haproxy_http.

All of this is just the tip of the iceberg.  Check out the ServerTemplates for yourself and let us know what you think.

Add Server and Add Server Array Assistants

First, we’ve created new assistants to simplify the process of creating a server or server array. It’s a big change, requested by our customers, designed to make your life easier in the day-to-day usage of the Dashboard. We worked with a few customers over the last few months to refine this flow, so we know it will be a welcome change. The previous process was getting a little disjointed after a few years of rapid cloud innovation! Learn more about the new assistants shown below.

Community Translations

Next, in May of this year, we launched our first language translation for the RightScale Dashboard: Japanese. Back then, we mentioned that we accomplished this through a platform we planned to make available to the community. That time is here. You will now notice a “Help Us Translate” link in the footer:

So how can you help us translate (and why would you)? First, the how. When you click on this link, you’ll be taken to a tool that will allow you to see all the phrases that need to be translated, translate these phrases, and vote on translations that others might have submitted. You can then link back to the Dashboard to see the translations in real-time! To get started, click on the link, read the instructions, and choose your language:

Now, why would you help us? Well, many of you have already offered out of the goodness of your heart, and we appreciate that. If you are someone that needs an incentive, we appreciate that too. That’s why we’re going to give the top translator for each of the following languages an Amazon Kindle: German, Chinese Simplified, Japanese, French, Spanish, and Korean. For how to get started, and for more information on this “Translation Showdown,” read the RightScale Dashboard Translation Guide.

New MultiCloud API

We’ve been incubating a new API with a few of our largest customers for over half a year now. This API is a complete redesign, and takes into account everything we have learned over the years on how to manage multiple clouds behind a single “pane of glass.” Or in this case, a single set of XML/JSON instructions.

We are making it available today as a public beta, supporting Cloud.com, Eucalyptus, and Rackspace. Not everything that is in API 1.0 is available in this new API yet, but it is burning a hole in our pocket, and will be extremely useful to our customers who want to begin automating their multi-cloud deployments. As we equalize the feature set between this new API and the 1.0 EC2 API, we will move AWS EC2 support into this new API and retire API 1.0.

One new feature here (available on all clouds) is the ability to provision and manage users via the API. You can now list all users in an account, add users, and set their permissions. Coming up in the October release, Enterprise plan customers will also be able to provision new accounts.

Learn more about the new API and how to get started with it.

Release Notes

As always, please read the Release Notes for a detailed list of changes made to the Dashboard and API.  Look out for the ServerTemplate & MultiCloud Image release next week – we have some great solutions coming up for both Linux and Windows cloud administrators.

Enjoy!

Security testing is one aspect of a security program that is often overlooked. Organizations who take security seriously understand that testing systems and applications is just smart business. We felt that one way we could help our customers is to describe the process, and nuances, that we go through during our testing. Since RightScale runs in the cloud, the information should help any RightScale customer accomplish the same tasks on their environment.

Our process is basically broken down into the following steps:

1. Identify instances and applications that will be tested
2. Select tools and systems that will be used to perform the testing
3. Coordinate with the cloud service provider to get authorization for testing
4. Execute the test
5. Communicate the results

Below I have outlined some of the practical details of each of these steps.

Identify Targets

Before we start testing, we identify what we want to test. For this particular test, we decided that we would include all of the systems that make up our platform, as well as the main dashboard application. Since we use RightScale to manage RightScale, and one of the main functions of our service is using ServerTemplates™ and RightScripts™ to ensure that systems are deployed consistently, there was a temptation to select a representative sample.

Since this was my first time testing RightScale since becoming the Director of Security and Compliance, we decided to test them all. We figured it is good practice, and provided a “validation” of sorts that we were following the practices we champion. We did however decide to limit the testing to publicly addressable AWS IP addresses. (Note: Anyone trying to be PCI compliant in AWS will likely need to test private IPs as well.)

As for the application, we decided on the entire dashboard, and not just a portion (mostly because I wanted a good overview to have as a baseline).

Select Testing Tools

Along with determining which systems/instances and applications we were testing, we selected tools that would help us automate the testing. We had agreed that a primarily automated vulnerability test (with manual validation) was acceptable, but that the application scanning would require a more manual approach given the complexity of our application. To that end, we had the following basic selection criteria:

• Vulnerability scanner: Number one criterion was its ability to appropriately identify vulnerabilities. We did not want a lot of false positives, but felt that false negatives would be much worse. A second criterion for the vulnerability scanner, was the flexibility of its reporting mechanism.
• Application testing: Number one criterion was our ability to use it, not what others think of it. A second criterion for the application testing tool was its ability to test against the framework of our application.

Given those “requirements” we chose three vulnerability scanners that we wanted to evaluate, in hopes of selecting one as the foundation for our ongoing testing program. Those were SAINT, NeXpose, and OpenVAS. Many will point out that there are other tools out there, and I agree, but these were tools I personally have history with, and one is free. We had to start somewhere.

As far as the application testing, I have used Burp Pro for a number of years and am a fan of it, and selected that as an application testing tool of choice. It should be noted that a number of other tools have recently come out that may rival Burp Pro in its functionality, but familiarity of use was important. We wanted to test the application, not the tool.

Where to Run Them?

Once we determined the tools that we wanted to use, we had to figure out where we wanted to run them:

• SaaS
• Instance in the same cloud
• Instance in a different cloud
• Traditional hosting environment
• Physical system on our network

We chose the “Instance in the same cloud” for a couple of reasons:

• Flexibility: We were able to install multiple tools to evaluate and test
• Eating our own dog food: RightScale is all about configuring and managing systems, so what better way for us to help our customers be able to deploy scanning systems than to do it ourselves
• Bandwidth cost: By using an instance within the same availability zones on AWS, bandwidth was not an issue
• Access to internal IPs: By running in the same cloud (AWS region) we can test internal IP addresses

Once we decided to build our own, we downloaded a trial version of SAINT, the community version of NeXpose, and followed the Ubuntu installation directions for OpenVAS. Then we wrote some RightScripts to automate the majority of the install and we were “cooking with gas” so to speak.

Get Authorization from Cloud Provider

Once we identified all our instances we were going to test, and had our testing sources (one in our case), per the AWS usage agreement, we needed to get authorization from AWS to perform the testing.
AWS provides a form that we filled out to request penetration testing of instances. We had to supply the AWS instance IDs and IPs that we obtained earlier, as well as the source of the testing. AWS uses this to create a ticket that AWS security team will get, and subsequently white list the account so the IDS systems are not triggering alerts during the testing. This prevents getting nasty emails about policy violation as well as port blocking, which would affect the test results.

AWS security responded back within a couple of days with approval for the scanning. It is interesting to note that it appears it is the vulnerability scanning that this applies to, for all intents and purposes you should make this request for application-based scanning as well, but it’s been my experience that testing the application does not cause abuse reports to be generated within AWS. During the testing, launching and relaunching of the scanner we did accidentally perform a number of scans from an IP address other than the one we provided to AWS and we did receive two abuse notices.

Probably the biggest point to note with respect to testing instances running in AWS is that instance size must be medium or greater. AWS policy does not allow pen testing, including port/service scanning, of smalls or below, presumably because they want to avoid that the testing degrades the other VMs on the same host. It should be noted, that we were just testing in AWS, depending on your cloud service provider, what you need to provide as far as what you are testing will vary. For AWS, we provided the instance ID as well as the public IP that will be tested, and the source of the testing.

For AWS, the quickest way to get the list of all AWS instance IDs and associated IPs is to use the rest_connection API. It can be used to programmatically generate a list of the instances and associated IP addresses that will be the targets of testing. We ignored the security groups in this test and hit all the “well known ports” that the tools scan. An alternative would be to only test the accessible ports.

Execute the Test

Once we obtained the authorization for the testing, we coordinated with the ops team to make sure they were ready for any potential problems. Once we got their “we are a go” signal, we commenced the testing. The general methodology looked something like this:

1. A sequential vulnerability scan, using each of the scanners. For both SAINT and NeXpose, we utilized the “exploit” portion of the tools (when it existed) on any noted vulnerability. (Note that we performed multiple scans with each scanner over the course of our 3 weeks of testing.)
2. General walk through and Burp Pro “passive” testing of the entire dashboard. Attempting to get an overall feel for the testing tool with the dashboard, and basically doing a full manual spider of the site.
3. Next we specifically performed testing of our session state mechanism, looking for entropy, manipulation, and injection flaws.
4. We then stepped through each of the dashboard’s main function areas, “Reports,” “Manage,” “Design,” “Clouds” and “Settings,” looking for well-known attack vectors. In particular focusing on identifying Cross Site Scripting and Request Forgers (XSS and CSRF), Injection, parameter manipulation, and other common web app exposures. See the OWASP testing guide for a good discussion of things that should be tested for in web applications.

Note that all testing we performed was done in both an authenticated state as well as an unauthenticated state.

As stated earlier, we made the decision that the vulnerability scanning portion of our testing would be mostly automated, and the application testing mostly manual. It took us approximately 3 weeks to identify the systems, get the authorization, and perform the testing. About 2 weeks of that was dedicated to the manual app testing.

A Bit More on the Application

It could be argued, that the bulk of “cloud” security testing should revolve around the application. This is not to say that making sure supporting services like Apache and MySQL versions are patched is not important (it is, just ask Sony), but meaning that much of the exposure to your data will come through the application. Taking the time to assess the mechanisms protecting the application is critical. For example:

• Are the security groups appropriate?
• Do you have appropriate controls on who can access API calls or make security related changes via the UI?
• Does your authorization mechanism enforce appropriate controls via all interfaces?

Items like these are things that will be critical for long-term protection of information. Make sure that you include them in your testing regiment.

Communicate Results

We are an Agile shop, so frequent communication is part of our culture, and we leveraged that to provide feedback from the testing to the appropriate engineering or ops teams as we uncovered potential threats. This allowed us to create records of our testing results, as well as provided timely information to be fed into our sprint process. At the completion of the testing, we wriote a summary report and included details of the vulnerabilities from each of the tools as appendices. Even though the information is already fed into the appropriate groups, including details along with the final report allowed stakeholders the ability to review the overall testing methodology and findings, as well as dig down into the details of any vulnerabilities found.

Your process may vary, and you may have a much more formal reporting requirement. The most important part is to get the appropriate information to the people who can get the system services or applications fixed in a timely manner.

Summary

The process of identifying targets, maintaining testing tools, coordinating with cloud service providers, and communicating those results should be formalized within your organization. Security testing should become an integral part of the IT culture. There will always be issues, as nothing is absolutely secure, but trying to stay ahead of the curve is a worthy cause. With a formal process, you can make it a regular occurrence, thus enhancing your security program and likely meeting many practical as well as compliance requirements.

One side note about the testing is that for all practical purposes, it was exactly the same methodology and tools that I have used previously in non-cloud environments. So I encourage you to roll up your sleeves and implement a testing program for your infrastructure and applications.