CloudFoundry Architecture

From a technical point of view I see two main innovations in Cloud Foundry. The first is that the software is released as an open source project with an Apache license, which gives users and third-parties access to make customizations and to operate Cloud Foundry on their own. The second is that Cloud Foundry is very modular and separates the data path from the control plane, i.e. the components that make user applications run from the ones that control Cloud Foundry itself and the deployment and scaling of user applications. The reason the latter innovation is significant is that it really opens up the door to innovate on the management of the PaaS as well as integrate it into existing frameworks such as RightScale’s Dashboard.

Enough prelude, the pieces that make up Cloud Foundry are:

• At the core the app execution engine is the piece that runs your application. It’s what launches and manages the Rails, Java, and other language app servers. As your app is scaled up more app execution engines will launch an app server with your code. The way the app execution engine is architected is nice in that it is fairly stand-alone. It can be launched on any suitably configured server, then it connects to the other servers in the PaaS and starts running user applications (the app execution engines can be configured to run a single app per server or multiple). This means that to scale up the PaaS infrastructure itself the primary method is to launch more suitably configured app execution engines, something that is easy to do in a RightScale server array!
• The request router is the front door to the PaaS: it accepts all the HTTP requests for all the applications running in the PaaS and routes them to the best app execution engine that runs the appropriate application code. In essence the request router is a load balancer that knows which app is running where. The request router needs to be told about the hostname used by each application and it keeps track of the available app execution engines for each app. Request routers are generally not scaled frequently, in part because DNS entries point to them and it’s good practice to keep DNS as stable as possible, and also because a small number of request routers go a long way compared to app execution engines. It is possible, however to place regular load balancers in front of the request routers to make it easy to scale them without DNS changes.
• The cloud controller implements the external API used by tools to load/unload apps and control their environment, including the number of app execution engines that should run each application. As part of taking in new applications it creates the bundles that app execution engines load to run an application. A nice aspect of the cloud controller is that it is relatively policy-free, meaning that it relies on external input to perform operations such as scaling how many app execution engines each application uses. This allows different management policies to be plugged-in.
• A set of services provide data storage and other functions that can be leveraged by applications. In analogy with operating systems these are the device drivers. Each service tends to consist of two parts: the application implementing the service itself, much as MySQL, MongoDB, redis, etc. and a Cloud Foundry management layer that establishes the connections between applications and the service itself. For example, in the MySQL case this layer creates a separate logical database for each application and manages the credentials such that each application has access to its database.
• A health manager responsible for keeping applications alive and ensuring that if an app execution engine crashes the applications it ran are restarted elsewhere.

All these parts are tied together using a simple message bus, which, among other things allows all the servers to find each other.

Auto-scaling Cloud Foundry

“So, does it auto-scale”? seems to be the question everyone asks. (I wonder who started this auto-scaling business? ) The answer is “no, but trivially so”. There are actually two levels at which Cloud Foundry scales, whether automatically or not. The first is at the Cloud Foundry infrastructure level, e.g. how many app execution engines, how many request routers, how many cloud controllers, and how many services there are. The second level is at the individual application level and is primarily expressed in how many app execution engines are “running” the application (really, how many have the application loaded and are accepting requests from the request router).

The first level of scaling the Cloud Foundry infrastructure is the responsibility of the PaaS operator. The operator needs to monitor the load on the various servers and launch additional or terminate idle ones as appropriate. In particular, there should always be a number of idle app execution engines that can accept the next application or that can be brought to bear on an application that needs more resources. This level of scaling can be performed relatively easily manually or automatically in RightScale. The app execution engines can be placed in a server array and scaled based on their load.

The second level of scaling is the responsibility of each application’s owner. The nice thing about the modularity of Cloud Foundry is that it exposes the necessary hooks to adding external application monitoring and scaling decisions. It is also interesting that Cloud Foundry in effect exposes the resource costs and lets the application owner decide how much to consume–and pay for. This is in contrast to other systems that make it difficult to limit the resources other than by setting quotas at which point an application is suspended as opposed to simply running slower.

What we envision in working with Cloud Foundry is simple: RightScale will be able to monitor the various servers in the Cloud Foundry cluster, and determine for example when it’s “slack pool” of warm, ready-to-go app execution engines has dropped below a given threshold (or exceeded an idle threshold), and either boot new servers to add to the “slack pool” or de-commission unnecessary ones to save on cost, as appropriate.

PaaS and IaaS Synergy

The benefits of PaaS come from defining a constrained application deployment environment. That makes it necessary for many applications to “punch out” and leverage services outside of the PaaS framework. In some cases this may be a simple service, like a messaging server or a special form of data storage. In other cases it will end up being almost a reversed situation where a large portion of the application runs outside of the PaaS and the portions in the PaaS are really just complements or front-ends for the main system. Cloud Foundry makes it relatively easy to make outside services available to applications in the PaaS, but these outside applications still need to be managed. This is where an IaaS management framework like RightScale is great because it can bring the whole infrastructure under one roof.

Some examples for this punching out:

• Databases from the SQL variety to NoSQL and other models. Accessing legacy databases as well as leveraging popular DB setups like our MySQL Manager, which provides master slave replication.
• Different load balancers in front of the request routers, perhaps with extensive caching features, global load balancing, or other goodies. Examples would be Zeus, Squid and many others.
• Legacy or licensed software, for example video encoding software or PDF generators.
• Special back-end services, such as a telephony server.

If there’s one thing I’ve learned about customers at RightScale it’s the incredible variety of needs, architectures, and software packages that are in use. For this reason alone I see PaaS as another very nice tool in the RightScale toolbox.

Can you run Cloud Foundry without RightScale? Of course. It certainly runs on raw servers. They can PXE boot a base image and join the PaaS in one of the above server roles. However in a mixed environment it is much more beneficial to run the Cloud Foundry roles within a managed infrastructure cloud.

It seems obvious from the traditional SaaS/PaaS/IaaS cloud diagrams that these different layers were made to interoperate. And that’s what we’ve already seen our customers doing: combining PaaS and IaaS in ways that meet their needs. There are a number of PaaS solutions in the market with more on the horizon. We will continue to support as many as we can and to the extent that their architectures allow it, because cloud is a heterogeneous world and customers want choice. In the case of Cloud Foundry, we have a particularly open architecture that provides a compelling fit – and we’re excited to see where our joint customers take us together.

The upshot is that there are a lot of moving parts! Each one of the subsystems consists of many servers and everything needs to scale-up as the load increases. What Animoto CTO Stevie Clifton did really well is to connect all the operations using queues, many of them in SQS. One queue contains work items that list photo URLs to fetch from other sites, such as Facebook, Flickr, etc., and that is processed by one array of worker instances. Another queue has the list of render jobs and each work item in there points to the set of photos sitting at the ready in S3 and at the music files also on S3. All of these queues are held in Amazon SQS and the arrays of worker instances are managed by RightScale. This allows the monitoring part of our service to detect when the queue gets too large and more instances need to be launched. What’s nice about using queues is that it decouples the various parts of the site, so if the renderers get backlogged the queue simply builds up and users have to wait a little longer for their video to be produced. Waiting is not good, but dropping requests on the floor is much worse!

Producing the videos takes 8-9 minutes on average and at peak Animoto has pumped over 450 render requests per minute into the queue. Last week we ended up with just under 3500 instances in the various Animoto deployments and tonight it was more than 4000 and it looks like it will not drop under 2000 instances through the night. Yikes! At peak RightScale was launching and configuring 40 new instances per minute pretty much sustained to handle the injection of thousands of render jobs that needed special handling. Mind-boggling stuff…

Lessons learned… First of all, when you scale 10x and then 10x again to run on thousands of servers every little problem turns into a large one. That insignificant error rate of 0.1% gets multiplied by 1000x per second and you end up with an error a second, and actually, the error rate typically increases in itself too because of the added load on the system. So suddenly it’s not something you can ignore anymore. An example for this was having exponential backoff for uploads to S3 when using curl, but forgetting that the 5th retry exceeds the S3 connection timeout. Normally, this happens only once in a blue moon, but when tens of uploader instances are banging hard on one S3 bucket the S3 error rate goes up a bit and suddenly uploads are failing left and right. Once we changed this to a constant retry timeout it all went smooth again.

Now does this mean that you should fix all the little issues before going live? Of course not: you can’t! What I’ve found to be most effective is to think about every little problem that you come across for a few minutes. Don’t just brush it aside as being insignificant. It is now, but it *will* trip you up tomorrow or the day after. So spend 5 minutes to troubleshoot and hypothesize as far as you can get. You don’t have to solve it immediately. Think up a work-around or how you would troubleshoot further, or perhaps how you’d fix it. Then move on. Come tomorrow, when and if the issue becomes big, you will have an invaluable head-start. Instead of being caught off-guard you’ll be able to immediately kick into action and solve the issue.

Another lesson learned is not to forget the manual overrides. Yup, I know, we have this super smart auto-scaling algorithm. But we also have manual overrides and when Animoto went from about 50 instances to 4000 instances we used it. We wanted to make sure the extra instances didn’t overload the database, the queue, and that everything was running smoothly (and, yes, to take a pause and fix some issues before scaling up further). Stevie and the Hungry Machines guys also had put in some overrides to queue-up automatically generated videos and let manually requested ones zip through. This was essential in keeping the active users happy when everything first exploded and the system had trouble keeping up with the load. A lot of the queued videos were processed a bit later when the load went back down. Automation is cool for the daily routine events but for something like this you want the overrides.

Animoto is a great example of leveraging the cloud for its strengths of instant availability and virtually limitless scope. Of course, most sites don’t need to launch 4000 servers in one go, but its nice to know you can if you need to. Whether the number is 4 or 40 or 4000 — getting the resources you need at the time you need them is a key benefit of “right-scaling” your deployment using the cloud. Looking at our database today I noticed that RightScale has launched, configured, and managed over 200,000 instances to date! That’s an impressive number — but as the Animoto scale-up proves, we’re only just beginning…