Cloud management over the cloud

One of the most important challenges we are facing at Radiojar, is the effective management of components that assemble this service platform. Radiojar as a platform virtualises all necessary components a web radio station needs to work. To name but a few: radio servers, radio automation, media library, virtual studio, streaming servers, webcasters, web apps etc. Each service has different needs in hardware resources. Most can work on their own but at the same time collaborate with other services. The architecture we envisioned would support independent services running from managed or unmanaged cloud instances and use a central “intelligent” system to control them, order them, check their health and expand them if needed.

This system has to be scalable, efficient and robust. Scalable means able to serve as many clients as demand dictates at any given time. Efficient means able to utilise all available resources to their full extent, which in turn leads to cost effectiveness, making the solution affordable and viable. Finally, robust means fail-proof, with low maintenance requirements, flexible configuration options and adaptability to various environments.

Service management in Radiojar

So, what we’re designing is a piece of software that runs on the web and can be triggered to perform certain tasks in given situations. Its major responsibilities are to track all running services and available resources, to expand the cloud with new hardware resources as needed, to register and unregister services and to delegate commands and pass configurations to running components. This is the component we’re calling Radiojar Manager.

The Radiojar Manager has a RESTful API that most components use for their communication. All services are unaware of their neighbours and there is no direct connection between them, they’re connected through the manager. Communication works both ways, components are able to receive and transmit status and commands related to their function. There are also some components that are completely unaware of being managed: these are what we call workers and they are designed to do specific tasks in a cloud computing environment but can be remotely turned off and on.

What we have achieved with the Radiojar Manager is to be able to have any number of components that we can combine to deliver a configurable solution. This solution can be robust, flexible and -most importantly- work in a self-managing manner, which is our ultimate goal for the Radiojar platform as a whole.

Chosing a hosting environment

With this concept in mind, we started looking for the best way to host the servers on which Radiojar’s infrastructure software will run. Research led us to Rackspace Cloud Servers. Its most appealing feature was the RESTful API, which allows for customised and automated management of our servers.

While these servers suit us for most of Radiojar’s components, there are cases where these virtual instances are not fit for our purposes. In such cases, we want to be to able to lease other servers from other providers. Of course we won’t be able to remotely manage the hardware of these servers but the services they run will be managed by the Radiojar Manager.

We chose to host the key component, the Radiojar Manager itself, on Google AppEngine which meets all aforementioned criteria. Infinite scalability, persistence and as reliability. It was a difficult and critical decision since we are creating another overhead which is security (yet another post to come). We also chose Java as a language of implementation instead of Python, mostly because of the open source resources that we found which could speed up our work.

Automatically expanding clouds

So we need to manage a cloud of servers (Rackspace Cloud Servers) from another cloud server (Google AppEngine). The goal is to create a system that will automate tasks that would otherwise have to be carried out by system administrators. When a new project has to be created, the system will communicate with Rackspace’s cloud service and automatically deploy and set up all necessary instances and services.

The connecting link between these platforms is Rackspace’s RESTful API. Since the Radiojar Manager is implemented as a Java project, we had to choose a framework that will make our lives easier when working with RESTful web services. Apache CXF was chosen for this purpose mainly because of our previous experience with it. We then created a library providing a Java binding for Rackspace’s API.

The most interesting feature of CXF was a WADL-based source code generator. Rackspace provides a WADL document for its web services and this served as a good test case for the generator. We ended up patching CXF’s source code to make it work, and submitted these changes contributing back to CXF (they are now part of the project). The result was worth the effort: instead of writing the required Java stub interfaces by hand, we generate them in seconds -and this can be done for any WADL-described RESTful web service. Of course, integration of the binding library into Google’s AppEngine had its quirks and in order to make everything work as expected we had to dedicate a considerable amount of time. In the spirit of open source code -and since most of our work is built over a collection of open source projects- we plan to release the binding library as an open source project too, after applying the necessary polish on the source code.

We are ready to jam

The Radiojar Manager is now in a condition to connect the dots between the numerous components in our platform. The amazing part of this component that acts as the heart of our platform, is that it has the intelligence to act according to feedback and demand, and at the same time it can be easily extended via web calls to include even more independent systems.

151 Notes/ Hide

  1. radiojar posted this