This paper outlines a proposed architecture for an elastic CI capsule that allows testing VMware virtual appliances on AWS using Ravello’s nested virtualization technology.
Range of CI implementation in the VMware technology partner community
VMware has over 2,000 ISV partners that provide everything from security appliances, WAN optimization appliances, backup/replication appliances etc. to the broader ecosystem. Most of these ISVs have implemented some form of automated testing – ranging from weekly builds that have to go through a series of automated tests to true CI where every commit triggers a build that goes through a range of automated tests in parallel.
Lab capacity shortages for advanced ISVs – Jenkins queue buildup
Today, once a build is triggered (the output is an image or an ISO file, installer, binary or .war file), Jenkins spins up a set of virtual machines (using the VMware vSphere™ API), boots the VMs from the new image, runs a battery of automated tests, returns the results and then shuts down the instances. This usually works well if the builds are not too frequent – say weekly or even daily. As these ISVs get more advanced and move towards true continuous integration where every commit triggers a build, they very quickly run out of on-premises lab capacity. This usually manifests itself in ever increasing Jenkins queues.
Nested virtualization enables using AWS for CI for VMware workloads
In an ideal world, ISVs would run their virtual appliances in AWS and run integration and system tests in the public cloud. This would allow them infinite capacity and a pay-per-use model that is ideal for testing, especially CI. However, VMware virtual appliances do not run natively in AWS.
Ravello is an overlay cloud service powered by nested virtualization. It allows ISVs to run VMware virtual machines on AWS or Google without any changes. It also includes an overlay network that exposes a clean L2 network to the virtual machines. This allows ISVs to define any network configuration – static IPs, DHCP, DNS, multicast, VLANs, etc. exactly as they would in their data center.
An elastic CI capsule for testing VMware workloads in AWS
From an overall architecture perspective, most ISVs have their source control repository and build server on-premises along with their CI server – in this case a Jenkins master. The test environments (the CI capsule) will run in the cloud (AWS or Google).
The capsule is an “application environment” in Ravello. It consists of a capsule controller and up to 500 capsule workers. The capsule controller consists of a Jenkins slave (and optionally a WAN optimization appliance to help reduce the amount of VM appliance/image data transferred on every build if needed). The Jenkins slave is responsible for starting the capsule worker VMs with the right build/image, executing the tests, returning the results and then shutting down the VM. The worker VMs are started on-demand/ just-in-time when they are needed providing the “elasticity” needed in the test environment to absorb variable load. The Jenkins master (sitting in the enterprise data center) can spin up additional CI capsules if more capacity (more than 500 capsule workers) is needed. Hence, elasticity is provided along two dimensions – first, within a capsule to turn on and off worker VMs as needed and second, the ability to have multiple capsules as required to scale beyond 500 VMs.
The end-to-end CI flow
The overall CI architecture is shown in the diagram below and explained in 5 steps.
Step 1: Code commit and build
A developer checks in code into source control. Jenkins triggers a build. The output of the build could be anything – in our case, let’s assume that its a yum repo (could be any file repository). The entire repo generated is about 5GB in size.
Step 2: transfer the files to the test environment
The WAN optimization appliance transfers just the changeset to the capsule in the cloud. Typically, the changeset is about 5-20MB in size. This data transfer takes just a few seconds.
Step 3: upload the image to the Ravello service
The Jenkins slave in the CI capsule creates the image from the latest yum repo (file repository) and uploads that to the Ravello service via the CLI uploader. An example script is provided here:
ravello import-disk -u firstname.lastname@example.org ~/disk.qcow --name vsh_ver12345
Note: Since the data transfer here is within the AWS cloud, it will be relatively fast compared to going over the WAN.
Step 4: Spin up the virtual appliance (DUT)
The Jenkins slave then creates a new VM in its capsule/application environment. The VM then boots from the image that has been recently uploaded to Ravello.
The following one-liner uses a utility which comes bundled with the Ravello Python SDK called ravello-create-nodes. This utility is a simple CLI which enables the creation of complex applications and addition of VMs to such applications.
In order to build a VM around the uploaded image, add it to the existing application and update the changes, you can simply run:
ravello-create-nodes -i <image id> -a <app name> 1
<image id> is the ID of the disk image we just uploaded in step 3 and
<app name> is the name or ID of the application in which we wish to add the VM. The trailing “1” instructs the utility to add a single copy of this VM to the application.
The VM will be created with a default hardware profile (1 vCPU, 2GB RAM, virtio devices for disks and network, DHCP based network, port-forwarding for external traffic and no supplied services). For creating a VM with a different profile, please refer to the documentation by running
In order for it to authenticate using the right user, make sure to set the environment variables
RAVELLO_PASSWORD to the correct values.
Step 5: execute automated tests
Once the VM has booted, the Jenkins slave can run a series of automated tests and return the results to the development team.
Step 6: Delete the DUT VM (or leave it on for debugging)
At this point, the Jenkins slave can delete the DUT VM or in case there were some errors, it can leave the VM running and send a link (public DNS name or IP address) to the developer or QA team to investigate further.
Ravello offers a straightforward and easy way to extend an on-premises VMWare virtual appliance CI lab environment to AWS or Google cloud. ISVs can continue to use their on-premises lab environment and burst as required to the public cloud to avoid Jenkins queue buildup.
About Ravello Systems
Ravello is the industry’s leading nested virtualization and software-defined networking SaaS. It enables enterprises to create cloud-based development, test, UAT, integration and staging environments by automatically cloning their VMware-based applications in AWS. Ravello is built by the same team that developed the KVM hypervisor in Linux.