Start Your Free Trial

How to build a 250 node VMware vSphere/ ESXi lab environment in AWS for testing

This blog was written with guidance from Scott Lowe around best practices of VMware data center design and automation.

As most of you are aware, we recently announced the public beta of a new feature that allows users to run VMware ESXi™ on AWS or Google cloud. Essentially, we have implemented Intel VT/ AMD-V functionality in software in our hypervisor, HVX. That makes the underlying cloud look like real x86 hardware – complete with silicon extensions required to run modern hypervisors like ESXi and KVM. In this blog, I am going to illustrate how to set up a large scale, 250-node VMware ESXi data center in AWS for less than $250/hr. We believe that this could be extremely useful for enterprises for upgrade testing their VMware vSphere™ environment or for new product and feature testing.

We recently held a webinar discussing how to build ESXi labs on AWS/ Google Cloud. Enjoy the webcast and slides…

ESXi-home-lab

Prerequisites

  1. If you do not already have an account in Ravello, please first open a trial account.
  2. A VMware ESXi VM image in Ravello Library (described in another document).
  3. A VMware vCenter™ server VM image in Ravello Library (described in another document).

The first ESXi host

Adding the first ESXi

  1. Create an empty application (do not use any blueprint) in Ravello and give it a name.
  2. Add the vCenter server VM from the library and publish the application (note – it takes a few minutes to publish and few more minutes for the vCenter and all its services to go up). While this is being done, please continue to the next item (adding ESXi machine).
  3. Add one ESXi VMs from the library and update the application (note – it takes a few minutes to this kind of update and few more minutes for the ESXi machine and all its services to go up) and name it “firstesxi”( You will need to change the hostname (only in the Ravello web – defined in “General” tab->”Hostnames”) to “firstesxi”)). While this is being done, please continue to the next item (adding NFS machine).
  4. Use the VM named “NFS” from Ravello library to add an NFS server to your application and update your application in order to publish the NFS machine (you will need to select a key pair for that). This NFS machine contains some images to later on create virtual machines in your cluster. Note – You can use your own private NFS image (there is another blog describes how to create such an NFS image). If you choose your own private NFS image please adjust later steps accordingly.
  5. After all machines are published and finished booting, login to the vCenter server web (HTTPS on port 9443). The credentials are the same as when you saved your original vCenter into the Ravello library.
  6. Add a new datacenter.
  7. Add the ESXi host to the datacenter using its Ravello hostname “firstesxi”.
    Add ESXi Host to Datacenter
    The credentials are the same as when you saved your original ESXi into the Ravello library. Note – you will see a yellow warning regarding your host because it does not have any datastore. Later on in this document, When you will add NFS datastore, this warning will disappear.

Configuring ESXi host to use NFS

  1. Select the ESXi machine.
  2. Browse to “Related Objects” tab.
  3. Browse to “Datastores” tab (see screenshot).
    Datastores
  4. Click the “Create a new datastore” button.
  5. When asked to select either VMFS or NFS datastore type, select NFS and click Next.
  6. In the “Server” edit-box insert the hostname of the NFS server (should be “lio1”).
  7. In the “Folder” edit-box insert “/nfs” (see screenshot) and click “Next”.
    NFS Folder
  8. Continue and finish the wizard.
  9. You can now see that the yellow warning sign regarding no local datastore has disappeared.

Optional – Setting virtual distributed switch

In order to configure virtual distributed switch, follow the instructions on this article. Note – each ESXi in your datacenter has 2 NICs:

  1. The first one is used for management.
  2. The second NIC is used for data.

It is recommended putting the 2nd interface on a virtual distributed switch.

Note – You can decide whatever your networking setting are preferred. Just please notice that some network configurations are not supported. If you prefer another networking configuration than described above, when it comes to Host Profiles and specifically Host Profile Remediation – it might fail/make your ESXi unreachable.

Extract host profile

  1. Right click on the ESXi host.
  2. “All vCenter Actions->”Host Profiles->”Extract Host Profile…”
    ESXi Host Profile
  3. Give the profile a name (for example “myprofile”) and click “Finish”.

Creating a VM to run on your ESXi

Deploy the VM

  1. Right click on the ESXi host.
  2. Select “Deploy OVF Template…”
  3. Select your OVA file
    Select OVA
  4. Complete the wizard and wait few minutes until VM is deployed.
  5. Power on the VM.
  6. Using vCenter web interface- open the VM’s console and login with user and password associated with the OVA template. Note – you might need to install a plugin for your browser for that. Pay attention to popups/blocked windows during the plugin installation.

Configure the network of the VM

Static IPs

  1. Need to configure the default gateway to Ravello’s default gateway on the network of the application (usually 10.0.0.2).
  2. Set the netmask/network in such a way it will be exactly the same as the Ravello’s network (For example 255.255.0.0).
  3. Set the static IP of the machine to a “high unique ip” that it will not conflict with other Ravello VMs in the same application (For example 10.0.100.1).
  4. Set the DNS to known public DNS (like 8.8.8.8).

To test the networking – Open the console and ping from your vm some address in the internet, like www.google.com.

DHCP

Not supported for now

Install VMware tools

If VMware tools is not installed on the VM, it is important you will first install VMware tools.

Create a cluster with many hosts and VMs

This is the most important part in this document. We will describe here how to build a big datacenter using scripts, automation and multi-selection actions in the GUI.

You can repeat the following steps few times and create several big clusters. Each cluster can contain tens of ESXi hosts and hundreds of VMs. Here are some nice screenshot of a datacenter with 4 clusters, 64 hosts per cluster and around 500 VMs in total.
Cluster with many hosts

Cluster with many VMs

Create a new Cluster

  1. Please give it a simple name. For example “ClusterA”.
  2. Turn on the cluster’s DRS (by default – Fully automated).

Add several ESXi hosts

Note: If your application includes multiple ESXi serevrs, please make sure you start them separately, one by one, and not all at the same time.

  1. First you need to know your application ID and your ESXi template vm id.
    1. Your application ID can be found in the URL of your Canvas view (for example) of your Ravello application:
      https://cloud.ravellosystems.com/#/apps/YOUR_APP_ID/canvas?isNew=true
    2. Your ESXi template ID can be found when also in the URL when browsing to “Library->VMs”
      https://cloud.ravellosystems.com/#/library/vms/?vmIds=;YOUR_TEMPLATE_ID
  2. Then, download publish_vms_from_template_into_existing_application.py script from Ravello’s GitHub (https://github.com/ravello/vmware-automation).
  3. Then, run the script to deploy several ESXi hosts. In this example I have created 16 ESXi machines named “esxiA1″..”esxiA16”.
    python publish_vms_from_template_into_existing_application.py -t 56232441 -u "ohad@ravellosystems.com" -a 56558267 -b esxiA -n 16
  4. Script should print “Success” within few few seconds. Then it takes ~5 minutes for ESXi machines to be deployed.

Add ESXi hosts to the cluster – Using PowerCLI

Install PowerCLI

Installation instructions here.

Connect to your vCenter

Connect-VIServer -Server vcenter -Protocol https -User root -Password vmware

(you can ignore the yellow warnings if such appear)

Whereas:

  1. vcenter – indicates the hostname of your vCenter machine.
  2. root and vmware are the credentials for your vCenter machine.

Add ESXi hosts to the cluster

1..16| Foreach { Add-VMHost esxiA$_ -Location (Get-Cluster -Name "ClusterA")[0] -User root -Password esxpassword -Force -RunAsync}

Whereas:

  1. 1..16 – indicates that we want to add 16 hosts.
  2. esxiA$_ – indicates that that hosts names to add are “esxiA1″..”esxiA16”.
  3. “ClusterA” is the name of the cluster in which hosts will be added.
  4. root and esxpassword are the credentials for your ESXi machine.

Apply host profile to the new cluster

  • Right click on the cluster – “Attach Host Profile…”
    Host Profile
  • Select “myprofile” profile and complete the wizard (might take a while for “Validation”)
  • Select all hosts in the cluster
    Select all hosts in the cluster
  • Right click and “Enter Maintenance Mode” (confirm whatever needed).
  • Again, select all hosts in the cluster. Right click and “All vCenter Actions->” “Host profiles->” “Remediate…” and then continue the wizard (might take a while for “Validation”)
    Select All Hosts in the Cluster Again
  • Wait a few minutes until host configuration tasks are completed.
  • Again, select all hosts in the cluster. Right click and “Exit Maintenance Mode” (confirm whatever needed).
  • Clone VM several times – Using PowerCLI

    Now, that you have a nice cluster, it is time to deploy some VMs on it. In this document we will describe how to fastly clone a VM into the cluster several times (using a “linked clone”).

    Install PowerCLI

    Installation instructions here.

    Connect to your vCenter

    Connect-VIServer -Server vcenter -Protocol https -User root -Password vmware
    

    (you can ignore the yellow warnings if such appear)

    Whereas:

    1. vcenter – indicates the hostname of your vCenter machine.
    2. root and vmware are the credentials for your vCenter machine.

    Clone VMs using a linked clone

    $sOriginVM="Ubuntu 14.04"
    $sOriginVMSnapshotName="mastervm_linkedclone_snap"
    $oVCenterFolder=(Get-VM $sOriginVM).Folder
    $oSnapShot=New-Snapshot -VM $sOriginVM -Name $sOriginVMSnapshotName -Description "Snapshot for linked clones" -Memory -Quiesce
    $oESXDatastore=Get-Datastore -Name "Datastore"
    $oResourcePool=(Get-ResourcePool -Location (Get-Cluster "ClusterA"))
    1..25| Foreach {New-VM -Name UbuntuCloned$_ -VM $sOriginVM -Location $oVCenterFolder -Datastore $oESXDatastore -ResourcePool $oResourcePool -LinkedClone -ReferenceSnapshot $oSnapShot}
    

    Whereas:

    1. 1..25 – indicates that we want to clone 25 vms.
    2. Ubuntu 14.04 – indicates that the original vm to clone is named “Ubuntu 14.04”.
    3. UbuntuCloned$_ – indicates the name of the target cloned VMs, “UbuntuCloned1”..”UbuntuCloned25″ .
    4. Datastore – indicates that the NFS datastore is named “Datastore”.
    5. ClusterA – indicates that the target location is in a cluster named “ClusterA”

    Then, Using vCenter web interface, select all vms and power on.

    Set the VM order accordingly in Ravello GUI

    Storage VM and vCenter VM need to start first (before all ESXi machines) and shut down last (after all ESXi machines).

    Optional – Monitoring crucial vms

    It is recommended that you will install a monitoring tool to monitor crucial machines in your datacenter, such as NFS and vCenter server. It helps a lot understanding how the datacenter behaves and how you can stretch its limits.

    I will demonstrate graphs with New Relic.
    New Relic Graphs

    Saving application to blueprint

    How to create a blueprint

    Known issues/limitations

    1. The ESXi network interfaces in Ravello are having some problems working with VMXNet3. Please use only E1000 for the VMs network interfaces and for the ESXi network interfaces.

    VMware product names, logos, brands, and other trademarks featured or referred to in the ravellosystems domain are the property of VMware. VMware is not affiliated with Ravello Systems or any of Ravello System’s employees or representatives. VMware does not sponsor or endorse the contents, materials, or processes discussed on the site.

    Note: If your application includes multiple ESXi serevrs, please make sure you start them separately, one by one, and not all at the same time.

    About Ravello Systems

    Ravello is the industry’s leading nested virtualization and software-defined networking SaaS. It enables enterprises to create cloud-based development, test, UAT, integration and staging environments by automatically cloning their VMware-based applications in AWS. Ravello is built by the same team that developed the KVM hypervisor in Linux.

    Check our product demo video

    How to build a 250 node VMware vSphere/ ESXi lab environment in AWS for testing