Start Your Free Trial

Building VMWare NSX lab on AWS and Google Cloud with Ravello Systems

Hi, that is my first blog post ever, so don’t judge too much. This post will introduce my journey in running VMware NSX™ lab setup on Ravello service. Firstly let’s get some background how I found Ravello.

Everything began with my willingness to learn new stuff and VMware NSX came to my point of interest. I started googling around checking where, what and why. I came across different bloggers who setup their own home labs and play/lab on it. Therefore, I thought in building my own home lab too, but after checking prices jumped of those thought since it was too expensive and time consuming. Next step was to pay for a dedicated server or to get equipment together with a friend, but with such solution, it is not 100% yours and you need to agree with friend on time slots etc. Therefore, I went back googling and while googling “Virtual Labs”, since after all that was I wanted to do, I came across Ravello.

Of course there are the “HandOnLabs” from VMware, but they are pre-configured and I wanted something that I can build from scratch.

Long story short, here began my journey with Ravello and NSX lab setup.

Lab requirements

NSX Edge (Large)

  • 1GB RAM
  • 512MB HD
  • 2 vCPU
VMware NSX
NSX Controller (each)

  • 4GB RAM
  • 20GB HD
  • 4 vCPU
NSX Manager

  • 12GB RAM
  • 60GB HD
  • 4 vCPU
Test VM

  • 1GB RAM
  • 512MB HD
  • 2 vCPU

Total with 3 Controllers, Manager, 2 Edges, Test VMs, Windows VM and vCetner

  • ~45GB RAM
  • ~200GB HD
  • ~20 vCPU

I decided not to invent the wheel and followed those great blogs during my installation and configuration of the NSX, I won’t go over it in my post:

So basically what I’ve build is a 3 tier setup with internet access over 6 ESXi Hosts, but of course less can be useful as well, for example 4 (you will understand why as I go deeper with the setup)

The journey begins

On the way of getting optimized configuration and setup I faced some issue and I hope readers won’t do same.

In my first attempt to setup NSX, I wanted to properly configure at least 3 ESXi hosts with VDS switch and move forward from that.

I installed 3 Hosts on Ravello, but decided to install vCenter on my local PC. That was a big mistake, since no matter what I was doing: configuring hosts to use VDS or any other configuration changes, hosts were always disconnecting from the vCenter leading to a horrible experience. No matter what networking configuration I tried on Ravello, DHCP, Static – the result was the same.

Network Connection

My next step was to get vCenter up and running on Ravello too, having it on the same LAN as Hosts.
Installation was successful, with only one note: during the boot there is an error.

Error

You can skip it and move on with booting vCenter, it doesn’t heart anything. The only problem that it must be done each time you boot vCenter.

After getting vCenter and Hosts up, I finally managed to setup VDS and everything looked just fine.

The next step was to deploy, or at least to try to deploy, NSX Manager – which ended up with failure. I suspect that due to a delay or other networking issue it didn’t want to upload image from my local PC to Amazon running through Ravello.

Therefore, my next step was to install some windows in order to get files on it and use it as my Workstation for vCenter, deployments and other stuff.

And that was a birth to my Lab setup 🙂

NSX Lab Application

Lab Setup and Hosts preparation

As you can see I have 6 Hosts (2 for management cluster, 2 for compute A and 2 for compute B). Now I can say that one host per compute A and B would be enough too, since test VMs do not use that much resources in my setup. Each ESXi host has, together with 300GB HD and 4vCPU – LSI Logic Parallel.

System Properties

vCenter (Linux appliance was used)

System Properties

And Windows with 50G IDE and 4vPCU

image10

In order to get everything properly working on the Hosts

  • Enable nesting on the ESXi Vms
  • Test powering on the VMs
  • Avoid other errors that can come with improper setup

Some configuration changes needed on the Hosts running ESXi themselves (I did just those bellow and nothing else)

Delete ESX UUID

  1. run “vi /etc/vmware/esx.conf”
  2. go to last line in the file in which “/system/uuid” is defined. Delete this line and save the file.

Set unique MAC addresses

  1. In Ravello GUI, in the Network tab of the VM, make sure “Auto MAC” is checked for both interfaces.
  2. run ‘esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1’

Enable nesting on all ESX guests

  1. This step is important in order to be able to power on VMs running on ESXi. It replace the need in configuring each guests with the ‘vmx.allowNested’ flag.
  2. run ‘vi /etc/vmware/config’.
  3. add the following to the file ‘vmx.allowNested = “TRUE”‘ and save.

Ensure changes are saved

  1. run ‘/sbin/auto-backup.sh’ (ignore warnings in its output if exist)

Here is the network configuration:

Network Configuration

After all NSX modules were installed, and I’m not going to go over step by step for that, there are many forums and blogs that do that, here is the topology I got:

Topology

Details

Windows and DNS – I didn’t setup Win Server with DNS, many of you would like to do that of course, but I decided to keep it simple and fast, I used haneWIN DNS Server 1.5.5 for my DNS settings and that worked just perfect without any issues. To copy files to windows SolarWinds-SFTP-SCP-Server was used and besides that I used FireFox to manage vCenter Web Client.

Hosts – As you can see I have 3 Controllers on the same Host, which is not good for redundancy, but for a lab setup it is enough. Manager and 2 edges reside on one Host too. Web, App and DB VMs each on separate Host

DNS – I have 2 DNS servers configured, one Win7 running DNS server and second Ravello DNS (Win for internal name resolving, Ravello for internet one).

NTP – NTP everywhere was setup to use public one – pool.ntp.org

And here is Logical Diagram after all configuration and manipulations:

Logical Diagram

Overview

Each VM (Web, App and DB) reside on different subnets, 192.168.10.0/24, 192.168.20.0/24 and 192.168.30.0/24.

Logical router makes possible for these VMs to speak with each other.

NSX Edge and Logical router has OSFP configured between them in order to redistribute Logical router’s connected subnets further, thus allowing internet connection for Web, App and DB VMs.

NSX edge also has ospf default-originate which is default GW for Logical router.

The last thing that makes internet connection work from VMs is a SNAT on the Edge Uplink interface.

Caveats

  1. vCenter must have DNS name and must be accessible from the browser with name and not IP. If vCenter reached over IP, you will not be possible to deploy NSX Manager ovf template. I overcome that by simply configuring hosts file on my local PC or Workstation from where I was accessing Web Client.
  2. DNS records needed for the hosts preparation after NSX Manager deployed, which is why I used simple DNS server on windows 7 just to get it working.
  3. DNS needed when you register NSX Manager with vCenter for SSO, it won’t work without that, but you can use IP during setup. That is funny because it is not needed for the actual Manager and vCenter registration, works without any DNS record.
  4. When you stop Ravello application, it takes time to have all back up online so here are few things you’ll need to do:
    1. Make all guests auto power on with Host’s power on.
    2. Once NSX manager is up, check the registration with SSO and vCenter.
    3. Once b completed and all good, from vCenter NSX menu – Node installation/preparation must be done again (sort of “resolve the issue”). Once that done, VXLAN and all other stuff will be online again.
    4. After “c” you’ll need to go to each host and reload netcpa process, without which inter-VXLAN connect won’t work.

VMware product names, logos, brands, and other trademarks featured or referred to in the ravellosystems domain are the property of VMware. VMware is not affiliated with Ravello Systems or any of Ravello System’s employees or representatives. VMware does not sponsor or endorse the contents, materials, or processes discussed on the site.

About Ravello Systems

Ravello is the industry’s leading nested virtualization and software-defined networking SaaS. It enables enterprises to create cloud-based development, test, UAT, integration and staging environments by automatically cloning their VMware-based applications in AWS. Ravello is built by the same team that developed the KVM hypervisor in Linux.

Check our product demo video

Building VMWare NSX lab on AWS and Google Cloud with Ravello Systems