NSX Logical Routing in the Homelab - OSPF Edition

NSX Logical Routing in the Homelab - OSPF Edition

· Read in about 14 min · (2922 words) ·

NSX ESG

Introduction

NSX is pretty freaking cool (both V and T; although we’re exclusively talking about V in this post). It’s pretty much the Swiss Army Knife of the modern Software Defined Datacenter. It’s time to value is totally unmatched; and the fact that you can consume such advanced services with such basic hardware requirements is unreal. Ever since NSX was added to VMUG Advantage; we’ve seen a drastic adoption of it in homelabs. The cool thing about deploying NSX in a homelab is that as long as you have a router that can handle OSPF of BGP as a dynamic routing protocol, many of the concepts actually will carry over to an enterprise deployment (albeit; at a drastically reduced scale; 3 servers absolutely does not equal 100 servers).

From a field perspective, when we start working with a customer on NSX - we’re most commonly looking to understand what they are looking to get started with specifically. Microsegmentation, or Logical Networking? Most people don’t realize how quickly you can get started with just the microsegmentation use case. Simply deploy the NSX manager and link it up to vCenter and you’re off to the races. Logical networking is a bit more complicated; but not by much. You need the 1600 MTU on your transport VLAN. You also need to solve for dynamic routing in/out, unless you’re going to do static routes. Fortunately - there’s a ton of hardware these days that can do OSPF and BGP for extremely cheap. In this article, I use the Ubiquiti Unified Security Gateway, a sub $125 router that is extremely popular in the homelab community. The Edgerouter Lite (also by Ubiquiti) is another common device with the same configuration.

Each time I talk about my NSX deployment, a common question comes up - How did you get the logical networking working in a lab? Drilling down into this, the real questions end up being…

  • What subnets should I configure?
  • How can I do dynamic routing?
  • OSPF or BGP?

In this post, I’ll take through the configuration I use in my lab, and how I landed here. I’ll also cover why Sjors and Tim both pick on me, in the hopes of eliciting some level of pity.

Lets jump in!

Networking Requirements

I’m not going to go through all of the virtual machine/appliance requirements for setup. We’re going to assume that you have those covered. At a high level…

  • Ensure you have a VDS setup between the hosts in your environment. I named mine “NSX-VDS”, clever eh?
  • Carve up appropriate VLANs that you want to use for logical separation USG vLAN

As you can see, I leveraged 3 /28 networks in my environment as the “core” configuration. Ultimately I could’ve made these smaller and tuned them better - but I felt like the simplicity of 3 /28’s was easy enough. Besides, at least its not 3 /24’s :). These networks are…

  • Management - For the NSX Manager and Controllers
  • VTEP - For the Virtual Tunnel Endpoint IPs, these are the VMKernels that are assigned to transport the VXLAN traffic (Transport VLAN)
  • Uplink - A physical VLAN where the Edge Services Gateway has an interface on, handling the ingress/egress between virtual and physical environments.

ESXi Configuration

Within ESXi/vCenter, I’ve configured my networking as follows

  • Standard Port Group created for NSX Management, using VLAN 600
  • Uplink Port Group Configured on my VDS, using VLAN 620 - Note, I’ve seen a lot of people place this on a Standard Port Group as well. It’s probably not a great idea to place this on the same VDS as your NSX is configured to use, but in a lab its fine
  • During my NSX configuration, I specified using VLAN 610 for the VTEP traffic - this automatically creates a Port Group on the VDS as a result; the following configuration occurred NSX VTEP
  • I also created an nsx-ha Port Group on the VDS to handle HA communication between the DLR Control VMs

Unified Security Gateway - OSPF Configuration

If you were following me on Twitter recently, you’ll notice a debate broke out around using BGP vs OSPF for peering between NSX and the physical network. Most of the time, best practice says that leveraging BGP gives you a lot more flexibility. I opted to leverage OSPF in my environment for simplicity, and due to how small my lab environment is. Configuring BGP isn’t much more difficult at this scale.

When using the Unified Security Gateway, in order to keep your settings persistent, you need to create a configuration JSON. I’m not going to cover that in this guide - there are plenty of guides out on the internet that cover that. What I will provide is the snipped of the configuration that is required for OSPF to work.

Unifi Controller gateway.config.json

{
	"interfaces": {
		"ethernet": {
			"eth1": {
				"mtu": "1600",
				"vif": {
					"620": {
						"ip": {
							"ospf": {
								"cost": "1",
								"dead-interval": "4",
								"hello-interval": "1",
								"priority": "128",
								"retransmit-interval": "5",
								"transmit-delay": "1"
							}
						}
					}
				}
			}
		}
	},
	"protocols": {
		"ospf": {
			"area": {
				"0": {
					"area-type": {
						"normal": "''"
					},
					"network": [
						"172.24.11.32/28"
					]
				}
			},
			"default-information": {
				"originate": {
					"metric-type": "2"
				}
			},
			"parameters": {
				"abr-type": "cisco"
			},
			"redistribute": {
				"connected": {
					"metric-type": "2"
				}
			}
		}
	}
}

By creating the gateway.config.json file and loading it onto my Unifi controller, this configuration is appended to the running configuration within the device. You can see that we set the MTU on the interface to 1600 as required, we then dive into the specific vlan (vif in Ubiquiti world) and configure OSPF. Take note of your configured interval and delay settings - you need those to match in NSX!. Also, ensure you update your network in the USG to match what you’ve configured from an uplink perspective (if you don’t use mine).

Next, we head in and configure the actual OSPF service on the router. We setup what area we are using (0 in my case), and define what type of area, and what network should be peering. This is the VLAN 620 network I configured; the Uplink network.

With this file in place, OSPF is running on my USG and I can start deploying/configuring my ESG’s and DLR’s in NSX. Note, we could’ve configured those first without any problems. Routing just wouldn’t happen dynamically until OSPF or BGP (or static routes) were setup to handle ingress/egress.

Deploying our Edge Services Gateway

The Edge Services Gateway (ESG) is a pretty powerful little beast. It handles bridging your physical and virtual networks, but also has a ton of additional features it can do. Routing, VPN, Load Balancing, DHCP Relay, as well as its own firewall capabilities. Again, think Swiss Army Knife.

In my environment, I’ve configured the ESG to deploy an HA pair. This isn’t mandatory for a lab, so if you’re resource constrained you can certainly do a single appliance. Deploying an ESG is very straightforward. Once you’re in the NSX interface within vCenter, select Edges on the left

Edges

Select the green plus symbol on the top, and ensure that Edge Services Gateway is selected (it is by default). Name our ESG, and provide it a hostname if you wish. The configuration steps are easy to decide on based on your environment for the most part. I usually keep SSH disabled since I can just use the vCenter console, plus then I don’t need to setup management IP’s.

When you reach the “Configure Interfaces” screen, you’ll want to pay close attention to what you set. This is where we are configuring our Uplink interfaces (as well as our Internal interface if we want to; but I typically do that later). You can see how I configured mine below:

NSX ESG

Note a few key items * Uplink selected for the network type * Connected to the Port Group I configured earlier * IP Addresses that are part of the VLAN I setup for that port group are specified in the primary/secondary IP Address boxes, along with the appropriate subnet mask in CIDR format.

Select OK when you are set, and move onto configuring your Default Gateway. Since you likely only created a single interface here, it should be selected by default. Plug in the Default Gateway on the network you created. In my case, it was 172.24.11.33.

In a lab environment, I set the default firewall to allow traffic initially so I don’t get stuck troubleshooting with the firewall enabled - but this obviously a security compromise. Your mileage may vary.

Finish up, and you’ll see an OVA deploy in your environment for the ESG. You’re well on your way! You’ll notice that you should be able to ping the interfaces you created after the configuration is complete. This is a good sign!

Now we can setup OSPF on the ESG…

Configuring OSPF on the ESG

Double click on the ESG to enter the configuration. Ensure you are on the Manage tab. You should see vNIC 0 assigned to the uplink we created during the deployment.

NSX ESG

Select the “Routing” tab. From a “Global Configuration” perspective. our Gateway should already be set from our earlier configuration. Under “Dynamic Routing Configuration” select the edit button. The drop down that displays will be the uplink interface you configured during the original setup. Select ok, and you’ll notice a green notification pop up on the top of the page allowing you to “Publish” the changes.

Select OSPF on the left. Select edit and check to Enable OSPF. In my environment, I deleted the NSSA area (51) leaving just area 0. Long term, we’ll end up adding another area here for the downstream peering with the DLR - but for now we can leave it as just the area 0 that we will be peering between the ESG and our USG Physical Router. Select the green plus symbol in the “Area to Interface Mapping” section. Select our uplink interface, are defined area, and set the appropriate intervals we set in our gateway.config.json.

"ospf": {
		"cost": "1",
		"dead-interval": "4",
		"hello-interval": "1",
		"priority": "128",
		"retransmit-interval": "5",
		"transmit-delay": "1"
	}

Once these are set, you can hit publish again.

Finally, select the “Route Redistribution” tab. Select edit and check the OSPF box. Select “Ok”. Under “Route Redistribution Table” select the plus symbol again. We’re not doing any BGP so we don’t need to check that box. We also (hopefully) aren’t doing any static routes so we don’t need to check that box. We will check connected routes, and select “OK” to move forward.

Route Redistribution

Hit publish changes and we’re done. OSPF should start peering between your physical router and the virtual infrastructure. You can verify this on the USG by SSH’ing to the Unified Security Gateway device and running a show ip ospf neighbor

Show IP OSPF Neighbor

We can also do a show ip route to see all the individual routes that are being shared via OSPF

Show IP OSPF Neighbor

Assuming you are seeing routes similar to the above, all is good! If you aren’t I’d suggest rechecking your settings, or doing a console to the ESG and running a debug ip ospf and seeing what errors pop up. When you’re done, you can do a no debug ip ospf to stop the messages from coming up. You should see a lot of messages within this screen that’ll help you understand what problems are happening. Most typical things I see are interval mismatches, MTU mismatches, or actual network routing issues.

From here, we can move on to deploying our DLR.

Deploying our Distributed Logical Router

Before we get started, we need to create a Logical Switch in NSX. This switch is going to effectively be our transport network for the communication between the ESG and DLR. The 2 devices will have 3 IPs total that facilitate communication. The ESG will have a downstream interface, while the DLR will have a Protocol IP Address and a Forwarder IP Address. We’ll get to that later.

For now, head into the “Logical Switches” tab in NSX. Create a network, in my environment I called it “transport-lswitch”. If you look at your VDS in vCenter, you’ll see a new network was created. Now we can start creating the DLR.

The process is similar to deploying an ESG. We move back to the Edges screen and select the green plus symbol again. This time, we select the “Logical Router” option. In my lab, I selected “Enable High Availability” which at a later screen required me to select an HA “Port Group” for the HA traffic to communicate over. In a lab, doing a single deployment is fine.

Move through the appliance deployment as typical. When you arrive at the interface configuration screen, make sure you set your HA network if you opted to go that route. Select the green plus symbol under “Configure Interfaces of This NSX Edge”. Name it appropriately, select your transport network on the “Connected To”. You can see an example of my settings below.

Create DLR

For our subnet, we’ll configure a smaller private one this time. For this interface, I used 172.24.12.3/29 as an uplink to the Transport Switch. Eventually, we’ll configure an “Internal” network on the ESG’s second NIC slot to tie these together. Submit this interface and select next. We don’t need to configure a default gateway this time. We’re not going to be routing out - so uncheck that box. Move forward and complete.

Summary DLR

The OVA should deploy successfully. We’re getting close to done. We can now configure our OSPF between the DLR and ESG.

Configuring OSPF on the DLR

Enter into our DLR from the NSX interface, and select the Routing tab. Under Global Configuration > Dynamic Routing Configuration. Select Edit next to it. This is similar to what we did on the ESG settings. Assign your Router ID to the interface that was configured. Select, and publish the settings. Notice how default gateway continues to be blank.

Routing Config

Select OSPF on the left, similar to on our ESG however the OSPF screen looks different this time. We have a Protocol Address and Forwarding Address to fill in. Select Edit, and change the drop down to the name of the uplink we created (esg-uplink). You’ll notice the “Forwarding Address” is automatically populated with the IP of the interface. For the protocol address we’ll use 172.24.12.2 and ensure the “Enable OSPF” box is checked.

Routing Config OSPF

Select OK. In my environment I deleted the default Not-So-Stubby-Area, 51. We will add a new area, for me I created an area id of 10 - leaving everything else as default. We also want to create an Area to Interface map for the uplink interface to match with the Area ID we just created.

Area to Interface

As before, select OK and publish our changes. Move on to Route Redistribution. It should already be configured. If its not, ensure you enable OSPF and then enable Route Redistribution over OSPF for Connected networks. Our DLR should be completely configured at this point.

Now we return to the ESG to tie up the last of the configuration.

Finalizing ESG Peering with DLR

There are a few items we need to setup on the ESG. Double click on the ESG, and navigate to Manage > Interfaces. Edit the 2nd VNIC (vNIC #1). We’re going to create an internal network for the communication. Set it to operate over the Transport-Switch we created earlier and create the actual IP address. I used 172.24.12.1/29. See my example below.

ESG Interface

With this in place, the communication pathway between our devices is set. The ESG has the network, and the DLR has to interfaces that live on that network. Change to the Routing > OSPF tab. We need to create the Area we created on the DLR. Add a new area by clicking the green plus symbol, as I mentioned above - in my environment it was “10”. After this is created, create a new “Area to Interface Mapping”.

DLR Interface

After hitting OK, as usual, go ahead and publish the changes.

Interfaces Configured

This completes our OSPF configuration. Lets create a network to test with!

Creating Our Network

In NSX head over to Logical Switches. Create a network, I named it “Test-Network” for a start. Same as when we created the Transport-Switch network; if you look in vCenter at your VDS - you’ll see a new network created. Head back into our DLR that we created and select Settings > Interfaces.

Select the green plus symbol. Name the interface, and select the Logical Switch we just configured. Select OK. Click the green plus symbol under “Configured Subnets” and add a network, for testings sake I did a 192.168.8.1/24 and hit OK.

Interfaces Configured

If we fire up a “PING” out to 192.168.8.1, and everything was configured correctly - you should get your responses!

PING Success

Also, if we SSH to our USG, we can do a show ip route again and see our connected networks.

Networks Connected

We’ve wired up our networking in our homelab! Success!

Conclusion

Getting started with NSX is easy; and the flexibility you get out of it is pretty incredible. Like I said earlier - Swiss Army Knife. This post should’ve exposed you to a lot of concepts, and hopefully you’ll have seen how straight forward it is to get started with some traditionally complicated concepts. There is still a lot to learn; and just because you can configure it doesn’t mean you completely understand it end to end - but at least you can get started with the functionality!

Part 2 of those post will be migrating our OSPF configuration to BGP instead, mostly so that Sjors and Tim will get off my back :)