NSX-T Transport VLANs and Inter TEP Communication | LAB2PROD
Inter TEP Communication, Sharing Transport VLANs: Unpacking One of The Latest Features In NSX-T 3.1
This post will include a brief recap of how Tunnel Endpoints (TEP) and Transport VLANs have had to be configured for communication in prior releases of NSX-T. It will then focus on how the release of NSX-T 3.1 has changed the way Tunnel Endpoints for Edge Appliances that reside on NSX-T hosts can be configured, and how inter TEP communication can be achieved.
Table of Contents
- What are TEPs and Transport VLANs?
- How they communicate: Both host to host and host to Edge Appliances
- How TEPs need to be configured when the Edge Appliances reside on a host transport node in releases prior to NSX-T 3.1
- A streamlined method to TEP configuration NSX-T 3.1
Note: All screenshots in this article have been taken in an NSX-T 3.1 environment.
The video below shows Tunnel Endpoint and Logical Routing operations, it also walks through some troubleshooting.
NSX-T Tunnel Endpoints (TEPs) and Logical Routing // DEEP DIVE!! // DC VMUG 2022
This is a demonstration of NSX-T Tunnel Endpoints (TEPs) and Logical Routing, delivered live at DC V...
What are Tunnel Endpoints (TEPs) and Transport VLANs?
TEP’s are a critical component of NSX-T, each hypervisor participating in NSX-T (known as a host transport node) and Edge Appliance will have at minimum one TEP IP address assigned to it. The host transport nodes and Edge Appliances may have two for load balancing traffic, this is generally referred to as multi-TEP.
Transport VLANs is just what NSX-T calls the VLAN used for the TEPs, these are mapped using profiles and then attached to the host transport nodes and edge nodes.
TEP IP Assignment:
- Host transport node TEP IP addresses can be assigned to hosts statically, using DHCP or using predefined IP Pools in NSX-T.
- Edge Appliances can have their TEP IP addresses assigned statically or using a pool.
TEP’s are the source and destination IP addresses used in the IP packet for communication between host transport nodes and Edge Appliances. Without correct design, you will more than likely have communication issues between your host transport nodes participating in NSX-T and your Edge Appliances, this will lead to the GENEVE tunnel between endpoints being down. If your GENEVE tunnels are down you will NOT have any communication between the endpoints on either side of the downed tunnel.
Notice how here we have a column for Inner MAC, Outer MAC and Outer IP under the LCP section, also how the MAC addresses have been spread across the two TEP IPs. If we dig a little deeper, we can start by identifying what exactly those inner MACs are, where they came from and then repeat the same for the outer MAC.
Now that we have the logical switch’s name and UUID we can dig a little deeper.
Wait a second.. why are there no entries, especially since we know there is workload on this segment. Chances are, either the Virtual Machine (VM) has not sent an Address Resolution Protocol (ARP) request or it had and the table has since been flushed. We can repopulate this table by simply initiating a ping from a VM on this segment.
Excellent, now we have an entry in the table, if we scroll up to the first image you can now see that the inner MAC is the same as the MAC address from this entry. So that means the inner MAC is the source VM’s vmnic’s MAC address.
That leaves us having to identify the remaining two outer MAC’s, this is pretty easy to do. There are a couple of ways to do this, but let’s continue with nsxcli and the logical switch we have been working with.
This time we can run ‘get logical-switch 6b151015-ab77-46bc-b4bd-62281ee7d6ec vtep-table’, you should see output similar to the below image.
The above output may not be enough to determine which outer MAC belongs to each host transport node. To verify this, you can exit out of nsxcli, type in ‘esxcfg-vmknic -l’, and see the below output. Here we can see the outer MAC’s are the MAC addresses of the vmkernel nics used for NSX-T, vmk10 and vmk11. Therefore, we can deduce that payload from within the NSX-T domain is encapsulated in a packet header that uses the source and destination TEP IP and MAC addresses.
How TEPs Communicate?
NSX-T TEPs communicate with each other by using their source and destination addresses. The TEPs can be in Transport VLANs (different subnets) and still communicate, if this is part of your requirements then you must ensure those Transport VLANs (subnets) are routable and can communicate.
Once the packet is decapsulated the payload is exposed and actioned accordingly. This article focusses on TEPs and will not go into logical routing and switching of payload data.
How TEPs need to be configured when Edge Appliances reside on a host transport node in prior releases NSX-T 3.1
This is a screenshot of the edge appliances TEP IP interfaces.
Host transport node with TEP IP’s in the same range.
vCenter view showing the Edge appliances are in fact sitting on the management host which is prepared for NSX-T.
Now you should have a clearer picture of how the environment is set up. The next part of the article will show you that with this setup, inter TEP communication does not work, i.e. the Edge Appliance sits on a host transport node and both are on the same Transport VLAN, and the packet is not routed externally and brought back into the edge.
This is a WireShark view of the same to make reading easier.
Testing the same configuration with the Edge Appliances in a different Transport VLAN
Immediately the tunnels come up.
The below packet capture on Edge also shows the control messages are passing correctly now and that the packets are no longer being dropped.
In summary, to configure TEPs when Edge Appliances reside on a host transport node in release prior to NSX-T 3.1, ensure they are not on the same Transport VLAN. However, if they are, ensure they are not on the same vDS/N-VDS to have the GENEVE tunnel up.
How has this changed in NSX-T 3.1
TEP interfaces on the edge appliance, showing the same IP range.
And now if we go back into the UI, the tunnels are up.
And here is a packet capture on the edge to show the BFD control messages coming through.
Hello, I really appreciate your work! it’s just Awesome.
Please tell me, there is ay requierement for the vmk10 about where to place it (in term of portgroup) and what could be the best design for that?
In factI’m using NSX-T3.0 , i had a communication problem between edge and TN and i changed the mapping of vmk10 from none to “Seg_TEP” (its and Overlay Seg). Due to that, Node tunnel gone down and now I’m not able to replace vkm10 to previous location (Port group= None) so that I can at least get TN Tunnel UP and theh apply your tutorial to make then communication with edge as you explained here.
Also, i’m using
Hi Wilfried,
Are you attempting to run a single VLAN for tunnel endpoints and configure inter tep?
If so, you will need to create a VLAN-backed segment and set it to trunked 0-4094. In vCenter, attach this new portgroup to the edge VM. Then tag the transport VLANs in the uplink profile for the transport node. If you run into an issue where the VLAN-backed segment is not showing in vCenter to be able to attach it to the edge node, this is likely because host transport node does not belong to the same VLAN transport zone you created the segment in.
If you are not attempting to configure inter TEP communication, the portgroup does not need to be a VLAN backed and can be a standard VDS portgroup.
So your edge appliances vNIC attachments should look like this for inter TEP communication;
– vnic0 – management portgroup
– vnic1 – VLAN-backed trunked portgroup
– vnic2 – VLAN-backed trunked portgroup
Edge nic assignment for standard TEP communication:
– vnic0 – management portgroup
– vnic1 – VDS trunked portgroup
– vnic2 – VDS trunked portgroup
Then in System > Fabric > Profiles > Uplink Profiles; create the uplink profiles for your edges and host transport nodes, both should specify the same VLAN if using inter TEP, if not configuring inter TEP specify the correct VLAN ID for each.
Hopefully this clears things up for you.
Great article Shank!!
In my lab environment I am using NSX-T 3.2 version with VDS prepared host transport nodes. Two of those are added to both type of TZs (Overlay and VLAN). The Edges is running on Host transport node and it’s Mgmt. and Uplink interfaces are connected to VDS created DVPG. But the Tunnel interface for TEP is connected to NSX-T created VLAN segment. Both Host TNs and Edge TNs are sharing the same TEP IP Pool since they are in the same VLAN. But still the tunnel status is showing down from edge and host TN side. Any input from your side would be much appreciated.
Thanks!
The Edge node should have 3 vNICs, one connected to a management portgroup and two additional vNIC interfaces that will be attached to trunking portgroups. Both of these interfaces should be attached to a VLAN-backed segment configured in trunking 0-4094 and should be created in NSX-T.
Once you complete that, continue to wire the Tier-0 as per normal, using separate VLAN-backed segments for T0 uplink connectivity. Do not use the trunking portgroups you created for the Edge vNICs.
Let me know if that helps, I will look to rewrite this article to include the Edge wiring.