VXLAN Deep Dive

I’ve been spending my free time digging into network virtualization and network overlays.  This is part 1 of a 2 part series, part 2 can be found here: http://www.definethecloud.net/vxlan-deep-divepart-2.  By far the most popular virtualization technique in the data center is VXLAN.  This has as much to do with Cisco and VMware backing the technology as the tech itself.  That being said VXLAN is targeted specifically at the data center and is one of many similar solutions such as: NVGRE and STT.)  VXLAN’s goal is allowing dynamic large scale isolated virtual L2 networks to be created for virtualized and multi-tenant environments.  It does this by encapsulating frames in VXLAN packets.  The standard for VXLAN is under the scope of the IETF NVO3 working group.

 

VxLAN Frame

The VXLAN encapsulation method is IP based and provides for a virtual L2 network.  With VXLAN the full Ethernet Frame (with the exception of the Frame Check Sequence: FCS) is carried as the payload of a UDP packet.  VXLAN utilizes a 24-bit VXLAN header, shown in the diagram, to identify virtual networks.  This header provides for up to 16 million virtual L2 networks.

Frame encapsulation is done by an entity known as a VXLAN Tunnel Endpoint (VTEP.)  A VTEP has two logical interfaces: an uplink and a downlink.  The uplink is responsible for receiving VXLAN frames and acts as a tunnel endpoint with an IP address used for routing VXLAN encapsulated frames.  These IP addresses are infrastructure addresses and are separate from the tenant IP addressing for the nodes using the VXLAN fabric.  VTEP functionality can be implemented in software such as a virtual switch or in the form a physical switch.

VXLAN frames are sent to the IP address assigned to the destination VTEP; this IP is placed in the Outer IP DA.  The IP of the VTEP sending the frame resides in the Outer IP SA.  Packets received on the uplink are mapped from the VXLAN ID to a VLAN and the Ethernet frame payload is sent as an 802.1Q Ethernet frame on the downlink.  During this process the inner MAC SA and VXLAN ID is learned in a local table.  Packets received on the downlink are mapped to a VXLAN ID using the VLAN of the frame.  A lookup is then performed within the VTEP L2 table using the VXLAN ID and destination MAC; this lookup provides the IP address of the destination VTEP.  The frame is then encapsulated and sent out the uplink interface.

image

Using the diagram above for reference a frame entering the downlink on VLAN 100 with a destination MAC of 11:11:11:11:11:11 will be encapsulated in a VXLAN packet with an outer destination address of 10.1.1.1.  The outer source address will be the IP of this VTEP (not shown) and the VXLAN ID will be 1001.

In a traditional L2 switch a behavior known as flood and learn is used for unknown destinations (i.e. a MAC not stored in the MAC table.  This means that if there is a miss when looking up the MAC the frame is flooded out all ports except the one on which it was received.  When a response is sent the MAC is then learned and written to the table.  The next frame for the same MAC will not incur a miss because the table will reflect the port it exists on.  VXLAN preserves this behavior over an IP network using IP multicast groups.

Each VXLAN ID has an assigned IP multicast group to use for traffic flooding (the same multicast group can be shared across VXLAN IDs.)  When a frame is received on the downlink bound for an unknown destination it is encapsulated using the IP of the assigned multicast group as the Outer DA; it’s then sent out the uplink.  Any VTEP with nodes on that VXLAN ID will have joined the multicast group and therefore receive the frame.  This maintains the traditional Ethernet flood and learn behavior.

VTEPs are designed to be implemented as a logical device on an L2 switch.  The L2 switch connects to the VTEP via a logical 802.1Q VLAN trunk.  This trunk contains an VXLAN infrastructure VLAN in addition to the production VLANs.  The infrastructure VLAN is used to carry VXLAN encapsulated traffic to the VXLAN fabric.  The only member interfaces of this VLAN will be VTEP’s logical connection to the bridge itself and the uplink to the VXLAN fabric.  This interface is the ‘uplink’ described above, while the logical 802.1Q trunk is the downlink.

image

Summary

VXLAN is a network overlay technology design for data center networks.  It provides massively increased scalability over VLAN IDs alone while allowing for L2 adjacency over L3 networks.  The VXLAN VTEP can be implemented in both virtual and physical switches allowing the virtual network to map to physical resources and network services.  VXLAN currently has both wide support and hardware adoption in switching ASICS and hardware NICs, as well as virtualization software.

GD Star Rating
loading...
VXLAN Deep Dive, 4.7 out of 5 based on 19 ratings

Comments

  1. One small thing, and I’m being picky. A frame that is using 802.1Q is ‘ Ethernet tagged’ not ‘encapsulated’ cf. VXLAN is a MAC in IP encapsulation. Tagging uses much less CPU than encapsulation and inserts less latency.

    Also, VXLAN is just one of many encapsulation protocols and primarily intended for use in Data Centers. Others include NVO3, NVGRE, STT and others. They all operate more or less the same as VXLAN.

    GD Star Rating
    loading...
  2. There is also a new one called DOVE from IBM that uses the VXLAN header but isn’t multi cast based.

    The NVo3 group is trying to come up with a Standard version that removes the need for customers to have to choose between VXLAN, STT, NVGRE, DOVE, .

    There is a good chance that NVo3 will support more than one encapsulation method, and that one of those methods will use the VXLAN form so as to not hurt customers that buy hardware now that supports VXLAN.

    (Disclaimer: I represent Brocade in the NVo3 WG)

    GD Star Rating
    loading...
  3. VXLAN incurs a potentially large TAX (performance cost) in vSphere 5.1 environments:
    http://www.vmware.com/files/pdf/techpaper/VMware-vSphere-VXLAN-Perf.pdf

    GD Star Rating
    loading...
    • that hardly looks to be a large TAX (performance cost).

      “or a small message size, throughput scaled linearly as we added more virtual
      machines. For large message sizes, we were able to achieve line rate for all configurations. There is some CPU
      overhead due to the use of VXLAN. This overhead varies based on the configuration and is due to lack of
      hardware offloads for encapsulated packets. “

      GD Star Rating
      loading...
  4. For example, you may write several articles for Associated Content that are related to each
    other. This could, however, estrange the relationship
    between the creators and the strategists or SEO. With online businesses,
    it’s very important to have good uptime, which means
    no downtime.

    GD Star Rating
    loading...
  5. Finding the right house for your family means finding the best nest will help you to take care of your kids better.
    Some companies will make an adjustable crankset for shorter or taller people too.
    Getting the services of Long Distance Moving Company make your moving easy and very much less taxing.

    GD Star Rating
    loading...
  6. Woah! I’m really enjoying the template/theme of this blog.
    It’s simple, yet effective. A lot of times it’s hard to get that “perfect balance” between usability and visual appearance.

    I must say that you’ve done a amazing job with this.
    Also, the blog loads very fast for me on Chrome. Exceptional
    Blog!

    GD Star Rating
    loading...
  7. Please let me know if you’re looking for a writer for your blog.
    You have some really good articles and I think I would be
    a good asset. If you ever want to take some of the load off, I’d
    really like to write some content for your blog in exchange for a link back to mine.
    Please shoot me an e-mail if interested. Regards!

    GD Star Rating
    loading...
  8. I am really loving the theme/design of your site.
    Do you ever run into any internet browser compatibility issues?
    A small number of my blog audience have complained
    about my blog not operating correctly in Explorer but looks great in Firefox.

    Do you have any advice to help fix this issue?

    GD Star Rating
    loading...
  9. Excellent post. Keep writing such kind of information on your page.
    Im really impressed by it.
    Hi there, You’ve performed a great job. I’ll definitely digg it and in my view suggest to my friends.
    I am sure they’ll be benefited from this website.

    GD Star Rating
    loading...

Trackbacks

  1. [...] Deep Dive – Part II November 13, 2012 By Joe Onisick Leave a Comment In part one of this post I covered the basic theory of operations and functionality of VXLAN [...]

  2. [...] November 19, 2012 By Joe Onisick Leave a Comment The most viable competitor to VXLAN is NVGRE which was proposed by Microsoft, Intel, HP and Dell.  It is another encapsulation [...]

Speak Your Mind

*