Access Layer Network Virtualization: VN-Tag and VEPA

One of the highlights of my trip to lovely San Francisco for VMworld was getting to join Scott Lowe and Brad Hedlund for an off the cuff whiteboard session.  I use the term join loosely because I contributed nothing other than a set of ears.  We discussed a few things, all revolving around virtualization (imagine that at VMworld.)  One of the things we discussed was virtual switching and Scott mentioned a total lack of good documentation on VEPA, VN-tag and the differences between the two.  I’ve also found this to be true, the documentation that is readily available is:

  • Marketing fluff
  • Vendor FUD
  • Standards body documents which might as well be written in a Klingon/Hieroglyphics slang manifestation

This blog is my attempt to demystify VEPA and VN-tag and place them both alongside their applicable standards, and by that I mean contribute to the extensive garbage info revolving around them both.  Before we get into them both we’ll need to understand some history and the problems they are trying to solve.

First let’s get physical.  Looking at a traditional physical access layer we have two traditional options for LAN connectivity: Top-of-Rack (ToR) and End-of-Row (EoR) switching topologies.  Both have advantages and disadvantages.


EoR topologies rely on larger switches placed on the end of each row for server connectivity.


  • Less Management points
  • Smaller Spanning-Tree Protocol (STP) domain
  • Less equipment to purchase, power and cool


  • More above/below rack cable runs
  • More difficult cable modification, troubleshooting and replacement
  • More expensive cabling


ToR utilizes a switch at the top of each rack (or close to it.)


  • Less cabling distance/complexity
  • Lower cabling costs
  • Faster move/add/change for server connectivity


  • Larger STP domain
  • More management points
  • More switches to purchase, power and cool

Now let’s virtualize.  In a virtual server environment the most common way to provide Virtual Machine (VM) switching connectivity is a Virtual Ethernet Bridge (VEB) in VMware we call this a vSwitch.  A VEB is basically software that acts similar to a Layer 2 hardware switch providing inbound/outbound and inter-VM communication.  A VEB works well to aggregate multiple VMs traffic across a set of links as well as provide frame delivery between VMs based on MAC address.  Where a VEB is lacking is network management, monitoring and security.  Typically a VEB is invisible and not configurable from the network teams perspective.  Additionally any traffic handled by the VEB internally cannot be monitored or secured by the network team.


  • Local switching within a host (physical server)
    • Less network traffic
    • Possibly faster switching speeds
  • Common well understood deployment
  • Implemented in software within the hypervisor with no external hardware requirements


  • Typically configured and managed within the virtualization tools by the server team
  • Lacks monitoring and security tools commonly used within the physical access layer
  • Creates a separate management/policy model for VMs and physical servers

These are the two issues that VEPA and VN-tag look to address in some way.  Now let’s look at the two individually and what they try and solve.

Virtual Ethernet Port Aggregator (VEPA):

VEPA is standard being lead by HP for providing consistent network control and monitoring for Virtual Machines (of any type.)  VEPA has been used by the IEEE as the basis for 802.1Qbg ‘Edge Virtual Bridging.’  VEPA comes in two major forms: a standard mode which requires minor software updates to the VEB functionality as well as upstream switch firmware updates, and a multi-channel mode which will require additional intelligence on the upstream switch.

Standard Mode:

The beauty of VEPA in it’s standard mode is in it’s simplicity, if you’ve worked with me you know I hate complex designs and systems, they just lead to problems.  In the standard mode the software upgrade to the VEB in the hypervisor simply forces each VM frame out to the external switch regardless of destination.  This causes no change for destination MAC addresses external to the host, but for destinations within the host (another VM in the same VLAN) it forces that traffic to the upstream switch which forwards it back instead of handling it internally, called a hairpin turn.)  It’s this hairpin turn that causes the requirement for the upstream switch to have updated firmware, typical STP behavior prevents a switch from forwarding a frame back down the port it was received on (like the saying goes, don’t egress where you ingress.)  The firmware update allows the negotiation between the physical host and the upstream switch of a VEPA port which then allows this hairpin turn.  Let’s step through some diagrams to visualize this.

image  image

Again the beauty of this VEPA mode is in its simplicity.  VEPA simply forces VM traffic to be handled by an external switch.  This allows each VM frame flow to be monitored managed and secured with all of the tools available to the physical switch.  This does not provide any type of individual tunnel for the VM, or a configurable switchport but does allow for things like flow statistic gathering, ACL enforcement, etc.  Basically we’re just pushing the MAC forwarding decision to the physical switch and allowing that switch to perform whatever functions it has available on each transaction.  The drawback here is that we are now performing one ingress and egress for each frame that was previously handled internally.  This means that there are bandwidth and latency considerations to be made.  Functions like Single Root I/O Virtualization (SR/IOV) and Direct Path I/O can alleviate some of the latency issues when implementing this.  Like any technology there are typically trade offs that must be weighed.  In this case the added control and functionality should outweigh the bandwidth and latency additions.

Multi-Channel VEPA:

Multi-Channel VEPA is an optional enhancement to VEPA that also comes with additional requirements.  Multi-Channel VEPA allows a single Ethernet connection (switchport/NIC port) to be divided into multiple independent channels or tunnels.  Each channel or tunnel acts as an unique connection to the network.  Within the virtual host these channels or tunnels can be assigned to a VM, a VEB, or to a VEB operating with standard VEPA.  In order to achieve this goal Multi-Channel VEPA utilizes a tagging mechanism commonly known as Q-in-Q (defined in 802.1ad) which uses a service tag ‘S-Tag’ in addition to the standard 802.1q VLAN tag.  This provides the tunneling within a single pipe without effecting the 802.1q VLAN.  This method requires Q-in-Q capability within both the NICs and upstream switches which may require hardware changes.

image VN-Tag:

The VN-Tag standard was proposed by Cisco and others as a potential solution to both of the problems discussed above: network awareness and control of VMs, and access layer extension without extending management and STP domains.  VN-Tag is the basis of 802.1qbh ‘Bridge Port Extension.’  Using VN-Tag an additional header is added into the Ethernet frame which allows individual identification for virtual interfaces (VIF.)


The tag contents perform the following functions:



Identifies the VN tag


Direction, 1 indicates that the frame is traveling from the bridge to the interface virtualizer (IV.)


Pointer, 1 indicates that a vif_list_id is included in the tag.


A list of downlink ports to which this frame is to be forwarded (replicated). (multicast/broadcast operation)


Destination vif_id of the port to which this frame is to be forwarded.


Looped, 1 indicates that this is a multicast frame that was forwarded out the bridge port on which it was received. In this case, the IV must check the Svif_id and filter the frame from the corresponding port.




Version of the tag


The vif_id of the source of the frame

The most important components of the tag are the source and destination VIF IDs which allow a VN-Tag aware device to identify multiple individual virtual interfaces on a single physical port.

VN-Tag can be used to uniquely identify and provide frame forwarding for any type of virtual interface (VIF.)  A VIF is any individual interface that should be treated independently on the network but shares a physical port with other interfaces.  Using a VN-Tag capable NIC or software driver these interfaces could potentially be individual virtual servers.  These interfaces can also be virtualized interfaces on an I/O card (i.e. 10 virtual 10G ports on a single 10G NIC), or a switch/bridge extension device that aggregates multiple physical interfaces onto a set of uplinks and relies on an upstream VN-tag aware device for management and switching.


Because of VN-tags versatility it’s possible to utilize it for both bridge extension and virtual networking awareness.  It also has the advantage of allowing for individual configuration of each virtual interface as if it were a physical port.  The disadvantage of VN-Tag is that because it utilizes additions to the Ethernet frame the hardware itself must typically be modified to work with it.  VN-tag aware switch devices are still fully compatible with traditional Ethernet switching devices because the VN-tag is only used within the local system.  For instance in the diagram above VN-tags would be used between the VN-tag aware switch at the top of the diagram to the VIF but the VN-tag aware switch could be attached to any standard Ethernet switch.  VN-tags would be written on ingress to the VN-tag aware switch for frames destined for a VIF, and VN-tags would be stripped on egress for frames destined for the traditional network. 

Where does that leave us?

We are still very early in the standards process for both 802.1qbh and 802.1Qbg, and things are subject to change.  From what it looks like right now the standards body will be utilizing VEPA as the basis for providing physical type network controls to virtual machines, and VN-tag to provide bridge extension.  Because of the way in which each is handled they will be compatible with one another, meaning a VN-tag based bridge extender would be able to support VEPA aware hypervisor switches.

Equally as important is what this means for today and today’s hardware.  There is plenty of Fear Uncertainty and Doubt (FUD) material out there intended to prevent product purchase because the standards process isn’t completed.  The question becomes what’s true and what isn’t, let’s take care of the answers FAQ style:

Will I need new hardware to utilize VEPA for VM networking?

No, for standard VEPA mode only a software change will be required on the switch and within the Hypervisor.  For Multi-Channel VEPA you may require new hardware as it utilizes Q-in-Q tagging which is not typically an access layer switch feature.

Will I need new hardware to utilize VN-Tag for bridge extension?

Yes, VN-tag bridge extension will typically be implemented in hardware so you will require a VN-tag aware switch as well as VN-tag based port extenders. 

Will hardware I buy today support the standards?

That question really depends on how much change occurs with the standards before finalization and which tool your looking to use:

  • Standard VEPA – Yes
  • Multi-Channel VEPA – Possibly (if Q-in-Q is supported)
  • VN-Tag – possibly

Are there products available today that use VEPA or VN-Tag?

Yes Cisco has several products that utilize VN-Tag: Virtual interface Card (VIC), Nexus 2000, and the UCS I/O Module (IOM.)  Additionally HP’s FlexConnect technology is the basis for multi-channel VEPA.


VEPA and VN-tag both look to address common access layer network concerns and both are well on their way to standardization.  VEPA looks to be the chosen method for VM aware networking and VN-Tag for bridge extension.  Devices purchased today that rely on pre-standards versions of either protocol should maintain compatibility with the standards as they progress but it’s not guaranteed.    That being said standards are not required for operation and effectiveness, and most start as unique features which are then submitted to a standards body.

GD Star Rating

HP Flex-10, Cisco VIC, and Nexus 1000v

When discussing the underlying technologies for cloud computing topologies virtualization is typically a key building block.  Virtualization can be applied to any portion of the data center architecture from load-balancers to routers, and from servers to storage.  Server virtualization is one of the most widely adopted virtualization technologies, and provides a great deal of benefits to the server architecture. 

One of the most common challenges with server virtualization is the networking.  Virtualized servers typically consist of networks of virtual machines that are configured by the server team with little to no management/monitoring possible from the network/security teams.  This causes inconsistent policy enforcement between physical and virtual servers as well as limited network functionality for virtual machines.

Virtual Networks


The separate network management models for virtual servers and physical servers presents challenges to: policy enforcement, compliance, and security, as well as adds complexity to the configuration and architecture of virtual server environments.  Due to this fact many vendors are designing products and solutions to help draw these networks closer together.

The following is a discussion of three products that can be used for this, HP’s Flex-10 adapters, Cisco’s Nexus 1000v and Cisco’s Virtual interface Card (VIC.) 

This is not a pro/con or discussion of which is better, just an overview of the technology and how it relates to VMware.

HP Flex-10 for Virtual Connect:

Using HP’s Virtual Connect switching modules for C-Class blades and either Flex-10 adapters or Lan-On-Motherboard (LOM) administrators can ‘partition the bandwidth of a single 10Gb pipeline into multiple “FlexNICs.” In addition, customers can regulate the bandwidth for each partition by setting it to a user-defined portion of the total 10Gb connection. Speed can be set from 100 Megabits per second to 10 Gigabits per second in 100 Megabit increments.’ (

This allows a single 10GE uplink to be presented to any operating system as 4 physical Network Interface Cards (NIC.)



In order to perform this interface virtualization FlexConnect uses internal VLAN mappings for traffic segregation within the 10GE Flex-10 port (mid-plane blade chassis connection from the Virtual Connect Flex-10 10GbE interconnect module and the Flex-10 NIC device.)  Each FlexNIC can present one or more VLANs to the installed operating system.

Some of the advantages with this architecture are:

  • A single 10GE link can be divided into 4 separate logical links each with a defined portion of the bandwidth.
  • More interfaces can be presented from fewer physical adapters which is extremely advantageous within the limited space available on blade servers.

When the installed operating system is VMware this allows for 2x10GE links to be presented to VMware as 8x separate NICs and used for different purposes such as vMotion, Fault Tolerance (FT), Service Console, VM kernel and data.

The requirements for Flex-10 as described here are:

  • HP C-Class blade chassis
  • VC Flex-10 10GE interconnect module (HP blade switches)
  • Flex-10 LOM and or Mezzanine cards

Cisco Nexus 1000v:

‘Cisco Nexus® 1000V Series Switches are virtual machine access switches that are an intelligent software switch implementation for VMware vSphere environments running the Cisco® NX-OS operating system. Operating inside the VMware ESX hypervisor, the Cisco Nexus 1000V Series supports Cisco VN-Link server virtualization technology to provide:

• Policy-based virtual machine (VM) connectivity

• Mobile VM security and network policy, and

• Non-disruptive operational model for your server virtualization, and networking teams’(

The Nexus 1000v is a Cisco software switch which is placed in the VMware environment and provides physical type network control/monitoring to VMware virtual networks.  The Nexus 1000v is comprised of two components the Virtual Supervisor Module (VSM) and Virtual Ethernet Module (VEM.)  The Nexus 1000v does not have hardware requirements and can be used with any standards compliant physical switching infrastructure.  Specifically the upstream switch should support 802.1q trunks and LACP.

Cisco Nexus 1000v


Using the Nexus 1000v Network teams have complete control over the virtual network and manage it using the same tools and policies used on the physical network.

Some advantages of the 1000v are:

  • Consistent policy enforcement for physical and virtual servers
  • vMotion aware policies migrate with the VM
  • Increased, security, visibility and control of virtual networks

The requirements for Cisco Nexus 1000v are:

  • vSphere 4.0 or higher
  • Enterprise + VMware license
  • Per physical host CPU VEM license
  • Virtual Center Server

Cisco Virtual interface Card (VIC):

The Cisco VIC provides interface virtualization similar to the Flex-10 adapter.  One 10GE port is able to be presented to an operating system as up to 128 virtual interfaces depending on the infrastructure. ‘The Cisco UCS M81KR presents up to 128 virtual interfaces to the operating system on a given blade. The virtual interfaces can be dynamically configured by Cisco UCS Manager as either Fibre Channel or Ethernet devices’ (

Fibre Channel interfaces are known as vFC and Ethernet interfaces are known as vEth, they can be used in any combination up to the architectural limits.  Currently the VIC is only available for Cisco UCS blades but will be supported on UCS rack mount servers as well by the end of 2010.  Interfaces are segregated using an internal tagging mechanism known as VN-Tag which does not use VLAN tags and operates independently of VLAN operation.

Virtual Interface Card


Each virtual interface acts as if directly connected to a physical switch port and can be configured in Access or Trunk mode using 802.1q standard trunking. These interfaces can then be used by any operating system or VMware.  For more information on their use see my post Defining VN-Link (

VIC Advantages:

  • Granular configuration of multiple Fibre Channel and Ethernet ports on one 10GE link.
  • Single point of network configuration handled by a network team rather than a server team.


  • Cisco UCS B-series blades (until C-Series support is released)
  • Cisco Fabric interconnect access layer switches/managers.


Each of these products has benefits in specific use cases and can reduce overhead and/or administration for server networks.  When combining one or more of these products you should carefully analyze the benefits of each and identify features that may be sacrificed by combining the two.  For instance using the Nexus 1000v along with FlexConnect adds a Server administered network management layer in between the physical network and virtual network.

Nexus 1000v with Flex-10


Comments and corrections are always welcome.

GD Star Rating

Post defining VN-Link

For the Cisco fans or those curious about Cisco’s VN-Link see my post on my colleagues Unified Computing Blog:

GD Star Rating