What Network Virtualization Isn’t

Brad Hedlund recently posted an excellent blog on Network Virtualization.  Network Virtualization is the label used by Brad’s employer VMware/Nicira for their implementation of SDN.  Brad’s article does a great job of outlining the need for changes in networking in order to support current and evolving application deployment models.  He also correctly points out that networking has lagged behind the rest of the data center as technical and operational advancements have been made. 

Network configuration today is laughably archaic when compared to storage, compute and even facilities.  It is still the domain of CLI wizards hacking away on keyboards to configure individual devices.  VMware brought advancements like resource utilization based automatic workload migration to the compute environment.  In order to support this behavior on the network an admin must ensure the appropriate configuration is manually defined on each port that workload may access and every port connecting the two.  This is time consuming, costly and error prone. Brad is right, this is broken.

Brad also correctly points out that network speeds, feeds and packet delivery are adequately evolving and that the friction lies in configuration, policy and service delivery.  These essential network components are still far too static to keep pace with application deployments.  The network needs to evolve, and rapidly, in order to catch up with the rest of the data center.

Brad and I do not differ on the problem(s), or better stated: we do not differ on the drivers for change.  We do however differ on the solution.  Let me preface in advance that Brad and I both work for HW/SW vendors with differing solutions to the problem and differing visions of the future.  Feel free to write the rest of this off as mindless dribble or vendor Kool Aid, I ain’t gonna hate you for it.

Brad makes the case that Network Virtualization is equivalent to server virtualization, and from this simple assumption he poses it as the solution to current network problems.

Let’s start at the top: don’t be fooled by emphatic statements such as Brad’s stating that network virtualization is analogous to server virtualization.  It is not, this is an apples and oranges discussion.  Network virtualization being the orange where you must peel the rind to get to the fruit.  Don’t take my word for it, one of Brad’s colleagues, Scott Lowe, a man much smarter then I says it best:

image

The issue is that these two concepts are implemented in a very different fashion.  Where server virtualization provides full visibility and partitioning of the underlying hardware, network virtualization simply provides a packet encapsulation technique for frames on the wire.  The diagram below better illustrates our two fruits: apples and oranges.

image

As the diagram illustrates we are not working with equivalent approaches.  Network virtualization would require partitioning of switch CPU, TCAM, ASIC forwarding, bandwidth etc. to be a true apples-to-apples comparison.  Instead it provides a simple wrapper to encapsulate traffic on the underlying Layer 3 infrastructure.  These are two very different virtualization approaches.

Brad makes his next giant leap in the “What is the Network section.”  Here he makes the assumption that the network consists of only virtual workloads “The “network” we want to virtualize is the complete L2-L7 services viewed by the virtual machines” and the rest of his blog focuses there.  This is fine for those data center environments that are 100% virtualized including servers, services and WAN connectivity and use server virtualization for all of those purposes.  Those environments must also lack PaaS and SaaS systems that aren’t built on virtual servers as those are also non-applicable to the remaining discussion.  So anyone in those environments described will benefit from the discussion, anyone <crickets>.

So Brad and, presumably VMware/Nicira (since network virtualization is their term), define the goal as taking “all of the network services, features, and configuration necessary to provision the application’s virtual network (VLANs, VRFs, Firewall rules, Load Balancer pools & VIPs, IPAM, Routing, isolation, multi-tenancy, etc.) – take all of those features, decouple it from the physical network, and move it into a virtualization software layer for the express purpose of automation.”  So if your looking to build 100% virtualized server environments with no plans to advance up the stack into PaaS, etc. it seems you have found your Huckleberry.

What we really need is not a virtualized network overlay running on top of an L3 infrastructure with no communication or correlation between the two.  What we really need is something another guy much smarter than me (Greg Ferro) described:

image

Abstraction, independence and isolation, that’s the key to moving the network forward.  This is not provided by network virtualization.  Network virtualization is a coat of paint on the existing building.  Further more that coat of paint is applied without stripping, priming, or removing that floral wall paper your grandmother loved.  The diagram below is how I think of it.

Network Virtualization

With a network virtualization solution you’re placing your applications on a house of cards built on a non-isolated infrastructure of legacy design and thinking.  Without modifying the underlying infrastructure, network virtualization solutions are only as good as the original foundation.  Of course you could replace the data center network with a non-blocking fabric and apply QoS consistently across that underlying fabric (most likely manually) as Brad Hedlund suggests below.

image

If this is the route you take, to rebuild the foundation before applying network virtualization paint, is network virtualization still the color you want?  If a refresh and reconfigure is required anyway, is this the best method for doing so? 

The network has become complex and unmanageable due to things far older than VMware and server virtualization.  We’ve clung to device centric CLI configuration and the realm of keyboard wizards.  Furthermore we’ve bastardized originally abstracted network constructs such as VLAN, VRF, addressing, routing, and security tying them together and creating a Frankenstein of a data center network.  Are we surprised the villagers are coming with torches and pitch forks?

So overall I agree with Brad, the network needs to be fixed.  We just differ on the solution, I’d like to see more than a coat of paint.  Put lipstick on a pig and all you get is a pretty pig.

lipstick pig

CloudStack Graduates to Top-Level Apache Project

The Apache Software Foundation announced in late March that CloudStack is now a top-level project. This is a promotion from CloudStack's incubator status, where it had lived after being released as open source by Citrix.

This promotion provides additional encouragement to companies and developers looking to contribute to the project, because it validates the CloudStack community and demonstrates ongoing support under the Apache Software Foundation. To read more visit the full article.

OpenStack Video Cage Match With Colin McNamara

This post is a little late, mainly because I’m both lazy and distracted.  That being said I hope you’ll enjoy this video of Colin McNamara (@colinmcnamara) and I debating the merits of OpenStack.  For more Engineer’s unplugged goodness from Amy Lewis (@commsninja) visit: http://blogs.cisco.com/datacenter/.

WWT GeekDay 2013

I had the privilege this week to attend the opening keynote and SDN panel of WWT’s Geek Day.  The SDN panel was made up of heavy hitters from Cisco, VMware, HP, and Embrane.  They each presented their vision and solutions for SDN, then teamed up for questions.  The session was very good and was recorded.  I highly recommend watching them at the links below.  For more info on attending or sponsoring this great event see www.geekday.com

Cisco’s Balaji Sivasubramanian:

 

VMware’s Brad Hedlund:

HP’s Mauicio Sanchez:

Embrane’s Tom Nosella:

Taking a Good Hard Look at SDN

SDN is sitting at the peak of it’s hype cycle (at least I hope it’s the peak.)  Every vendor has a definition and a plan.  Most of those definitions and plans focus around protecting their existing offerings and morphing those into some type of SDN vision.  Products and entire companies have changed their branding from whatever they were to SDN and the markets flooded with SDN solutions that solve very different problems.  This post will take a deep dive into the concepts around SDN and the considerations of a complete solution.  As always with my posts this is focused on the data center network, because I can barely spell WAN, have never spent time on a campus and have no idea what magic it is that service providers do.

The first question anyone considering SDN solutions needs to ask is: What problem(s) am I trying to solve.  Start with the business drivers for the decision.  There are many that SDN solutions look to solve, a few examples are:

That leaves a lot of areas with room for improvement in order to accomplish those tasks.  That’s one of the reasons the definition is so loose and applied to such disparate technologies.  In order to keep the definition generic enough to encompass a complete solution there are three major characteristics I prefer for defining an SDN architecture:

The Complete Picture:

In looking for a complete solution for Software Defined data center network it’s important to assess all aspects required to deliver cohesive network services and packet delivery:

Depending on your overall goals you may not have requirements in each of these areas but you’ll want to analyze that carefully based on growth expectations.  Don’t run your data center like congress kicking the can (problem) down the road.  The graphic below shows the various layers to be considered when looking at SDN solutions.

image

Current Options:

The current options for SDN typically provide solutions for one or more of these issues but not all.  The chart below takes a look at some popular options.

 

 

VLAN Scale

L4-7

Bare Metal Support

Physical Network Node MGMT

KVM

VMware

Xen

HyperV

L3

Flow MGMT

Nicira/VMware X 3rd Party *   X * X   3rd Party X
Overlays X       X X X X    
OpenFlow     X   X X X X X X
Midokura X X     X   X   X  

X = Support

* = Future Support
This chart is not intended to be all encompassing or to compare all features of equal products (obviously an overlay doesn’t compete with a Nicira or Midokura solution, and each of those rely on overlays of some type.)  Instead it’s intended to show that the various solutions lumped into SDN provide solutions for different areas of the data center network.  One or more tools may be necessary to deploy a full SDN architecture and even then there may be gaps in areas like bare metal support, integration of standalone network appliances and provisioning/monitoring/troubleshooting of physical switch nodes (yes that all still matters.)
API Model:
Another model lumped into SDN is northbound APIs for network devices.  Several networking vendors are in various stages of support for this model.  This model does provide programmability but I would argue against it’s scale.  Using this model requires top down management systems that understand each device, its capabilities and its API.  To scale this type of management system and program network flows this way is not easy and will be error prone.  Additionally this model does not provide any additional functionality, visibility or holistic programmability, simply a better way to configure individual devices. That being said managing via APIs is light years ahead of screen scrapes and CLI scripting.
Hardware Matters:
Let me preface with what I’m not saying: I’m not saying that hardware will/won’t be commoditized, and I’m not saying that custom silicon or merchant silicon is better or worse.
I am saying that the network hardware you choose will matter.  Table sizes, buffer space, TCAM size will all factor in, and depending on your deployment model will be a major factor.  The hardware will also need to provide maximum available bandwidth and efficient ECMP load-balancing for network throughput.  This load-balancing can be greatly affected by the overlay method chosen based on available header information for hashing algorithms.  Additionally your hardware must support the options of the SDN model you choose.  For example in a Nicira/VMware deployment you’ll have future support for management of switches running OVS, you may want these to tie in physical servers, etc.  The same would apply if you choose OpenFlow.  You’ll need switch hardware that provides OpenFlow support, additionally it will need to support your deployment model hybrid or pure OpenFlow.
The hardware also matters in configuration, management, and troubleshooting.  While there is a lot of talk of “We just need any IP connectivity” that IP network still has to be configured and managed.  Layer 2/3 constructs must be put in place, ports must be configured.  This hardware will also have to be monitored, and troubleshot when things fail.  This will be more difficult in cases where the overlay is unknown to the L3 infrastructure at which point two separate independent networks will be involved: physical and logical.
Management Model:

There are several management models to choose from and two examples in the choices I compared above.  OpenFlow uses a centralized top down approach with the controller pushing flows to all network elements and handling policy for new flows forwarded from those devices.  The Nicira/VMware solution uses the same model as OpenFlow.  Midokura on the other hand takes a play from distributed systems and pushes intelligence to the edges in that fashion.  Each model offers various pros/cons and will play a major role in the scale and resiliency of your SDN deployment.

Northbound API:

The Northbound API is different than the device APIs mentioned below.  This API opens the management of your SDN solution as whole up to higher level systems.  Chances are you’re planning to plug your infrastructure into an automation/orchestration solution or cloud platform.  In order to do this you’ll want a robust northbound API for your infrastructure components, in this case your SDN architecture.  If you have these systems in place, or have already picked your horse you’ll want to ensure compatibility with the SDN architectures you consider.  Not all APIs are created equal, and they are far from standardized so you’ll want to know exactly what you’re getting from a functionality perspective and ensure the claims match your upper layer systems needs.

Additional Considerations:

There are several other considerations which will effect both the options chosen and the architecture used some of those:

Summary:

SDN should be putting the application back in focus and providing tools for more robust and rapid application deployment/change.  In order to effectively do this an SDN architecture should provide functionality for the full life of the packet on the data center network.  The architecture should also provide tools for the scale you forecast as you grow.  Because of the nature of the ecosystem you may find more robust deployment options the more standardized your environment is (I’ve written about standardization several times in the past for example:http://www.networkcomputing.com/private-cloud-tech-center/private-cloud-success-factor-standardiza/231500532 .)  You can see examples of this in the hypervisor support shown in the chart above.

While solutions exist for specific business use cases the market is far from mature.  Products will evolve and as lessons are learned and roadmaps executed we’ll see more robust solutions emerge.  In the interim choose technologies that meet your specific business drivers and deploy them in environments with the largest chance of success, low hanging fruit.  It’s prudent to move into network virtualization in the same fashion you moved into server virtualization, with a staged approach.

Why We Need Network Abstraction

The move to highly virtualized data centers and cloud models is straining the network. While traditional data center networks were not designed to support the dynamic nature of today's workloads, the fact is, the emergence of highly virtualized environments is merely exposing issues that have always existed within network constructs. VLANs, VRFs, subnets, routing, security, etc. have been stretched well beyond their original intent. The way these constructs are currently used limits scale, application expansion, contraction and mobility.  To read the full article visit: http://www.networkcomputing.com/next-gen-network-tech-center/why-we-need-network-abstraction/240142588

Data Center Overlays 101

I've been playing around with Show Me (www.showme.com) as a tool to add some white boarding to the blog.  Here’s my first crack at it covering Data Center Network overlays.

Stateless Transport Tunneling (STT)

STT is another tunneling protocol along the lines of the VXLAN and NVGRE proposals.  As with both of those the intent of STT is to provide a network overlay, or virtual network running on top of a physical network.  STT was proposed by Nicira and is therefore not surprisingly written from a software centric view rather than other proposals written from a network centric view.  The main advantage of the STT proposal is it’s ability to be implemented in a software switch while still benefitting from NIC hardware acceleration.  The other advantage of STT is its use of a 64 bit network ID rather than the 32 bit IDs used by NVGRE and VXLAN.

The hardware offload STT grants relieves the server CPU of a significant workload in high bandwidth systems (10G+.)  This separates it from it’s peers that use an IP encapsulation in the soft switch which negate the NIC’s LSO and LRO functions.   The way STT goes about this is by having the software switch inserts header information into the packet to make it look like a TCP packet, as well as the required network virtualization features.  This allows the guest OS to send frames up to 64k to the hypervisor which are encapsulated and sent to the NIC for segmentation.  While this does allow for the HW offload to be utilized it causes several network issues due to it’s use of valid TCP headers it causes issues for many network appliances or “middle boxes.” 

STT is not expected to be ratified and is considered by some to have been proposed for informational purposes, rather than with the end goal of a ratified standard.  With its misuse of a valid TCP header it would be hard pressed for ratification.  STT does bring up the interesting issue of hardware offload.  The IP tunneling protocols mentioned above create extra overhead on host CPUs due to their inability to benefit from NIC acceleration techniques.  VXLAN and NVGRE are intended to be implemented in hardware to solve this problem.  Both VXLAN and NVGRE use a 32 bit network ID because they are intended to be implemented in hardware, this space provides for 16 million tenants.  Hardware implementation is coming quickly in the case of VXLAN with vendors announcing VXLAN capable switches and NICs. 

VXLAN Deep Dive

I’ve been spending my free time digging into network virtualization and network overlays.  This is part 1 of a 2 part series, part 2 can be found here: http://www.definethecloud.net/vxlan-deep-divepart-2.  By far the most popular virtualization technique in the data center is VXLAN.  This has as much to do with Cisco and VMware backing the technology as the tech itself.  That being said VXLAN is targeted specifically at the data center and is one of many similar solutions such as: NVGRE and STT.)  VXLAN’s goal is allowing dynamic large scale isolated virtual L2 networks to be created for virtualized and multi-tenant environments.  It does this by encapsulating frames in VXLAN packets.  The standard for VXLAN is under the scope of the IETF NVO3 working group.

 

VxLAN Frame

The VXLAN encapsulation method is IP based and provides for a virtual L2 network.  With VXLAN the full Ethernet Frame (with the exception of the Frame Check Sequence: FCS) is carried as the payload of a UDP packet.  VXLAN utilizes a 24-bit VXLAN header, shown in the diagram, to identify virtual networks.  This header provides for up to 16 million virtual L2 networks.

Frame encapsulation is done by an entity known as a VXLAN Tunnel Endpoint (VTEP.)  A VTEP has two logical interfaces: an uplink and a downlink.  The uplink is responsible for receiving VXLAN frames and acts as a tunnel endpoint with an IP address used for routing VXLAN encapsulated frames.  These IP addresses are infrastructure addresses and are separate from the tenant IP addressing for the nodes using the VXLAN fabric.  VTEP functionality can be implemented in software such as a virtual switch or in the form a physical switch.

VXLAN frames are sent to the IP address assigned to the destination VTEP; this IP is placed in the Outer IP DA.  The IP of the VTEP sending the frame resides in the Outer IP SA.  Packets received on the uplink are mapped from the VXLAN ID to a VLAN and the Ethernet frame payload is sent as an 802.1Q Ethernet frame on the downlink.  During this process the inner MAC SA and VXLAN ID is learned in a local table.  Packets received on the downlink are mapped to a VXLAN ID using the VLAN of the frame.  A lookup is then performed within the VTEP L2 table using the VXLAN ID and destination MAC; this lookup provides the IP address of the destination VTEP.  The frame is then encapsulated and sent out the uplink interface.

image

Using the diagram above for reference a frame entering the downlink on VLAN 100 with a destination MAC of 11:11:11:11:11:11 will be encapsulated in a VXLAN packet with an outer destination address of 10.1.1.1.  The outer source address will be the IP of this VTEP (not shown) and the VXLAN ID will be 1001.

In a traditional L2 switch a behavior known as flood and learn is used for unknown destinations (i.e. a MAC not stored in the MAC table.  This means that if there is a miss when looking up the MAC the frame is flooded out all ports except the one on which it was received.  When a response is sent the MAC is then learned and written to the table.  The next frame for the same MAC will not incur a miss because the table will reflect the port it exists on.  VXLAN preserves this behavior over an IP network using IP multicast groups.

Each VXLAN ID has an assigned IP multicast group to use for traffic flooding (the same multicast group can be shared across VXLAN IDs.)  When a frame is received on the downlink bound for an unknown destination it is encapsulated using the IP of the assigned multicast group as the Outer DA; it’s then sent out the uplink.  Any VTEP with nodes on that VXLAN ID will have joined the multicast group and therefore receive the frame.  This maintains the traditional Ethernet flood and learn behavior.

VTEPs are designed to be implemented as a logical device on an L2 switch.  The L2 switch connects to the VTEP via a logical 802.1Q VLAN trunk.  This trunk contains an VXLAN infrastructure VLAN in addition to the production VLANs.  The infrastructure VLAN is used to carry VXLAN encapsulated traffic to the VXLAN fabric.  The only member interfaces of this VLAN will be VTEP’s logical connection to the bridge itself and the uplink to the VXLAN fabric.  This interface is the ‘uplink’ described above, while the logical 802.1Q trunk is the downlink.

image

Summary

VXLAN is a network overlay technology design for data center networks.  It provides massively increased scalability over VLAN IDs alone while allowing for L2 adjacency over L3 networks.  The VXLAN VTEP can be implemented in both virtual and physical switches allowing the virtual network to map to physical resources and network services.  VXLAN currently has both wide support and hardware adoption in switching ASICS and hardware NICs, as well as virtualization software.

Much Ado About Something: Brocade’s Tech Day

Yesterday I had the privilege of attending Brocade’s Tech Day for Analysts and Press.  Brocade announced the new VDX 8770, discussed some VMware announcements, as well as discussed strategy, vision and direction.  I’m going to dig in to a few of the topics that interested me, this is no way a complete recap.

First in regards to the event itself.  My kudos to the staff that put the event together it was excellent from both a pre-event coordination and event staff perspective.  The Brocade corporate campus is beautiful and the EBC building was extremely well suited to such an event.  The sessions went on smoothly, the food was excellent and overall it was a great experience.  I also want to thank Lisa Caywood (@thereallisac) for pointing out that my tweets during the event were more inflammatory then productive and outside the lines of ‘guest etiquette.’  She’s definitely correct and hopefully I can clear up some of my skepticism here in a format left open for debate, and avoid the same mistake in the future.  That being said I had thought I was quite clear going in on who I was and how I write.  To clear up any future confusion from anyone:  if you’re not interested in my unfiltered, typically cynical, honest opinion don’t invite me, I won’t take offense.  Even if you’re a vendor with products I like I’ve probably got a box full of cynicism for your other product lines.

During the opening sessions I observed several things that struck me negatively:

On the positive side Brocade has some vision that’s quite interesting as well as some areas where they are leading by filling gaps in industry offerings.

On the financial side Brocade has been looking good and climbed over $6.00 a share.  There are plenty of conversations stating some of this may be due to upcoming shifts at the CEO level.  They’ve reported two great quarters and are applying some new focus towards federal government and other areas lacking in recent past. I didn’t dig further into this discussion.

During lunch I was introduced to one of the most interesting Brocade offerings I’d never heard of: ‘Brocade Network Subscription”: http://www.brocade.com/company/how-to-buy/capital-solutions/index.page.  Basically you can lease your on-prem network from Brocade Capitol.  This is a great idea for customers looking to shift CapEx to OpEx which can be extremely useful.  I also received a great explanation for the value of a fabric underneath an SDN network from Jason Nolet (VP of Data Center Networking Group.)  Jason’s position (summarized) is that implementing SDN adds a network management layer, rather than removing one.  With that in mind the more complexity we remove from the physical network the better off we are.  What we’ll want for our SDN networks is fast, plug-and-play functionality with max usable links and minimal management.  Brocade VCS fabric fits this nicely.  While I agree with that completely I ‘d also say it’s not the only way to skin that particular cat.  More to come on that.

For the last few years I’ve looked at Brocade as a company lacking innovation and direction.  They clung furiously to FC while the market began shifting to Ethernet, ignored cloud for quite a while, etc.  Meanwhile they burned down deals to purchase them and ended up where they’ve been.  The overall messaging, while nothing new, did have undertones of change as a whole and new direction.  That’s refreshing to hear.  Brocade is embracing virtualization and cloud architectures without tying their cart to a single hypervisor horse.  They are positioning well for SDN and the network market shifts.  Most impressively they are identifying gaps in the spaces they operate and executing on them both from a business and technology perspective.  Examples of this are Brocade Network Subscription and the VXLAN gateway functionality respectively.

Things are looking up and there is definitely something good happening at Brocade.  That being said they aren’t out of the woods yet.  For them, as a company, purchase is far fetched as the vendors that would buy them already have networking plays and would lose half of Brocade’s value by burning OEM relationships with the purchase.  The only real option from a sale perspective is for investors looking to carve them up and sell off pieces individually.  A scenario like this wouldn’t bode well for customers.  Brocade has some work to do but they’ve got a solid set of products and great direction.  We’ll see how it pans out.  Execution is paramount for them at this point.

Final Note:  This blog was intended to stop there but this morning I received an angry accusatory email from Brocade’s head of corporate communications who was unhappy with my tweets.  I thought about posting the email in full, but have decided against it for the sake of professionalism.  Overall his email was an attack based on my tweets.  As stated my tweets were not professional, but this type of email from someone in charge of corporate communications is well over the top in response.  I forwarded the email to several analyst and blogger colleagues, a handful of whom had similar issues with this individual.  One common theme in social media is that lashing out at bad press never does any good, a senior director in this position should know such, but instead continues to slander and attack.  His team and colleagues seem to understand social media use as they’ve engaged in healthy debate with me in regards to my tweets, it’s a shame they are not lead from the front.