Network Management Needs New Ideas

As networks have grown, the industry has sought better ways in which to manage them at scale. Traditional network management systems are typically device-centric, particularly for network infrastructure. These systems take a top-down management approach and use a central server to push configuration into devices and to manage device state. With few exceptions, this approach provides no additional abstraction or functionally and fundamentally becomes a GUI representation of CLI configuration…

To see the full post visit:

GD Star Rating

Network Abstraction and Virtualization: Where to Start?

Network Abstraction and Virtualization: Where to Start?

With the growth of server virtualization network designs and the associated network management constructs have been stretched beyond their intended uses. This has brought about data center networks that are unmanageable and slow to adapt to change. While servers and storage can be rapidly provisioned to bring on new services the network itself has become a bottleneck of required administrative changes and inflexible constructs limiting scalability and speed of adoption.

These constraints of modern data center networks have motivated network architects to look for workarounds of which one current proposal is ‘network virtualization’ which looks to apply the benefits of server virtualization to the network. Conceptually network virtualization is the use of encapsulation techniques to create virtual overlays on existing network infrastructure. These methods use technologies such as VxLAN, STT, NVGRE, and others to wrap machine traffic in virtual IP overlays which can be transported across any Layer 3 infrastructure.

1. A primary benefit of these overlay techniques is the ability to scale beyond the limits of VLANs for network segmentation. Virtualization and multi-tenancy caused an explosion of network segments that strain traditional isolation techniques. With VLANs we are limited at 4096 segments or less depending on implementation. Other methods exist, such as placing ACLs within the Hypervisor but these also suffer limits in configuration and CPU overhead. The purpose of these techniques is creating application/tenant segmentation without security implications between segments. As the number of services and tenants grows these limits quickly become restrictive.

2. Another advantage of the network virtualization overlay is the ability to place workloads independent of physical locality and underlying topology. As long as IP connectivity is available the encapsulation handles delivery to end-point workloads. This provides greater flexibility in deployment, especially for virtualized workloads which receive encapsulation within the hypervisor switch. The operational benefit of this effect is the ability to place workloads where there is available capacity without restrictions from underlying network constructs.

Network virtualization does not come without drawbacks. The act of layering virtual networks over existing infrastructure puts an opaque barrier between the virtual workloads and the operation of the underlying infrastructure. This brings on issues with performance, quality of service (QoS) and network troubleshooting. Unlike server virtualization this limitation is not seen with compute hypervisors which are tightly coupled with the hardware maintaining visibility at both levels. The diagram below shows the relationship between network and server virtualization.


1. This lack of cross-visibility between the logical networks carrying production application traffic, and the physical network providing the packet delivery, leads to issues with application performance and system troubleshooting. With SDN techniques based on network virtualization through encapsulation, the packet delivery infrastructure is completely obfuscated by the encapsulation. This can lead to performance issues arising from lack of quality of service, altered multi-pathing ability, and others within the underlying network. This separation is shown in the diagram below.


2. Additionally these logical networks add a point of management to the network architecture. While they can hide the complexity of the underlying network for the purposes of application deployment, the network underneath still exists. The switching infrastructure must still be configured, managed and deployed as usual. All of the constructs shown above must still be architected and pushed into device configuration. Network virtualization provides perceived independence from the infrastructure but does not provide a means to manage the network as a whole.

3. The last challenge for network virtualization techniques is the ability to tie overlays back to traditional networking constructs understood by the network switches below. Switch hardware and software is designed to use VLANs which are tied to IP subnets and stitch security and services to these constructs. The overlay created by encapsulation does not alleviate these issues.

For example encapsulation techniques such as VxLAN provide far greater logical network scalability upwards of 16 million virtual networks. This logical scalability does not currently stitch into traditional switching equipment that assumes VLANs are global. Tighter cohesion will be required between physical switching infrastructure and hypervisor based access layers to provide robust services to real-world heterogeneous environments.

While overlay techniques provide separate namespace and therefore a means for overlapping IP addressing there will still be a need to architect the routing that handles this. In order to accomplish this network functionality such as Virtual Routing and Forwarding (VRF) must be configured on the switching infrastructure, or virtual routers deployed in the hypervisor. VRF scalability is greatly limited by hardware implementation and will be far less than VxLAN scalability, while virtualized routers will consume CPU overhead and require additional architectural considerations. Without techniques in place network tenants will require non-overlapping IP space.

Making a case for true abstraction

With network virtualization alone being overlaid onto existing infrastructure we just add layers of complexity. This occurs without correcting the issues that have arisen in traditional networking constructs; just adding network virtualization will do no more than amplify existing problems. A parallel can be drawn to server virtualization where the more rapid pace of server provisioning quickly brought out problems in underlying architecture and processes.

The underlying network consists of hardware, cabling, and Layer 2 / Layer 3 topologies that dictate traffic flow and potential application throughput. These layers have their own limitations and stability issues which are not addressed by network virtualization. Think of the OSI model in terms of building a house, the bottom layers (1-3) create a foundation, a frame, and a structure. Issues in those foundational layers will be exacerbated at each additional layer added on top.

Rather than applying an overlay technique such as a virtualization layer on top of existing architecture, IT architects will benefit greatly from abstracting the network constructs from the ground up first. Separating out logical and physical constructs, security, services, etc. prior to layering on overlays will provide a clean canvas on which to paint the future’s scalable feature rich networks. Virtualization must be built into the network from the ground up rather than layered on top. Again this parallels server virtualization where the greatest success has been seen in full virtualization of the hardware platform and tight integration down to bare metal. The end goal is addressing the underlying network issues rather than mask them with a virtualization layer.

The ties between network constructs such as VLAN, IP subnet, security, load-balancing etc. have placed constraints on the scalability and agility of the network. Each VLAN is provided an IP subnet, security and network services are then tied to these constructs. Addressing and location become the identifying characteristics of the network rather than the application requirements. This is not optimal behavior for a network responsible for elastic business services, workload anywhere designs, and ever increasing connectivity needs. These attributes and capabilities of connectivity must be abstracted in a new way to allow us to move beyond the constraints we have imposed by overloading or misusing these basic network constructs.

imageRather than starting with a new coat of paint on a peeling building, abstraction takes a ground up approach. By looking at the purpose of each construct: VLAN = Broadcast domain, IP = addressing mechanism, etc. we can redesign with a goal to alleviate the unnecessary constraints that have been placed on today’s networks. With these constructs separated we can provide a transport capable of maximizing the performance, security and scalability of the applications using it.

Take a step back from traditional network thinking and think in terms of application needs without consideration of current deployment methodology. Think through the following questions leaving out concepts like: VLAN, Subnet, IP addressing, etc.:

  • How would you tie application tiers together?
  • How would you group like services?
  • What policies would be required between application tiers?
  • What services are required for a given application?
  • How does that application connect to the intranet and internet?

Separating out the applications and services required from the underlying architecture is not possible with today’s networks, virtualized or not. Overlay network virtualization alone may hide some of the complexities but does not provide tools for optimizing the delivery and holistic design. The conversation must include addressing, VLAN construct, location and service insertion. If these constructs are instead abstracted from one another, and the architecture, the conversation can revolve around application requirements rather than network restrictions.


While network virtualization provides a set of tools for gaining greater network scale and application deployment flexibility, it is not a complete solution. Without true network abstraction and tools for visibility between the logical and physical network virtualization does no more than add complexity to existing problems. As was seen with server virtualization, layering virtualization on infrastructure issues and bad processes exponentially increases the complexity and room for error.

In order to truly scale networks in a sustainably manageable fashion we need to remove the ties of disparate network constructs by abstracting them out. Once these constructs operate independently of one another we’re provided a flexible architecture that removes the inherent complexity rather than leaving the problems and compounding them through layers of virtualization.

To build networks that meet current demands while being able to support the rapid scale and emerging requirements we need to rethink network design as a whole. Taking a top down look at what we need from the network without tying ourselves to the way in which we use the constructs today allows us to design towards the future and apply layers of abstraction down the stack to meet those goals.

Thinking about your network today, is virtualization alone solving the problems or adding a layer?

Network virtualization without network abstraction – results in short term patching with limited control of longer term operational complexity.

Network virtualization based on an abstracted network – results in effective control of both capital and operational expenses.

GD Star Rating

What Network Virtualization Isn’t

Brad Hedlund recently posted an excellent blog on Network Virtualization.  Network Virtualization is the label used by Brad’s employer VMware/Nicira for their implementation of SDN.  Brad’s article does a great job of outlining the need for changes in networking in order to support current and evolving application deployment models.  He also correctly points out that networking has lagged behind the rest of the data center as technical and operational advancements have been made. 

Network configuration today is laughably archaic when compared to storage, compute and even facilities.  It is still the domain of CLI wizards hacking away on keyboards to configure individual devices.  VMware brought advancements like resource utilization based automatic workload migration to the compute environment.  In order to support this behavior on the network an admin must ensure the appropriate configuration is manually defined on each port that workload may access and every port connecting the two.  This is time consuming, costly and error prone. Brad is right, this is broken.

Brad also correctly points out that network speeds, feeds and packet delivery are adequately evolving and that the friction lies in configuration, policy and service delivery.  These essential network components are still far too static to keep pace with application deployments.  The network needs to evolve, and rapidly, in order to catch up with the rest of the data center.

Brad and I do not differ on the problem(s), or better stated: we do not differ on the drivers for change.  We do however differ on the solution.  Let me preface in advance that Brad and I both work for HW/SW vendors with differing solutions to the problem and differing visions of the future.  Feel free to write the rest of this off as mindless dribble or vendor Kool Aid, I ain’t gonna hate you for it.

Brad makes the case that Network Virtualization is equivalent to server virtualization, and from this simple assumption he poses it as the solution to current network problems.

Let’s start at the top: don’t be fooled by emphatic statements such as Brad’s stating that network virtualization is analogous to server virtualization.  It is not, this is an apples and oranges discussion.  Network virtualization being the orange where you must peel the rind to get to the fruit.  Don’t take my word for it, one of Brad’s colleagues, Scott Lowe, a man much smarter then I says it best:


The issue is that these two concepts are implemented in a very different fashion.  Where server virtualization provides full visibility and partitioning of the underlying hardware, network virtualization simply provides a packet encapsulation technique for frames on the wire.  The diagram below better illustrates our two fruits: apples and oranges.


As the diagram illustrates we are not working with equivalent approaches.  Network virtualization would require partitioning of switch CPU, TCAM, ASIC forwarding, bandwidth etc. to be a true apples-to-apples comparison.  Instead it provides a simple wrapper to encapsulate traffic on the underlying Layer 3 infrastructure.  These are two very different virtualization approaches.

Brad makes his next giant leap in the “What is the Network section.”  Here he makes the assumption that the network consists of only virtual workloads “The “network” we want to virtualize is the complete L2-L7 services viewed by the virtual machines” and the rest of his blog focuses there.  This is fine for those data center environments that are 100% virtualized including servers, services and WAN connectivity and use server virtualization for all of those purposes.  Those environments must also lack PaaS and SaaS systems that aren’t built on virtual servers as those are also non-applicable to the remaining discussion.  So anyone in those environments described will benefit from the discussion, anyone <crickets>.

So Brad and, presumably VMware/Nicira (since network virtualization is their term), define the goal as taking “all of the network services, features, and configuration necessary to provision the application’s virtual network (VLANs, VRFs, Firewall rules, Load Balancer pools & VIPs, IPAM, Routing, isolation, multi-tenancy, etc.) – take all of those features, decouple it from the physical network, and move it into a virtualization software layer for the express purpose of automation.”  So if your looking to build 100% virtualized server environments with no plans to advance up the stack into PaaS, etc. it seems you have found your Huckleberry.

What we really need is not a virtualized network overlay running on top of an L3 infrastructure with no communication or correlation between the two.  What we really need is something another guy much smarter than me (Greg Ferro) described:


Abstraction, independence and isolation, that’s the key to moving the network forward.  This is not provided by network virtualization.  Network virtualization is a coat of paint on the existing building.  Further more that coat of paint is applied without stripping, priming, or removing that floral wall paper your grandmother loved.  The diagram below is how I think of it.

Network Virtualization

With a network virtualization solution you’re placing your applications on a house of cards built on a non-isolated infrastructure of legacy design and thinking.  Without modifying the underlying infrastructure, network virtualization solutions are only as good as the original foundation.  Of course you could replace the data center network with a non-blocking fabric and apply QoS consistently across that underlying fabric (most likely manually) as Brad Hedlund suggests below.


If this is the route you take, to rebuild the foundation before applying network virtualization paint, is network virtualization still the color you want?  If a refresh and reconfigure is required anyway, is this the best method for doing so? 

The network has become complex and unmanageable due to things far older than VMware and server virtualization.  We’ve clung to device centric CLI configuration and the realm of keyboard wizards.  Furthermore we’ve bastardized originally abstracted network constructs such as VLAN, VRF, addressing, routing, and security tying them together and creating a Frankenstein of a data center network.  Are we surprised the villagers are coming with torches and pitch forks?

So overall I agree with Brad, the network needs to be fixed.  We just differ on the solution, I’d like to see more than a coat of paint.  Put lipstick on a pig and all you get is a pretty pig.

lipstick pig

GD Star Rating