WWT GeekDay 2013

I had the privilege this week to attend the opening keynote and SDN panel of WWT’s Geek Day.  The SDN panel was made up of heavy hitters from Cisco, VMware, HP, and Embrane.  They each presented their vision and solutions for SDN, then teamed up for questions.  The session was very good and was recorded.  I highly recommend watching them at the links below.  For more info on attending or sponsoring this great event see www.geekday.com

Cisco’s Balaji Sivasubramanian:

 

VMware’s Brad Hedlund:

HP’s Mauicio Sanchez:

Embrane’s Tom Nosella:

GD Star Rating
loading...

Network Overlays: An Introduction

While network overlays are not a new concept, they have come back into the limelight, thanks to drivers brought on by large-scale virtualization. Several standards have been proposed to enable virtual networks to be layered over a physical network infrastructure: VXLAN, NVGRE, and SST. While each proposed standard uses different encapsulation techniques to solve current network limitations, they share some similarities. Let’s look at how network overlays work in general…

To see the full article visit: http://www.networkcomputing.com/next-gen-network-tech-center/network-overlays-an-introduction/240144228

GD Star Rating
loading...

Why We Need Network Abstraction

The move to highly virtualized data centers and cloud models is straining the network. While traditional data center networks were not designed to support the dynamic nature of today’s workloads, the fact is, the emergence of highly virtualized environments is merely exposing issues that have always existed within network constructs. VLANs, VRFs, subnets, routing, security, etc. have been stretched well beyond their original intent. The way these constructs are currently used limits scale, application expansion, contraction and mobility.  To read the full article visit: http://www.networkcomputing.com/next-gen-network-tech-center/why-we-need-network-abstraction/240142588

GD Star Rating
loading...

VXLAN Deep Dive

I’ve been spending my free time digging into network virtualization and network overlays.  This is part 1 of a 2 part series, part 2 can be found here: http://www.definethecloud.net/vxlan-deep-divepart-2.  By far the most popular virtualization technique in the data center is VXLAN.  This has as much to do with Cisco and VMware backing the technology as the tech itself.  That being said VXLAN is targeted specifically at the data center and is one of many similar solutions such as: NVGRE and STT.)  VXLAN’s goal is allowing dynamic large scale isolated virtual L2 networks to be created for virtualized and multi-tenant environments.  It does this by encapsulating frames in VXLAN packets.  The standard for VXLAN is under the scope of the IETF NVO3 working group.

 

image

The VXLAN encapsulation method is IP based and provides for a virtual L2 network.  With VXLAN the full Ethernet Frame (with the exception of the Frame Check Sequence: FCS) is carried as the payload of a UDP packet.  VXLAN utilizes a 24-bit VXLAN header, shown in the diagram, to identify virtual networks.  This header provides for up to 16 million virtual L2 networks.

Frame encapsulation is done by an entity known as a VXLAN Tunnel Endpoint (VTEP.)  A VTEP has two logical interfaces: an uplink and a downlink.  The uplink is responsible for receiving VXLAN frames and acts as a tunnel endpoint with an IP address used for routing VXLAN encapsulated frames.  These IP addresses are infrastructure addresses and are separate from the tenant IP addressing for the nodes using the VXLAN fabric.  VTEP functionality can be implemented in software such as a virtual switch or in the form a physical switch.

VXLAN frames are sent to the IP address assigned to the destination VTEP; this IP is placed in the Outer IP DA.  The IP of the VTEP sending the frame resides in the Outer IP SA.  Packets received on the uplink are mapped from the VXLAN ID to a VLAN and the Ethernet frame payload is sent as an 802.1Q Ethernet frame on the downlink.  During this process the inner MAC SA and VXLAN ID is learned in a local table.  Packets received on the downlink are mapped to a VXLAN ID using the VLAN of the frame.  A lookup is then performed within the VTEP L2 table using the VXLAN ID and destination MAC; this lookup provides the IP address of the destination VTEP.  The frame is then encapsulated and sent out the uplink interface.

image

Using the diagram above for reference a frame entering the downlink on VLAN 100 with a destination MAC of 11:11:11:11:11:11 will be encapsulated in a VXLAN packet with an outer destination address of 10.1.1.1.  The outer source address will be the IP of this VTEP (not shown) and the VXLAN ID will be 1001.

In a traditional L2 switch a behavior known as flood and learn is used for unknown destinations (i.e. a MAC not stored in the MAC table.  This means that if there is a miss when looking up the MAC the frame is flooded out all ports except the one on which it was received.  When a response is sent the MAC is then learned and written to the table.  The next frame for the same MAC will not incur a miss because the table will reflect the port it exists on.  VXLAN preserves this behavior over an IP network using IP multicast groups.

Each VXLAN ID has an assigned IP multicast group to use for traffic flooding (the same multicast group can be shared across VXLAN IDs.)  When a frame is received on the downlink bound for an unknown destination it is encapsulated using the IP of the assigned multicast group as the Outer DA; it’s then sent out the uplink.  Any VTEP with nodes on that VXLAN ID will have joined the multicast group and therefore receive the frame.  This maintains the traditional Ethernet flood and learn behavior.

VTEPs are designed to be implemented as a logical device on an L2 switch.  The L2 switch connects to the VTEP via a logical 802.1Q VLAN trunk.  This trunk contains an VXLAN infrastructure VLAN in addition to the production VLANs.  The infrastructure VLAN is used to carry VXLAN encapsulated traffic to the VXLAN fabric.  The only member interfaces of this VLAN will be VTEP’s logical connection to the bridge itself and the uplink to the VXLAN fabric.  This interface is the ‘uplink’ described above, while the logical 802.1Q trunk is the downlink.

image

Summary

VXLAN is a network overlay technology design for data center networks.  It provides massively increased scalability over VLAN IDs alone while allowing for L2 adjacency over L3 networks.  The VXLAN VTEP can be implemented in both virtual and physical switches allowing the virtual network to map to physical resources and network services.  VXLAN currently has both wide support and hardware adoption in switching ASICS and hardware NICs, as well as virtualization software.

GD Star Rating
loading...

Something up Brocade’s Sleeve, and it looks Good

Brocade’s got some new tricks up their sleeve and they look good.  For far too long Brocade fought against convergence to protect its FC install base and catch up.  This bled over into their Ethernet messaging and hindered market growth and comfort levels there.  Overall they appeared as a company missing the next technology waves and clinging desperately to the remnants of a fading requirement: pure storage networks.  That has all changed, Brocade is embracing Ethernet and focusing on technology innovation that is relevant to today’s trends and business.

The Hardware:

Brocade’s VDX 8770 (http://www.brocade.com/downloads/documents/data_sheets/product_data_sheets/vdx-8770-ds.pdf) is their flagship modular switch for Brocade VCS fabrics.  While at first I scoffed at the idea of bigger chassis switches for fabrics, it turns out I was wrong (happens often.)  I forgot about scale.  These fabrics will typically be built in core/edge or spine leaf/designs, often using End of Row (EoR) rather than Top of Rack (ToR) designs to reduce infrastructure.  This leaves max scalability bound by a combination of port count and switch count dependent on several factors such as interconnect ports.  Switch count will typically be limited by fabric software limitations either real or due to testing and certification processes.  Having high density modular fabric-capable switches helps solve scalability issues.

Some of the more interesting features:

  • Line-rate 40GE
  • “Auto-trunking” ISLs (multiple links between switches will bond automatically.)
  • Multi-pathing at layers 1, 2 and 3
  • Dynamic port-profile configuration and migration for VM mobility
  • 100GE ready
  • 4us latency with 4TB switching capacity
  • Support for 384,000 MAC addresses per fabric for massive L2 scalability
  • Support for up to 8000 ports in a VCS fabric
  • 4 and 8 slot chassis options
  • Multiple default gateways for load-balancing routing

The Software:

The real magic is Brocade’s fabric software.  Brocade looks at the fabric as the base on which to build an intelligent network, SDN or otherwise.  As such the fabric should be: resilient, scalable and easy to manage.  In several conversations with people at Brocade it was pointed out that SDN actually adds a management layer.  No matter how you slice it the SDN software overlays a physical network that must be managed.  Minimizing configuration requirements at this level simplifies the network overall.  Additionally the fabric should provide multi-pathing without link blocking for maximum network throughput. 

Brocade executes on this with VCS fabric.  VCS provides an easy to set up and manage fabric model.  Operations like adding a link for bandwidth are done with minimal configuration through tools like “auto-trunking.’  Basically ports identified as fabric ports will be built into the network topology automatically.  They also provide impressive scalability numbers with support for 384,000 MACs, 352,000 IPv4 routes, 88,000 IPv6 routes, and 8000 ports.

One surprise to me was that Brocade is doing this using custom silicon.  With companies like Arista and Nicira (now part of VMware) touting commodity hardware as the future, why is Brocade spending money on silicon?  The answer is in latency.  If you want to do something at line-rate it must be implemented in hardware.  Merchant silicon is adept at keeping cutting edge at things like switching latency and buffering but is slow to implement new features.  This is due to addressable market.  Merchant silicon manufacturers want to ensure that the cost of hardware design and manufacturing will be recouped through bulk sale to multiple manufacturers.  This means features must have wide applicability and typically be standards driven before being implemented.

Brocade saw the ability to innovate with features while maintaining line-rate as an advantage worth the additional cost.  This allows Brocade to differentiate themselves, and their fabric, from vendors relying solely on merchant silicon.  Additionally they position they’re fabric as enough of an advantage to be worth the additional cost when implementing SDN for reasons listed above.

Summary:

Brocade is making some very smart moves and coming out from under the FC rock.  The technology is relevant and timely, but they will still have an uphill battle gaining the confidence of network teams.  They will have to rely on their FC data center heritage to build that confidence and expand their customer base.  The key now will be in execution, it will be an exciting ride.

GD Star Rating
loading...

Much Ado About Something: Brocade’s Tech Day

Yesterday I had the privilege of attending Brocade’s Tech Day for Analysts and Press.  Brocade announced the new VDX 8770, discussed some VMware announcements, as well as discussed strategy, vision and direction.  I’m going to dig in to a few of the topics that interested me, this is no way a complete recap.

First in regards to the event itself.  My kudos to the staff that put the event together it was excellent from both a pre-event coordination and event staff perspective.  The Brocade corporate campus is beautiful and the EBC building was extremely well suited to such an event.  The sessions went on smoothly, the food was excellent and overall it was a great experience.  I also want to thank Lisa Caywood (@thereallisac) for pointing out that my tweets during the event were more inflammatory then productive and outside the lines of ‘guest etiquette.’  She’s definitely correct and hopefully I can clear up some of my skepticism here in a format left open for debate, and avoid the same mistake in the future.  That being said I had thought I was quite clear going in on who I was and how I write.  To clear up any future confusion from anyone:  if you’re not interested in my unfiltered, typically cynical, honest opinion don’t invite me, I won’t take offense.  Even if you’re a vendor with products I like I’ve probably got a box full of cynicism for your other product lines.

During the opening sessions I observed several things that struck me negatively:

  • A theme (intended or not) that Brocade was being lead into new technologies by their customers.  Don’t get me wrong, listening to your customers and keeping your product in line with their needs is key to success.  That being said if your customers are leading you into new technology you’ve probably missed the boat.  In most cases they’re being lead there by someone else and dragging you along for the ride, that’s not sustainable.  IT vendors shouldn’t need to be dragged kicking and screaming into new technologies by customers.  This doesn’t mean chase every shiny object (squirrel!) but major trends should be investigated and invested in before you’re hearing enough customer buzz to warrant it.  Remember business isn’t just about maintaining current customers it’s about growing by adopting new ones.  Especially for public companies stagnant is as good as dead.
  • The term “ Ethernet Fabric” which is only used by Brocade, everyone else just calls it fabric.  This ties in closely with the next bullet.
  • A continued need to discuss commitment to pure Fibre Channel (FC) storage.  I don’t deny that FC will be around for quite some time and may even see some growth as customers with it embedded will expand.  That being said customers with no FC investment should be avoiding it like the plague and as vendors and consultants we should be pushing more intelligent options to those customers.  You can pick apart technical details about FC vs. anything all day long, enjoy that on your own, the fact is two fold: running two separate networks is expensive and complex, the differences in reliability, performance, etc. are fading if not gone.  Additionally applications are being written in more intelligent ways that don’t require the high availability, low latency silo’d architecture of yester year.  Rather than clinging to FC like a sinking ship vendors should be protecting customer investment while building and positioning the next evolution.  Quote of the day during a conversation in the hall: “Fibre channel is just a slightly slower melting ice cube then we expected.’
  • An insistence that Ethernet fabric was a required building block of SDN.  I’d argue that while it can be a component it is far from required, and as SDN progresses it will be irrelevant completely.  More on this to come.
  • A stance that the network will not be commoditized was common throughout the day.  I’d say that’s either A) naïve or B) posturing to protect core revenue.  I’d say we’ll see network commoditization occur en mass over the next five years.  I’m specifically talking about the data center and a move away from specialized custom built ASICS, not the core routers, and not  the campus.  Custom silicon is expensive and time-consuming to develop, but provides performance/latency benefits and arguable some security benefits.  As the processor and off the shelf chips continue to increase exponentially this differentiator becomes less and less important.  What becomes more important is rapid adaption to new needs.  SDN as a whole won’t rip and replace networking in the next five years but it’s growth and the concepts around it will drive commoditization.  It happened with servers, then storage while people made the same arguments.  Cheaper, faster to produce and ‘good-enough’ consistently wins out.

On the positive side Brocade has some vision that’s quite interesting as well as some areas where they are leading by filling gaps in industry offerings.

  • Brocade is embracing the concept of SDN and understands a concept I tweeted about recently: ‘Revolutions don’t sell.’  Customers want evolutionary steps to new technology.  Few if any customers will rip and replace current infrastructure to dive head first into SDN.  SDN is a complete departure from the way we network today, and will therefore require evolutionary steps to get there. This is shown in their support of ‘hybrid’ open flow implementations on some devices.  This means that OpenFlow implementations can run segregated alongside traditional network deployments.  This allows for test/dev or roll-out of new services without an impact on production traffic.  This is a great approach where other vendors are offering ‘either or’ options.
  • There was discussion of Brocade’s VXLAN gateway which was announced at VMworld.  To my knowledge this is the first offering in this much needed space.  Without a gateway VXLAN is limited to virtual only environments. This includes segregation from services provided by physical devices.  The Brocade VXLAN gateway will allow the virtual and physical networks to be bridged. (http://newsroom.brocade.com/press-releases/brocade-adx-series-to-unveil-vxlan-gateway-and-app-nasdaq-brcd-0923542) To dig deeper on why this is needed check out Ivan’s article: http://blog.ioshints.info/2011/10/vxlan-termination-on-physical-devices.html.
  • The new Brocade VDX 8770 is one bad ass mamma jamma.  With industry leading latency and MAC table capacity, along with TRILL based fabric functionality, it’s built for large scalable high-density fabrics.  I originally tweeted “The #BRCD #VDX8770 is a bigger badder chassis in a world with less need for big bad chassis.” After reading Ivan’s post on it I stand corrected (this happens frequently.)  For some great perspective and a look at specs take a read: http://blog.ioshints.info/2012/09/building-large-l3-fabrics-with-brocade.html.

On the financial side Brocade has been looking good and climbed over $6.00 a share.  There are plenty of conversations stating some of this may be due to upcoming shifts at the CEO level.  They’ve reported two great quarters and are applying some new focus towards federal government and other areas lacking in recent past. I didn’t dig further into this discussion.

During lunch I was introduced to one of the most interesting Brocade offerings I’d never heard of: ‘Brocade Network Subscription”: http://www.brocade.com/company/how-to-buy/capital-solutions/index.page.  Basically you can lease your on-prem network from Brocade Capitol.  This is a great idea for customers looking to shift CapEx to OpEx which can be extremely useful.  I also received a great explanation for the value of a fabric underneath an SDN network from Jason Nolet (VP of Data Center Networking Group.)  Jason’s position (summarized) is that implementing SDN adds a network management layer, rather than removing one.  With that in mind the more complexity we remove from the physical network the better off we are.  What we’ll want for our SDN networks is fast, plug-and-play functionality with max usable links and minimal management.  Brocade VCS fabric fits this nicely.  While I agree with that completely I ‘d also say it’s not the only way to skin that particular cat.  More to come on that.

For the last few years I’ve looked at Brocade as a company lacking innovation and direction.  They clung furiously to FC while the market began shifting to Ethernet, ignored cloud for quite a while, etc.  Meanwhile they burned down deals to purchase them and ended up where they’ve been.  The overall messaging, while nothing new, did have undertones of change as a whole and new direction.  That’s refreshing to hear.  Brocade is embracing virtualization and cloud architectures without tying their cart to a single hypervisor horse.  They are positioning well for SDN and the network market shifts.  Most impressively they are identifying gaps in the spaces they operate and executing on them both from a business and technology perspective.  Examples of this are Brocade Network Subscription and the VXLAN gateway functionality respectively.

Things are looking up and there is definitely something good happening at Brocade.  That being said they aren’t out of the woods yet.  For them, as a company, purchase is far fetched as the vendors that would buy them already have networking plays and would lose half of Brocade’s value by burning OEM relationships with the purchase.  The only real option from a sale perspective is for investors looking to carve them up and sell off pieces individually.  A scenario like this wouldn’t bode well for customers.  Brocade has some work to do but they’ve got a solid set of products and great direction.  We’ll see how it pans out.  Execution is paramount for them at this point.

Final Note:  This blog was intended to stop there but this morning I received an angry accusatory email from Brocade’s head of corporate communications who was unhappy with my tweets.  I thought about posting the email in full, but have decided against it for the sake of professionalism.  Overall his email was an attack based on my tweets.  As stated my tweets were not professional, but this type of email from someone in charge of corporate communications is well over the top in response.  I forwarded the email to several analyst and blogger colleagues, a handful of whom had similar issues with this individual.  One common theme in social media is that lashing out at bad press never does any good, a senior director in this position should know such, but instead continues to slander and attack.  His team and colleagues seem to understand social media use as they’ve engaged in healthy debate with me in regards to my tweets, it’s a shame they are not lead from the front.

GD Star Rating
loading...

Thoughts From a Tech Leadership Summit

This week I attended a tech leadership Summit in Vail Colorado for the second time.  The event is always a fantastic series of discussions and brings some of the top minds in the technology industry.  Here are some thoughts on the trends and thinking that were common at the event.

Virtualization and VDI:

There was a lot less talk of VDI and virtualization then in 2011.  These conversations were replaced with more conversations about cloud and app delivery.  Overall the consensus felt to be that getting the application to the right native environment on a given device was a far better approach then getting the desktop there.

Hypervisors were barely mentioned except in a recurring theme that the hypervisor itself has hit commodity.  This means that management and upper layer feature set are the differentiators.  Parallel to this thought was that VMware no longer has the best hypervisor yet their management system is still far superior to the competition (KVM was touted as the best hypervisor several times.)

The last piece of the virtualization discussion was around VMware’s acquisition of Nicira.  Some bullet points on that:

  • VMware paid too much for Nicira but that was unavoidable for the startup-to-be in the valley and it’s a great acquisition overall.
  • It’s no surprise VMware moved into networking everyone is moving that way.
  • While this is direct competition with Cisco it is currently in a small niche of service provider business.  Nicira’s product requires significant custom integration to deploy and will take time for VMware to productize it in a fashion usable for the enterprise.  Best guess: two years to real product. \
  • Overall the Cisco VMware partnership is very lucrative on both sides and should not be effected by this in the near term.
  • A seldom discussed portion of this involves the development expertise that comes with the acquisition.  With the hypervisor being commodity, and differentiation moving into the layers above that, we’ll see more and more variety in hypervisors.  This means multi-hypervisor support will be a key component of the upper level management products where virtualization vendors will compete.  Nicira’s team has proven capabilities in this space and can accelerate VMware’s multi-hypervisor strategy.

Storage:

There was a lot of talk about both the vision and execution of EMC over the past year or more.  I personally used ‘execution machine’ more than once to describe them (coming from a typically non-EMC Kool-Aid guy.)  Some key points that resonated over past few days:

  • EMC’s execution on the VNX/VNXe product lines is astounding.  EMC launched a product and went on direct attack into a portion of NetApp’s business that nobody could really touch.  Through both sales and marketing excellence they’ve taken an increasingly large chunk out of this portion of the market.  This shores up a breech in their product line NetApp was using to gain share.
  • EMC’s Isilon acquisition was not only a fantastic choice, but was quickly integrated well.  Isilon is a fantastic product and has big data potential which is definitely a market that will generate big revenue in coming years.
  • EMC’s cloud vision is sound and they are executing well on it.  Additionally they were ahead of their pack of hardware vendor peers in this regard. EMC is embracing a software defined future.

I also participated in several discussions around flash and flash storage.  Some highlights:

  • PCIe based flash storage is definitely increasing in sales and enterprise consumption.  This market is expected to continue to grow as we strive to move the data closer to the processor.  There are two methods for this: storage in the server, servers in the storage.  PCIe flash plays in the server side and EMC Isilon will eventually play on the storage side.  Also look for an announcement in the SMB storage space around this during VMworld.
  • One issue in this space is that the expensive fast server based flash becomes trapped capacity if a server can’t drive enough I/O to it.  Additionally there are data loss concerns with this data trapped in the server.
  • Both of these issues are looking to be solved by EMC and IBM who intend to add server based flash into the tiering of shared storage.
  • Most traditional storage vendors flash options are ‘bolt-ons’ to traditional array architecture.  This can leave the expensive flash I/O starved, limiting it’s performance benefit.  Several all flash startups intend to use this as an inflection point with flash based systems designed from the ground up for the performance the disk offers.
  • Flash is still not an answer to every problem, and never will be.

The last point that struck me was a potential move from shared storage as a whole.  Microsoft would rather have you use local storage, clusters and big data apps like Hadoop thrive on local storage and one last big shared storage draw is going away: vMotion.  Once shared storage is no longer need for live virtual machine migration there will be far less draw for expensive systems.

Cloud:

The major cloud discussion I was a part of (mainly observer) involved OpenStack.  Overall OpenStack has a ton of buzz, and a plethora of developers.  What it’s lacking is customers, leadership and someone driving it who can lead a revolution.  Additionally it’s suffering from politics and bureaucracy.  It was described as impossible to support by one individual who would definitely know one way or another.  My thinking is that if you have CloudStack sitting there with real customers, an easily deployed system, support and leadership why waste cycles continuing down the OpenStack path?  The best answer I heard for that: Ego.  Everyone wants to build the next Amazon and CloudStack is too baked to make as much of a mark.

Overall it’s an interesting topic but my thought is: with limited developers the industry should be getting behind the best horse and working together.

Big Data:

Big Data was obviously another fun topic.  The quote of the week was ‘There are ten people, not companies, that understand Big Data.  6 of them are at Cloudera and the other 4 are locked in Google writing their own checks.’  Basically Big Data knowledge is rare and hiring consultants is not typically a viable option because you need people holding three things: Knowledge of big data processing, knowledge of your data, and knowledge of your business.  These data scientists aren’t easy to come by.  Additionally contrary to popular hype, Hadoop is not the end-all be-all of big data, it’s a tool in a large tool chest.  Especially when talking about real-time you’ll need to look elsewhere.  The consensus was that we are with big data where we were with cloud 2-3 years ago.  That being said CIO’s may still need to show big data initiatives (read: spend) so you should see $$ thrown at well packaged big data solutions geared toward plug-n-play in the enterprise.

All in all it was an excellent event and I was humbled as usual to participate in great conversations with so many smart people who are out there driving the future of technology.  What I’ve written here is a a summary from my perspective on the one summit portion I had time to participate in.  There is always a good chance I misquoted/misunderstood something so feel free to call me out.  As always I’d love your feedback, contradictions or hate mail comments.

GD Star Rating
loading...

SDN – Centralized Network Command and Control

Software Defined Networking (SDN) is a hot topic in the data center and cloud community.  The geniuses <sarcasm> over at IDC predict a $2 billion market by 2016 (expect this number to change often between now and then, and look closely at what they count in the cost.) The concept has the potential to shake up the networking business as a whole (http://www.networkcomputing.com/next-gen-network-tech-center/240001372) and has both commercial and open source products being developed and shipping, but what is it, and why?

Let’s start with the why by taking a look at how traditional networking occurs.

 

Traditional Network Architecture:

 

image

 

The most important thing to notice in the graphic above is the separate control and data planes.  Each plane has separate tasks that provide the overall switching/routing functionality.  The control plane is responsible for configuration of the device and programming the paths that will be used for data flows.  When you are managing a switch you are interacting with the control plane.  Things like route tables and Spanning-Tree Protocol (STP) are calculated in the control plane.   This is done by accepting information frames such as BPDUs or Hello messages and processing them to determine available paths.  Once these paths have been determined they are pushed down to the data plane and typically stored in hardware.  The data lane then typically makes path decisions in hardware based on the latest information provided by the control plane.  This has traditionally been a very effective method.  The hardware decision making process is very fast, reducing overall latency while the control plane itself can handle the heavier processing and configuration requirements.

This method is not without problems, the one we will focus on is scalability.  In order to demonstrate the scalability issue I find it easiest to use Quality of Service (QoS) as an example.  QoS allows forwarding priority to be given to specific frames for scheduling purposes based on characteristics in those frames.  This allows network traffic to receive appropriate treatment in times of congestion.  For instance latency sensitive voice and video traffic is typically engineered for high priority to ensure the best user experience.  Traffic prioritization is typically based on tags in the frame known as Class of Service (CoS) and or Differentiated Services Code Point (DSCP.)  These tags must be marked consistently for frames entering the network and rules must then be applied consistently for their treatment on the network. This becomes cumbersome in a traditional multi-switch network because the configuration must be duplicated in some fashion on each individual switching device.

An easier example of the current administrative challenges consider each port in the network a management point, meaning each port must be individually configured.  This is both time consuming and cumbersome.

Additional challenges exist in properly classifying data and routing traffic.  A fantastic example of this would be two different traffic types, iSCSI and voice.  iSCSI is storage traffic and typically a full size packet or even jumbo frame while voice data is typically transmitted in a very small packet.  Additionally they have different requirements, voice is very latency sensitive in order to maintain call quality, while iSCSI is less latency sensitive but will benefit from more bandwidth.  Traditional networks have few if any tools to differentiate these traffic types and send them down separate paths which are beneficial to both types.

These types of issues are what SDN looks to solve.

The Three Key Elements of SDN:

  • Ability to manage the forwarding of frames/packets and apply policy
  • Ability to perform this at scale in a dynamic fashion
  • Ability to be programmed

Note: In order to qualify as SDN an architecture does not have to be Open, standard, interoperable, etc.  A proprietary architecture can meet the definition and provide the same benefits.  This blog does not argue for or against either open or proprietary architectures.

An SDN architecture must be able to manipulate frame and packet flows through the network at large scale, and do so in a programmable fashion.  The hardware plumbing of an SDN will typically be designed as a converged (capable of carrying all data types including desired forms of storage traffic) mesh of large lower latency pipes commonly called a fabric.  The SDN architecture itself will in turn provide a network wide view and the ability to manage the network and network flows centrally.

This architecture is accomplished by separating the control plane from the data plane devices and providing a programmable interface for that separated control plane.  The data plane devices receive forwarding rules from the separated control plane and apply those rules in hardware ASICs.  These ASICs can be either commodity switching ASICs or customized silicone depending on the functionality and performance aspects required.  The diagram below depicts this relationship:

image

In this model the SDN controller provides the control plane and the data plane is comprised of hardware switching devices.  These devices can either be new hardware devices or existing hardware devices with specialized firmware.  This will depend on vendor, and deployment model.  One major advantage that is clearly shown in this example is the visibility provided to the control plane.  Rather than each individual data plane device relying on advertisements from other devices to build it’s view of the network topology a single control plane device has a view of the entire network.  This provides a platform from which advanced routing, security, and quality decisions can be made, hence the need for programmability.  Another major capability that can be drawn from this centralized control is visibility.  With a centralized controller device it is much easier to gain usable data about real time flows on the network, and make decisions (automated or manual) based on that data.

This diagram only shows a portion of the picture as it is focused on physical infrastructure and serves.  Another major benefit is the integration of virtual server environments into SDN networks.  This allows centralized management of consistent policies for both virtual and physical resources.  Integrating a virtual network is done by having a Virtual Ethernet Bridge (VEB) in the hypervisor that can be controlled by an SDN controller.  The diagram below depicts this:

image

This diagram more clearly depicts the integration between virtual networking systems and physical networking systems in order to have cohesive consistent control of the network.  This plays a more important role as virtual workloads migrate.  Because both the virtual and physical data planes are managed centrally by the control plane when a VM migration happens it’s network configuration can move with it regardless of destination in the fabric.  This is a key benefit for policy enforcement in virtualized environments because more granular controls can be placed on the VM itself as an individual port and those controls stick with the VM throughout the environment.

Note: These diagrams are a generalized depiction of an SDN architecture.  Methods other than a single separated controller could be used, but this is the more common concept.

With the system in place to have centralized command and control of the network through SDN and a programmable interface more intelligent processes can now be added to handle complex systems.  Real time decisions can be made for the purposes of traffic optimization, security, outage, or maintenance.  Separate traffic types can be run side by side while receiving different paths and forwarding that can respond dynamically to network changes.

Summary:

Software Defined Networking has the potential to disrupt the networking market and move us past the days of the switch/router jockey.  This shift will provide extreme benefits in the form of flexibility, scalability and traffic performance for datacenter networks.  While all of the aspects are not yet defined SDN projects such as OpenFlow (www.openflow.org) provide the tools to begin testing and developing SDN architectures on supported hardware.  Expect to see lots of changes in this eco system and many flavors in the vendor offerings.

GD Star Rating
loading...

Why Software-Defined Networking Could Revolutionize Networking

Software-defined networking (SDN) is a hot topic in the network community. Vendors such as Big Switch, Brocade, Cisco and HP are getting into the mix, and just about anyone in networking is making announcements. Additionally, OpenFlow is an open-source option for building an SDN control plane. To read the full article visit: http://www.networkcomputing.com/next-gen-network-tech-center/240001372

GD Star Rating
loading...

Server Networking With gen 2 UCS Hardware

** this post has been slightly edited thanks to feedback from Sean McGee**

In previous posts I’ve outlined:

If you’re not familiar with UCS networking I suggest you start with those for background.  This post is an update to those focused on UCS B-Series server to Fabric Interconnect communication using the new hardware options announced at Cisco Live 2011.  First a recap of the new hardware:

The UCS 6248UP Fabric Interconnect

The 6248 is a 1RU model that provides 48 universal ports (1G/10G Ethernet or 1/2/4/8G FC.)  This provides 20 additional ports over the 6120 in the same 1RU form factor.  Additionally the 6248 is lower latency at 2.0us from 3.2us previously.

The UCS 2208XP I/O Module

The 2208 doubles the total uplink bandwidth per I/O module providing a total of 160Gbps total throughput per 8 blade chassis.  It quadruples the number of internal 10G connections to the blades allowing for 80Gbps per half-width blade.

UCS 1280 VIC

The 1280 VIC provides 8x10GE ports total, 4x to each IOM for a total of 80Gbps per half-width slot (160 Gbs with 2x in a full-width blade.)  It also double the VIF numbers of the previous VIC allowing for 256 (theoretical)  vNICs or vHBAs.  The new VIC also supports port-channeling to the UCS 2208 IOM and iSCSI boot.

The other addition that affects this conversation is the ability to port-channel the uplinks from the 2208 IOM which could not be done before (each link on a 2104 IOM operated independently.)  All of the new hardware is backward compatible with all existing UCS hardware.  For more detailed information on the hardware and software announcements visit Sean McGee’s blog where I stole these graphics: http://www.mseanmcgee.com/2011/07/ucs-2-0-cisco-stacks-the-deck-in-las-vegas/.

Let’s start by discussing the connectivity options from the Fabric Interconnects to the IOMs in the chassis focusing on all gen 2 hardware.

There are two modes of operation for the IOM: Discrete and Port-Channel.  in both modes it is possible to configure 1, 2 , 4, or 8 uplinks from each IOM in either Discrete mode (non-bundled) or port-channel mode (bundled.)

UCS 2208 fabric Interconnect Failover

image

Discrete Mode:

In discrete mode a static pinning mechanism is used mapping each blade to a given port dependent on number of uplinks used.  This means that each blade will have an assigned uplink on each IOM for inbound and outbound traffic.  In this mode if a link failure occurs the blade will not ‘re-pin’ on the side of the failure but instead rely on NIC-Teaming/bonding or Fabric Failover for failover to the redundant IOM/Fabric.  The pinning behavior is as follows with the exception of 1-Uplink (not-shown) in which all blades use the only available Port:

2 Uplinks

Blade

Port 1

Port 2

Port 3

Port 4

Port 5

Port 6

Port 7

Port 8

1

image

2

image

3

image

4

image

5

image

6

image

7

image

8

image

4 Uplinks

Blade

Port 1

Port  2

Port 3

Port 4

Port 5

Port 6

Port 7

Port 8

1

image

2

image

3

image

4

image

5

image

6

image

7

image

8

image

8 Uplinks

Blade

Port 1

Port2

Port 3

Port 4

Port 5

Port 6

Port 7

Port 8

1

image

2

image

3

image

4

image

5

image

6

image

7

image

8

image

The same port-pinning will be used on both IOMs, therefore in a redundant configuration each blade will be uplinked via the same port on separate IOMs to redundant fabrics.  The draw of discrete mode is that bandwidth is predictable in link failure scenarios.  If a link fails on one IOM that server will fail to the other fabric rather than adding additional bandwidth draws on the active links for the failure side.  In summary it forces NIC-teaming/bonding or Fabric Failover to handle failure events rather than network based load-balancing.  The following diagram depicts the failover behavior for server three in an 8 uplink scenario.

Discrete Mode Failover

image

In the previous diagram port 3 on IOM A has failed.  With the system in discrete mode NIC-teaming/bonding or Fabric Failover handles failover to the secondary path on IOM B (which is the same port (3) based on static-pinning.)

Port-Channel Mode:

In Port-Channel mode all available links are bonded and a port-channel hashing algorithm (TCP/UDP + Port VLAN, non-configurable) is used for load-balancing server traffic.  In this mode all server links are still ‘pinned’ but they are pinned to the logical bundle rather than individual IOM uplinks.  The following diagram depicts this mode.

Port-Channel Mode

image

In this scenario when a port fails on an IOM port-channel load-balancing algorithms handle failing the server traffic flow to another available port in the channel.  This failover will typically be faster than NIC-teaming/bonding failover.  This will decrease the potential throughput for all flows on the side with a failure, but will only effect performance if the links are saturated.  The following diagram depicts this behavior.

image

In the diagram above Blade 3 was pinned to Port 1 on the A side.  When port 1 failed port 4 was selected (depicted in green) while fabric B port 6 is still active leaving a potential of 20 Gbps.

Note: Actual used ports will vary dependent on port-channel load-balancing.  These are used for example purposes only.

As you can see the port-channel mode enables additional redundancy and potential per-server bandwidth as it leaves two paths open.  In high utilization situations where the links are fully saturated this will degrade throughput of all blades on the side experiencing the failure.  This is not necessarily a bad thing (happens with all port-channel mechanisms), but it is a design consideration.  Additionally port-channeling in all forms can only provide the bandwidth of a single link per flow (think of a flow as a conversation.)  This means that each flow can only utilize 10Gbps max even though 8x10Gbps links are bundled.  For example a single FTP transfer would max at 10Gbps bandwidth, while 8xFTP transfers could potentially use 80Gbps (10 per link) dependent on load-balancing.

Next lets discuss server to IOM connectivity (yes I use discuss to describe me monologuing in print, get over it, and yes I know monologuing isn’t a word) I’ll focus on the new UCS 1280 VIC because all other current cards maintain the same connectivity.  the following diagram depicts the 1280 VIC connectivity.

image

The 1280 VIC utilizes 4x10Gbps links across the mid-pane per IOM to form two 40Gbps port-channels.  This provides for 80Gbps total potential throughput per card.  This means a half-width blade has a total potential of 80Gbps using this card and a full-width blade can receive 160Gbps (of course this is dependent upon design.)  As with any port-channel, link-bonding, trunking or whatever you may call it, any flow (conversation) can only utilize the max of one physical link (or back plane trace) of bandwidth.  This means every flow from any given UCS server has a max potential bandwidth of 10Gbps, but with 8 total uplinks 8 different flows could potentially utilize 80Gbps.

This becomes very important with things like NFS-based storage within hypervisors.  Typically a virtualization hypervisor will handle storage connectivity for all VMs.  This means that only one flow (conversation) will occur between host and storage.  In these typical configurations only 10Gbps will be available for all VM NFS data traffic even though the host may have a potential 80Gbps bandwidth.  Again this is not necessarily a concern, but a design consideration as most current/near-future hosts will never use more than 10Gbps of storage I/O.

Summary:

The new UCS hardware packs a major punch when it comes to bandwidth, port-density and failover options.  That being said it’s important to understand the frame flow, port-usage and potential bandwidth in order to properly design solutions for maximum efficiency.  As always comments, complaints and corrections are quite welcome!

GD Star Rating
loading...