What’s the deal with Quantized Congestion Notification (QCN)

For the last several months there has been a lot of chatter in the blogosphere and Twitter about FCoE and whether full scale deployment requires QCN.  There are two camps on this:

  1. FCoE does not require QCN for proper operation with scale.
  2. FCoE does require QCN for proper operation and scale.

Typically the camps break down as follows (there are exceptions) :

  1. HP camp stating they’ve not yet released a suite of FCoE products because QCN is not fully ratified and they would be jumping the gun.  The flip side of this is stating that Cisco did jumped the gun with their suite of products and will have issues with full scale FCoE.
  2. Cisco camp stating that QCN is not required for proper FCoE frame flow and HP is using the QCN standard as an excuse for not having a shipping product.

For the purpose of this post I’m not camping with either side, I’m not even breaking out my tent.  What I’d like to do is discuss when and where QCN matters, what it provides and why.  The intent being that customers, architects, engineers etc. can decide for themselves when and where they may need QCN.

QCN: QCN is a form of end-to-end congestion management defined in IEEE 802.1.Qau.  The purpose of end-to-end congestion management is to ensure that congestion is controlled from the sending device to the receiving device in a dynamic fashion that can deal with changing bottlenecks.  The most common end-to-end congestion management tool is TCP Windows sizing.

TCP Window Sizing:

With window sizing TCP dynamically determines the number of frames to send at once without an acknowledgement.  It continuously ramps this number up dynamically if the pipe is empty and acknowledgements are being received.  If a packet is dropped due to congestion and an acknowledgement is not received TCP halves the window size and starts the process over.  This provides a mechanism in which the maximum available throughput can be achieved dynamically.

Below is a diagram showing the dynamic window size (total packets sent prior to acknowledgement) over the course of several round trips.  You can see the initial fast ramp up followed by a gradual increase until a packet is lost, from there the window is reduced and the slow ramp begins again.

image If you prefer analogies I always refer to TCP sliding windows as a Keg Stand (http://en.wikipedia.org/wiki/Keg_stand.)

File:Kegstand147.jpg

In the photo we see several gentleman surrounding a keg, with one upside down performing a keg stand.

To perform a keg stand:

  • Place both hands on top of the keg
  • 1-2 Friend(s) lift your feet over your head while you support your body weight on locked-out arms
  • Another friend places the keg’s nozzle in your mouth and turns it on
  • You swallow beer full speed for as long as you can

What the hell does this have to do with TCP Flow Control? I’m so glad you asked.

During a keg stand your friend is trying to push as much beer down your throat as it can handle, much like TCP increasing the window size to fill the end-to-end pipe.  Both of your hands are occupied holding your own weight, and your mouth has a beer hose in it, so like TCP you have no native congestion signaling mechanism.  Just like TCP the flow doesn’t slow until packets/beer drops, when you start to spill they stop the flow.

So that’s an example of end-to-end congestion management.  Within Ethernet and FCoE specifically we don’t have any native end-to-end congestion tools (remember TCP is up on L4 and we’re hanging out with the cool kids at L2.)  No problem though because We’re talking FCoE right?  FCoE is just a L1-L2 replacement for Fibre Channel (FC) L0-L1, so we’ll just use FC end-to-end congestion management… Not so fast boys and girls, FC does not have a standard for end-to-end congestion management, that’s right our beautiful over engineered lossless FC has no mechanism for handling network wide, end-to-end congestion.  That’s because it doesn’t need it.

FC is moving SCSI data, and SCSI is sensitive to dropped frames, latency is important but lossless delivery is more important.  To ensure a frame is never dropped FC uses a hop-by-hop flow control known as buffer-to-buffer (B2B) credits. At a high level each FC device knows the amount of buffer spaces available on the next hop device based on the agreed upon frame size (typically 2148 bytes.)  This means that a device will never send a frame to a next hop device that cannot handle the frame.  Let’s go back to the world of analogy.

Buffer-to-buffer credits:

The B2B credit system works in the same method you’d have 10 Marines offload and stack a truckload of boxes (‘fork-lift, we don’t need no stinking forklift.’)  The best system to utilize 10 Marines to offload boxes is to line them up end-to-end one in the truck and one on the other end to stack.  Marine 1 in the truck initiates the send by grabbing a box and passing it to Marine 2, the box moves down the line until it gets to the target Marine 10 who stacks it.  Before any Marine hands another Marine a box they look to ensure that Marines hands are empty verifying they can handle the box and it won’t be dropped.  Boxes move down the line until they are all offloaded and stacked.  If anyone slows down or gets backed up each marine will hold their box until the congestion is relieved.

In this analogy the Marine in the truck is the initiator/server and the Marine doing the stacking is the target/storage with each Marine in between being a switch.

When two FC devices initiate a link they follow the Link-Initialization-Protocols (LIP.)  During this process they agree on an FC frame size and exchange the available dedicated frame buffer spaces for the link.  A sender is always keeping track of available buffers on the receiving side of the link.  The only real difference between this and my analogy is each device (Marine) is typically able to handle more than one frame (box) at once.

So if FC networks operate without end-to-end congestion management just fine why do we need to implement a new mechanism in FCoE, well there-in lies the rub.  Do we need QCN?  The answer is really Yes and No, and it will depend on design.  FCoE today provides the exact same flow control as FC using two standards defined within Data Center Bridging (DCB) these are Enhanced Transmission Selection (ETS) and Priority-Flow Control (PFC) for more info on theses see my DCB blog: http://www.definethecloud.net/?p=31.)  Basically ETS provides a bandwidth guarantee without limiting and PFC provides lossless delivery on an Ethernet network.

Why QCN:

The reason QCN was developed is the differences between the size, scale, and design of FC and Ethernet networks.  Ethernet networks are usually large mesh or partial mesh type designs with multiple switches.  FC designs fall into one of three major categories Collapsed core (single layer), Core edge (two layer) or in rare cases for very large networks edge-core-edge (three layer.)  This is because we typically have far fewer FC connected devices than we do Ethernet (not every device needs consolidated storage/backup access.)

If we were to design our FCoE networks where every current Ethernet device supported FCoE and FCoE frames flowed end-to-end QCN would be a benefit to ensure point congestion didn’t clog the entire network.  On the other hand if we maintain similar size and design for FCoE networks as we do FC networks, there is no need for QCN.

Let’s look at some diagrams to better explain this:

image

 image In the diagrams above we see a couple of typical network designs.  The Ethernet diagram shows Core at the top, aggregation in the middle, and edge on the bottom where servers would connect.  The Fibre Channel design shows a core at the top with an edge at the bottom.  Storage would attach to the core and servers would attach at the bottom.  In both diagrams I’ve also shown typical frame flow for each traffic type.  Within Ethernet, servers commonly communicate with one another as well network file systems, the WAN etc.  In an FC network the frame flow is much more simplistic, typically only initiator target (server to storage) communication occurs.  In this particular FC example there is little to no chance of a single frame flow causing a central network congestion point that could effect other flows which is where end-to-end congestion management comes into play.

What does QCN do:

QCN moves congestion from the network center to the edge to avoid centralized congestion on DCB networks.  Let’s take a look at a centralized congestion example (FC only for simplicity):

image In the above example two 2Gbbps hosts are sending full rate frame flows to two storage devices.  One of the storage devices is a 2Gbps device and can handle the full speed, the other is a 1Gbps device and is not able to handle the full speed. If these rates are sustained switch 3’s buffers will eventually fill and cause centralized congestion effecting frame flows to both switch 4, and 5.  This means that the full rate capable devices would be affected by the single slower device.  QCN is designed to detect this type of congestion and push it to the edge, therefore slowing the initiator on the bottom right avoiding overall network congestion.

This example is obviously not a good design and is only used to illustrate the concept.  In fact in a properly designed FC network with multiple paths between end-points central congestion is easily avoidable.

When moving to FCoE if the network is designed such that FCoE frames pass through the entire full-mesh network shown in the Common Ethernet design above, there would be greater chances of central congestion.  If the central switches were DCB capable but not FCoE Channel Forwarders (FCF) QCN could play a part in pushing that congestion to the edge.

If on the other hand you design FCoE in a similar fashion to current FC networks QCN will not be necessary.  An example of this would be:

imageThe above design incorporates FCoE into the existing LAN Core, Aggregation, Edge design without clogging the LAN core with unneeded FCoE traffic.  Each server is dual connected to the common Ethernet mesh, and redundantly connected to FCoE SAN A and B.  This design is extremely scalable and will provide more than enough ports for most FCoE implementations.

Summary:

QCN like other congestion management tools before it such as FECN and BECN have significant use cases.  As far as FCoE deployments go QCN is definitely not a requirement and depending on design will provide no benefit for FCoE.  It’s important to remember that the DCB standards are there to enhance Ethernet as a whole, not just for FCoE.  FCoE utilizes ETS and PFC for lossless frame delivery and bandwidth control, but the FCoE standard is a separate entity from DCB.

Also remember that FCoE is an excellent tool for virtualization which reduces physical server count.  This means that we will continue to require less and less FCoE ports overall especially as 40Gbps and 100Gbps are adopted.  Scaling FCoE networks further than today’s FC networks will most likely not be a requirement.

GD Star Rating
loading...
What’s the deal with Quantized Congestion Notification (QCN), 4.8 out of 5 based on 13 ratings

Comments

  1. I enjoyed your post. It’s a very thoughtful examination of QCN and the potential problems it was designed to solve. In my experience, the design where QCN is needed isn’t realistic in the enterprise datacenter networks I design and support. In fact I would consider that a poor design. The future embrace of DCB/TRILL may indeed foster these type of designs. I would think we’d see more East/West scaling rather than adding additional tiers, but I could be wrong. Just because you can doesn’t mean you should. :-) QCN seems to me a solution in search of a problem.

    Also, just as a point of reference, Cisco did introduce FCC (Fibre Channel Congestion Control) some years back. This is pretty much the FC equivalent of QCN and nobody much cared. I realize that is anecdotal but food for thought nonetheless.

    Regards.

    GD Star Rating
    loading...
    • Thanks for the comment and thoughts. I definitely agree that QCN is a solution in search of a problem, much like FCC was. Thanks for reading.

      GD Star Rating
      loading...
  2. Nobody cared because FC has it’s own congestion control. Buffer to buffer ;)

    GD Star Rating
    loading...
  3. Part of the challenge with promoting QCN / ETS /PCB is that QOS is only needed when a congestion event occurs. This means that bandwidth must be oversubscribed at some point in the network.

    For the solutions available today, and in very general terms, it is effectively impossible for FCoE to oversubscribe a single link from server to switch as FCoE remains a single hop solution. Therefore customers are not experiencing QoS issues and may regard it an unnecessary.

    But a few months will see the FCoE mulithop once FIP gets going and products make it available. As such, collapsed tree oversubscription becomes more likely and people will take more time to understand the need for QoS.

    But until then, it’s a tough set of concepts to promote.

    Oh yeah, and QCN is very latency sensitive on 10GbE circuits so we won’t see particularly large networks using FCoE multihop. Perhaps just a few hops will be the maximum, but that should work for most people.

    greg

    GD Star Rating
    loading...
    • Greg,

      Great comments and additional information, much appreciated. The thing I try and remember and discuss whenever discussing FCoE multi-hop topologies is that in most cases it will be best to dsign them in a similar fashion to the FC topologies they represent. For most customers nothing more than Core/Edge designs will ever be needed, and FC traffic will remain initiator/target ie. north/south based. This is a topology in which QCN might add value as the parachute during sustained periods of congestion but may also end up not being needed.

      Looking forward to seeing how it all pans out as we progress.

      Joe

      GD Star Rating
      loading...
  4. joe smith says:

    Hi. Appreciate the attempt to explain QCN, but something about the explanation doesnt make sense.

    In your FC example, where switch 3 gets congested, it wouldnt get congested. Thats the whole point of FC’s B2B credit system. If an initiator wants to send FC traffic at a rate of 2G and the array has 1G capability, the initiator will eventually be forced to throttle back without congestion developing at switch 3.

    Moreover, switch 3 has separate downlinks and ports to each server and and separate uplinks and ports to each array, so the path from the 2G init to the 2G array will be uneffected.

    Theres no oversubscription of head of line blocking going on because there are two separate data paths, regardless of the fact that switch 3 is a common switch.

    GD Star Rating
    loading...
    • Joe,

      While the example is not ideal considering modern switch hardware but without a form of VoQ in place within switch hardware the example would create HoL blocking at the schedular causing network congestion. B2B credits would ensure lossless delivery but scheduling congestion would cause backups. Depending on the level of congestion those backups could eventually cause lost frames due to FC timers.

      The point of that section is to illustrate where a system such as QCN comes into play.

      Please be sure to include your affiliation when commenting so that we’re aware of where your perspective is coming from.

      Joe

      GD Star Rating
      loading...
  5. Hello, i think that i saw you visited my blog thus i came to “return the favor”.I’m
    attempting to find things to enhance my site!I suppose its ok to
    use a few of your ideas!!

    GD Star Rating
    loading...
  6. Every weekend i used to pay a visit this site, because i wish for enjoyment, since this this website conations actually good funny material too.

    GD Star Rating
    loading...
  7. Admiring the dedication you put into your site
    and detailed information you offer. It’s nice to come across a
    blog every once in a while that isn’t the same outdated rehashed material.
    Excellent read! I’ve saved your site and I’m including your RSS
    feeds to my Google account.

    GD Star Rating
    loading...
  8. How will they compare in performance, durability,
    uses, pricing, etc. Take time for it to look at terms and conditions with
    a potential contract, ask a great deal of questions and shop to get a
    reputable company. One in the tips that lots of home buyers are seeking is a home warranty.

    GD Star Rating
    loading...
  9. Someone necessarily lend a hand to make significantly posts I would state.
    That is the first time I frequented your website page and so far?
    I amazed with the analysis you made to make this particular
    publish amazing. Magnificent activity!

    GD Star Rating
    loading...
  10. When I initially commented I appear to have clicked on the -Notify
    me when new comments are added- checkbox and from now on whenever a
    comment is added I recieve four emails with the exact same comment.
    Perhaps there is a way you are able to remove me from that service?

    Many thanks!

    GD Star Rating
    loading...
  11. I blog often and I genuinely thank you for your content.

    The article has really peaked my interest. I am going to book
    mark your site and keep checking for new information about once
    a week. I opted in for your Feed too.

    GD Star Rating
    loading...
  12. Hey there are using WordPress for your site platform?
    I’m new to the blog world but I’m trying to get started and
    set up my own. Do you need any coding knowledge to make your own blog?

    Any help would be greatly appreciated!

    GD Star Rating
    loading...
  13. Pretty! This wwas an extrremely wonderful post.
    Many thanks for providing this info.

    GD Star Rating
    loading...
  14. My brother recommended I would possibly like this blog.
    He used to be entirely right. This post truly made my day.
    You can not believe simply how so much time I had spent for this
    info! Thank you!

    GD Star Rating
    loading...
  15. It is not my first time to pay a quick visit this web site,
    i am browsing this web site dailly and obtain fastidious data from here all the time.

    GD Star Rating
    loading...
  16. I’m not sure exactly why but this site is loading incredibly slow
    for me. Is anyone else having this problem or
    is it a issue on my end? I’ll check back later and see if the problem still exists.

    GD Star Rating
    loading...
  17. Hi everyone, it’s my first pay a quick visit at this site,
    and post is in fact fruitful in support of me, keep up posting these content.

    GD Star Rating
    loading...
  18. Great blog here! Also your web site loads up very fast!

    What host are you using? Can I get your affiliate link to your host?
    I wish my web site loaded up as fast as yours lol

    GD Star Rating
    loading...
  19. In fact no matter if someone doesn’t understand afterward its up to other viewers that they will help, so here it takes place.

    GD Star Rating
    loading...
  20. celine nano luggage handbags
    Way cool! Some very valid points! I appreciate you writing this write-up and
    the rest of the website is very good.

    GD Star Rating
    loading...
  21. Good day! Do you know if they make any plugins
    to assist with SEO? I’m trying to get my blog to rank for some targeted keywords but I’m not seeing very good
    gains. If you know of any please share. Appreciate it!

    GD Star Rating
    loading...
  22. magnificent points altogether, you just gained a logo
    new reader. What could you recommend in regards to your publish that you made a few days ago?
    Any certain?

    GD Star Rating
    loading...
  23. Greetings! I know this is kind of off topic but I was wondering which blog platform are you using for this site?
    I’m getting fed up of WordPress because I’ve had issues with hackers and I’m looking at
    alternatives for another platform. I would be awesome if you could point me
    in the direction of a good platform.

    GD Star Rating
    loading...
  24. Have you ever considered creating an e-book or guest authoring on other
    websites? I have a blog centered on the same topics you
    discuss and would love to have you share some stories/information. I
    know my audience would enjoy your work. If you are even remotely
    interested, feel free to send me an e-mail.

    GD Star Rating
    loading...
  25. I think this is among the most vital info for me.
    And i’m glad reading your article. But should remark on few general
    things, The site style is wonderful, the articles is really excellent :
    D. Good job, cheers

    GD Star Rating
    loading...
  26. Hello, i think that i saw you visited my weblog thus i came to “return the favor”.I am trying to find things
    to improve my website!I suppose its ok to use some of your ideas!!

    GD Star Rating
    loading...
  27. What i do not understood is in reality how you’re not really a lot more well-favored than you
    might be now. You’re so intelligent. You recognize
    therefore considerably in the case of this matter,
    made me personally consider it from so many various angles.
    Its like men and women don’t seem to be interested except it’s
    one thing to do with Lady gaga! Your personal stuffs great.
    At all times handle it up!

    GD Star Rating
    loading...
  28. What’s up to all, how is all, I think every one is getting more from this web page, and your
    views are nice in support of new people.

    GD Star Rating
    loading...
  29. Your style is really unique compared to other people I’ve read
    stuff from. Thank you for posting when you have the opportunity, Guess I will just book mark this blog.

    GD Star Rating
    loading...
  30. Hmm it seems like your blog ate my first comment (it was
    super long) so I guess I’ll just sum it up what I wrote
    and say, I’m thoroughly enjoying your blog. I as well am an aspiring blog writer
    but I’m still new to the whole thing. Do you have any points
    for inexperienced blog writers? I’d certainly appreciate it.

    GD Star Rating
    loading...
  31. ATTENTION -FOR ADULTS ONLY!!!Just letting you know this great new
    app with sex videos and picutures, NO ads, NO bullshit- Dirty Monika
    Android App. Get it for free with the link here. Enjoy responsibly

    GD Star Rating
    loading...
  32. Great goods from you, man. I’ve understand your stuff previous to and you’re just extremely great.
    I really like what you have acquired here, really like what you are stating and the way in which
    you say it. You make it entertaining and you still care for to keep it smart.

    I can not wait to read much more from you. This is actually a tremendous site.

    GD Star Rating
    loading...
  33. Hi there very nice web site!! Man .. Beautiful .. Amazing ..
    I will bookmark your blog and take the feeds also? I am glad to search
    out numerous useful info right here within the publish,
    we need develop more techniques in this regard, thanks for sharing.
    . . . . .

    GD Star Rating
    loading...
  34. I visited several websites except the audio quality for audio songs
    present at this site is genuinely wonderful.

    GD Star Rating
    loading...

Trackbacks

  1. [...] This post was mentioned on Twitter by unix player, unix player, Mitchell Muco, Andy Sholomon, angryjesters and others. angryjesters said: RT @jonisick: What's the Deal With Quantized Congestion Notification (QCN) http://bit.ly/aBnHqw < love the keg stand reference [...]

  2. [...] him the expertise to clearly articulate the challenges and benefits to practitioners.  Joe’s recent post on QCN is a good example of a neutral examination of the technology and how it applies to the FCoE [...]

  3. [...] been freed.  For more information on this see the buffer credits section of my previous post: http://www.definethecloud.net/whats-the-deal-with-quantized-congestion-notification-qcn.&#160; This mechanism ensures that a device never sends a frame that the receiving device does not [...]

  4. [...] blog battle over FCoE’s need for QCN rages on. Joe Onisick does a good job of explaining QCN and why it might/might not be necessary, so if you’re unfamiliar with the debate that’s a good place to start. Ivan Pepelnjak [...]

  5. [...] was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such [...]

  6. [...] management, and congestion management. With the first two of these now widely available and the third following shortly, Ethernet is ready to take center [...]

  7. zappos womens sandals size 6

    What’s the deal with Quantized Congestion Notification (QCN) — Define The Cloud

    GD Star Rating
    loading...

Speak Your Mind

*