What’s the deal with Quantized Congestion Notification (QCN)

For the last several months there has been a lot of chatter in the blogosphere and Twitter about FCoE and whether full scale deployment requires QCN.  There are two camps on this:

  1. FCoE does not require QCN for proper operation with scale.
  2. FCoE does require QCN for proper operation and scale.

Typically the camps break down as follows (there are exceptions) :

  1. HP camp stating they’ve not yet released a suite of FCoE products because QCN is not fully ratified and they would be jumping the gun.  The flip side of this is stating that Cisco did jumped the gun with their suite of products and will have issues with full scale FCoE.
  2. Cisco camp stating that QCN is not required for proper FCoE frame flow and HP is using the QCN standard as an excuse for not having a shipping product.

For the purpose of this post I’m not camping with either side, I’m not even breaking out my tent.  What I’d like to do is discuss when and where QCN matters, what it provides and why.  The intent being that customers, architects, engineers etc. can decide for themselves when and where they may need QCN.

QCN: QCN is a form of end-to-end congestion management defined in IEEE 802.1.Qau.  The purpose of end-to-end congestion management is to ensure that congestion is controlled from the sending device to the receiving device in a dynamic fashion that can deal with changing bottlenecks.  The most common end-to-end congestion management tool is TCP Windows sizing.

TCP Window Sizing:

With window sizing TCP dynamically determines the number of frames to send at once without an acknowledgement.  It continuously ramps this number up dynamically if the pipe is empty and acknowledgements are being received.  If a packet is dropped due to congestion and an acknowledgement is not received TCP halves the window size and starts the process over.  This provides a mechanism in which the maximum available throughput can be achieved dynamically.

Below is a diagram showing the dynamic window size (total packets sent prior to acknowledgement) over the course of several round trips.  You can see the initial fast ramp up followed by a gradual increase until a packet is lost, from there the window is reduced and the slow ramp begins again.

image If you prefer analogies I always refer to TCP sliding windows as a Keg Stand (http://en.wikipedia.org/wiki/Keg_stand.)

File:Kegstand147.jpg

In the photo we see several gentleman surrounding a keg, with one upside down performing a keg stand.

To perform a keg stand:

  • Place both hands on top of the keg
  • 1-2 Friend(s) lift your feet over your head while you support your body weight on locked-out arms
  • Another friend places the keg’s nozzle in your mouth and turns it on
  • You swallow beer full speed for as long as you can

What the hell does this have to do with TCP Flow Control? I’m so glad you asked.

During a keg stand your friend is trying to push as much beer down your throat as it can handle, much like TCP increasing the window size to fill the end-to-end pipe.  Both of your hands are occupied holding your own weight, and your mouth has a beer hose in it, so like TCP you have no native congestion signaling mechanism.  Just like TCP the flow doesn’t slow until packets/beer drops, when you start to spill they stop the flow.

So that’s an example of end-to-end congestion management.  Within Ethernet and FCoE specifically we don’t have any native end-to-end congestion tools (remember TCP is up on L4 and we’re hanging out with the cool kids at L2.)  No problem though because We’re talking FCoE right?  FCoE is just a L1-L2 replacement for Fibre Channel (FC) L0-L1, so we’ll just use FC end-to-end congestion management… Not so fast boys and girls, FC does not have a standard for end-to-end congestion management, that’s right our beautiful over engineered lossless FC has no mechanism for handling network wide, end-to-end congestion.  That’s because it doesn’t need it.

FC is moving SCSI data, and SCSI is sensitive to dropped frames, latency is important but lossless delivery is more important.  To ensure a frame is never dropped FC uses a hop-by-hop flow control known as buffer-to-buffer (B2B) credits. At a high level each FC device knows the amount of buffer spaces available on the next hop device based on the agreed upon frame size (typically 2148 bytes.)  This means that a device will never send a frame to a next hop device that cannot handle the frame.  Let’s go back to the world of analogy.

Buffer-to-buffer credits:

The B2B credit system works in the same method you’d have 10 Marines offload and stack a truckload of boxes (‘fork-lift, we don’t need no stinking forklift.’)  The best system to utilize 10 Marines to offload boxes is to line them up end-to-end one in the truck and one on the other end to stack.  Marine 1 in the truck initiates the send by grabbing a box and passing it to Marine 2, the box moves down the line until it gets to the target Marine 10 who stacks it.  Before any Marine hands another Marine a box they look to ensure that Marines hands are empty verifying they can handle the box and it won’t be dropped.  Boxes move down the line until they are all offloaded and stacked.  If anyone slows down or gets backed up each marine will hold their box until the congestion is relieved.

In this analogy the Marine in the truck is the initiator/server and the Marine doing the stacking is the target/storage with each Marine in between being a switch.

When two FC devices initiate a link they follow the Link-Initialization-Protocols (LIP.)  During this process they agree on an FC frame size and exchange the available dedicated frame buffer spaces for the link.  A sender is always keeping track of available buffers on the receiving side of the link.  The only real difference between this and my analogy is each device (Marine) is typically able to handle more than one frame (box) at once.

So if FC networks operate without end-to-end congestion management just fine why do we need to implement a new mechanism in FCoE, well there-in lies the rub.  Do we need QCN?  The answer is really Yes and No, and it will depend on design.  FCoE today provides the exact same flow control as FC using two standards defined within Data Center Bridging (DCB) these are Enhanced Transmission Selection (ETS) and Priority-Flow Control (PFC) for more info on theses see my DCB blog: http://www.definethecloud.net/?p=31.)  Basically ETS provides a bandwidth guarantee without limiting and PFC provides lossless delivery on an Ethernet network.

Why QCN:

The reason QCN was developed is the differences between the size, scale, and design of FC and Ethernet networks.  Ethernet networks are usually large mesh or partial mesh type designs with multiple switches.  FC designs fall into one of three major categories Collapsed core (single layer), Core edge (two layer) or in rare cases for very large networks edge-core-edge (three layer.)  This is because we typically have far fewer FC connected devices than we do Ethernet (not every device needs consolidated storage/backup access.)

If we were to design our FCoE networks where every current Ethernet device supported FCoE and FCoE frames flowed end-to-end QCN would be a benefit to ensure point congestion didn’t clog the entire network.  On the other hand if we maintain similar size and design for FCoE networks as we do FC networks, there is no need for QCN.

Let’s look at some diagrams to better explain this:

image

 image In the diagrams above we see a couple of typical network designs.  The Ethernet diagram shows Core at the top, aggregation in the middle, and edge on the bottom where servers would connect.  The Fibre Channel design shows a core at the top with an edge at the bottom.  Storage would attach to the core and servers would attach at the bottom.  In both diagrams I’ve also shown typical frame flow for each traffic type.  Within Ethernet, servers commonly communicate with one another as well network file systems, the WAN etc.  In an FC network the frame flow is much more simplistic, typically only initiator target (server to storage) communication occurs.  In this particular FC example there is little to no chance of a single frame flow causing a central network congestion point that could effect other flows which is where end-to-end congestion management comes into play.

What does QCN do:

QCN moves congestion from the network center to the edge to avoid centralized congestion on DCB networks.  Let’s take a look at a centralized congestion example (FC only for simplicity):

image In the above example two 2Gbbps hosts are sending full rate frame flows to two storage devices.  One of the storage devices is a 2Gbps device and can handle the full speed, the other is a 1Gbps device and is not able to handle the full speed. If these rates are sustained switch 3’s buffers will eventually fill and cause centralized congestion effecting frame flows to both switch 4, and 5.  This means that the full rate capable devices would be affected by the single slower device.  QCN is designed to detect this type of congestion and push it to the edge, therefore slowing the initiator on the bottom right avoiding overall network congestion.

This example is obviously not a good design and is only used to illustrate the concept.  In fact in a properly designed FC network with multiple paths between end-points central congestion is easily avoidable.

When moving to FCoE if the network is designed such that FCoE frames pass through the entire full-mesh network shown in the Common Ethernet design above, there would be greater chances of central congestion.  If the central switches were DCB capable but not FCoE Channel Forwarders (FCF) QCN could play a part in pushing that congestion to the edge.

If on the other hand you design FCoE in a similar fashion to current FC networks QCN will not be necessary.  An example of this would be:

imageThe above design incorporates FCoE into the existing LAN Core, Aggregation, Edge design without clogging the LAN core with unneeded FCoE traffic.  Each server is dual connected to the common Ethernet mesh, and redundantly connected to FCoE SAN A and B.  This design is extremely scalable and will provide more than enough ports for most FCoE implementations.

Summary:

QCN like other congestion management tools before it such as FECN and BECN have significant use cases.  As far as FCoE deployments go QCN is definitely not a requirement and depending on design will provide no benefit for FCoE.  It’s important to remember that the DCB standards are there to enhance Ethernet as a whole, not just for FCoE.  FCoE utilizes ETS and PFC for lossless frame delivery and bandwidth control, but the FCoE standard is a separate entity from DCB.

Also remember that FCoE is an excellent tool for virtualization which reduces physical server count.  This means that we will continue to require less and less FCoE ports overall especially as 40Gbps and 100Gbps are adopted.  Scaling FCoE networks further than today’s FC networks will most likely not be a requirement.

GD Star Rating
loading...
What’s the deal with Quantized Congestion Notification (QCN), 4.8 out of 5 based on 12 ratings

Comments

  1. Financial position is getting tougher day-by-day even for the people with great credit record.
    You must decide if you want to make long term income, by
    working with a company that has products that create
    residual income or if you just want to find quick ways to make money.
    Note that this is not a comprehensive list by any means,
    and you could probably convert any special skill you have into a freelancing career.

    GD Star Rating
    loading...
  2. Hello there, just became aware of your blog through Google, and
    found that it is truly informative. I’m gonna watch out for brussels.

    I’ll appreciate if you continue this in future. Numerous people will be benefited from your writing.
    Cheers!

    GD Star Rating
    loading...
  3. I just like the helpful information you supply for your articles.

    I’ll bookmark your blog and check once more right here regularly.

    I am slightly certain I’ll learn lots of new stuff right here!
    Good luck for the following!

    GD Star Rating
    loading...
  4. I am in fact thankful to the owner of this web site who has shared this wonderful article at at this time.

    GD Star Rating
    loading...
  5. After looking at a handful of the articles on your web page, I
    truly appreciate your way of writing a blog. I book marked it to my bookmark site list
    and will be checking back in the near future. Please visit my web site as well and tell me how you feel.

    GD Star Rating
    loading...
  6. Today, while I was at work, my cousin stole my iphone and tested to see
    if it can survive a forty foot drop, just so she can be a youtube sensation.
    My iPad is now destroyed and she has 83 views. I know this is completely off topic but I had to share it with someone!

    GD Star Rating
    loading...
  7. It acts immediately on all your garage concerns as quickly
    as possible. Steel garage doors cost about $600 for
    the most basic variety and there are many more styles which are priced accordingly.
    If your garage door is old and worn out, a new one can do a lot to improve the look of your home.

    GD Star Rating
    loading...
  8. Hello! I could have sworn I’ve been to this site before but after reading through some of the post
    I realized it’s new to me. Anyhow, I’m definitely delighted I found it and I’ll be book-marking and checking back often!

    GD Star Rating
    loading...
  9. Terrific work! This is the type of info that are
    meant to be shared across the web. Disgrace on Google for
    not positioning this submit upper! Come on over and
    visit my website . Thank you =)

    GD Star Rating
    loading...
  10. Hi would you mind letting me know which web host you’re using?

    I’ve loaded your blog in 3 different web browsers and I must
    say this blog loads a lot quicker then most.
    Can you suggest a good internet hosting provider at a fair price?
    Cheers, I appreciate it!

    GD Star Rating
    loading...
  11. You made your point!

    GD Star Rating
    loading...
  12. Good day! Would you mind if I share your blog with my facebook
    group? There’s a lot of people that I think would really enjoy your content.
    Please let me know. Thanks

    GD Star Rating
    loading...
  13. When someone writes an post he/she keeps the image of a user
    in his/her brain that how a user can know it.
    Therefore that’s why this paragraph is outstdanding.

    Thanks!

    GD Star Rating
    loading...
  14. Regardless of what style you prefer, the quality is something that really counts when keeping your home safe and secure.

    One of its advantages is that it is less expensive than the three other types of garage
    doors. Many home improvement stores carry Wayne Dalton parts and doors so they
    would have additional resources for you.

    GD Star Rating
    loading...
  15. Comment Porter Les Baskets Isabel Marant Mercato ha più
    di 40 fornitori che offrono una varietà di un’impresa fresco, cibo di produzione locale a
    base, elementi hndlagde, e una concessione stare quel ristorante diritto fondamentale, gli uomini hanno
    un sapore di casa. Il mercato comprende spettacoli gratuiti, dimostrazioni
    Guest Chef con ingredienti freschi della fattoria, attività per bambini e corsi di salute, ernring
    e giardinaggio. Isabel Marant Trainers Uk Rebagz Tutti i messaggeri sono in grado di ospitare la maggior parte dei computer
    portatili con schermo fino a 11, con dimensioni vano notebook di 13 x 12 x 4 Un altro motivo sono ottimi modi per borse per notebook funky per
    gli uomini è la pletora di tasche e slot, sia all’interno che fuori.
    Inoltre, considerare il fatto che essi sono progettati sacchetti di riso riciclati
    a 100 chilogrammi di riso erano per mantenere puliti e asciutti, sono fatti in modo
    da sapere che queste borse per notebook funky Phillipino sono costruiti per durare nonostante
    il fatto che pesa solo solo il 14 grammi.. Isabel Marant Wedge Sneakers Cheap Sono deluso che il senatore Nelson mette
    lealtà verso l’ala liberale del suo partito di fronte Florida.

    Il [House] bolletta tagliare spese inutili, dare un sollievo importante senza aumentare il deficit ed
    evitare un arresto di governo e il senatore Nelson ha votato contro perché taglia in finanziamenti per programmi come Solyndra debacle?.
    isabel marant sneakers sale Dopo aver lavorato per diversi mesi,
    George ha deciso che gli piaceva la campagna della Virginia, e aveva
    alcuni diritti quatters terra? insieme a diversi altri nuovi arrivati.
    La sua casa di registro è stata attaccata più volte da indiani,
    per lo più a caccia squadre in sei-otto uomini, ma George è stato
    in grado di ottenere il suo fucile a pietra focaia e sparare un colpo contro di loro.
    Isabel Marant Sneakers Original Tutto ciò che può realizzare un profitto
    in modo che non dipendono da un uomo. L’unica cosa Desember sente manca un padre di chiamarli.
    New Isabel Marant Sneakers 2014 Ralph Rucci, un artista che succede a lavorare nel campo
    della moda, vincendo uno dei pochi sollievo applaudirono in settimana eccezionale per fare strutture complesse e la sua bellezza estetica speciale completamente soggezione ispiratore.
    (Le donne si lamentano di caduta reale collezioni ecstasy).
    Isabel Marant Uk Sale Quindi cestino di Pasqua ha cose come cerotti, garze,
    nastro, iodio, alcool, Benedryl, impacchi di ghiaccio ghiaccio istantaneo può schiacciare e sono fredde, una piastra elettrica, medicine per la tosse, strofinare vapore Vicks (che non è migliorata con questo?) E un pianta
    di aloe. CVS si amano vedermi battere le sue porte ed è
    molto comodo cestino è sicuro di durare più di caramelle.
    Isabel Marant Sandals Copy Mi limito il mio
    latticini e l’assunzione di uova in modo significativo e li mangio solo in casi molto rari.
    Ho un certo numero di motivi per fare queste scelte, soprattutto perché mi oppongo alle grandi metodi di coltivazione commerciali utilizzati per la
    produzione della nostra carne, uso del suolo
    irresponsabile e danno ambientale, per non parlare di danni
    economici LongRange che è stato l’agricoltore famiglia.

    Isabel Marant Boots Replica Etoile Isabel Marant Lederjacke

    GD Star Rating
    loading...
  16. Regardless of what style you prefer, the quality is something that really counts when keeping your home
    safe and secure. One of its advantages is that it is
    less expensive than the three other types of garage
    doors. Rolling shutters of ROMA have a patented sound body and
    are therefore pleasantly quiet.

    GD Star Rating
    loading...
  17. minecraft gift code minecraft pobierz za darmo chomikuj

    GD Star Rating
    loading...
  18. These mirrors would extend almost to the top of the parlor wall
    to the crown moldings. The Palette Car Pendants Socks Christmas
    stocking is among the many stockings available and comes in the best design and color which is
    red and sparkly green. Christmas is the time when many of us like to decorate as much as we can.

    GD Star Rating
    loading...
  19. Thanks for sharing your info. I truly appreciate
    your efforts and I am waiting for your next post thank you once again.

    GD Star Rating
    loading...
  20. Inside the eye, Vitamin C acts as a natural UV filter, while the macula
    lutea (‘yellow spot’) and the aging crystalline lens (which yellows
    with age) provide blue-blocking light filtration which may naturally slow the retinal aging process, as anti-oxidant defences dwindle.
    Cevitamic acid content of the blood and urine of subjects with senile cataracts.
    Pharmaceutical grade fish oil is not all that it
    has been cracked up to be, and I wouldn.

    GD Star Rating
    loading...
  21. Hi my friend! I want to say that this post is amazing, nice written and
    come with almost all significant infos. I’d like to see extra posts like this .

    GD Star Rating
    loading...
  22. It’s impressive that you are getting ideas from this piece of writing as well as from our
    dialogue made at this place.

    GD Star Rating
    loading...
  23. I all the time used to study piece of writing in
    news papers but now as I am a user of web thus from
    now I am using net for posts, thanks to web.

    GD Star Rating
    loading...
  24. Go to “File” select “Preferences,” go to “Landscape Movement” and change
    the sensitivity multiplier to 5. You now have all the objects and cells from
    the world, and you can start exploring the Render, Object and the Cell View windows.
    The rewards of increased tennis racket length are enhanced
    reach and elevated power.

    GD Star Rating
    loading...
  25. Why people still make use of to read news papers when in this technological world
    everything is existing on net?

    GD Star Rating
    loading...
  26. Hi there! Do you know if they make any plugins to assist
    with Search Engine Optimization? I’m trying to get my blog to
    rank for some targeted keywords but I’m not seeing very good success.
    If you know of any please share. Appreciate it!

    GD Star Rating
    loading...
  27. Its like you read my mind! You seem to know a lot about this,
    like you wrote the book in it or something. I think that you could do with a few pics to drive the message home a bit, but other than that, this is fantastic blog.
    A fantastic read. I’ll definitely be back.

    GD Star Rating
    loading...
  28. When we get there, even though I have a gang of band dudes with me, my Italian friend jumps out
    again, points to my guitar case and says:. Many NH roofers usually design flat roofs with fall arrest lifelines.

    The music itself is sincere, I understand the essence even
    though the syntax doesn’t inspire me.

    GD Star Rating
    loading...
  29. The Bible tells us that Israel marched to songs in the wilderness immediately after fleeing
    Egypt. Or was it instead a happy dark night where you sat near a bonfire
    with your friends eating marshmallows. The comprehensive lyrics database
    is an opportunity to learn the lyrics of your favorite song.

    GD Star Rating
    loading...
  30. chanel Handbags buy
    Ahaa, its nice conversation regarding this piece of writing at this place at this
    website, I have read all that, so now me also commenting at this place.

    GD Star Rating
    loading...
  31. Hello there! I could have sworn I’ve visited your blog before but after going through some of the posts I
    realized it’s new to me. Anyhow, I’m certainly happy I discovered it
    and I’ll be bookmarking it and checking back frequently!

    GD Star Rating
    loading...
  32. I constantly emailed this webpage post page to all my contacts, as if like to read it
    next my links will too.

    GD Star Rating
    loading...
  33. Hi there, its fastidious post on the topic of media print,
    we all understand media is a wonderful source of facts.

    GD Star Rating
    loading...
  34. You are so awesome! I do not suppose I’ve truly read anything like this before.
    So nice to find somebody with original thoughts on this issue.

    Really.. thanks for starting this up. This web site is one thing that is required on the internet, someone with a little
    originality!

    GD Star Rating
    loading...
  35. I think this is one of the most vital info for me. And i’m glad reading your article.
    But wanna remark on few general things, The web site style is great, the articles is really
    great : D. Good job, cheers

    GD Star Rating
    loading...
  36. Pretty! This has been an extremely wonderful post. Thank you for supplying
    this information.

    GD Star Rating
    loading...
  37. Nice post. I was checking constantly this blog and I’m impressed!
    Very helpful info specifically the last part :) I care for such information a lot.
    I was looking for this particular info for a
    long time. Thank you and good luck.

    GD Star Rating
    loading...
  38. I’m not sure exactly why but this website is loading incredibly slow for me.
    Is anyone else having this problem or is it a problem on my end?
    I’ll check back later and see if the problem still exists.

    GD Star Rating
    loading...
  39. It is addressed to both men and women in good health who want to increase their
    muscle mass. Hidden lactose intolerance may also cause flare-ups.
    Here’s some great foods that contain ‘healthy fats’ that
    you can incorporate into your diet.

    GD Star Rating
    loading...
  40. My relatives every time say that I am killing my time here
    at web, however I know I am getting knowledge daily by reading thes fastidious articles.

    GD Star Rating
    loading...
  41. hi!,I like your writing very much! percentage we be in contact
    more about your article on AOL? I need an expert in this
    house to solve my problem. May be that is you! Looking forward to look you.

    GD Star Rating
    loading...
  42. Hi there, I read your blogs like every week.
    Your humoristic style is awesome, keep it up!

    GD Star Rating
    loading...
  43. If there is any lime accumulation on the head of Viserys Targaryen
    causing him to writhe and scream in unspeakable pain.
    These woods are durable and spray concealed batteries easy to use.
    I feel guilty for being so naive. You can also feel which
    pipe is broken by feeling whether the water comes out fast and
    splatters everywhere.

    GD Star Rating
    loading...
  44. I am truly grateful to the owner of this web site who has shared this enormous paragraph at here.

    GD Star Rating
    loading...
  45. Greetings from Idaho! I’m bored to tears at work so I decided to check out your website on my iphone during lunch break.
    I love the info you present here and can’t wait to take a look when I get home.
    I’m surprised at how fast your blog loaded on my phone
    .. I’m not even using WIFI, just 3G .. Anyways, great blog!

    GD Star Rating
    loading...
  46. If you are going for most excellent contents like I do, simply pay a quick visit this website everyday since it presents feature contents, thanks

    GD Star Rating
    loading...
  47. I pay a quick visit everyday some web sites and websites to read posts, but this weblog
    offers feature based writing.

    GD Star Rating
    loading...
  48. I’m really impressed with your writing skills as well as with the
    layout on your blog. Is this a paid theme or did you modify it
    yourself? Either way keep up the nice quality writing, it is rare to see a nice blog like this one these days.

    GD Star Rating
    loading...
  49. Hello! I just wish to give you a big thumbs up for the great information you’ve got here on this post.
    I am returning to your site for more soon.

    GD Star Rating
    loading...
  50. Excellent write-up. I definitely appreciate this website.
    Keep it up!

    GD Star Rating
    loading...

Trackbacks

  1. [...] This post was mentioned on Twitter by unix player, unix player, Mitchell Muco, Andy Sholomon, angryjesters and others. angryjesters said: RT @jonisick: What's the Deal With Quantized Congestion Notification (QCN) http://bit.ly/aBnHqw < love the keg stand reference [...]

  2. [...] him the expertise to clearly articulate the challenges and benefits to practitioners.  Joe’s recent post on QCN is a good example of a neutral examination of the technology and how it applies to the FCoE [...]

  3. [...] been freed.  For more information on this see the buffer credits section of my previous post: http://www.definethecloud.net/whats-the-deal-with-quantized-congestion-notification-qcn.&#160; This mechanism ensures that a device never sends a frame that the receiving device does not [...]

  4. [...] blog battle over FCoE’s need for QCN rages on. Joe Onisick does a good job of explaining QCN and why it might/might not be necessary, so if you’re unfamiliar with the debate that’s a good place to start. Ivan Pepelnjak [...]

  5. [...] was surprised to see that Brocade and QLogic list 802.1Qau Congestion Management (QCN) as supported. I didn’t realize anyone supported this spec yet, or indeed that there was such [...]

  6. [...] management, and congestion management. With the first two of these now widely available and the third following shortly, Ethernet is ready to take center [...]

Speak Your Mind

*