FCoE initialization Protocol (FIP) Deep Dive

In an attempt to clarify my future posts I will begin categorizing a bit.  The following post will be part of a Technical Deep Dive series.

Fibre Channel over Ethernet (FCoE) is a protocol designed to move native Fibre Channel over 10 Gigabit Ethernet and above links, I’ve described the protocol in a previous post (http://www.definethecloud.net/?p=80.)  In order for FCoE to work we need a mechanism to carry the base Fibre Channel port / device login mechanisms over Ethernet.  These are the processes for a port to login and obtain a routable Fibre Channel Address.  Let’s start with some background and definitions:

DCB Data Center Bridging
FC Native Fibre Channel Protocol
FCF Fibre Channel Forwarder (an Ethernet switch capable of handling Encapsulation/De-encapsulation of FCoE frames and some or all FC services)
FCID Fibre Channel ID (24 Bit Routable address)
FCoE Fibre Channel over Ethernet
FC-MAP A 24-Bit value identifying an individual fabric
FIP FCoE Initialization Protocol
FLOGI FC Fabric Login
FPMA Fabric Provided MAC Address
PLOGI FC Port Login
PRLI Process Login
SAN Storage Area Network (switching infrastructure)
SCSI Small Computer Systems Interface
 
Now for the background, you’ll never grasp FIP properly if you don’t first get the fundamentals of FC:
 
N_Port Initialization
image

 

When a node comes online it’s port is considered an N_port.  When an N_port connects to the SAN it will connect to a switch port defined as a Fabric Port F_Port (this assumes your using a switched fabric.)  All N_ports operate the same way when they are brought online:

  1. FLOGI – Used to obtain a routable FCID for use in FC frame exchange.  The switch will provide the FCID during a FLOGI exchange.
  2. PLOGI – Used to register the N_Port with the FC name server

At this point a targets (disk or storage array) job is done, they can now sit and wait for requests.  An initiator (server) on the other hand needs to perform a few more tasks to discover available targets:

  1. Query – Request available targets from the FC name server, zoning will dictate which targets are available.
  2. PLOGI – A 2nd port Login, this time into the target port.
  3. PRLI – Process login to exchange supported upper layer protocols (ULP) typically SCSI-3.

Once this process has been completed the initiator can exchange frames with the target, i.e. the server can write to disk.

FIP:

The reason the FC login process is key to understanding FIP is that this is the process that FIP is handling for FCoE networks.  FIP allows an Ethernet attached FC node (Enode) to discover existing FCFs and supports the FC login procedure over 10+GE networks.  Rather than just providing an FCID, FIP will provide an FPMA which is a MAC address comprised of two parts: FC-MAP and FCID.

48 bit FCMAP (Mac Address)

image

FIP

image

So FIP provides an Ethernet MAC address used by FCoE to traverse the Ethernet network which contains the FCID required to be routed on the FC network.  FIP also passes the query and query response from the FC name server.  FIP uses a separate Ethertype from FCoE and its frames are standard Ethernet size (1518 Byte 802.1q frame) whereas FCoE frames are 2242 Byte Jumbo Frames.

FIP Snooping:

FIP snooping is used in multi-hop FCoE environments.  FIP snooping is a frame inspection method that can be used by FIP snooping capable DCB devices to monitor FIP frames and apply policies based on the information in those frames.  This allows for:

  • Enhanced FCoE security (Prevents FCoE MAC spoofing.)
  • Creates FC point-to-point links within the Ethernet LAN
  • Allows auto-configuration of ACLs based on name server information read in the FIP frames

FIP Snooping

image

Summary:

FIP snooping uses dynamic Access Control Lists to enforce Fibre Channel rules within the DCB Ethernet network.  This prevents Enodes from seeing or communicating with other Enodes without first traversing an FCF.

Feedback, corrections, updates, questions?

GD Star Rating
loading...

Fibre Channel over Ethernet

Fibre Channel over Ethernet (FCoE) is a protocol standard ratified in June of 2009.  FCoE provides the tools for encapsulation of Fibre Channel (FC) in 10 Gigabit Ethernet frames.  The purpose of FCoE is to allow consolidation of low-latency, high performance FC networks onto 10GE infrastructures.  This allows for a single network/cable infrastructure which greatly reduces switch and cable count, lowering the power, cooling, and administrative requirements for server I/O.

FCoE is designed to be fully interoperable with current FC networks and require little to no additional training for storage and IP administrators. FCoE operates by encapsulating native FC into Ethernet frames.  Native FC is considered a ‘lossless’ protocol, meaning frames are not dropped during periods of congestion.  This is by design in order to ensure the behavior expected by the SCSI payloads.  Traditional Ethernet does not provide the tools for lossless delivery on shared networks so enhancements were defined by the IEEE to provide appropriate transport of encapsulated Fibre Channel on Ethernet networks.  These standards are known as Data Center Bridging (DCB) which I’ve discussed in a previous post (http://www.definethecloud.net/?p=31.)  These Ethernet enhancements are fully backward compatible with traditional Ethernet devices, meaning DCB capable devices can exchange standard Ethernet frames seamlessly with legacy devices.  The full 2148 Byte FC frame is encapsulated in an Ethernet jumbo frame avoiding any modification/fragmentation of the FC frame.

FCoE itself takes FC layers 2-4 and maps them to Ethernet layers 1-2, this replaces the FC-0 Physical layer, and FC-1 Encoding Layer.  This mapping between Ethernet and Fibre Channel is done through a Logical End-Point (LEP) which can by thought of as a translator between the two protocols.  The LEP is responsible for providing the appropriate encoding and physical access for frames traveling from FC nodes to Ethernet nodes and vice versa.  There are two devices that typically act as FCoE LEPs: Fibre Channel Forwarders (FCF) which are switches capable of both Ethernet and Fibre Channel, and Converged Network Adapters (CNA) which provide the server-side connection for a FCoE network.  Additionally the LEP operation can be done using a software initiator and traditional 10GE NICs but this places extra workload on the server processor rather than offloading it to adapter hardware.

One of the major advantages of replacing FC layers 0-1 when mapping onto 10GE is the encoding overhead.  8GB Fibre Channel uses an 8/10 bit encoding which adds 25% protocol overhead, 10GE uses a 64/64 bit encoding which has about 2% overhead, dramatically reducing the protocol overhead and increasing throughput.  The second major advantage is that FCoE maintains FC layers 2-4 which allows seamless integration with existing FC devices and maintains the Fibre Channel tool set such as zoning, LUN masking etc.  In order to provide FC login capabilities, multi-hop FCoE networks, and FC zoning enforcement on 10GE networks FCoE relies on another standard set known as Fibre Channel initialization Protocol (FIP) which I will discuss in a lter post.

Overall FCoE is one protocol to choose from when designing converged networks, or cable-once architectures.  The most important thing to remember is that a true cable-once architecture doesn’t make you choose your Upper Layer Protocol (ULP) such as FCoE, only your underlying transport infrastructure.  If you choose 10GE the tools are now in place to layer any protocol of your choice on top, when and if you require it.

Thanks to my colleagues who recently provided a great discussion on protocol overhead and frame encoding…

GD Star Rating
loading...

Data Center Bridging Exchange

Data Center Bridging Exchange (DCBX) is one of the components of the DCB standards.  These standards offer enhancements to standard ethernet which are backwards compatible with traditional Ethernet and provide support for I/O Consolidation (http://www.definethecloud.net/?p=18.)  The three purposes of DCBX are:

Discovery of DCB capability:

The ability for DCB capable devices to discover and identify capabilities of DCB peers as well as identify non-DCB capable legacy devices.  You can find more information on DCB in a previous post (http://www.definethecloud.net/?p=31.)

Identification of misconfigured DCB features:

The ability to discover misconfiguration of features that require symmetric configuration between DCB peers.  Some DCB features are asymmetric meaning they can be configured differently on each end of a link, other features must match on both sides to be effective (symmetric.)  This functionality allows detection of configuration errors for these symmetric features.

Configuration of Peers:

A capability allowing DCBX to pass configuration information to a peer.  For instance a DCB capable switch can pass Priority Flow Control (PFC) information on to a Converged Network Adapter (CNA) to ensure FCoE traffic is appropriately tagged and pause is enabled for the chosen Class of Service (CoS) value.  This PFC exchange is a symmetric exchange and must match on both sides of the link.  DCB features such as Enhanced Transmission Selection (ETS) otherwise known as bandwidth management can be configured asymmetrically (different on each side of the link.)

DCBX relies on Link Level Discovery Protocol (LLDP) in order to pass this information and configuration.  LLDP is an industry standard version of Cisco Discovery Protocol (CDP) which allows devices to discover one another and exchange information about basic capabilities.  Because DCBX relies on LLDP and is an acknowledged protocol (2 way communication) any link which is intended to support DCBX must have LLDP enabled on both sides of the link for Tx/Rx.  When a port has LLDP disabled for either Rx or Tx DCBX is disabled on the port and DCBX Type Length Values (TLV) within received LLDP frames will be ignored.

DCBX capable devices should have DCBX enabled by default with the ability to administratively disable it.  This allows for more seamless deployments of DCB networks with less tendency for error.

GD Star Rating
loading...