Fibre Channel over Ethernet (FCoE) is a protocol standard ratified in June of 2009. FCoE provides the tools for encapsulation of Fibre Channel (FC) in 10 Gigabit Ethernet frames. The purpose of FCoE is to allow consolidation of low-latency, high performance FC networks onto 10GE infrastructures. This allows for a single network/cable infrastructure which greatly reduces switch and cable count, lowering the power, cooling, and administrative requirements for server I/O.
FCoE is designed to be fully interoperable with current FC networks and require little to no additional training for storage and IP administrators. FCoE operates by encapsulating native FC into Ethernet frames. Native FC is considered a ‘lossless’ protocol, meaning frames are not dropped during periods of congestion. This is by design in order to ensure the behavior expected by the SCSI payloads. Traditional Ethernet does not provide the tools for lossless delivery on shared networks so enhancements were defined by the IEEE to provide appropriate transport of encapsulated Fibre Channel on Ethernet networks. These standards are known as Data Center Bridging (DCB) which I’ve discussed in a previous post (http://www.definethecloud.net/?p=31.) These Ethernet enhancements are fully backward compatible with traditional Ethernet devices, meaning DCB capable devices can exchange standard Ethernet frames seamlessly with legacy devices. The full 2148 Byte FC frame is encapsulated in an Ethernet jumbo frame avoiding any modification/fragmentation of the FC frame.
FCoE itself takes FC layers 2-4 and maps them to Ethernet layers 1-2, this replaces the FC-0 Physical layer, and FC-1 Encoding Layer. This mapping between Ethernet and Fibre Channel is done through a Logical End-Point (LEP) which can by thought of as a translator between the two protocols. The LEP is responsible for providing the appropriate encoding and physical access for frames traveling from FC nodes to Ethernet nodes and vice versa. There are two devices that typically act as FCoE LEPs: Fibre Channel Forwarders (FCF) which are switches capable of both Ethernet and Fibre Channel, and Converged Network Adapters (CNA) which provide the server-side connection for a FCoE network. Additionally the LEP operation can be done using a software initiator and traditional 10GE NICs but this places extra workload on the server processor rather than offloading it to adapter hardware.
One of the major advantages of replacing FC layers 0-1 when mapping onto 10GE is the encoding overhead. 8GB Fibre Channel uses an 8/10 bit encoding which adds 25% protocol overhead, 10GE uses a 64/64 bit encoding which has about 2% overhead, dramatically reducing the protocol overhead and increasing throughput. The second major advantage is that FCoE maintains FC layers 2-4 which allows seamless integration with existing FC devices and maintains the Fibre Channel tool set such as zoning, LUN masking etc. In order to provide FC login capabilities, multi-hop FCoE networks, and FC zoning enforcement on 10GE networks FCoE relies on another standard set known as Fibre Channel initialization Protocol (FIP) which I will discuss in a lter post.
Overall FCoE is one protocol to choose from when designing converged networks, or cable-once architectures. The most important thing to remember is that a true cable-once architecture doesn’t make you choose your Upper Layer Protocol (ULP) such as FCoE, only your underlying transport infrastructure. If you choose 10GE the tools are now in place to layer any protocol of your choice on top, when and if you require it.
Thanks to my colleagues who recently provided a great discussion on protocol overhead and frame encoding…