Up and Down the Network Stack

10 Gigabit Ethernet and VMware - A Match Made in Heaven

Thanks to Steve Grantham, who pointed me at a nice blog entry of the above title, which provides a good overview of the technology trends being driven by virtualisation.

Virtual Geek (aka Chad Sakac) says:

Back when I was in the valley (so this was 4 years ago), a buddy of mine worked at a tiny IP (not Internet Protocol - Intellectual Property) company focused on high-end IP blocks for networking and storage (going after Hi/fn and the others in that space).   He showed me their A0 spin, and told me "This chip will do 10GbE BaseT over Cat6 cables, full TCP offload including segment offload, and all iSCSI offload".  COOL. "oh and we think we can mass produce it for $25 per chip".   DROOL.   

Well, fast forward 4 years, and they are out of business :-)

..... GOOD news is that another company has been able to realise this device.

Go Green with Server Consolidation

You may have taken the glossy adverts with something of a pinch of salt, but PG&E (utility company) is serious about offering businesses rebates based reductions in power consumption by virtue of server consolidation. This rebate currently stands at 8c per kilowatt-hour reduction. See here for more details.

PG&E stipulate that servers must be decommissioned in order to gain the rebate. Perhaps they should look more closely at a scheme whereby the servers are re-provisioned either within the organisation or elsewhere in order to avoid the scheme being energy neutral (energy savings balanced by the cost of early replacement of existing servers).

BTW I'm hearing more and more that organizations are employing the migration feature provided by server virtualization in order to load balance for energy efficiency. Live migration of a complete operating system over a 10GBASE-T Ethernet network in total takes order 5seconds to complete  with a down-time measured in milli-seconds - clearly a powerful tool.

FCoE or iSCSI which way are you converging?

The recent FCOE announcements have clearly stirred the pot; with plenty of debate on whether FCOE is simply a smash and grab layer violation, or else is a technology with lasting utility. Thankfully I don’t need to take a public stance on the matter since Solarflare by definition supports all upper layer protocols over Ethernet and in particular supports the Data Center Ethernet (also known as CEE) protocols required for software based implementations of FCOE within its Solarstorm Ethernet controller products.

See:

http://www.eetimes.com/showArticle.jhtml?articleID=207100764

http://interconnects.blogspot.com/2008/04/moving-at-10g-rates.html

http://blocksandfiles.com/article/4773

http://blogs.cisco.com/datacenter/2008/04/the_antifcoe_sentiment.html

http://en.wikipedia.org/wiki/OSI_model

http://www.open-fcoe.org/

http://www.fcoe.com

It is however worth understanding the architectural differences between the current crop of FCOE announcements and also the difference between network convergence and unified networks. First, a quick look at the two FCOE architectures:

 

1. Software based FCOE is the natural model to adopt for a commodity LAN vendor who is converging on storage. Here the goal may be to inter-network with legacy FC equipment by running  a FC protocol stack on top of a regular Ethernet driver in a commodity  host operating system. The Ethernet controller looks just like a regular Ethernet controller, except that it is able to partake in the new Ethernet  congestion avoidance protocols currently being defined in the IEEE. The FCOE driver is handed an FCOE frame and needs to perform FC protocol.      Performance is likely to be worse than a hardware based solution because software is now doing FC protocol which would previously have been done by the FC controller and also because FC data integrity is now being done in software. (As an aside, data-integrity performance is becoming less of an issue with this architecture due to protocol neutral CRC support which is increasingly being found in hardware.) Also end customer acceptance is likely to be slow on the uptake because the software FC stack is immature and the FC community is rightly very conservative regarding system reliability and cross-vendor compatibility. However, software based FCOE has the      promise of highly integrated commodity silicon (for the server at least).

 

2. Hardware based FCOE is the natural model to adopt for a storage HBA vendor who is converging on LAN. Here the goal may be to cost-optimize a rack solution by running both FC and LAN      traffic using a single HBA and wire to a top of rack switch which connects out to FC and regular Ethernet networks. (Yes I know that IEEE is also defining inter-switch congestion avoidance protocols which would enable a multi-tier converged Ethernet network, but frankly these are a long, long way from any sort of maturity). The converged HBA looks at the hardware level like two distinct hardware devices: an Ethernet controller and an FC controller with the advantage that the software running on the server can be the existing mature (and loved) driver stacks of the respective Ethernet and FC controller vendors. (Yes, unexpected alliances between FC HBA vendors and well known Ethernet vendors are required for this solution unless customers are to be persuaded that FC vendors overnight have their own mature Ethernet controllers and software). So the advantage of this model is that it can be available as a deployable product in short order. However not easily integrated within volume silicon. Of course if you are an end customer paying HBA prices, then this might be acceptable.

As well as the implementation and business differences between the FCOE architectures, both FCOE and iSCSI illustrate the difference between two different approaches to the unification of networks:

 

1. iSCSI is an example of network convergence, where different network functions are Layered into one protocol stack architecture. Ethernet implementing OSI Layers 1, 2 TCP/IP implementing Layers 3, 4+ and iSCSI implementing Layer 5. By converging on a single protocol stack architecture the features and abstractions of the underlying layers are built upon. For example iSCSI sessions may be routed over different underlying IP networks.

 

2. Whereas FCOE is an example of a unified network in that two different network protocol stack architectures are carried by a single Layer 2 network (Ethernet in this case). By maintaining two distinct network protocol stacks, a unified network is able to support applications which require both network protocol stacks while transitioning the network at a lower layer.

 
These two different approaches are the same double-edged sword for both iSCSI and FCOE and will be the cause of significant continuing religious debate. Understanding the difference is key to deciding whether one or the other is the right approach for a given enterprise to take.

Efficiency and data rates

A quick Google search of 10GBASE-T and power will find a large number of articles and pundits decrying that the technology uses too much power to be energy efficient.  Generally, the more recent and relevant articles will compare today's Nth-generation 1000BASE-T devices, which consume around 500mW to first-generation shipping 10GBASE-T PHYs, which consume in the neighborhood of 10 Watts.  Even considering that the next generation of 10GBASE-T PHYs will  be roughly half the power of today's, this makes it look like a long way to go for 10GBASE-T to be energy efficient; however, that is, until you put it in perspective.

Now, when a 10Gb link is used to carry traffic that could easily be sent on a 1Gb link, without any extra hardware (e.g., extra NICs, aggregation hardware, switching ports & fabric), the analysis above makes sense.  However, that would be like comparing the energy efficiency of a train carrying a single passenger to a small car driving the same route.  The train is efficient simply because it CAN carry more.  When it is below a critical carrying capacity, its efficiency is handicapped.  With this metric in hand, if other-than-PHY hardware is considered (server CPUs, MACs, Switch ports), first generation 10GBASE-T links will already be more efficient when fully loaded, and, by the second generation, 10GBASE-T PHYs by themselves will be efficient on a Watt / (Gbit/second) basis than 1000BASE-T PHYs.  The question is, "How well are 10Gb links going to be utilized?"

K. Lloyd recently posted in the Intel "Server Room"  (http://communities.intel.com/openport/blogs/server/2008/01/29/almost-free-data-center-capacity)  that today's data centers are often 5-15% utilized, but that a conceivable target for next generation data centers would be as high as 75%.  Looking back at my last posting, I mentioned the macro approach of consolidating many links into one, driven by data center virtualization - consolidating multiple servers (and switch ports) together.  A quick look at Cisco's data center visions (http://blogs.cisco.com/datacenter/2008/02/expecting_and_getting_more_fro.html)  in addition to Intel's (http://communities.intel.com/openport/blogs/server/2008/02/13/data-center-fabric) expands further on this vision, by consolidating not only traditional ethernet applications but also bringing in storage, currently carried today on fibre channel and other networks.  The result, I believe, is that you can expect to see links filled to the breaking point, fully amortizing each link.

In comparison, when 1000BASE-T was adopted, it began around 6 Watts per port, at a time when 100BASE-T PHYs could be found around 300mW , the same roughly 20:1 ratio seen today.  1000BASE-T came down first by a factor of 2 and then more incrementally, but after that first step, the big change in efficiency comes from the ability to carry 10X the traffic, even without the consolidation we see at 10Gb today.

In short, energy efficiency in Ethernet comes directly out of one of the mantra's of for Ethernet success: growing at 10X the performance with a moderate increase in cost, which would include energy.  Already in the early generations we are in sight of energy parity on a Gigabit-per-second per Watt basis for heavily loaded links, and with all the changes happening in the network driving 10G, we can expect the efficiency curve to run rapidly.