Up and Down the Network Stack

10 Gigabit Ethernet and VMware - A Match Made in Heaven

Thanks to Steve Grantham, who pointed me at a nice blog entry of the above title, which provides a good overview of the technology trends being driven by virtualisation.

Virtual Geek (aka Chad Sakac) says:

Back when I was in the valley (so this was 4 years ago), a buddy of mine worked at a tiny IP (not Internet Protocol - Intellectual Property) company focused on high-end IP blocks for networking and storage (going after Hi/fn and the others in that space).   He showed me their A0 spin, and told me "This chip will do 10GbE BaseT over Cat6 cables, full TCP offload including segment offload, and all iSCSI offload".  COOL. "oh and we think we can mass produce it for $25 per chip".   DROOL.   

Well, fast forward 4 years, and they are out of business :-)

..... GOOD news is that another company has been able to realise this device.

Go Green with Server Consolidation

You may have taken the glossy adverts with something of a pinch of salt, but PG&E (utility company) is serious about offering businesses rebates based reductions in power consumption by virtue of server consolidation. This rebate currently stands at 8c per kilowatt-hour reduction. See here for more details.

PG&E stipulate that servers must be decommissioned in order to gain the rebate. Perhaps they should look more closely at a scheme whereby the servers are re-provisioned either within the organisation or elsewhere in order to avoid the scheme being energy neutral (energy savings balanced by the cost of early replacement of existing servers).

BTW I'm hearing more and more that organizations are employing the migration feature provided by server virtualization in order to load balance for energy efficiency. Live migration of a complete operating system over a 10GBASE-T Ethernet network in total takes order 5seconds to complete  with a down-time measured in milli-seconds - clearly a powerful tool.

Simon / Xen / Cambridge / Citrix / 10GBASE-T

Hey did you know Simon Crosby is one of the most influential people in IT. It's official.

Tesla and Computing for the Future of the Planet

I've just finished reading a "Tesla, a Man Out of Time" which is a rip-roaring read on the great AC vs DC wars which took place just before the turn of the last century, together with lots of great  material for the conspiracy theorists.

Of course, Tesla was a truly great mind of his time credited with inventing the AC electric motor as well as radio transmission.

One of Tesla's dreams was to transmit energy wirelessly. While never confused between the transmission of information and energy it is likely true that much of the capital and time spent on his dual wireless energy / information transmission experiments could have been more profitably spent had he concentrated purely on what we now know as radio.  Tesla watched as his commercial lead was lost to Marconi and others and was unable to defend his radio patents during his lifetime (he was awarded this posthumously).

All this resonates (no pun intended) with recent directions taken by Andy Hopper at the Cambridge University Computer Laboratory. From the principle that transmission of information is much more efficient than the transmission of energy, his group are exploring new techniques to explore the tradeoffs between physical and virtual resources particularly as this relates to digital-enablement for the developing world and classic data-center energy-efficiency applications. Take a look  here  and at his Google Tech Talk (where Solarflare gets a mention for high-efficiency migration of Citrix-Xen computations using 10G Ethernet).

 

OpenOnload 2.2.125

Dev-snapshot is available from www.openonload.org.

Solarflare and the Art of Virtualised IO

Many Ethernet controller companies have been talking the talk about accelerating virtualised IO. Not many are able to walk the walk. Solarflare by contrast has quietly been working at making high-speed networking a reality for the Citrix Xen virtual environment.

Following our architectural proposal  at the Xen 2006 summit, Solarflare developed a new API for the Xen hypervisor to support direct guest acceleration. This work was accepted into xen-devel and results including  accelerated virtualised iSCSI initiator performance were presented at the Xen 2007 summit and elsewhere.

More recently, Solarflare has submitted its accelerated driver set which implements the Citrix Xen direct guest acceleration API for the SFC4000 controller range.

High performance virtualised network IO takes us one step closer to ubiquitous virtualisation.

Thanks to Kieran, Greg and David for reminding me that our Euro-Par '07 paper provides a good write-up of the 2006 work: Download SF-101106-TC-4.pdf

Overhead and OpenOnload

If you read my previous tongue in cheek post on QoS and latency, you will appreciate Dave in our recent OpenOnload talk talk at Google somehow managing to represent that all networking woes are actually down to overhead.

Efficiency and data rates

A quick Google search of 10GBASE-T and power will find a large number of articles and pundits decrying that the technology uses too much power to be energy efficient.  Generally, the more recent and relevant articles will compare today's Nth-generation 1000BASE-T devices, which consume around 500mW to first-generation shipping 10GBASE-T PHYs, which consume in the neighborhood of 10 Watts.  Even considering that the next generation of 10GBASE-T PHYs will  be roughly half the power of today's, this makes it look like a long way to go for 10GBASE-T to be energy efficient; however, that is, until you put it in perspective.

Now, when a 10Gb link is used to carry traffic that could easily be sent on a 1Gb link, without any extra hardware (e.g., extra NICs, aggregation hardware, switching ports & fabric), the analysis above makes sense.  However, that would be like comparing the energy efficiency of a train carrying a single passenger to a small car driving the same route.  The train is efficient simply because it CAN carry more.  When it is below a critical carrying capacity, its efficiency is handicapped.  With this metric in hand, if other-than-PHY hardware is considered (server CPUs, MACs, Switch ports), first generation 10GBASE-T links will already be more efficient when fully loaded, and, by the second generation, 10GBASE-T PHYs by themselves will be efficient on a Watt / (Gbit/second) basis than 1000BASE-T PHYs.  The question is, "How well are 10Gb links going to be utilized?"

K. Lloyd recently posted in the Intel "Server Room"  (http://communities.intel.com/openport/blogs/server/2008/01/29/almost-free-data-center-capacity)  that today's data centers are often 5-15% utilized, but that a conceivable target for next generation data centers would be as high as 75%.  Looking back at my last posting, I mentioned the macro approach of consolidating many links into one, driven by data center virtualization - consolidating multiple servers (and switch ports) together.  A quick look at Cisco's data center visions (http://blogs.cisco.com/datacenter/2008/02/expecting_and_getting_more_fro.html)  in addition to Intel's (http://communities.intel.com/openport/blogs/server/2008/02/13/data-center-fabric) expands further on this vision, by consolidating not only traditional ethernet applications but also bringing in storage, currently carried today on fibre channel and other networks.  The result, I believe, is that you can expect to see links filled to the breaking point, fully amortizing each link.

In comparison, when 1000BASE-T was adopted, it began around 6 Watts per port, at a time when 100BASE-T PHYs could be found around 300mW , the same roughly 20:1 ratio seen today.  1000BASE-T came down first by a factor of 2 and then more incrementally, but after that first step, the big change in efficiency comes from the ability to carry 10X the traffic, even without the consolidation we see at 10Gb today.

In short, energy efficiency in Ethernet comes directly out of one of the mantra's of for Ethernet success: growing at 10X the performance with a moderate increase in cost, which would include energy.  Already in the early generations we are in sight of energy parity on a Gigabit-per-second per Watt basis for heavily loaded links, and with all the changes happening in the network driving 10G, we can expect the efficiency curve to run rapidly.

Diverging Roads to Energy Efficiency?

As one involved in the IEEE 802.3az Task Force for Energy Efficient Ethernet, as well as working towards virtualized server networking, I often find the two appearing to be at odds. On one hand, Energy Efficient Ethernet counts on bursty network traffic with significant periods of idle. Energy consumed in the network elements is conserved by either reducing the speed or idling the network interfaces when no traffic is offered. On the other hand, virtualized servers and data center infrastructures are aimed at fully loading each processing element, so that the idle times are minimized.

Both of these approaches would conserve energy, and both can exist in the same network. Clearly, using fewer servers through virtualization by actually reducing the number of units used will be the most effective approach, provided that the virtualization is introduced with minimal overhead. By eliminating equipment, the energy consumption associated with power distribution, internal power supplies, processor cycles, as well as any idle network interface time, will be eliminated. 10 Gigabit Ethernet provides the high bandwidth links necessary to increase the overall system utilization to these levels. Generally you would expect these links to be fairly highly utilized. However, they are rarely used to full capacity, and generally only in special cases (like performance tests) are they near 100% for a sustained period of time.

Energy consumption of fully utilized network interfaces will be lowered by market forces and technical advances.  Already the power on 10GBASE-T interfaces is expected to decrease by 50% or more.  This parallels 1000BASE-T where the power consumption rapidly decreased from over 6W at introduction to less than 0.5W today. (Those of us with enough gray hairs will remember many saying the power would never decrease below various thresholds that got lower with time).  However, I digress.

Because peak consumption will be driven by market forces, true energy efficiency will need complementary approaches to both increase utilization and decrease the power consumption during underutilized periods. By allowing a low-power idle of interfaces during periods of inactivity, Energy Efficient Ethernet will reduce the energy consumption in all but the 100% utilization case. If Energy Efficient Ethernet goes the way of low-power idles, as some proponents have it, systems will quickly inject their traffic into the network and then go quiet. The real savings will happen within the network, as network elements will be able to respond by processing less traffic offered, but be able to maintain the high delivery rate (and energy efficiency per bit offered) that the high bandwidth interfaces afford.  During periods of low utilization, this can offer substantial savings; adding to the savings provided by consolidation and fully loading the interface in the first place.

The first and biggest step is to get those high bandwidth interfaces deployed and loaded up. Even at 7-10W per port, a fully-loaded 10GBASE-T interface offers a very competitive 0.7 to 1Watt per Gigabit transferred. Compared to multiple servers running multiple gigabit ports, this is an energy bargain. By loading up high bandwidth interfaces, we can make a big dent in Ethernet energy consumption.  After that, trimming the remaining underutilized interface time down to 15-20% of its power consumption will be icing on the cake.

Virtualization and iSCSI simplify IT

Yesterday Dell completed its acquisition of Equal Logic, together with a quote from Michael Dell:

Virtualization and iSCSI are two keys to simplifying IT. Enterprises are creating data and consuming power at an exponential rate, driving up IT cost and complexity.

When you combine this with Dell's Veso announcement at the '07 VMworld, a picture of ubiquitous virtualization combined with simple easy to use (commoditized) iSCSI emerges. This environment is something that Solarflare has been actively supporting - our SFC4000 10GBASE-T controller has been designed specifically to support iSCSI within a virtualized environment. For some more details, see our see our  presentation  at the Xen Nov 07 summit.

-- Thanks to Robert Stonehouse