« January 2008 | Main | March 2008 »
An update for those interested in tracking the Moore's Law curve is that bi-directional line-rate is now achievable with very modest CPU utilization using a commodity Lin/Win/Tel server and standard (1500) Ethernet frames.
Take a look at these videos of a couple of model engines:
2. steam engine
Two very different approaches to solving a similar problem and a good reminder that achieving energy efficiency designs often requires you to think outside the box.
BTW go buy both they're fantastic for big and small kids (1 , 2)
Many Ethernet controller companies have been talking the talk about accelerating virtualised IO. Not many are able to walk the walk. Solarflare by contrast has quietly been working at making high-speed networking a reality for the Citrix Xen virtual environment.
Following our architectural proposal at the Xen 2006 summit, Solarflare developed a new API for the Xen hypervisor to support direct guest acceleration. This work was accepted into xen-devel and results including accelerated virtualised iSCSI initiator performance were presented at the Xen 2007 summit and elsewhere.
More recently, Solarflare has submitted its accelerated driver set which implements the Citrix Xen direct guest acceleration API for the SFC4000 controller range.
High performance virtualised network IO takes us one step closer to ubiquitous virtualisation.
Thanks to Kieran, Greg and David for reminding me that our Euro-Par '07 paper provides a good write-up of the 2006 work: Download SF-101106-TC-4.pdf
If you read my previous tongue in cheek post on QoS and latency, you will appreciate Dave in our recent OpenOnload talk talk at Google somehow managing to represent that all networking woes are actually down to overhead.
A quick Google search of 10GBASE-T and power will find a large number of articles and pundits decrying that the technology uses too much power to be energy efficient. Generally, the more recent and relevant articles will compare today's Nth-generation 1000BASE-T devices, which consume around 500mW to first-generation shipping 10GBASE-T PHYs, which consume in the neighborhood of 10 Watts. Even considering that the next generation of 10GBASE-T PHYs will be roughly half the power of today's, this makes it look like a long way to go for 10GBASE-T to be energy efficient; however, that is, until you put it in perspective.
Now, when a 10Gb link is used to carry traffic that could easily be sent on a 1Gb link, without any extra hardware (e.g., extra NICs, aggregation hardware, switching ports & fabric), the analysis above makes sense. However, that would be like comparing the energy efficiency of a train carrying a single passenger to a small car driving the same route. The train is efficient simply because it CAN carry more. When it is below a critical carrying capacity, its efficiency is handicapped. With this metric in hand, if other-than-PHY hardware is considered (server CPUs, MACs, Switch ports), first generation 10GBASE-T links will already be more efficient when fully loaded, and, by the second generation, 10GBASE-T PHYs by themselves will be efficient on a Watt / (Gbit/second) basis than 1000BASE-T PHYs. The question is, "How well are 10Gb links going to be utilized?"
K. Lloyd recently posted in the Intel "Server Room" (http://communities.intel.com/openport/blogs/server/2008/01/29/almost-free-data-center-capacity) that today's data centers are often 5-15% utilized, but that a conceivable target for next generation data centers would be as high as 75%. Looking back at my last posting, I mentioned the macro approach of consolidating many links into one, driven by data center virtualization - consolidating multiple servers (and switch ports) together. A quick look at Cisco's data center visions (http://blogs.cisco.com/datacenter/2008/02/expecting_and_getting_more_fro.html) in addition to Intel's (http://communities.intel.com/openport/blogs/server/2008/02/13/data-center-fabric) expands further on this vision, by consolidating not only traditional ethernet applications but also bringing in storage, currently carried today on fibre channel and other networks. The result, I believe, is that you can expect to see links filled to the breaking point, fully amortizing each link.
In comparison, when 1000BASE-T was adopted, it began around 6 Watts per port, at a time when 100BASE-T PHYs could be found around 300mW , the same roughly 20:1 ratio seen today. 1000BASE-T came down first by a factor of 2 and then more incrementally, but after that first step, the big change in efficiency comes from the ability to carry 10X the traffic, even without the consolidation we see at 10Gb today.
In short, energy efficiency in Ethernet comes directly out of one of the mantra's of for Ethernet success: growing at 10X the performance with a moderate increase in cost, which would include energy. Already in the early generations we are in sight of energy parity on a Gigabit-per-second per Watt basis for heavily loaded links, and with all the changes happening in the network driving 10G, we can expect the efficiency curve to run rapidly.
As one involved in the IEEE 802.3az Task Force for Energy Efficient Ethernet, as well as working towards virtualized server networking, I often find the two appearing to be at odds. On one hand, Energy Efficient Ethernet counts on bursty network traffic with significant periods of idle. Energy consumed in the network elements is conserved by either reducing the speed or idling the network interfaces when no traffic is offered. On the other hand, virtualized servers and data center infrastructures are aimed at fully loading each processing element, so that the idle times are minimized.
Both of these approaches would conserve energy, and both can exist in the same network. Clearly, using fewer servers through virtualization by actually reducing the number of units used will be the most effective approach, provided that the virtualization is introduced with minimal overhead. By eliminating equipment, the energy consumption associated with power distribution, internal power supplies, processor cycles, as well as any idle network interface time, will be eliminated. 10 Gigabit Ethernet provides the high bandwidth links necessary to increase the overall system utilization to these levels. Generally you would expect these links to be fairly highly utilized. However, they are rarely used to full capacity, and generally only in special cases (like performance tests) are they near 100% for a sustained period of time.
Energy consumption of fully utilized network interfaces will be lowered by market forces and technical advances. Already the power on 10GBASE-T interfaces is expected to decrease by 50% or more. This parallels 1000BASE-T where the power consumption rapidly decreased from over 6W at introduction to less than 0.5W today. (Those of us with enough gray hairs will remember many saying the power would never decrease below various thresholds that got lower with time). However, I digress.
Because peak consumption will be driven by market forces, true energy efficiency will need complementary approaches to both increase utilization and decrease the power consumption during underutilized periods. By allowing a low-power idle of interfaces during periods of inactivity, Energy Efficient Ethernet will reduce the energy consumption in all but the 100% utilization case. If Energy Efficient Ethernet goes the way of low-power idles, as some proponents have it, systems will quickly inject their traffic into the network and then go quiet. The real savings will happen within the network, as network elements will be able to respond by processing less traffic offered, but be able to maintain the high delivery rate (and energy efficiency per bit offered) that the high bandwidth interfaces afford. During periods of low utilization, this can offer substantial savings; adding to the savings provided by consolidation and fully loading the interface in the first place.
The first and biggest step is to get those high bandwidth interfaces deployed and loaded up. Even at 7-10W per port, a fully-loaded 10GBASE-T interface offers a very competitive 0.7 to 1Watt per Gigabit transferred. Compared to multiple servers running multiple gigabit ports, this is an energy bargain. By loading up high bandwidth interfaces, we can make a big dent in Ethernet energy consumption. After that, trimming the remaining underutilized interface time down to 15-20% of its power consumption will be icing on the cake.
I was thinking of photographing some old networking hardware I've knocking about. Here are some fascinating archive images from:
Cambridge Computer Laboratory
David Greaves (Cambridge Ring, CFR, and ATM)
More to follow ...
A lot has been and is being said about power and going to 10 gigabits. Pundits and conventional wisdom commonly focus on the PHY power as the power per port. Ultimately, when switch designs get to much higher densities than they are today, the PHY power will be limiting. Today, however, recent announcements show that the hardest part is still amortizing the power required to process line-rate packets at speed, but things are changing, both on the PHY and within switch designs.
Last week Cisco announced a doubling in the number of 10gigabit ports for line cards in their catalyst 6500 platform ( http://newsroom.cisco.com/dlls/2008/prod_012808.html ). According to the release, the new module “can help reduce power consumption by up to 50 percent per port.” Now, barring a new port type (the release mentions none, and the data sheet description specifies X2 modules), it is fair to assume that the power consumption of these line cards must therefore be primarily the power consumed by the switch fabric. More detailed power calculations will confirm this fact. This is not an unusual situation in enterprise class switches today. The fact that Cisco is bringing the port density of Catalyst 6500 line cards up is very good for 10gigE, simply because it allows a better amortization of the power requirements for switching high-performance 10GigE traffic. The simple fact is that like on TCP Offload and iWarp NICS (http://www.eetimes.com/news/design/showArticle.jhtml;jsessionid=0MIIR0LRC0MQOQSNDLOSKHSCJUNN2JVN?articleID=205918831 ), in switches the majority of the power consumption per port is still going . When you double the port density (and keep the fabric roughly the same). Unfortunately for the switch vendors, getting beyond 16 ports in a standard line card form factor is going to require a shift away from the fairly versatile X2 modules that are currently shipping into the market. Soldered-down solutions such as 10GBASE-T, in addition to yet another optical module form factor shift will allow densities to look more like gigabit ethernet does today, physically allowing 48 or more ports in the same front-panel space. That transition is happening today, and the higher density can be seen even in the companion Cisco announcement (Nexus 7000 series) and in announced designs from SMC http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=8&scid=107&pid=1646 ), Arastra ( http://www.arastra.com/media/2007-11-05/ ) and others using soldered down 10GBASE-T silicon, new module form factors, and new switch silicon designs.
These are further enabled because, silicon designers, both internal to the OEMs and in merchant silicon teams have been working the switch density and power consumption problem just as PHY vendors have been diligently making solutions, such as 10GBASE-T, which will supplant the X2 and relieve the front-panel real estate crisis. At the same time as the power budget begins to realign, PHY powers are entering These designs are showing that this year, densities of 10gigabit platforms, enabled by 10GBASE-T and new optical module designs are finally providing the cost and power efficiencies necessary for mass deployment of high-performance 10gig networking.
I plan to be writing in the next few days about a number of topics in energy efficiency, including two trends that on the surface seem to be at odds – increased utilization of resources, including high speed links through virtualization and other means (e.g., http://www.cisco.com/en/US/solutions/ns708/networking_solutions_products_genericcontent0900aecd806fd32a.pdf) and the desire to rapidly turn down underutilized links (“Energy Efficient Ethernet” – e.g., http://www.ieee802.org/3/az/index.html ). So please send thoughts on what might be of interest.
