Up and Down the Network Stack

Auto-Negotiation Part 3 - Why 100Mbps is Important on a 10G PHY

Why are multi-speed 10G/1G/100BASE-T PHYs important?

In prior generations of Ethernet, this proved to be the path towards rapid provisioning of new speeds.  With multi-speed 10GBASE-T NICs in their servers, an end-user may begin to provision 10GBASE-T capable servers prior to upgrading the switching infrastructure and later install the 10GBASE-T switch and upgrade the entire set of servers’ speeds.  This allows for much easier and more rapid provisioning, without a “forklift” upgrade, where entire infrastructures are changed at once.

A second reason, growing in importance, for multi-speed PHYs is power management.  I’ve already said much about Energy Efficient Ethernet, but that is not what I mean.  Here, I am talking about “wake on LAN” capability.   I will say more about this subject separately, but putting it simply, today’s version of server power management puts idle servers into a low-speed mode (100BASE-TX usually), and waits for a “Magic Packet” to “Wake on LAN”.  This way, idle machines can dramatically lower their power consumption.  With the increasing EPA and Industry focus on making devices consume minimal power when not doing work, this feature is a must for servers.  Eventually, we’ll have to see how this plays out with regard to Energy Efficient Ethernet, but that won’t occur for years.  Today, if you want energy efficiency when a server is idle, you support Wake-on-LAN functionality.

Simple Auto-Negotiation enables some pretty important benefits which have helped the dominance of BASE-T copper Ethernet.  However, it is not to be forgotten that this is because it allows new generations to build on the prior generations technological and installed infrastructure base of BASE-T networks.  Now that 10GBASE-T is offering the right lower speeds of Ethernet, we will begin to see the power of Auto-Negotiation as it allows IT managers to asynchronously upgrade the speeds of their servers, AND still support power management (through Wake-on-LAN) for idle systems.

Auto-Negotiation – a simple but powerful idea, often misunderstood

Almost every copper Ethernet device supports “Auto-Negotiation”, or clause 28 of IEEE Std. 802.3.  However, based on questions I’ve been asked, it seems that this simple protocol is often misunderstood. This is the first part of a 3 part series to answer some of the more common questions.

What exactly is Auto-Neg?

Simply put, support for Auto-Negotiation means that two devices can talk with each other and communicate what Ethernet standards each is capable of supporting.  They then agree on the highest common speed, which may be none (they agree to disagree).  There are many Ethernet standards left on the dustbin of history, not offered in modern devices.  The Auto-Negotiation capability is one part of an equation that has allowed twisted-pair copper (BASE-T) devices to dominate the Ethernet market.  The other part is the ability to communicate over a common connector (RJ-45) and media type (UTP copper with structured cabling rules).  The most misunderstood part of Auto-Negotiation is that it does not imply support for any speeds.

Does support for Auto-Neg mean Multi-Speed?

The initial versions of 10GBASE-T PHYs were all single speed, largely because the switch interfaces could only support the 10 Gigabit XAUI interface, and could not make use of a multi-speed PHY.  They supported Auto-Negotiation, but not multiple speeds.  That is changing this year.  The coming generation of 10GBASE-T NICs will support 100Mbps and 1000Mbps in addition to 10Gigabit speeds.  They will Auto-Negotiate with a switch port and provide the highest possible speed, but allowing connection at lower speeds where appropriate.

Energy Efficient Ethernet Update: Low Power Idle baselined at all speeds, Editorial team begins work

At the May Interim meeting of the Energy Efficient Ethernet task force (IEEE P802.3az), the group reached achieved two important milestones.  The IEEE P802.3at Task Force voted to approve a baseline "low power idle" proposal at the 10 Gbps speed, making a consistent low power idle suite for BASE-T transceivers from 100Mbps through 10 Gbps. The consensus proposal, authored by Solarflare, Teranetics and Aquantia representatives allows 10GBASE-T devices to rapidly restore to transmiting data (in <5usec) and refresh their cancellers while minimizing consumed power in the receivers (http://www.ieee802.org/3/az/public/may08/taich_02_0508.pdf ).  Because of this capability, I expect that the generation of 10GBASE-T transceivers employing EEE technology, once it is standardized, (somewhere around 2011) will operate at the same average power consumption as 1000BASE-T transceivers when 1Gbps traffic loads are offered.  To add to that, they will offer the ability to burst at 10Gbps, which will aid storage, video and other high bandwidth tasks. We expect this to dramatically accelerate the proliferation of 10GBASE-T into workstation, client and applications beyond those using 10Gbps Ethernet today.

The completion of a baseline suite for BASE-T transceivers positioned the group to embark on the next phase of standards development: commissioning the editorial team to put together the first draft of the new standard.  Because EEE touches many different speeds and types of Ethernet (from 10BASE-T to 10GBASE-T), the bulk of the standards work will be done by a large number of individual editors from various companies. Solarflare's Gavin Parnaby, has taken on the work of drafting the text.  In July, the Task Force is expected to begin the work of reviewing the draft text and focusing on filling in the remaining TBD items on their march towards a specification which can be balloted.  There's still a long way to go, but this is great progress.

Go Green with Server Consolidation

You may have taken the glossy adverts with something of a pinch of salt, but PG&E (utility company) is serious about offering businesses rebates based reductions in power consumption by virtue of server consolidation. This rebate currently stands at 8c per kilowatt-hour reduction. See here for more details.

PG&E stipulate that servers must be decommissioned in order to gain the rebate. Perhaps they should look more closely at a scheme whereby the servers are re-provisioned either within the organisation or elsewhere in order to avoid the scheme being energy neutral (energy savings balanced by the cost of early replacement of existing servers).

BTW I'm hearing more and more that organizations are employing the migration feature provided by server virtualization in order to load balance for energy efficiency. Live migration of a complete operating system over a 10GBASE-T Ethernet network in total takes order 5seconds to complete  with a down-time measured in milli-seconds - clearly a powerful tool.

Tesla and Computing for the Future of the Planet

I've just finished reading a "Tesla, a Man Out of Time" which is a rip-roaring read on the great AC vs DC wars which took place just before the turn of the last century, together with lots of great  material for the conspiracy theorists.

Of course, Tesla was a truly great mind of his time credited with inventing the AC electric motor as well as radio transmission.

One of Tesla's dreams was to transmit energy wirelessly. While never confused between the transmission of information and energy it is likely true that much of the capital and time spent on his dual wireless energy / information transmission experiments could have been more profitably spent had he concentrated purely on what we now know as radio.  Tesla watched as his commercial lead was lost to Marconi and others and was unable to defend his radio patents during his lifetime (he was awarded this posthumously).

All this resonates (no pun intended) with recent directions taken by Andy Hopper at the Cambridge University Computer Laboratory. From the principle that transmission of information is much more efficient than the transmission of energy, his group are exploring new techniques to explore the tradeoffs between physical and virtual resources particularly as this relates to digital-enablement for the developing world and classic data-center energy-efficiency applications. Take a look  here  and at his Google Tech Talk (where Solarflare gets a mention for high-efficiency migration of Citrix-Xen computations using 10G Ethernet).

 

Progress in energy efficiency

This past week I've been off meeting with the IEEE task force on energy efficient ethernet (802.3az), something I've mentioned before.  The group is working hard to define operating modes for systems during periods of low link utilization.  Now, while most of us think of our electric bills on the scale of days, weeks and months, the EEE group is looking to optimize utilization at the time scale of microseconds.  A longtime industry veteran in the group pointed out that this is a  return to the old days of ethernet when the only time transmission occured was when there was data, and a preamble signalled the receiver and facilitated its wakeup.

One of the most interesting discussions was around congested networks in the data center.  Within this group, it was somewhat universally stated that this was no longer an issue for gigabit networks, because those were being scaled to 10gigabits when congestion raised its ugly head.  A scant few years ago, I can recall that not being the case.   The future looks bright for 10 Gigabit.

In addition, the group is making progress towards a specification.  Proposals were baselined for low-power idles for 100 and 1000BASE-T, and proposals were made for a low-power idle mode for 10GBASE-T, which promises to cut power during low-utilization periods by up to 80%.  This has the potential to rapidly accelerate 10gigabit to the desktop,  by allowing quick bursts of speed when required, without the power penalty. 

Link: Up and Down the Network Stack.

It's all about using the right engine for the job ..

Take a look at these videos of a couple of model engines:

1.  Stirling engine

2.  steam engine 

Two very different approaches to solving a similar problem and a good reminder that achieving energy efficiency designs often requires you to think outside the box.

BTW go buy both they're fantastic for big and small kids (1   , 2) 

Efficiency and data rates

A quick Google search of 10GBASE-T and power will find a large number of articles and pundits decrying that the technology uses too much power to be energy efficient.  Generally, the more recent and relevant articles will compare today's Nth-generation 1000BASE-T devices, which consume around 500mW to first-generation shipping 10GBASE-T PHYs, which consume in the neighborhood of 10 Watts.  Even considering that the next generation of 10GBASE-T PHYs will  be roughly half the power of today's, this makes it look like a long way to go for 10GBASE-T to be energy efficient; however, that is, until you put it in perspective.

Now, when a 10Gb link is used to carry traffic that could easily be sent on a 1Gb link, without any extra hardware (e.g., extra NICs, aggregation hardware, switching ports & fabric), the analysis above makes sense.  However, that would be like comparing the energy efficiency of a train carrying a single passenger to a small car driving the same route.  The train is efficient simply because it CAN carry more.  When it is below a critical carrying capacity, its efficiency is handicapped.  With this metric in hand, if other-than-PHY hardware is considered (server CPUs, MACs, Switch ports), first generation 10GBASE-T links will already be more efficient when fully loaded, and, by the second generation, 10GBASE-T PHYs by themselves will be efficient on a Watt / (Gbit/second) basis than 1000BASE-T PHYs.  The question is, "How well are 10Gb links going to be utilized?"

K. Lloyd recently posted in the Intel "Server Room"  (http://communities.intel.com/openport/blogs/server/2008/01/29/almost-free-data-center-capacity)  that today's data centers are often 5-15% utilized, but that a conceivable target for next generation data centers would be as high as 75%.  Looking back at my last posting, I mentioned the macro approach of consolidating many links into one, driven by data center virtualization - consolidating multiple servers (and switch ports) together.  A quick look at Cisco's data center visions (http://blogs.cisco.com/datacenter/2008/02/expecting_and_getting_more_fro.html)  in addition to Intel's (http://communities.intel.com/openport/blogs/server/2008/02/13/data-center-fabric) expands further on this vision, by consolidating not only traditional ethernet applications but also bringing in storage, currently carried today on fibre channel and other networks.  The result, I believe, is that you can expect to see links filled to the breaking point, fully amortizing each link.

In comparison, when 1000BASE-T was adopted, it began around 6 Watts per port, at a time when 100BASE-T PHYs could be found around 300mW , the same roughly 20:1 ratio seen today.  1000BASE-T came down first by a factor of 2 and then more incrementally, but after that first step, the big change in efficiency comes from the ability to carry 10X the traffic, even without the consolidation we see at 10Gb today.

In short, energy efficiency in Ethernet comes directly out of one of the mantra's of for Ethernet success: growing at 10X the performance with a moderate increase in cost, which would include energy.  Already in the early generations we are in sight of energy parity on a Gigabit-per-second per Watt basis for heavily loaded links, and with all the changes happening in the network driving 10G, we can expect the efficiency curve to run rapidly.

Diverging Roads to Energy Efficiency?

As one involved in the IEEE 802.3az Task Force for Energy Efficient Ethernet, as well as working towards virtualized server networking, I often find the two appearing to be at odds. On one hand, Energy Efficient Ethernet counts on bursty network traffic with significant periods of idle. Energy consumed in the network elements is conserved by either reducing the speed or idling the network interfaces when no traffic is offered. On the other hand, virtualized servers and data center infrastructures are aimed at fully loading each processing element, so that the idle times are minimized.

Both of these approaches would conserve energy, and both can exist in the same network. Clearly, using fewer servers through virtualization by actually reducing the number of units used will be the most effective approach, provided that the virtualization is introduced with minimal overhead. By eliminating equipment, the energy consumption associated with power distribution, internal power supplies, processor cycles, as well as any idle network interface time, will be eliminated. 10 Gigabit Ethernet provides the high bandwidth links necessary to increase the overall system utilization to these levels. Generally you would expect these links to be fairly highly utilized. However, they are rarely used to full capacity, and generally only in special cases (like performance tests) are they near 100% for a sustained period of time.

Energy consumption of fully utilized network interfaces will be lowered by market forces and technical advances.  Already the power on 10GBASE-T interfaces is expected to decrease by 50% or more.  This parallels 1000BASE-T where the power consumption rapidly decreased from over 6W at introduction to less than 0.5W today. (Those of us with enough gray hairs will remember many saying the power would never decrease below various thresholds that got lower with time).  However, I digress.

Because peak consumption will be driven by market forces, true energy efficiency will need complementary approaches to both increase utilization and decrease the power consumption during underutilized periods. By allowing a low-power idle of interfaces during periods of inactivity, Energy Efficient Ethernet will reduce the energy consumption in all but the 100% utilization case. If Energy Efficient Ethernet goes the way of low-power idles, as some proponents have it, systems will quickly inject their traffic into the network and then go quiet. The real savings will happen within the network, as network elements will be able to respond by processing less traffic offered, but be able to maintain the high delivery rate (and energy efficiency per bit offered) that the high bandwidth interfaces afford.  During periods of low utilization, this can offer substantial savings; adding to the savings provided by consolidation and fully loading the interface in the first place.

The first and biggest step is to get those high bandwidth interfaces deployed and loaded up. Even at 7-10W per port, a fully-loaded 10GBASE-T interface offers a very competitive 0.7 to 1Watt per Gigabit transferred. Compared to multiple servers running multiple gigabit ports, this is an energy bargain. By loading up high bandwidth interfaces, we can make a big dent in Ethernet energy consumption.  After that, trimming the remaining underutilized interface time down to 15-20% of its power consumption will be icing on the cake.

Hot topics in energy efficient switches

A lot has been and is being said about power and going to 10 gigabits. Pundits and conventional wisdom commonly focus on the PHY power as the power per port. Ultimately, when switch designs get to much higher densities than they are today, the PHY power will be limiting. Today, however, recent announcements show that the hardest part is still amortizing the power required to process line-rate packets at speed, but things are changing, both on the PHY and within switch designs.

Last week Cisco announced a doubling in the number of 10gigabit ports for line cards in their catalyst 6500 platform  ( http://newsroom.cisco.com/dlls/2008/prod_012808.html ). According to the release, the new module “can help reduce power consumption by up to 50 percent per port.” Now, barring a new port type (the release mentions none, and the data sheet description specifies X2 modules), it is fair to assume that the power consumption of these line cards must therefore be primarily the power consumed by the switch fabric. More detailed power calculations will confirm this fact. This is not an unusual situation in enterprise class switches today. The fact that Cisco is bringing the port density of Catalyst 6500 line cards up is very good for 10gigE, simply because it allows a better amortization of the power requirements for switching high-performance 10GigE traffic. The simple fact is that like on TCP Offload and iWarp NICS (http://www.eetimes.com/news/design/showArticle.jhtml;jsessionid=0MIIR0LRC0MQOQSNDLOSKHSCJUNN2JVN?articleID=205918831 ), in switches the majority of the power consumption per port is still going . When you double the port density (and keep the fabric roughly the same). Unfortunately for the switch vendors, getting beyond 16 ports in a standard line card form factor is going to require a shift away from the fairly versatile X2 modules that are currently shipping into the market. Soldered-down solutions such as 10GBASE-T, in addition to yet another optical module form factor shift will allow densities to look more like gigabit ethernet does today, physically allowing 48 or more ports in the same front-panel space. That transition is happening today, and the higher density can be seen even in the companion Cisco announcement (Nexus 7000 series) and in announced designs from SMC http://www.smc.com/index.cfm?event=viewProduct&localeCode=EN_USA&cid=8&scid=107&pid=1646  ), Arastra ( http://www.arastra.com/media/2007-11-05/ ) and others using soldered down 10GBASE-T silicon, new module form factors,  and new switch silicon designs.

These are further enabled because, silicon designers, both internal to the OEMs and in merchant silicon teams have been working the switch density and power consumption problem just as PHY vendors have been diligently making solutions, such as 10GBASE-T, which will supplant the X2 and relieve the front-panel real estate crisis. At the same time as the power budget begins to realign, PHY powers are entering  These designs are showing that this year, densities of 10gigabit platforms, enabled by 10GBASE-T and new optical module designs are finally providing the cost and power efficiencies necessary for mass deployment of high-performance 10gig networking.

I plan to be writing in the next few days about a number of topics in energy efficiency, including two trends that on the surface seem to be at odds – increased utilization of resources, including high speed links through virtualization and other means (e.g., http://www.cisco.com/en/US/solutions/ns708/networking_solutions_products_genericcontent0900aecd806fd32a.pdf) and the desire to rapidly turn down underutilized links (“Energy Efficient Ethernet” – e.g., http://www.ieee802.org/3/az/index.html ). So please send thoughts on what might be of interest.