Up and Down the Network Stack

Auto-Negotiation Part 3 - Why 100Mbps is Important on a 10G PHY

Why are multi-speed 10G/1G/100BASE-T PHYs important?

In prior generations of Ethernet, this proved to be the path towards rapid provisioning of new speeds.  With multi-speed 10GBASE-T NICs in their servers, an end-user may begin to provision 10GBASE-T capable servers prior to upgrading the switching infrastructure and later install the 10GBASE-T switch and upgrade the entire set of servers’ speeds.  This allows for much easier and more rapid provisioning, without a “forklift” upgrade, where entire infrastructures are changed at once.

A second reason, growing in importance, for multi-speed PHYs is power management.  I’ve already said much about Energy Efficient Ethernet, but that is not what I mean.  Here, I am talking about “wake on LAN” capability.   I will say more about this subject separately, but putting it simply, today’s version of server power management puts idle servers into a low-speed mode (100BASE-TX usually), and waits for a “Magic Packet” to “Wake on LAN”.  This way, idle machines can dramatically lower their power consumption.  With the increasing EPA and Industry focus on making devices consume minimal power when not doing work, this feature is a must for servers.  Eventually, we’ll have to see how this plays out with regard to Energy Efficient Ethernet, but that won’t occur for years.  Today, if you want energy efficiency when a server is idle, you support Wake-on-LAN functionality.

Simple Auto-Negotiation enables some pretty important benefits which have helped the dominance of BASE-T copper Ethernet.  However, it is not to be forgotten that this is because it allows new generations to build on the prior generations technological and installed infrastructure base of BASE-T networks.  Now that 10GBASE-T is offering the right lower speeds of Ethernet, we will begin to see the power of Auto-Negotiation as it allows IT managers to asynchronously upgrade the speeds of their servers, AND still support power management (through Wake-on-LAN) for idle systems.

Auto-Negotiation Part 2 – Auto-Neg vs. Speed Switching

Would someone use Auto-Negotiation to switch speeds on-the-fly?

No.  The switching of speeds via Auto-Negotiation occurs at the initiation of a physical-layer link.  It is not something done on-the-fly.  Auto-Negotiation is useful for network speed transitions that happen over long durations in computer time (at least minutes long, since the transition takes a few seconds).  Unlike Energy Efficient Ethernet, which I’ve discussed earlier, the kind of speed transitions that might be facilitated through Auto-Negotiation, would not be expected to be buffered up in a network, as this would require billions and billions of bits stored.

Auto-Negotiation – a simple but powerful idea, often misunderstood

Almost every copper Ethernet device supports “Auto-Negotiation”, or clause 28 of IEEE Std. 802.3.  However, based on questions I’ve been asked, it seems that this simple protocol is often misunderstood. This is the first part of a 3 part series to answer some of the more common questions.

What exactly is Auto-Neg?

Simply put, support for Auto-Negotiation means that two devices can talk with each other and communicate what Ethernet standards each is capable of supporting.  They then agree on the highest common speed, which may be none (they agree to disagree).  There are many Ethernet standards left on the dustbin of history, not offered in modern devices.  The Auto-Negotiation capability is one part of an equation that has allowed twisted-pair copper (BASE-T) devices to dominate the Ethernet market.  The other part is the ability to communicate over a common connector (RJ-45) and media type (UTP copper with structured cabling rules).  The most misunderstood part of Auto-Negotiation is that it does not imply support for any speeds.

Does support for Auto-Neg mean Multi-Speed?

The initial versions of 10GBASE-T PHYs were all single speed, largely because the switch interfaces could only support the 10 Gigabit XAUI interface, and could not make use of a multi-speed PHY.  They supported Auto-Negotiation, but not multiple speeds.  That is changing this year.  The coming generation of 10GBASE-T NICs will support 100Mbps and 1000Mbps in addition to 10Gigabit speeds.  They will Auto-Negotiate with a switch port and provide the highest possible speed, but allowing connection at lower speeds where appropriate.

Energy Efficient Ethernet Update: Low Power Idle baselined at all speeds, Editorial team begins work

At the May Interim meeting of the Energy Efficient Ethernet task force (IEEE P802.3az), the group reached achieved two important milestones.  The IEEE P802.3at Task Force voted to approve a baseline "low power idle" proposal at the 10 Gbps speed, making a consistent low power idle suite for BASE-T transceivers from 100Mbps through 10 Gbps. The consensus proposal, authored by Solarflare, Teranetics and Aquantia representatives allows 10GBASE-T devices to rapidly restore to transmiting data (in <5usec) and refresh their cancellers while minimizing consumed power in the receivers (http://www.ieee802.org/3/az/public/may08/taich_02_0508.pdf ).  Because of this capability, I expect that the generation of 10GBASE-T transceivers employing EEE technology, once it is standardized, (somewhere around 2011) will operate at the same average power consumption as 1000BASE-T transceivers when 1Gbps traffic loads are offered.  To add to that, they will offer the ability to burst at 10Gbps, which will aid storage, video and other high bandwidth tasks. We expect this to dramatically accelerate the proliferation of 10GBASE-T into workstation, client and applications beyond those using 10Gbps Ethernet today.

The completion of a baseline suite for BASE-T transceivers positioned the group to embark on the next phase of standards development: commissioning the editorial team to put together the first draft of the new standard.  Because EEE touches many different speeds and types of Ethernet (from 10BASE-T to 10GBASE-T), the bulk of the standards work will be done by a large number of individual editors from various companies. Solarflare's Gavin Parnaby, has taken on the work of drafting the text.  In July, the Task Force is expected to begin the work of reviewing the draft text and focusing on filling in the remaining TBD items on their march towards a specification which can be balloted.  There's still a long way to go, but this is great progress.

10 Gigabit Ethernet and VMware - A Match Made in Heaven

Thanks to Steve Grantham, who pointed me at a nice blog entry of the above title, which provides a good overview of the technology trends being driven by virtualisation.

Virtual Geek (aka Chad Sakac) says:

Back when I was in the valley (so this was 4 years ago), a buddy of mine worked at a tiny IP (not Internet Protocol - Intellectual Property) company focused on high-end IP blocks for networking and storage (going after Hi/fn and the others in that space).   He showed me their A0 spin, and told me "This chip will do 10GbE BaseT over Cat6 cables, full TCP offload including segment offload, and all iSCSI offload".  COOL. "oh and we think we can mass produce it for $25 per chip".   DROOL.   

Well, fast forward 4 years, and they are out of business :-)

..... GOOD news is that another company has been able to realise this device.

Go Green with Server Consolidation

You may have taken the glossy adverts with something of a pinch of salt, but PG&E (utility company) is serious about offering businesses rebates based reductions in power consumption by virtue of server consolidation. This rebate currently stands at 8c per kilowatt-hour reduction. See here for more details.

PG&E stipulate that servers must be decommissioned in order to gain the rebate. Perhaps they should look more closely at a scheme whereby the servers are re-provisioned either within the organisation or elsewhere in order to avoid the scheme being energy neutral (energy savings balanced by the cost of early replacement of existing servers).

BTW I'm hearing more and more that organizations are employing the migration feature provided by server virtualization in order to load balance for energy efficiency. Live migration of a complete operating system over a 10GBASE-T Ethernet network in total takes order 5seconds to complete  with a down-time measured in milli-seconds - clearly a powerful tool.

Tesla and Computing for the Future of the Planet

I've just finished reading a "Tesla, a Man Out of Time" which is a rip-roaring read on the great AC vs DC wars which took place just before the turn of the last century, together with lots of great  material for the conspiracy theorists.

Of course, Tesla was a truly great mind of his time credited with inventing the AC electric motor as well as radio transmission.

One of Tesla's dreams was to transmit energy wirelessly. While never confused between the transmission of information and energy it is likely true that much of the capital and time spent on his dual wireless energy / information transmission experiments could have been more profitably spent had he concentrated purely on what we now know as radio.  Tesla watched as his commercial lead was lost to Marconi and others and was unable to defend his radio patents during his lifetime (he was awarded this posthumously).

All this resonates (no pun intended) with recent directions taken by Andy Hopper at the Cambridge University Computer Laboratory. From the principle that transmission of information is much more efficient than the transmission of energy, his group are exploring new techniques to explore the tradeoffs between physical and virtual resources particularly as this relates to digital-enablement for the developing world and classic data-center energy-efficiency applications. Take a look  here  and at his Google Tech Talk (where Solarflare gets a mention for high-efficiency migration of Citrix-Xen computations using 10G Ethernet).

 

Two 10GBASE-T Hopefuls Pass On

Please pause for a moment of silence.  Last month, quietly, the 10GBASE-T vendor community lost two early players.  Both were banking on short-reach 10GBASE-T technology, figuring that patch-cord length solutions would be good enough.  Keyeye Communications closed its doors (http://www.entrepreneur.com/localnews/1610167.html), and Vativ Technologies, which had marketed a proprietary, 10gigabit ethernet patch-cord length connection sold its assets to Entropic Communications for a fraction of the money invested (http://www.primenewswire.com/newsroom/news.html?d=139528).

Both of these teams had star players.  I worked with Hiroshi Takatori, founder of Keyeye, a few years back as a partner in developing HDSL2 transceivers,  and I worked alongside, Mike McConnell, well-known in the Ethernet world, and Albert Vareljian in the IEEE standards.  I’ve had some interaction with Sreen Ragavan’s team from ComCore and National, from which he seeded Vativ, and they are not slouches either.  It is with some sadness that I see these engineers and technology marketers depart from the 10GBASE-T scene. It goes to show, as I’ve said a few times before, 10GBASE-T is an extremely hard problem to solve.  It just took more careful systems planning, intellectual property development, and patience than most will go for.  Both teams made compromises on distance or standards compliance, and attempted to go for less than 100 meter reach, in Vativ’s case, a few meters of patch cord, and in Keyeye’s case, aligned with a “short reach test mode” which was inserted into the 802.3an-2006 10GBASE-T standard as an optional mode to allow full-100m-capable devices to conserve power when they were connected on shorter links.  Relaxing the demands of the technical problem does make 10GBASE-T amenable to analog and other simpler techniques, but it also makes it a lot less useful.

Now that full performance devices have come to market, and it is clear that we have begun dramatically bringing the power down (see http://www.networkworld.com/news/2008/041408-solarflare-halves-10gbase-ts-power.html?code=nlnetarch133353 ), as was done for 1000BASE-T before, I guess that it was hard to generate enough greed in investors to overcome the fear that a short-reach technology would be relegated to a small niche, while traditional, full-100m devices with power management capabilities would once again fill the vast majority of slots.  Fortunately, Solarflare and I began this journey with a team and investors who were in it for the long haul, too.

Still, I wish the teams from Keyeye and Vativ well.  I’m already starting to see them show up in other places, helping contribute to other new technologies.

10GBASE-T Comes to the Fore

Sometimes I look back and find it’s amazing how constant the demand for the simplicity and utility of 10GBASE-T has been over the past 7 years since Solarflare began.  Never is it more apparent than in the responses to the launch of our recent single-chip, sub-6 watt transceiver.  It’s easy to say that it’s just history repeating itself, but a lot of hard work went into it.  10GBASE-T is about engineers putting the complexity into the silicon and firmware so that the user doesn’t have to.  Users have realized, and said to me, since 2000, that if they had a 10 Gigabit solution that ran on UTP copper (100 meters), they would adopt it, just as they had 1000BASE-T.  Today, the response is the same.  Speaking the Rick Merritt in EE Times (4/14), Dante Malagrino, director of product marketing for data center solutions at Nuova Systems, which was acquired last week by Cisco, said that “the 10GBASE-T technology "is great in terms of compatibility and simplicity,”.  Cisco has understood the importance of this technology since the beginning, it was their help in driving the 802.3an (10GBASE-T) standard, particularly on issues of power/performance tradeoffs and latency, that have resulted in the available parts today.   Recognizing, as Steve Pope has, that latency is mostly in the system, and that the PHY contributes little, Cisco helped set the 2.5usec latency that is standard for 10GBASE-T (http://www.ieee802.org/3/an/public/jul05/comments_3_0705.pdf ).

The question marks for 10GBASE-T have always been around 3 things: (1) can it be done, which was proven with the SFX7101, shipping since August 2006, (2), will the power come down (with the SFT9001 it has, dramatically), and (3) is there really demand for 10 gigabit. If you’re reading this blog, you are likely already aware that 10gigabit technology is (finally) beginning to take off in the market, and that it is driven by a variety of applications, including storage, unified fabrics, and virtualization.  These in turn, are not simply ends in themselves, but are driven by operational and energy efficiencies, major economic factors that are becoming increasingly important in today’s economy.  In relation to 10 gigabit networking, Renato Recio, a chief engineer for server networking at IBM Corp. said to EE Times (http://www.eetimes.com/news/latest/showArticle.jhtml?articleID=207200193), "Somewhere between 50 and 70 percent of the Fortune 1000 companies are going to be building data centers in the next three years," "They are looking for technologies to make them more green, and this network convergence group has that value--this rings for customers”.

Rick Merritt, writing for EE Times, understands the operational importance of 10GBASE-T to network convergence, when he writes, “The work on the 10GBASE-T standard for Ethernet over copper lines only indirectly fuels the network convergence. Its primary aim is to lower the cost of and expand the market for 10-Gbit Ethernet, which has been limited to expensive optical and short-reach copper cables to date.”  Lower cost is extremely important, because customers will only buy these more efficient technologies at a reasonable price, and that is where the new generation of 10GBASE-T transceivers comes in. The fully-integrated SFT9001 now brings the promise of low-cost 10 Gigabit Ethernet to fruition. Again from EE Times, Recio added, "In my opinion, 10GBASE-T is a very important piece because it significantly reduces my price point to use copper," "I'd rather not use fiber in a rack, and it's an even better deal if my end-of-row switch can use copper."

These new parts will take some months for OEMs to integrate into end-user products, and, expect to see a lot of conversation in the meantime, touting the benefits of optics or short-reach copper solutions.  Having watched the constancy of the 10GBASE-T market demand for a number of years, it looks like the promise has finally be prepared and the tidal wave of 10 Gigabit copper is coming. 

Efficiency and data rates

A quick Google search of 10GBASE-T and power will find a large number of articles and pundits decrying that the technology uses too much power to be energy efficient.  Generally, the more recent and relevant articles will compare today's Nth-generation 1000BASE-T devices, which consume around 500mW to first-generation shipping 10GBASE-T PHYs, which consume in the neighborhood of 10 Watts.  Even considering that the next generation of 10GBASE-T PHYs will  be roughly half the power of today's, this makes it look like a long way to go for 10GBASE-T to be energy efficient; however, that is, until you put it in perspective.

Now, when a 10Gb link is used to carry traffic that could easily be sent on a 1Gb link, without any extra hardware (e.g., extra NICs, aggregation hardware, switching ports & fabric), the analysis above makes sense.  However, that would be like comparing the energy efficiency of a train carrying a single passenger to a small car driving the same route.  The train is efficient simply because it CAN carry more.  When it is below a critical carrying capacity, its efficiency is handicapped.  With this metric in hand, if other-than-PHY hardware is considered (server CPUs, MACs, Switch ports), first generation 10GBASE-T links will already be more efficient when fully loaded, and, by the second generation, 10GBASE-T PHYs by themselves will be efficient on a Watt / (Gbit/second) basis than 1000BASE-T PHYs.  The question is, "How well are 10Gb links going to be utilized?"

K. Lloyd recently posted in the Intel "Server Room"  (http://communities.intel.com/openport/blogs/server/2008/01/29/almost-free-data-center-capacity)  that today's data centers are often 5-15% utilized, but that a conceivable target for next generation data centers would be as high as 75%.  Looking back at my last posting, I mentioned the macro approach of consolidating many links into one, driven by data center virtualization - consolidating multiple servers (and switch ports) together.  A quick look at Cisco's data center visions (http://blogs.cisco.com/datacenter/2008/02/expecting_and_getting_more_fro.html)  in addition to Intel's (http://communities.intel.com/openport/blogs/server/2008/02/13/data-center-fabric) expands further on this vision, by consolidating not only traditional ethernet applications but also bringing in storage, currently carried today on fibre channel and other networks.  The result, I believe, is that you can expect to see links filled to the breaking point, fully amortizing each link.

In comparison, when 1000BASE-T was adopted, it began around 6 Watts per port, at a time when 100BASE-T PHYs could be found around 300mW , the same roughly 20:1 ratio seen today.  1000BASE-T came down first by a factor of 2 and then more incrementally, but after that first step, the big change in efficiency comes from the ability to carry 10X the traffic, even without the consolidation we see at 10Gb today.

In short, energy efficiency in Ethernet comes directly out of one of the mantra's of for Ethernet success: growing at 10X the performance with a moderate increase in cost, which would include energy.  Already in the early generations we are in sight of energy parity on a Gigabit-per-second per Watt basis for heavily loaded links, and with all the changes happening in the network driving 10G, we can expect the efficiency curve to run rapidly.