Up and Down the Network Stack

It’s All About Utility – part 1 - 10Gig Data Centers vs. 10Gig Racks

Every so often, those of us who worry about the esoteric details of packet error rates, signal-to-noise ratios, and virtualized operating systems need stick our heads up and see what happens day-to-day as people install our equipment.  Some interesting things are happening with Data Center infrastructure these days, and it’s showing up in the near-term deployment of 10Gig in the data center.

To date, 10 Gigabit deployments have been focused around isolated applications, and the types of solutions offered have been suited towards those small deployments.  Much of 10 Gigabit in the data center today has been limited reach 10GBASE-CX4 and now emerging as SFP+ (including passive copper solutions).  These have been focused in such a way as to promote within-rack interconnects, to make islands of 10Gigabit performance.  Racks and systems have been architected around “top-of-rack” switching, to make the rack, not the network equipment unit itself, the building block.  (http://www.internetworkexpert.org/2008/04/08/cisco-nexus-5000-announced-today/).  Like CX4 before it, SFP+ passive copper is architected for within the rack, as Brad Hedlund states, “This means your SFP+ twin ax copper will remain within the rack as it does not have the distance to travel throughout the data center.”  Thus, 10 Gigabit Ethernet today gets installed in hot-spots (both by traffic needs AND thermally), to build 10 Gigabit Clusters, with SFP+ taking over from 10GBASE-CX4.

One might ask whether data centers are going to a rack-by-rack architecture.  For small, early applications, this may happen, but in the long run, I don’t think so.  Unfortunately, rack-by-rack cabling goes against the last 10 years of cabling practices, known in the industry as structured cabling.  It allows network managers to upgrade and migrate cabling and active network infrastructures independently.  Structured cabling allowed the transition to fast Ethernet and gigabit Ethernet.  I checked back into the trenches, in case one might think that the data center physical layout is changing to a rack-by-rack format, but my friends in the cabling industry tell me that they are increasingly putting in Category 6a cabling as people look towards 10GBASE-T.   

10GBASE-T systems, on the other hand, are oriented towards full data centers, with multi-rack interconnects.  They will use the centralized (and often already installed) infrastructure for connecting large numbers of 10Gigabit ports.  Within the next 12 months, we expect to see the current 10 Gigabit islands giving way to full 10Gig-oriented data centers, utilizing the structured cabling infrastructure, which are enabled by 10GBASE-T.

In the next 2 parts, we’ll look at how applications relate to physical connections, and how physical connectivity relates to energy efficiency; with some results that run counter to conventional wisdom.

Auto-Negotiation Part 3 - Why 100Mbps is Important on a 10G PHY

Why are multi-speed 10G/1G/100BASE-T PHYs important?

In prior generations of Ethernet, this proved to be the path towards rapid provisioning of new speeds.  With multi-speed 10GBASE-T NICs in their servers, an end-user may begin to provision 10GBASE-T capable servers prior to upgrading the switching infrastructure and later install the 10GBASE-T switch and upgrade the entire set of servers’ speeds.  This allows for much easier and more rapid provisioning, without a “forklift” upgrade, where entire infrastructures are changed at once.

A second reason, growing in importance, for multi-speed PHYs is power management.  I’ve already said much about Energy Efficient Ethernet, but that is not what I mean.  Here, I am talking about “wake on LAN” capability.   I will say more about this subject separately, but putting it simply, today’s version of server power management puts idle servers into a low-speed mode (100BASE-TX usually), and waits for a “Magic Packet” to “Wake on LAN”.  This way, idle machines can dramatically lower their power consumption.  With the increasing EPA and Industry focus on making devices consume minimal power when not doing work, this feature is a must for servers.  Eventually, we’ll have to see how this plays out with regard to Energy Efficient Ethernet, but that won’t occur for years.  Today, if you want energy efficiency when a server is idle, you support Wake-on-LAN functionality.

Simple Auto-Negotiation enables some pretty important benefits which have helped the dominance of BASE-T copper Ethernet.  However, it is not to be forgotten that this is because it allows new generations to build on the prior generations technological and installed infrastructure base of BASE-T networks.  Now that 10GBASE-T is offering the right lower speeds of Ethernet, we will begin to see the power of Auto-Negotiation as it allows IT managers to asynchronously upgrade the speeds of their servers, AND still support power management (through Wake-on-LAN) for idle systems.

Auto-Negotiation Part 2 – Auto-Neg vs. Speed Switching

Would someone use Auto-Negotiation to switch speeds on-the-fly?

No.  The switching of speeds via Auto-Negotiation occurs at the initiation of a physical-layer link.  It is not something done on-the-fly.  Auto-Negotiation is useful for network speed transitions that happen over long durations in computer time (at least minutes long, since the transition takes a few seconds).  Unlike Energy Efficient Ethernet, which I’ve discussed earlier, the kind of speed transitions that might be facilitated through Auto-Negotiation, would not be expected to be buffered up in a network, as this would require billions and billions of bits stored.

Auto-Negotiation – a simple but powerful idea, often misunderstood

Almost every copper Ethernet device supports “Auto-Negotiation”, or clause 28 of IEEE Std. 802.3.  However, based on questions I’ve been asked, it seems that this simple protocol is often misunderstood. This is the first part of a 3 part series to answer some of the more common questions.

What exactly is Auto-Neg?

Simply put, support for Auto-Negotiation means that two devices can talk with each other and communicate what Ethernet standards each is capable of supporting.  They then agree on the highest common speed, which may be none (they agree to disagree).  There are many Ethernet standards left on the dustbin of history, not offered in modern devices.  The Auto-Negotiation capability is one part of an equation that has allowed twisted-pair copper (BASE-T) devices to dominate the Ethernet market.  The other part is the ability to communicate over a common connector (RJ-45) and media type (UTP copper with structured cabling rules).  The most misunderstood part of Auto-Negotiation is that it does not imply support for any speeds.

Does support for Auto-Neg mean Multi-Speed?

The initial versions of 10GBASE-T PHYs were all single speed, largely because the switch interfaces could only support the 10 Gigabit XAUI interface, and could not make use of a multi-speed PHY.  They supported Auto-Negotiation, but not multiple speeds.  That is changing this year.  The coming generation of 10GBASE-T NICs will support 100Mbps and 1000Mbps in addition to 10Gigabit speeds.  They will Auto-Negotiate with a switch port and provide the highest possible speed, but allowing connection at lower speeds where appropriate.

Energy Efficient Ethernet Update: Low Power Idle baselined at all speeds, Editorial team begins work

At the May Interim meeting of the Energy Efficient Ethernet task force (IEEE P802.3az), the group reached achieved two important milestones.  The IEEE P802.3at Task Force voted to approve a baseline "low power idle" proposal at the 10 Gbps speed, making a consistent low power idle suite for BASE-T transceivers from 100Mbps through 10 Gbps. The consensus proposal, authored by Solarflare, Teranetics and Aquantia representatives allows 10GBASE-T devices to rapidly restore to transmiting data (in <5usec) and refresh their cancellers while minimizing consumed power in the receivers (http://www.ieee802.org/3/az/public/may08/taich_02_0508.pdf ).  Because of this capability, I expect that the generation of 10GBASE-T transceivers employing EEE technology, once it is standardized, (somewhere around 2011) will operate at the same average power consumption as 1000BASE-T transceivers when 1Gbps traffic loads are offered.  To add to that, they will offer the ability to burst at 10Gbps, which will aid storage, video and other high bandwidth tasks. We expect this to dramatically accelerate the proliferation of 10GBASE-T into workstation, client and applications beyond those using 10Gbps Ethernet today.

The completion of a baseline suite for BASE-T transceivers positioned the group to embark on the next phase of standards development: commissioning the editorial team to put together the first draft of the new standard.  Because EEE touches many different speeds and types of Ethernet (from 10BASE-T to 10GBASE-T), the bulk of the standards work will be done by a large number of individual editors from various companies. Solarflare's Gavin Parnaby, has taken on the work of drafting the text.  In July, the Task Force is expected to begin the work of reviewing the draft text and focusing on filling in the remaining TBD items on their march towards a specification which can be balloted.  There's still a long way to go, but this is great progress.

10 Gigabit Ethernet and VMware - A Match Made in Heaven

Thanks to Steve Grantham, who pointed me at a nice blog entry of the above title, which provides a good overview of the technology trends being driven by virtualisation.

Virtual Geek (aka Chad Sakac) says:

Back when I was in the valley (so this was 4 years ago), a buddy of mine worked at a tiny IP (not Internet Protocol - Intellectual Property) company focused on high-end IP blocks for networking and storage (going after Hi/fn and the others in that space).   He showed me their A0 spin, and told me "This chip will do 10GbE BaseT over Cat6 cables, full TCP offload including segment offload, and all iSCSI offload".  COOL. "oh and we think we can mass produce it for $25 per chip".   DROOL.   

Well, fast forward 4 years, and they are out of business :-)

..... GOOD news is that another company has been able to realise this device.

Internet - the future of Broadcasting

I had lunch last weekend with the Director of a Hong Kong based TV station, who asserted that the most important (by viewers) distribution medium for his company was the Internet.

Given that the HK population is generally ahead of the tech-curve, it probably safe to assume that other data-points, such as this , which predict 50% of Internet traffic will be video by 2012 are entirely credible.

For those outside the UK, it's also worth pointing out that last year the BBC started it's own service called iplayer , which is a free (for those with a UK based ISP) 7 day archive of all the BBCs UK broadcast. If you're unlucky enough to be outside of the UK (long summer evenings, punting, etc) then don't worry, there are plenty of resources on the web such as this video to enable your quality viewing experience.

Go Green with Server Consolidation

You may have taken the glossy adverts with something of a pinch of salt, but PG&E (utility company) is serious about offering businesses rebates based reductions in power consumption by virtue of server consolidation. This rebate currently stands at 8c per kilowatt-hour reduction. See here for more details.

PG&E stipulate that servers must be decommissioned in order to gain the rebate. Perhaps they should look more closely at a scheme whereby the servers are re-provisioned either within the organisation or elsewhere in order to avoid the scheme being energy neutral (energy savings balanced by the cost of early replacement of existing servers).

BTW I'm hearing more and more that organizations are employing the migration feature provided by server virtualization in order to load balance for energy efficiency. Live migration of a complete operating system over a 10GBASE-T Ethernet network in total takes order 5seconds to complete  with a down-time measured in milli-seconds - clearly a powerful tool.

Simon / Xen / Cambridge / Citrix / 10GBASE-T

Hey did you know Simon Crosby is one of the most influential people in IT. It's official.

10Gb/s Ethernet with commodity hardware (Revisited)

To update from my somewhat opaque post in February.

Please take a look at this  independent test report from the Broad-Band Test Labs, with some interesting perspective from Steve Broadhead in Computer Weekly

Bottom line is that if we're reporting exactly what's what for 10G Ethernet performance for a decent set of classic micro-benchmarks as well as for virtualization and storage deployments. So download and look no further. We hope that the Solarstorm controller available from SMC speaks for itself.

SMC 10GPCIe-10BT (10GBASE-T Server Adaptor Card)