Up and Down the Network Stack

Whence, Whither Ethernet: 400G or Terabit

Guest Blog by Bruce Tolley, VP Marketing Solarflare Communications

Last week, I attended the Ethernet Alliance's first Technology Exploration Forum which was ably moderated by John D'Ambrosia, a Force10 employee and chair of the IEEE 802.3 40/100G task force (also known as IEEE P802.3ba). The day started with a key note by by David Law of 3Com, currently Chairman of the IEEE 802.3 Working Group.  Mr. Law gave an excellent historical and standards perspective on Ethernet, observed that the 40/100G is standardizing no fewer than eight physical layers to cover various applications of 40G and 100G including 1 meter backplane connections, discussed how the Ethernet group needs a supermajority of 75% of the votes to achieve consensus on technical issues, and outlined current standards work defining converged Ethernet such as data center bridging. 

IEEE 802.3 is the standards development organization (SDO) that first standardized Ethernet in the 1980s following up on the pioneering specmanship work of Digital Electronics, Intel and Xerox (the so-called DIX document).  

Although the ink is not yet dry on the 40/100G spec, many of the sessions sounded like preliminary versions of what IEEE 802.3 committee calls “calls for interest” on 25 G serial for backplanes and 40G serial on duplex fiber for network interconnect.  Looking even further into the future, Chris Cole, an engineer with Finisar, presented his ideas on 400G optical interfaces for another yet to be started Ethernet standard.  

Rising above Layer 1 PHY issues, there were noteworthy presentations by IT professionals designing and operating real networks at Facebook and Google, just to name two. Both Facebook and Google today are installing servers that attach at PCIe gen 1 and gen 2 speeds of up to 20Gbps bidirectionally. The need to aggregate and switch thousands of 20Gbps attached network servers increases the pressure for a higher capacity switching fabric. 

Donn Lee, a network engineer for Facebook (Palo Alto, Calif.), said his company's rapidly expanding network could use multiple 100G Ethernet systems today if he could get his hands on them. The standard is in a late stage of being finalized by the IEEE 802.3ba group.   Facebook is building server clusters to support the estimated 300 million Facebook users. Mr. Lee reported that his bandwidth requirements for the network in Facebook's data centers more than double every year. Lee said his company has considered building its own Ethernet switches while it waits for vendors to deliver switches that go beyond today's 400G switching capacity.  Both the presenter from Facebook and from Google expressed desire for network interfaces at speeds greater than 100G.   Deutsche Telekom agreed that bandwidth needs are growing and called for a terabit Ethernet effort to start soon. 

By show of hands, about a third of those attending the meeting said they would support work on a 400G Ethernet standard.   While the Google architect showed how he was using fat trees to build capacity into his network,  there was no technical presentation by a systems OEM on the ability to build multi terabit-switches that could support  multiple 400G links in the foreseeable future

In addition to the exploration of higher speed technologies, Mike Bennett, chair of the Energy Efficient Ethernet task force,  discussed protocols for energy efficiency.  The IEEE 802.3 group is soon to publish a protocol that enables 100BASE-T, 1000BASE-T, and 10GBASE-T Ethernet switches and NICs to signal when they have little or no traffic on the wire and drop into a state that consumes substantially less power.  I’ll report on that in a future post.

Solarflare / SFP+ / STAC

Lot of folks seem to think that Solarflare is 10GBASE-T only. So here's a picture of a nice clean <7W SFP+ NIC. Incidentally this little number is kicking butt on the Street. Here's a STAC report which shows particularly good latency and jitter performance in conjunction with 29West/Cisco/IBM.

The Evolution of Data Networks on Wall Street

I’ve recently finished reading an outstanding book about the history of Wall Street, called “The Great Game: The Emergence of Wall Street as a World Power: 1653-2000”, written by John Steele Gordon. Gordon provides a detailed analysis of the true factors behind the emergence of New York’s Wall Street as the capital of the financial world. Included in his analysis are how the Erie Canal opened the west to trade down the Hudson and how the natural harbors of New York City made trade conducive.

For myself, the most fascinating elements of Wall Street’s rise to greatness was the sheer determination of the “Street” to improve the speed of information from which they traded equities and commodities. From ships that carried trading prices from the colonies to England, to today’s modern Internet, the goal has been the same, increase the volume and decrease the latency of information. So relentless in this task were the pioneers of Wall Street, that during the Civil War, the “Street” sent agents to the battlefields to monitor the outcomes. As the north printed Greenbacks and the South printed Confederate States of America Dollars, the Street was trading on the relative values depending on how the war was shaping up. Their network of agents was so efficient that they were able to relay the outcome of the Battle of Gettysburg even before President Lincoln knew. From The Great Game… Lincoln wished he could take those gold speculators betting on Union defeats at Gettysburg and Chancellorsville and "have their devilish heads shot off". Of course, this sentiment, for some, remains strong on Wall Street today.

 
Before New York City’s rise on Wall Street, Philadelphia was actually the financial center. Trading stocks between the two cities was highly desirable and first accommodated by horseback. But ingenuity and perhaps some ancient techniques took hold. From the roof of the Exchange in New York, men with telescopes and semaphore flags would relay stock prices from hilltop to hilltop and could reach Philadelphia in just 30 minutes, a bizarrely efficient network over 100 miles!

Well it’s obvious that the Telegraph, and perhaps more importantly, Samuel Morse’s practical implementation commissioned in 1844, made a quantum leap in the speed and cost of networking. In 1866, the first sustainable transcontinental telegraph emerged. Not surprisingly, the predominant use of this new network was not sending messages to Grandma, but rather to transmit and receive trading data. In fact, within weeks of laying that cable, Wall Street was actively trading with the London Stock Exchange. With each and every advance in information speed, the “Street” grew in volume and reach.

Fast forwarding through over a century and a half of astounding innovations in computers, semiconductors, and networking technologies, the trading battlefield has merely changed in form not function. Today, economists, physicists, mathematicians and computer scientists combine to create automated computer based trading systems that decide and trade on stocks in just milliseconds, thousands of times faster than any human can react.

This arms race for speed, transaction rate and latency has created a fertile ground for the expansion of 10Gb/sec Ethernet and related technologies that improve the performance of this new generation of trading systems. While many proprietary and niche network concepts, like Infiniband and iWarp, have attempted to address this issue, financial services IT users continually migrate towards plain, easy to use, low cost Ethernet.

OpenOnload, the open source tool for removing network I/O bottlenecks, used with Solarflare’s Solarstorm controllers and 10GBASE-T 10Xpress PHYs have hit a “Trifecta” of value and performance for today’s modern trading company. While fully protecting the customer’s investment with 10GBASE-T’s backward compatibility and increasing their network speeds tenfold, Solarstorm controllers used in conjunction with OpenOnload deliver an unparalleled level of performance and low power consumption. Customers have achieved 50%-300% performance gains in transaction rates while reducing latency and latency jitter by an order of magnitude. All of this is achieved while maintaining exact Ethernet TCP/UDP compatibility on the wire, absolutely no new protocols or application rewrites required. The only application changes we expect are those that remove their own limitations from taking advantage of this blazing performance improvement.

Solarstorm and 10Xpress products are the result of 20 years of research and development in the performance of high-speed networks. Realized in Solarflare’s current product line up of products, these technologies will soon be available as a single integrated device for deployment in low cost Server Adapter Cards and LAN on Motherboard implementations.

Russell Stern

CEO, Solarflare Communications

Xen 10GE Bidirectional Line Rate with Direct Guest Access

I've just finished the Iliad and moved straight onto Morte Arthure from that great 2nd hand bookshop in RDU. Get the theme yet? Yes of course about the only constant theme on this blog - the principle that conclusions on system performance should be deduced from a trajectory of data over time.

So here's another data point regarding accelerated virtualized IO performance. Recall that giving a guest safe, direct access to I/O hardware can remove many of the overheads associated with virtualization. We published plenty of data on older chipsets, here's some recent data taken using a Nehalem processor:

Experiment is 2 physical servers, A,B, connected back to back with 10G Ethernet. Each server is running XenServer 5.0 Update 3  with a single 8 logical core Nehalem CPU.

Method is to run pairs NetPerf TCP_STREAM between guests (Linux RHEL5) and measure aggregate throughput with and without acceleration.

Non accelerated
=============
4 guests A-B + 4 guests B-A 

(A -> B) 1094 + 1068 + 1046 + 1128 = 4336 Mbps
(B -> A) 1019 + 1028 + 1050 + 1021 = 4118 Mbps
Total: 8.45 Gbps
Bottleneck: both dom0 CPU

Accelerated
==========
4 guests A-B + 4 guests B-A

(A->B) 2355 + 2318 + 2296 + 2289 = 9258 Mbps
(B->A) 2285 + 2295 + 2315 + 2350 = 9245 Mbps 
Total: 18.50 Gbps 

There you go. Now you may argue that the non accelerated case could have been faster if we'd used a more recent version of Xen. But that wasn't my point. Point was that now you can hit bidirectional line rate at 1500MTU while virtualised.

That's a world first and you read it here ...

Back to Morte Arthure - get past the chivalry, this guy makes Atilla the Hun look tame.

[BTW I'm not talking about Thomas Malory's Le Morte d'Arthur,  I'm referring to the earlier alliterative and stanzaic texts -  more realistic ]

t Thomas Malory's Le Morte d'Arthur

Emerging trends in high-speed interfaces

I had the pleasure a couple of weeks back to participate in a forum on high-speed standardized interfaces at ISSCC.  Within this forum, most of the talks were of SERDES-like interfaces (PCIe, SATA, USB3, and SFP+), and then there was 10GBASE-T.   In the course of discussion, it struck me how different the approach to infrastructure is for 10GBASE-T than for the other technologies in the forum.

 

Like many engineers, I start with the constraints of the problem.  I consider it good analytical technique to look first at what the differing assumptions and constraints are, in order to compare technologies in their form, fit and function.

10GBASE-T, as we have written from time to time, begins with the constraint of preserving infrastructure practices in the enterprise.  Structured cabling (up to 100 meters, up to 4-connector, twisted pair) is the medium of choice in this environment, and, while incremental upgrades are allowed, backwards compatibility and maintenance of practice is the rule.  This allows spread of the technology with a minimum of retraining of the thousands of operational staff in the enterprise IT environment.

 

On the other hand, within-the-box technologies, such as PCIe & SATA often re-invent cabling to suit the PHY or other physical requirements.  Shielded cabling is made specifically for the application, and this is perfectly acceptable, since it comes and goes with new hardware purchases.

 

What brought all this to a sharp contrast were the two other technologies.  USB3 had to deal with something very similar to 10GBASE-T’s challenge: maintaining backwards compatibility with an installed base of USB devices, and existing USB cabling.  While USB cabling is not infrastructure, it is quite common, and the connector shares the same familiarity as 10GBASE-T’s RJ45.  USB3 addresses backward compatibility by inserting an additional transmission channel (shielded twin ax) onto the familiar USB connector.  USB3 devices connected to older cables will not work at the high speed, and I expect some level of cable-confusion will result, with two new high-speed USB3 devices running as USB2 because the cable (which connects just fine) is not also high speed. How this approach plays in the market will be interesting for those that follow Ethernet.

 

On the other hand, SFP+ tries to take Ethernet into the “within-the-box” realm of PCIe and SATA.  SFP+ direct-attach copper connections are within-rack and engineered media may work fine when the OEMs provide fully configured racks, and even full rooms which are brought in by truck.  The operations staff never touches the cabling, and the operational configuration looks more like a campus network than a LAN.   This happens often in “mega datacenters” (e.g., Google, Microsoft, AOL, Yahoo) which get a lot of press.   See February’s IEEE Spectrum (http://www.spectrum.ieee.org/feb09/7327) for details.    However, in enterprise data centers, which are smaller in scale, but far more numerous, we have yet to see if this approach will fly.

 

We may be witnessing a segmentation of the “data center” as a market, into a specialized mega data center and the traditional data center market.  But more on that later…

 

10GBASE-T - "Yes You Can"

Here's three reasons to say no to expensive, unhealthy and unsustainable travel and yes to 10GBASE-T

1. Airports are ugly.

Having had the annual Christmas travel virus experience with the family - infecting ourselves and F&F in 2 continents, I was particularly interested in this report in the Jan 13 2009 - Proc Nat Academy of Sciences

In short, scientists have identified a set of three genes from the 1918 "Spanish flu" - a virus that killed between 20 and 50 million people. When spliced onto contemporary flu, the resulting virus was shown to replicate in deep lung tissue just as the 1918 virus was reported doing.

Pretty sobering to think about as you sit down next to that passenger with the rheumy eyes.

2. The World is Warming.

Ok everyone, you all listened to Obama's inauguration speech. The USA is no longer in denial.

3. The Sky is Falling.

World is in probably the worst recession since the 1930s. Save money and invest wisely in future proof technologies.

Happy New Year

Sometimes you have to invent...

There have been a lot of articles of late about deficiencies of our patent system and calls for reform.  In my career, I’ve never been one to “paper up” every design feature that came across my desk.  Many appear to be clever, but obvious applications of basic engineering principles, sometimes cast into a different nomenclature.  However, in the pursuit of new problems, often some of the more difficult technical challenges result in new inventions, and I, and the engineers who work with me, will file them as patents.  When my thesis advisor and friend, Dr. Edward Posner, was killed in an untimely accident, one of the things I received from his widow was his copy of the 1950’s-era Bell Labs patent manual – oddly, a very thin book.  It explained how patents were central, not only to the Bell System, but to society, as a way of passing on technical progress.  They are so central, that in fact, that they are established in the United States Constitution:

“Congress shall have power . . . To promote the progress of science and useful arts, by securing for limited times to authors and inventors the exclusive right to their respective writings and discoveries.”

So it is with a recent patent issued to Solarflare (http://www.hpcwire.com/topic/interconnects/Solarflare-Granted-Patent-for-Low-Power-10GBASE-T-Silicon-36178704.html).

  In our early explorations of the 10GBASE-T problem (back before work really even began on the standard), it became apparent that silicon processes and state-of-the-art analog components would be greatly challenged to meet both the noise and echo rejection requirements for 10GBASE-T.  This was reflected in the discussions of the 10GBASE-T study group, with calls of “impossible” from many participants who had shown their skill in 1000BASE-T designs. 

As silicon processes improved, other vendors went ahead with traditional implementations of “replica DACs” as in 1000BASE-T.  The resulting solutions used active, power consuming components, and require a separate filter element for exclusion of many types of EMI.

In contrast, to solve the problem at an earlier node, Solarflare engineers simply had to invent a solution which both allowed low power, high echo rejection, and a tuned receiver filter.  Chris Pagnanelli came up with the solution, and, to his credit, has now been awarded a United States Patent for his original work.  The result has proven itself to us over time as providing not only an inherent power advantage (by replacing active, power-consuming circuitry with passive board traces), but also robust performance on long cabling and particularly in the presence of cellphones and WiFi access points.  This kind of innovation is what the patent system was enshrined in the constitution for – teaching the industry new and better ways of getting the job done.

10GBASE-T Here Now ...

Wow - we're all still high from sampling the production spin of our new 10GBASE-T PHY.  This one is better than half the power of our closest competitors and should be great for high density 10GBASE-T switches available in 2009.

Take a look at:
  http://www.solarflare.com/news/news_press_show.php?release=20081203

BTW you may have noticed the comments regarding our operation on unshielded Cat 5e cables and without susceptibility to cell phone interference .... that's also something unique from Solarflare.

Why Apps Can't Run Faster

This post by Ed Sperling is well worth a read. He points out that if your application doesn't parallelize well (as most don't) then a multi-core machine isn't going to help. Some additional comments on this are:

- Increasing utilization (number of VMs per machine) means that Server I/O will continue to increase. Expect to see 10GBASE-T technology as a major enabler for multi-core machines which would otherwise be horribly I/O bottlenecked (aka How many 1G links do you currently have from the back of your virtualized servers!?).

- Increasing the utilization of virtual OS deployments means that MTBF for virtualized x86 machines needs to come under the spotlight. Therefore virtual OS providers need to keep code size under control and therefore limit feature additions.

- The lack of increasing single x86 core performance means that the days of code bloat for operating systems and applications are also finally over.

- Expect to see more specialized hardware on the die in place of some of the cores, targeted at important vertical markets which otherwise would not be able to take advantage of multi-core.

For more (still light) information on this topic, read  Virtualization and multicore x86 CPUs

OpenOnload at Supercomputing

There will be a 10GBASE-T cluster at SC' 08 in the SMC booth running OpenOnload.
See for yourselves the benefits of commodity high-performance networking.

openonload 2.3.0013 latency

Martin asked me to give the UDP latency from hpcbench.

24.8us of the RTT is the switch. Gives a back to back UDP RTT/2 latency of 6.8us.

------

onload  ./udptest -ah 192.168.10.1
udptest[20598]: Copyright Solarflare Communications 2006-2008, Level 5 Networks 2002-2005 v2.3.0013
UDP Round Trip Time (1) : 38.514 usec
UDP Round Trip Time (2) : 38.529 usec
UDP Round Trip Time (3) : 38.498 usec
UDP Round Trip Time (4) : 38.507 usec
UDP Round Trip Time (5) : 38.510 usec
UDP Round Trip Time (6) : 38.519 usec
UDP Round Trip Time (7) : 38.508 usec
UDP Round Trip Time (8) : 38.499 usec
UDP Round Trip Time (9) : 38.492 usec
UDP Round Trip Time (10) : 38.499 usec
10 trials with message size 64 Bytes.
UDP RTT (64-byte) min/avg/max = 38.492/38.508/38.529 usec