Up and Down the Network Stack

10Gig Tipping Point

It's been a long time.  No, I'm not talking about my writing, but rather the wait for 10Gig to get to that magic "10X the bandwidth for 3X the price" that everyone talks about.  It's finally here and we're seeing end-user adoption.  See

http://www.networkcomputing.com/data-networking-management/10gige-tipping-point-reached---problems-remain.php

for an end-user's description of the compelling 10Gig upgrade.  Funny as it is, it seems at least this one blogger is not focused on new protocols or new paradigms of wiring datacenters, but rather simply getting more bandwidth over plain old ethernet.  Sometimes those of us close to the technology get enchanted by the new stuff, and we need a user to ground us in the important things.  Now that the tipping point is reached, that is going to be happening more and more.  New storage licenses for FCoE or special wiring for SFP+ still get in the way, but more on that later. 

It seems he's waiting for the 10GBASE-T value proposition to solve some of the mass-adoption issues, but that's right around the corner:

"Someday soon 10gigE will be as simple as 1gig with 10GBASE-T competing with SFP+ passive cables for short haul and SFP+ optics for long. 10GBASE-T uses familiar, if higher quality, cables and allows for 100/1000/10G trispeed ports, but today's 10GBASE-T solutions use 5W or more per port and cost a bit more than SFP+ cables."  - I suspect the power and cost comparisons come straight from the SFP+ salesment - they're not from box-level products, where the difference isn't material, and the costs he quotes on cabling are greater than anything coming from 10GBASE-T.

Expect  a lot 10GBASE-T switch and NIC announcements in the coming year, and a lot of interesting action...

Awesome New Dual-Port SFP+ Low-Power Low-Latency NIC from Solarflare

And for the real performance freaks out there,  we of course have an SFP+ variant also now sampling.

 

Again, very clean. Notice that we're driving the SFP+ ports directly from a single piece of silicon. Saves a lot of power and latency.

• Low power:                              < 5W for the NIC while under load
• Low latency:                             2us wire to app latency with full TCP/UDP POSIX compliance
• High Transaction rates:            millions of msg/s per CPU core for market data applications
• Deep Buffers:                           maintain performance during micro-bursts
• High throughput:                      fully saturating both ports Linux and Windows
• Low CPU overhead:                   40Gb/s @ < 25% of a 2.6Ghz DP server
• Consistent Performance:           perfect for deterministic / low-jitter application requirements
• Massively scalable virtualization:  PCeGen2 5GT/s, 256 IOV Virtual Functions, 2048 virtual NIC interfaces  - Rich L3 internal packet routing capabilities - Hard QoS shaping - Full suite of offloads and onloads including TCP, UDP, iSCSI and multicast acceleration

Contact sales@solarflare.com for more information.

Awesome New Dual-Port 10GBASE-T Low-Power NIC from Solarflare

Here's a sneak preview of Solarflare's new Dual-Port 10GBASE-T NIC

As you can see, very sweet looking, but there's also a lot under the hood:

• Low power - 12W for the NIC while under load
• Low latency - < 8us app to app in 10GBASE-T mode (RTT _1/2 using NetPerf _RR )
• High throughput - no problem fully saturating both ports in both directions on Linux and Windows
• Low CPU overhead - gone are the days when 10G was considered "hard for a server"
• Massively scalable virtualization - PCeGen2 5GT/s, 256 IOV Virtual Functions, 2048 virtual NIC interfaces  - Rich L3 internal packet routing capabilities - Hard QoS shaping - Full suite of offloads and onloads including TCP, UDP, iSCSI and multicast acceleration
• The NIC has backwards compatibility with existing 1000BASE-T plant and is engineered using Solarflare's rock-solid PHY technology.

If you want to get in the allocation queue for this or our SFP+ varients, contact sales@solarflare.com

Record Breaking Cloud Compute Platform

Check out Simon Crosby's post

"....  what may well be a virtualization record for 2009 - the results of the performance testing undertaken by a cloud customer of ours, nicknamed Project Monterey", that has delivered what I believe to be a record in performance for a service provider cloud offering for system I/O - using Solarflare IOV NICs, switching from Arista, and XenServer."

This effort was using the Citrix Xen IO device acceleration pluggin framework which was contributed by Solarflare and previously reported here and here. Anyone interested in the architectural details can take a look at the EuroPar07 paper

Very nice to see this technology entering the cloud space and more significantly this activity marks the end of the era when high-performance IO was considered a bottleneck when virtualized.

Press Release is at:
http://www.solarflare.com/news/news_press_show.php?release=20091207a

Whence, Whither Ethernet: 400G or Terabit

Guest Blog by Bruce Tolley, VP Marketing Solarflare Communications

Last week, I attended the Ethernet Alliance's first Technology Exploration Forum which was ably moderated by John D'Ambrosia, a Force10 employee and chair of the IEEE 802.3 40/100G task force (also known as IEEE P802.3ba). The day started with a key note by by David Law of 3Com, currently Chairman of the IEEE 802.3 Working Group.  Mr. Law gave an excellent historical and standards perspective on Ethernet, observed that the 40/100G is standardizing no fewer than eight physical layers to cover various applications of 40G and 100G including 1 meter backplane connections, discussed how the Ethernet group needs a supermajority of 75% of the votes to achieve consensus on technical issues, and outlined current standards work defining converged Ethernet such as data center bridging. 

IEEE 802.3 is the standards development organization (SDO) that first standardized Ethernet in the 1980s following up on the pioneering specmanship work of Digital Electronics, Intel and Xerox (the so-called DIX document).  

Although the ink is not yet dry on the 40/100G spec, many of the sessions sounded like preliminary versions of what IEEE 802.3 committee calls “calls for interest” on 25 G serial for backplanes and 40G serial on duplex fiber for network interconnect.  Looking even further into the future, Chris Cole, an engineer with Finisar, presented his ideas on 400G optical interfaces for another yet to be started Ethernet standard.  

Rising above Layer 1 PHY issues, there were noteworthy presentations by IT professionals designing and operating real networks at Facebook and Google, just to name two. Both Facebook and Google today are installing servers that attach at PCIe gen 1 and gen 2 speeds of up to 20Gbps bidirectionally. The need to aggregate and switch thousands of 20Gbps attached network servers increases the pressure for a higher capacity switching fabric. 

Donn Lee, a network engineer for Facebook (Palo Alto, Calif.), said his company's rapidly expanding network could use multiple 100G Ethernet systems today if he could get his hands on them. The standard is in a late stage of being finalized by the IEEE 802.3ba group.   Facebook is building server clusters to support the estimated 300 million Facebook users. Mr. Lee reported that his bandwidth requirements for the network in Facebook's data centers more than double every year. Lee said his company has considered building its own Ethernet switches while it waits for vendors to deliver switches that go beyond today's 400G switching capacity.  Both the presenter from Facebook and from Google expressed desire for network interfaces at speeds greater than 100G.   Deutsche Telekom agreed that bandwidth needs are growing and called for a terabit Ethernet effort to start soon. 

By show of hands, about a third of those attending the meeting said they would support work on a 400G Ethernet standard.   While the Google architect showed how he was using fat trees to build capacity into his network,  there was no technical presentation by a systems OEM on the ability to build multi terabit-switches that could support  multiple 400G links in the foreseeable future

In addition to the exploration of higher speed technologies, Mike Bennett, chair of the Energy Efficient Ethernet task force,  discussed protocols for energy efficiency.  The IEEE 802.3 group is soon to publish a protocol that enables 100BASE-T, 1000BASE-T, and 10GBASE-T Ethernet switches and NICs to signal when they have little or no traffic on the wire and drop into a state that consumes substantially less power.  I’ll report on that in a future post.

Solarflare / SFP+ / STAC

Lot of folks seem to think that Solarflare is 10GBASE-T only. So here's a picture of a nice clean <7W SFP+ NIC. Incidentally this little number is kicking butt on the Street. Here's a STAC report which shows particularly good latency and jitter performance in conjunction with 29West/Cisco/IBM.

The Evolution of Data Networks on Wall Street

I’ve recently finished reading an outstanding book about the history of Wall Street, called “The Great Game: The Emergence of Wall Street as a World Power: 1653-2000”, written by John Steele Gordon. Gordon provides a detailed analysis of the true factors behind the emergence of New York’s Wall Street as the capital of the financial world. Included in his analysis are how the Erie Canal opened the west to trade down the Hudson and how the natural harbors of New York City made trade conducive.

For myself, the most fascinating elements of Wall Street’s rise to greatness was the sheer determination of the “Street” to improve the speed of information from which they traded equities and commodities. From ships that carried trading prices from the colonies to England, to today’s modern Internet, the goal has been the same, increase the volume and decrease the latency of information. So relentless in this task were the pioneers of Wall Street, that during the Civil War, the “Street” sent agents to the battlefields to monitor the outcomes. As the north printed Greenbacks and the South printed Confederate States of America Dollars, the Street was trading on the relative values depending on how the war was shaping up. Their network of agents was so efficient that they were able to relay the outcome of the Battle of Gettysburg even before President Lincoln knew. From The Great Game… Lincoln wished he could take those gold speculators betting on Union defeats at Gettysburg and Chancellorsville and "have their devilish heads shot off". Of course, this sentiment, for some, remains strong on Wall Street today.

 
Before New York City’s rise on Wall Street, Philadelphia was actually the financial center. Trading stocks between the two cities was highly desirable and first accommodated by horseback. But ingenuity and perhaps some ancient techniques took hold. From the roof of the Exchange in New York, men with telescopes and semaphore flags would relay stock prices from hilltop to hilltop and could reach Philadelphia in just 30 minutes, a bizarrely efficient network over 100 miles!

Well it’s obvious that the Telegraph, and perhaps more importantly, Samuel Morse’s practical implementation commissioned in 1844, made a quantum leap in the speed and cost of networking. In 1866, the first sustainable transcontinental telegraph emerged. Not surprisingly, the predominant use of this new network was not sending messages to Grandma, but rather to transmit and receive trading data. In fact, within weeks of laying that cable, Wall Street was actively trading with the London Stock Exchange. With each and every advance in information speed, the “Street” grew in volume and reach.

Fast forwarding through over a century and a half of astounding innovations in computers, semiconductors, and networking technologies, the trading battlefield has merely changed in form not function. Today, economists, physicists, mathematicians and computer scientists combine to create automated computer based trading systems that decide and trade on stocks in just milliseconds, thousands of times faster than any human can react.

This arms race for speed, transaction rate and latency has created a fertile ground for the expansion of 10Gb/sec Ethernet and related technologies that improve the performance of this new generation of trading systems. While many proprietary and niche network concepts, like Infiniband and iWarp, have attempted to address this issue, financial services IT users continually migrate towards plain, easy to use, low cost Ethernet.

OpenOnload, the open source tool for removing network I/O bottlenecks, used with Solarflare’s Solarstorm controllers and 10GBASE-T 10Xpress PHYs have hit a “Trifecta” of value and performance for today’s modern trading company. While fully protecting the customer’s investment with 10GBASE-T’s backward compatibility and increasing their network speeds tenfold, Solarstorm controllers used in conjunction with OpenOnload deliver an unparalleled level of performance and low power consumption. Customers have achieved 50%-300% performance gains in transaction rates while reducing latency and latency jitter by an order of magnitude. All of this is achieved while maintaining exact Ethernet TCP/UDP compatibility on the wire, absolutely no new protocols or application rewrites required. The only application changes we expect are those that remove their own limitations from taking advantage of this blazing performance improvement.

Solarstorm and 10Xpress products are the result of 20 years of research and development in the performance of high-speed networks. Realized in Solarflare’s current product line up of products, these technologies will soon be available as a single integrated device for deployment in low cost Server Adapter Cards and LAN on Motherboard implementations.

Russell Stern

CEO, Solarflare Communications

Xen 10GE Bidirectional Line Rate with Direct Guest Access

I've just finished the Iliad and moved straight onto Morte Arthure from that great 2nd hand bookshop in RDU. Get the theme yet? Yes of course about the only constant theme on this blog - the principle that conclusions on system performance should be deduced from a trajectory of data over time.

So here's another data point regarding accelerated virtualized IO performance. Recall that giving a guest safe, direct access to I/O hardware can remove many of the overheads associated with virtualization. We published plenty of data on older chipsets, here's some recent data taken using a Nehalem processor:

Experiment is 2 physical servers, A,B, connected back to back with 10G Ethernet. Each server is running XenServer 5.0 Update 3  with a single 8 logical core Nehalem CPU.

Method is to run pairs NetPerf TCP_STREAM between guests (Linux RHEL5) and measure aggregate throughput with and without acceleration.

Non accelerated
=============
4 guests A-B + 4 guests B-A 

(A -> B) 1094 + 1068 + 1046 + 1128 = 4336 Mbps
(B -> A) 1019 + 1028 + 1050 + 1021 = 4118 Mbps
Total: 8.45 Gbps
Bottleneck: both dom0 CPU

Accelerated
==========
4 guests A-B + 4 guests B-A

(A->B) 2355 + 2318 + 2296 + 2289 = 9258 Mbps
(B->A) 2285 + 2295 + 2315 + 2350 = 9245 Mbps 
Total: 18.50 Gbps 

There you go. Now you may argue that the non accelerated case could have been faster if we'd used a more recent version of Xen. But that wasn't my point. Point was that now you can hit bidirectional line rate at 1500MTU while virtualised.

That's a world first and you read it here ...

Back to Morte Arthure - get past the chivalry, this guy makes Atilla the Hun look tame.

[BTW I'm not talking about Thomas Malory's Le Morte d'Arthur,  I'm referring to the earlier alliterative and stanzaic texts -  more realistic ]

t Thomas Malory's Le Morte d'Arthur

Emerging trends in high-speed interfaces

I had the pleasure a couple of weeks back to participate in a forum on high-speed standardized interfaces at ISSCC.  Within this forum, most of the talks were of SERDES-like interfaces (PCIe, SATA, USB3, and SFP+), and then there was 10GBASE-T.   In the course of discussion, it struck me how different the approach to infrastructure is for 10GBASE-T than for the other technologies in the forum.

 

Like many engineers, I start with the constraints of the problem.  I consider it good analytical technique to look first at what the differing assumptions and constraints are, in order to compare technologies in their form, fit and function.

10GBASE-T, as we have written from time to time, begins with the constraint of preserving infrastructure practices in the enterprise.  Structured cabling (up to 100 meters, up to 4-connector, twisted pair) is the medium of choice in this environment, and, while incremental upgrades are allowed, backwards compatibility and maintenance of practice is the rule.  This allows spread of the technology with a minimum of retraining of the thousands of operational staff in the enterprise IT environment.

 

On the other hand, within-the-box technologies, such as PCIe & SATA often re-invent cabling to suit the PHY or other physical requirements.  Shielded cabling is made specifically for the application, and this is perfectly acceptable, since it comes and goes with new hardware purchases.

 

What brought all this to a sharp contrast were the two other technologies.  USB3 had to deal with something very similar to 10GBASE-T’s challenge: maintaining backwards compatibility with an installed base of USB devices, and existing USB cabling.  While USB cabling is not infrastructure, it is quite common, and the connector shares the same familiarity as 10GBASE-T’s RJ45.  USB3 addresses backward compatibility by inserting an additional transmission channel (shielded twin ax) onto the familiar USB connector.  USB3 devices connected to older cables will not work at the high speed, and I expect some level of cable-confusion will result, with two new high-speed USB3 devices running as USB2 because the cable (which connects just fine) is not also high speed. How this approach plays in the market will be interesting for those that follow Ethernet.

 

On the other hand, SFP+ tries to take Ethernet into the “within-the-box” realm of PCIe and SATA.  SFP+ direct-attach copper connections are within-rack and engineered media may work fine when the OEMs provide fully configured racks, and even full rooms which are brought in by truck.  The operations staff never touches the cabling, and the operational configuration looks more like a campus network than a LAN.   This happens often in “mega datacenters” (e.g., Google, Microsoft, AOL, Yahoo) which get a lot of press.   See February’s IEEE Spectrum (http://www.spectrum.ieee.org/feb09/7327) for details.    However, in enterprise data centers, which are smaller in scale, but far more numerous, we have yet to see if this approach will fly.

 

We may be witnessing a segmentation of the “data center” as a market, into a specialized mega data center and the traditional data center market.  But more on that later…

 

10GBASE-T - "Yes You Can"

Here's three reasons to say no to expensive, unhealthy and unsustainable travel and yes to 10GBASE-T

1. Airports are ugly.

Having had the annual Christmas travel virus experience with the family - infecting ourselves and F&F in 2 continents, I was particularly interested in this report in the Jan 13 2009 - Proc Nat Academy of Sciences

In short, scientists have identified a set of three genes from the 1918 "Spanish flu" - a virus that killed between 20 and 50 million people. When spliced onto contemporary flu, the resulting virus was shown to replicate in deep lung tissue just as the 1918 virus was reported doing.

Pretty sobering to think about as you sit down next to that passenger with the rheumy eyes.

2. The World is Warming.

Ok everyone, you all listened to Obama's inauguration speech. The USA is no longer in denial.

3. The Sky is Falling.

World is in probably the worst recession since the 1930s. Save money and invest wisely in future proof technologies.

Happy New Year