Up and Down the Network Stack

Ethernet over Ethernet

There’s a great dustbin in the sky for network protocols that never went anywhere. Sadly there’s often been a lot of money spent implementing them. Good examples are:

Ethernet over ATM (aka LANE) – Are there any unfortunates out there still using it?

iSER – Who’s dumb idea was that anyway?

iWARP – Well at least it was implemented over TCP, but with all those overheads did anyone ever expect to get good performance out of it?

I fully expect the old bit bucket to be joined by a couple of new friends.

RoCEE/RoE  -A single vendor, dual-ended technology requiring applications use non-POSIX interfaces in order to get not quite as good performance as Infiniband? 

FCoE -  So you can’t route it, you need special, yet to be invented hardware to operate over a multi-tier network and worse of all (like RoE), you need to make your Ethernet network lossless.

Is there a message here?

Goes something like this … “Ma’am, I’d like Ethernet over my Ethernet please …”

Do yourself a favor and avoid Layer Violations. Stick to regular Ethernet, IP, TCP and UDP in your data-center. If you need super-high performance, go buy a super-high-performance network like Infiniband.  If you want to transition off Fibre Channel, just start buying some iSCSI based network storage which performs surprisingly well.

There are two papers worth reading.

TCP Offload is a dumb idea whose time has come

Read just to remind you that our industry does try to regularly market some spectacularly dumb ideas.

And

 When the CRC and TCP Checksum and disagree

Absolutely recommended reading for anyone who does not understand the dangers of using RDMA over networks which were not designed from the ground up with strict end-end integrity.

 

--- update 28/5/2010 ---

Don't confuse my XXXX-poor attempts at humor with  MAC-in-MAC encapsulations!

OpenOnload for MPI

I've been playing with Pallas/IMB with OpenOnload for a few MPI stacks. Seeing some nice speedups.

For example the following graph shows MPICH2 pretty much hitting bidirectional line rate for the Exchange2 test. This is about 2x the performance of the kernel stack.

OpenOnload is compatible with all known MPI stacks. Serious beer to anyone who can find a breakage: MPICH1/ MPICH2 / openMPI / aka HPMPI / aka SCALI

Also note the <10us latency.  Factor of 2x improvement. Just the ticket for HPC clusters that want bandwidth . latency but don't want to the pain of non-Ethernet.

Using the following settings in my .openonload.sh

export EF_INT_REPRIME=0      
export EF_POLL_USEC=200
export EF_TCP_RCVBUF=512000 

Next-up port striping ..  (for all those who remember what EF_STRIPE_NETMASK) was all about....

-- update 2/7/2010 --

Oh and for anyone who's thinking that <10 isn't exactly sparkling on the latency front, I should have said that the above was measured on a 10GBASE-T network!