Main -> System expanding -> Infiniband




Using Infiniband for ultra-high speed "traditional" network


In some cases "traditional" 1GB network connection is not enough. What it might be substituted by?

Of cause there is a new 10GB Ethernet, but is it the best choice?

Let us compare the 10GB Ethernet and Infiniband:

Topology

The Ethernet has a tree-like switching topology. This has some advantages (a bit less cabling) and disadvantages (a single router/switch breakdown turns down a full downstream segment or even the full network).

The Infiniband has switching fabric topology, where each node/router/switch may be connected to several others and the breakdown of a single router/switch nearly would not affect the network work (well, it just will work a bit slower)

Also some older 1GB Ethernet switches turn to work with the speed of the lowest-speed adapter, so a single 100Mb or 10MB card may degrade the speed of the whole network segment. Infiniband will work with each node at it maximum speed

Cabling

The following table presents most used 10GB Ethernet and (most popular) x4 Infiniband cables connections:

Cable types CX4 Cat. 6 RJ45 Active CX4 QSFP SR fiber optics LR fiber optics Glass optics
Cable max lenght See below 100 m See below Depends on the
cable used 		300 m 10 Km Unlimited
10GB Ethernet 15 m + 30 m - + + +
10GB Infiniband 15 m - 30 m + + + -
20GB Infiniband 15 m - 20 m + + + -
40GB Infiniband 7 m - 15 m + + + -


There is the difference in direct node-to-node connection: 10GB Ethernet cable should usually be cross-wired, while Infiniband work nicely in the "wrong connected" case

Adapters

Generally the two networks use different network-specific adapters, but some Infiniband adapters may work both with Infiniband and 10GB Ethernet switches

The most common for 10GB Ethernet are single-port adapters, though 2-port cards are not seldom met. 4-ports adapters also exist.

Single-port adapters use x4 or (more often) x8 PCIe v. 1.1, dual-ports - x8 PCIe v. 1.1, quad port - x8 PCIe v. 1.1 (do not think it is reasonable - except in the case when the card may not really support the 10Gb/s speed).

The most common for Infiniband are dual-port x4 cards for x8 PCIe v. 1.1/2.0, (single-port ones also exist - both x4 PCIe and x8 PCIe)

The average 10GB Ethernet and Infiniband cards have about the same prices - from $500 to $1200 for copper versions, $800-2500 for fiber optics

But here the similarities end.

10GB Ethernet port max throughput is 10Gb/s. For more bandwidth you'll need more ports

Infiniband is more versatile - except for 10GB adapters exist (more common now) 20GB and 40GB cards. Of cause the additional cards may also be installed. And the price of the 40GB card is no bigger than 1.5 of the 10GB one.

The adapter kind price range as follows: CX4, Cat. 6 RJ45, QSFP, SR fiber optics, LR fiber optics, Glass optics

Switches

Here the difference is much greater - the cheapest CX4 10GB Ethernet switch cost about $500 per port, single-port 10GB module for 1Gb switch - from $1000. Compare with $3500-4000 for CX4 40GB 36-port Infiniband switch. Similar proportion for more expensive fiber optics switches (about $1000-1500 per port for 10GB Ethernet, about $6000 40GB 36-port Infiniband switch). For bigger ports count switches the difference is lesser.

Both 10GB Ethernet and Infiniband CX4 switches may be connected to fiber optics by special (not cheap!) adapters. High-speed (20GB and 40GB) Infiniband CX4 may also use active cables (copper to fiber) for up to 15 meters lenght (not too expencive)

Switching/bridging to 1GB Ethernet

Here standard Infiniband solutions are losing to 10GB Ethernet ones - any 10GB network may connect to 1GB network by $3000-5000 1GB 24-48 ports switch with 2-4 10GB links (well, adding $1200-1500 per each active link), cheapest Infiniband to 10GB Ethernet bridge (Mellanox BridgeX BX4000 - 2 20/40GB Infiniband links / 6 10GB Ethernet ports) will add $10000 .

A dedicated server with 2-4 Infiniband (or 10 GB) ports and 10-18 1GB ports will cost no more than $3000-4000 with the same functionality.

Software

10GB Ethernet uses standard Ethernet drivers (usually coming with OS)

For network substitution Infiniband needs Infiniband basic drivers and IP-over-Infiniband drivers installation.
For faster NAS solution IB supports iSER– iSCSI Extensions for RemoteDMA driver that allows direct drivers interaction w/o the use of the IP-over-Infiniband transport layer (reduces protocol data overhead)
For nod-to-nod connection a subnet manager (OpenSM) should be installed in one of the systems (swithes usually come with "firmwired" subnet managers, though some cheap version may also need subnet manager installed at one of the nodes)

The Ethernet and IP-over-Infiniband should be in different subnets to avoid collisions for computers/servers connected to both.

Data transfer rate

With IP-over-Infiniband the 10GB InfiniBand is 5-10% slower than 10GB LAN (due to IB layer data overhead).
When working with iSER drivers it is, on the contrary, may be about 50% faster - depending on the SAN server / client processors/memory speed (IB layer is less "talkative" than IP layer and direct memory access allowes much faster drive-to-requestor path).
For typical read operation with IP connection request on the client side processed by IP-driver, passes the server IP driver, analysed by processor and then goes to disk driver memory. With iSER connection the request on the client side is processed by IB-driver and (smaller in size) goes direct to server disk driver memory. Same is with the responce path.
Conclusion

Current prices nearly totally kill the idea of 10GB Ethernet:
Minimal adapter prices are $500(CX4) / $1200 (fiber optics)
Cheapest 12-port 10GB switch - $6000.
24 ports 1000/100/10 switch with 2-4 pluggable 10GB ports - $3000 plus ~$2000 (card plus adapters) per each 10GB port

Infiniband pricing is a bit more democratic:
Minimal 10/20GB adapter prices are $600 (CX4) / $1100 (fiber optics).
Average 24-ports switch - from $2500 (10GB) / $3500 (20-40GB) ("Democratic" 10GB 8-port switch - $750)
One of the added servers might be used as a gateway: ten 1GB connections to the existing network (two 4-port 1GB LAN cards plus two on-board connections) will cost about $750. Separate gateway server (2 x 20GB Infiniband ports, 14-18 1GB LAN ports) - about $3500-$4000
Use of 1GB switch with pluggable 10GB ports - same as for 10GB Ethernet

For small high-speed storage add-on this might be not too big difference, but for a big network even the use of cheap Cat.6 cabling would not save the situation - even at 10Gb/s connection speed at both networks at least 10% (much more really) price difference will tell for itself.

If to remember that Infiniband may work at 20-40GB with not a big price difference, the price/performance ratio will come to 2-3/1 in favour of Infiniband (even adding a single dedicated server by direct link will cost about 1.5 cheaper for the same bandwidth).

PS

I got a reply to this article, telling that older (TopSpin 90 aka Cisco SFS 3001) Infiniband multiswitches may nicely by used as both low-speed (10GB) Infiniband switches and LAN bridges (12 IB ports plus 4 1Gb/s LAN ports). New devices might be bought at $3000-3500, used - from $1000.