Main -> Wish list -> Infiniband for small office




Infiniband for small office


The main reason for Infiniband equipment high prices is its niche positioning.

At present Infiniband is used mainly for building supercomputer clusters and as Fibre Channel and 10GB Ethernet replacement in data centers. The medium/small office and consumer use is nearly zero.

But if to compare with both of them, it has serious advantages:
1. It is faster - up to 40Gb/s over most common 4x connection (compare to 10Gb/s max for Ethernet and Fibre Channel).
2. It is more flexible - 1x, 2x, 4x, 12x connections are standard at 2.5/5/10Gb/s over each link in the connection.
3. Unlike Fibre Channel loop / tree-like switching or Ethernet broadcast tree-like structure, it is a point-to-point packets multi-switch fabric connection. This means less demanding requirement for cables compared to Ethernet (as they pass only dedicated portions of information) and more redundant overall structure (with properly built fabric a failure of a switch may only slow down the connections - not cause the whole segment breakdown, failure of a nod is just a failure of a nod - not of the whole loop).
4. It is cheaper. The host cards cost about similar for FC/10GB/IB, but the switches cost is more demonstrative.
- 8/10GB Fibre Channel switch cost $2000-4000 per port in the "democratic" (8-20 port) version. Corporate Director switches are two-three times more costly.
- 10GB Ethernet switches in democratic (8-20 ports) version are $750-1500 per port, Director switches double-triple the price.
- Infiniband democratic 40Gb/s switch will cost $100-200 per port, up to $500-1000 with Director switches.
As Director-class switches are mot of rear need for a small/medium business use, Infiniband switches are quite affordable, unlike their competitors.

The only Infiniband drawback is the problem of connection of an Infiniband fabric to Ethernet network.

The cheapest switch-based solution today cost about $13000-14000 (about $10000 Mellanox BridgeX BX4010 (2 x 20/40GB Infiniband to 6 x 10GB Ethernet), $3000 - 1GB LAN switch (10-40ports) with two-four slots for 10GB ports and about $1000 per connect (cable, adapters, port)) - way too much for a small office solution.

But take in mind that $3000 plus about $700-1500 per port is the price for 10GB/1GB Internet bridging - not cheap too.

For the price it quite possible to build a home-made switch - a server with a couple of 10GB LAN ports or 40GB IB port and 14-18 1GB ports (three-four PCIe x4 LAN cards plus two on-board connectors), so really the price of connecting either IB or 10GB to the existing LAN may be no more than $3000.

Note: after this article for published, I got a notion that an outdated TopSpin 90 / Cisco SFS 3001 (12 10GB IB ports plus 4 1Gb/s LAN ports) might be used as a bridge too

So there is a need for special small/medium office solution

Wanted Infiniband / 1GB LAN bridge/switch

The appliance built over the combination of Mellanox InfiniScale switch, one or more Mellanox BridgeX VPI and one or more 10GB/1GB Ethernet switches (Fulcrum FM4112 or Broadcom BCM56524) in one box.

The possible solutions:
Entry: 8-10 4X 20/40GB 4x CX4 Infiniband ports, 16-24 10/100/1000BaseT LAN ports, minimal "dump-proof" software, very reasonable price
Medium: 12-24 20/40GB 4x CX4 IB ports, 24-48 10/100/1000BaseT LAN ports, minimal "dump-proof" software, reasonable price
Adwanced: 24-30 20/40GB 4x CX4 IB ports, 48-72 10/100/1000BaseT LAN ports, optional possibility to add IB/LAN ports, more or less "professional" software
High-end: 30-90 20/40GB 4x CX4 IB ports, 72-144 10/100/1000BaseT LAN ports, possibility to add IB/LAN ports, "professional" software

I'd like to have a following system:
Mellanox InfiniScale switch, two IB 40GB 4x links to Mellanox BridgeX VPI, 2 x 3 10GigE links to two Fulcrum FM4112 switches, 2 x 10GigE interconnect between Fulcrum chips, needed for functionality additional hardware;
12 x 40GB 4x CX4 IB ports, 4 x 40GB 4x QSFP IB ports;
Two CX4 plus two SFP+ 10GB LAN expansion ports (one CX4 and one SFP+ from each LAN switch chip), 32 1GB LAN ports

Cheaper Infiniband cards

Well, I think that with the above-mentioned appliances handy, the higher demand will take the prices down, as now low demand is one of the causes for a bit too high pricing for them.

Do-it yourself Infiniband-to-LAN bridge

A sample of relativly inexpencive 40Gb/s Infiniband-to-LAN bridge server (two 10GB LAN or one 40Gb Infiniband connection, 10 x 1GB LAN connections

Supermicro X8ST3-F motherboard ($380-440)
One Intel Xeon E5502 processor ($180-210)
6GB (6x1GB) PC3-8500 (1066Mhz) memory ($180-230)
Mellanox MHJH29-XTC Dual Port universal 40Gb InfiniBand / 10GB LAN card ($1000-1200)
- Single-port version (for Infiniband only) - $750-900
2 x 4-port 1GB LAN card ($350-450)
Case, PSU, HDD, DVD, monitor, keybord, mouse (~200-600 - monitor, keybord, mouse are needed only if there is no KVM in the server room, I'd better add the latter)
Linux or Solaris OS, open-source drivers

Total for the bridge: $2700-4500