Main -> Ultra workstation

Ultra workstation

A challenge for Intel (others are invited too)

This is a description of an ultra workstation that may be built using modern technologies.
(updated 29.03.2010)

Unlike the "Super workstation" it needs some not yet produced components (but not outside the reach of existing technologies).
The components in question are:
- Intel Nehalem EX processors (already annonced, but not yet in supply).
- Intel Nehalem(Westmere)-based RAID controllers - somewhat in between Xeon 7500 series processors and the already announced Jasper Forest processors - 4-8 cores, 4-way (same with the main system speed) QPI, on-chip RAID functions, recognised by the system as a RAID co-processor (allowing to build QPI-speed RAID arrays).
- Intel Nehalem(Westmere)-based graphics decomposition/composition conductor (similar in functions with Lucid Hydra but much faster and more versatile - should be able to support up to 12 GPU cards) - in principle the same as 7500 series processor, but recognised by the system as a graphics co-processor - with 4-way (same with the main system speed) QPI (allowing to build QPI-speed graphics subsystems).
- SATA 6G SSD controller (Indilinx Jet Stream or some other) supporting up to 500GB/s throughput and SuperSSDs built on it.
- Infiniband VPI QDR x8 or ODR x4 controller (PCIe x16 link to one QDR x8 or ODR x4 Infiniband port plus 40/10GB LAN port), allowing ultra-high computers interconnect

Otherwise this is a much more powerful and more compact "Super workstation" system. So only key components are described here, all other are generally the same with the "Super workstation" project.

Specification

Dual-case system: rackmount main system plus desktop I/O subsystem

Main system

Chassis

16U high, 36" deep, 19" water-cooled rackmount chassis.
Rear (bottom to top): three 1U PSUs, 9U motherboard conpartment, 6U cooling compartment
Front (bottom to top): 4U 5.25" bays and controls comparment (8 x 5.25" bays), two x 3U 120mm fans compartments, one 4U 24 x 3.5" bays hot-swap "SSD superdrives" cage, one 2U 12x3.5" bays hot-swap HHDcage

PSU

3 x 2000W 1U redundant PSU units

Motherboard

Dual-board system.
Main board:
16.4" x around 27-30" board size.
Three-processors Nehalem-EX QPI ring (all three processors must be installed), six QPI connections
UEFI BIOS (also provides "driver OS" for Graphics and RAID co-processors)
Graphics and RAID Nehalem-EX- based co-processors
48 (6x8) main memory slots (using six 8-slots memory risers - up to 768GB of memory), eight auxilary (Graphics and RAID co-processors) memory slots
7.6" wide I/O panel, 11 expansion slots (10 at the face side, two in-line at the rear side)
Secondary (graphics) board:
16.4" x 12" board size.
10 dual-wide expansion slots at the face side, two in-line upstream slots at the rear side

Main board graphics co-processor and chipset

- Graphics co-processor: Intel Xeon 7500 6-core processor based, optimized for graphics decomposition/composition tasks, two upstream QPI links, two downstream QPI links.
- Four memory slots (up to 64GB Graphics co-processor dedicated memory);
- Intel 7500 providing two PCIe x16 links;
- PLX PEX 8664 switch - one (primary) PCIe x16 into three PCIe x16;

Main board RAID co-processor and chipset

- RAID co-processor: based on 6-core Intel Xeon 7500 "Jasper Forest"-like processor, optimized for RAID tasks, two downstream, two upstream QPI links
- Four memory slots (up to 64GB RAID-dedicated memory)
- Two daisy-chaned Intel 7500 providing eight PCIe x8 links and two PCIe x4 links
- Eight LSI LSISAS2116 6Gb/s SAS chips, providing 24 internal and 8 external miniSAS HD connections.
- Two LSI LSISAS2004 6Gb/s SAS chips, providing 2 internal miniSAS HD connections.

Basic chipset

Main board:
- Two Intel 7500 northbridges, providing two PCIe v.2.0 x16 links, four PCIe v.2.0 x8 links, two PCIe v.2.0 x4 links and one PCIe v.1.1 x4 link.
- Intel ICH10R southbridge (six PCI Express x1 v.1.1 links, 12 USB 2.0 ports, 6 SATA II Ports)
- PLX PEX 8614 switch - one PCIe v.2.0 x4 into four PCIe x1;
- PLX PEX 8625 switch - one PCIe x8 into one PCIe x8 and one PCIe x4;
- Intel 41210 bridge: PCIe x8 link into two PCI-X buses (wired to two PCI-X 133MHz slots) - using PEX 8625 PCIe x8 link
Secondary (graphics) board:
- PLX PEX 8680 switch - one (primary) PCIe x16 into four PCIe x16;
- PLX PEX 8647 switch - one PCIe x16 into two PCIe x16;
- Two PLX PEX 8664 switches - one PCIe x16 into three PCIe x16 each;

Main board on-board network

- Two Infiniband VPI QDR x8 or ODR x4 controllers: provide two pairs of (QDR x8 or ODR x4 80GB InfiniBand port and 40/10GB LAN port) - using two x16 links;
- Intel 82580 Gigabit Ethernet controller - providing four RJ45 LAN ports - using PLX PEX 8625 PCIe x4 link

Main board main system on-board storage

- Two internal SATA 6G ports from Intel ICH10R;
- Four internal SATA II ports from Intel ICH10R;
- JMicrom JMB363 controller: two eSATA ports, one IDE (2 devices) port - using Intel ICH10R PCIe x1 link;

Main board on-board USB/FireWire

- Three NEC µPD720200 USB 3.0 2-ports Host Controllers - five external, one internal USB 3.0 ports - using three PLX PEX 8614 PCIe x1 links;
- 8 internal plus 6 external USB 2.0 ports (Intel ICH10R);
- Texas Instruments TSB83AA23 1394b chip: 2 external and 1 internal IEEE 1394b ports - using Intel ICH10R PCIe x1 link;

Main board on-board Audio

- Realtec ALC888 codec - provides optical and copper S/DPIF, 6-jacks copper I/O and on-board stereo sound I/O - Intel 7500SB+ audio channel;

Main board super I/O

- VIA VT1211 legacy ports chip - FDD port, serial port, infrared port - Intel 7500SB+ chip legacy port;

Main board IPMI (optional)

- Winbond WPCM450 BMC with Realtek RTL8201N PHY - independent IPMI LAN port

Main board video (optional)

- Matrox G200eW - on-board video;

Main board I/O panel

- Two PCIe x8 external connectors
- Two QDR x8 or ODR x4 InfiniBand connectors
- Two 40/10GB LAN connectors
- 8 mini-SAS HD connectors
- Four RJ45 LAN ports
- Two eSATA ports
- Five USB 3.0 ports
- Six USB 2.0 ports
- Two IEEE 1394b ports (with 9-to 6 and 9-to-4 pin adapters)
- Optical S/PDIF Out port
- Copper S/PDIF Out port
- 6-jack ports of audio I/O
- COM port
- Optional IPMI RJ45 LAN port
- Optional VGA port

Expansion slots

Main board (left to right - rear view):
Graphics subsystem:
- PCIe x16 slot
- Two in-line PCIe x16 slots (on the board rear side)
- PCIe x16 slot
Main system:
- PCIe x1 slot (PLX PEX 8614)
- PCIe v.2.0 x8 slot
- PCIe v.2.0x8 slot
- PCIe v.2.0 x4 slot
- PCIe v.2.0 x8 slot
- Two PCI-X 133MHz slots
Secondary (graphics) board:
- Ten dual-width PCIe x16 slots on the face side and two in-line PCIe x16 slots on the rear side (opposite to main board slots)

Main board on-board connections

Main system:
- Two SATA 6G ports
- Six SATA II ports
- USB 3.0 port header
- 4 x 2 port USB 2.0 headers (8 ports)
- IEEE 1394b header
- CD audio in connector
- Front panel audio connector
- 1 x consumer infrared port header
RAID subsystem:
- 24 (8 x 3) internal miniSAS HD connectors

Processors:

- Three fastest Intel Xeon EX 6/8-core processors overclocked to 4-5GHz (water-cooled)
- Graphics co-processor: Intel Xeon 7500 6/8-core based, optimized for graphics decomposition/composition tasks, 4 QPI links, overclocked to 4-5GHz (water-cooled)
- RAID co-processor: Intel Xeon 7500 6/8-core based "Jasper Forest"-like processor, optimized for RAID tasks, 4 QPI links, overclocked to 4-5GHz (water-cooled)

Memory:

Main processors: 192GB - 24 x 8GB DDR3 1333MHz 240pin ECC Registered CL9 memory - about 40-60GB of working memory, about 130-150GB RAM-drive
Graphics co-processor: 8-32GB DDR3 1333MHz 240pin ECC Registered CL9 memory
RAID co-processor: 8-32GB DDR3 1333MHz 240pin ECC Registered CL9 memory

Expansion cards:

Main Board:
- Two GPU cards, each supporting 6-8 miniDisplayPort connectors.
- Other needed expansion cards
Secondary board:
- Up to ten GPU/GPGPU cards w/o video connectors.

Storage

180GB SSD superdrive

3.5" HDD form-factor enclosure with six 3GB/s SSD in each (three plates with two drives circuits on each), LSISAS2008-based SAS-to-SAS RAID controller (forming 6-drives RAID 0 array), one miniSAS connector.
Drives number and arrays speed calculations are based on the best so far 3G SSD - 30GB version of OCZ Vertex EX (about 230 MB/s sustained sequential read, about 100 MB/s sustained sequential write, up to 12500 IOPS)

768GB SSD superdrive

3.5" HDD form-factor enclosure with six 6GB/s SSD in each (three plates with two drives circuits on each), LSISAS2008-based SAS-to-SAS RAID controller (forming 4-drives RAID 0 array), one miniSAS connector.
Drives number and arrays speed calculations are based on the fastest modern 6G SSD - Crucial RealSSD-C300 (about 350 MB/s sustained sequential read, about 200 MB/s sustained sequential write, up to 8500 IOPS)

Storage configuration

- 3.6TB SSD main system/working storage (up to 12GB/s read/write, up to 1 500 000 IOPS): "Jasper Forest" hardware RAID 6 over (22 + 1 hot-spare) x 180GB SSD superdrives - using eight LSISAS2116 controllers
- 12TB SSD working/archive storage (up to 12GB/s read/write, up to 900 000 IOPS): "Jasper Forest" hardware RAID 6 over (8 + 1 cold-spare) x 2TB HDD drives - using LSISAS2004 controllers.

Front I/O

- DVD drive,
- FDD drive,
- 5.25" front connections hub: USB 3.0 to four USB 3.0 (using VIA VL810), USB 2.0 to four USB 2.0 (using NEC µPD720114), IEEE 1394b to two IEEE 1394b (using TI TSB81BA3), SATA II to three SATA II (using Silicon Image SiI3726)
- up to three other 5.25" (or 3.5") needed front I/O devices

External storage

Twelve free miniSAS connectors allow connection of any needed external storage

External expansion

Desktop subsystem - fiber optic connection using PCIe x8 external connector.
One more free PCIe x8 external connector allows any other needed external expansion

Cooling

Approximate heat dissipation:
5 overclocked processors - 250W x 5 = 1250W
12 GPU processors - 250 x 12 = 3000W
That will need an external outdoor cooling system with internal and external loops and heat extenger
Draft sample internal loop presented below:
6 x Koolance PMP-450S pumps with Koolance OV-RP450 Nozzle & Reservoir Base (each serves up to three CPU/GPU)
6 x Peltier chillers (two Koolance HD-60 HDD coolers with ~ 400-600W Peltier chip between them - installed between pumps outlets and hot water reservoir inlets - chills water to a bit lower than ambient temperature)
Cold and hot water reservoirs (common for all pumps)
3 x Koolance HXP-193 Plate Heat Exchangers (connection to external cooling system)
3 x Koolance HX-CU1320V 4-fans radiators (cools water in the case of external cooling fail - up to 5KW heat dissipation)
Needed control subsystem
External 8-10KW cooling system (1-1½" tubing) with parallel distribution to four Plate Heat Exchangers

Desktop subsystem

Case

12 x 5.25" bays full tower case with 500W Power Supply

Backplane

ATX form-factor expansion board
- PCIe v.2.0 x8 upstream connector
- PLX PEX 8625 switch - one PCIe x8 into two PCIe x4 and eight PCIe x1 links;
- Intel 41110 bridge: PCIe x4 link into 66MHz PCI-X bus (wired to two PCI-X slots and one on-board PCI controller)
- LSISAS1068E controller: eight internal SAS/SATA ports - using PCIe x4 link;
- JMicrom JMB362 controller: two SATA ports - using PCIe x1 link;
- JMicrom JMB363 controller: two eSATA ports, one IDE (2 devices) port - using PCIe x1 link;
- NEC µPD720200 USB 3.0 2-ports Host Controllers - one external, one internal USB 3.0 ports - using PCIe x1 link;
- NEC µPD720101 USB 2.0 5-ports Host Controller - two external, three internal USB 2.0 ports - using PCI-X bus; - Texas Instruments TSB83AA23 1394b chip: 2 external and 1 internal IEEE 1394b ports - using PCIe x1 link;
- One PCIe x4 slot
- Four PCIe x1 slots
- Two PCI-X 66MHz slots
- Two eSATA ports
- One USB 3.0 port
- Two USB 2.0 port
- Two IEEE 1394b port

Front I/O devices and connections

- 5.25" front connections hub: USB 3.0 to four USB 3.0 (using VIA VL810), USB 2.0 to four USB 2.0 (using NEC µPD720114), IEEE 1394b to two IEEE 1394b (using TI TSB81BA3), SATA II to three SATA II (using Silicon Image SiI3726)
- Other needed devices

Add-on cards and rear expansion connections

Needed set of cards and connectors

External I/O devices

Monitors

Up to six 2560x1600 monitors

Other I/O devices

Needed set of I/O devices