Main -> Ultra workstation

Ultra workstation

A challenge for Intel (others are invited too)

Updated 30.05.2010
This is a description of an "ultra" workstation that may be built using modern technologies.
Yes, this IS a monster, most probably not needed by any application "in a full stand", but, on the other hand, the ideas described here will allow to build nearly any really needed system.

Unlike the "Super workstation" it needs not yet produced processors and chipsets (but not outside the reach of existing technologies).

The components in question are:

- Intel workstation-class processors, based on "extended" Westmere-EX architecture: 32nm production, 8 processor cores, 4-way memory, 6 (six!) 4GHz (8GT/s) QPI interfaces, at least 3.3GHz processor clock.
Current 4 QPI interfaces at Nehalem-EX processors are not sufficient for fast workstation systems. Intel has shown that more is quite possible with latest Itaniums.
Such a processor might be a bit too hot (over usual 130W limit, used by Intel), but not too hot to be unstable under proper cooling.

- Quad channel memory controller with a 6-way QPI switch (in principle, the same processor chip w/o (or with blocked non-functional) processor cores - QPI router with six 20-lanes QPI PHY's and two dual SMI channels) - will allow to add more (slower) memory (up to 256GB per controller)
IBM already has a somewhat similar chip, allowing to add up to 32 1066MHz memory modules (up to 512GB per controller) with four QPI links (two upstream, two for further expansion) and three IBM propriatary links (allowing to connect one more IBM memory controller)

- Independent 6-way QPI router with six 20-lanes QPI PHY's (one more step back from the proposed processor - QPI router only) - will allow better interconnection abilities, including the use as a qluing element for multi-processors clasters.

- Intel RAID co-processors (enhanced "Jasper Forest" processors, based on the same 6 QPI "extended" Westmere-EX architecture): 32nm production, 6 processor cores, 2 XOR engines, 4-way memory, six 4GHz (8GT/s) QPI interfaces (up to two should be able to work at non-transparent mode), at least 3.0GHz processor clock, recognised by the system as a RAID co-processor (allowing to build QPI-speed RAID subsystems).
Think IBM (already having QPI add-on memory solution) may propose it's own solution, based on their Power 7 architecture with embed QPI interconnect.

- Intel "special functions" co-processors (based on the same 6 QPI "extended" Westmere-EX architecture - in the case in question such a chip shoud be recognised as a graphics co-processor): 32nm production, 4-8 processor cores, 4-way memory, six 4GHz (8GT/s) QPI interfaces (up to two should be able to work at non-transparent mode), at least 3.0GHz processor clock (working under Lucid Hydra-like OS/driver as the graphics decomposition/composition conductor - allowing to build QPI-speed graphics/GPGPU subsystems).
Such a processor should have a special programmed register, showing the function it is used for (graphics or some other special function co-processor).
Again, IBM may propose it's own solution, based on their Power 7 architecture with embed QPI interconnect.

- QPI cables, cable connector, interboard connectors with neccesary addition signalling lanes (IBM already use them in x5 series servers).

- 64 lanes PCIe v.2.0 Intel northbridges with 2-way 4GHz (8GT/s) QPI links: 3 x PCIe v.2.0 x16 links (each x16 link configurable as two independent x8 or four independent x4 links), one PCIe v.2.0 x8 link (configurable as two independent x4, four independent x2 or eight independent x1 links), and PCIe v.2.0 x8 southbridge link (used as two independent x4 links on non-legasy chips) - such a northbridge is needed for optimal utilisation of QPI speed by graphics cards and better southbridge performance.

- 48 lanes PCIe v.2.0 Intel northbridges with 2-way 4GHz (8GT/s) QPI links: 2 x PCIe v.2.0 x16 links (each x16 link configurable as two independent x8 or four independent x4 links), one PCIe v.2.0 x8 link (configurable as two independent x4, four independent x2 or eight independent x1 links or their mix), and PCIe v.2.0 x8 southbridge link (used as two independent x4 links on non-legasy chips) - such a northbridge is needed for better southbridge performance.

- 32 lanes PCIe v.3.0 Intel northbridges with 2-way 4GHz (8GT/s) QPI links: one PCIe v.3.0 x16 link (configurable as two independent x8 or four independent x4 links), one PCIe v.3.0 x8 link (configurable as two independent x4, four independent x2 or eight independent x1 links), and PCIe v.2.0 x8 southbridge link (used as two additional PCIe x4 links on non-legasy chips) - such a northbridge is needed for installation of PCIe v.3.0 RAID and IB/LAN cards.
Despite all nVidia babbling about the need of PCIe v.3.0 for graphics, I can not name any graphics card needing more than 1.2GB/s PCIe bandwidth even in multy-card arrangment (SLI or CrossFireX), which is really even less than PCIe v.1.0 may provide.

- PCIe v.2.0 extended performance Intel southbridge:
            - PCIe v.2.0 x8 northbridge link,
            - two on-chip PCIe x4 SATA 6G 4-port RAID 0/1/10 controllers, two on-chip 1-to-4 SATA 6G RAID 0/1/10 port multipliers (using two second controller links - may be set on/off by motherboard vendor), providing (8 x SATA 6G) ports or (7 x SATA 6G + 4 x limited speed SATA 6G) ports or (6 x SATA 6G + 8 x limited speed SATA 6G) ports;
            - two on-chip PCIe x4 USB 3.0 4-port controllers, two on-chip (one USB 3.0 to seven USB 2.0) hubs (using two second controller links - may be set on/off by motherboard vendor), providing (8 x USB 3.0) ports or (7 x USB 3.0 + 7 x USB 2.0) ports or (6 x USB 3.0 + 14 x USB 2.0) ports;
            - two on-chip 1GB LAN controllers;
            - two PCIe 2.0 (2.5Mb/s) x4 links (configurable as four independent x2 or eight independent x1 links);
            - High Definition Audio Interface, Low Pin Count Interface (legacy ports link), UEFI BIOS support, other needed connections.

- PCIe v.3.0 x4/x8 HBA/RAID controllers and based on them 4/8/16 SAS 6G channels PCIe v.3.0 x4 RAID/HBA cards and 8/16/32 SAS 6G channels PCIe v.3.0 x8 RAID/HBA cards

- PCI slot 24/36 ports SAS 6G SAS expanders and 8/16 ports SAS-to-SAS 6G RAID controllers (PMC-Sierra and LSI chips for them already exist)

- SATA 6G and SAS 6G SSDs based on new 6G controllers (no worse than up to 450GB/s throughput and up to 25000 IOPS on 60GB drives).

- 5-ports SATA 6G Port Multipliers (no less than 500MB/s upstream throughput), supporting up to 5-drives JBOD, 3/4 drives RAID 5 (with support of one hot-spare drive), 3/4 drives RAID 10 (with support of one hot-spare drive), RAID 0/1 on 2 drives

- 2-ports SATA 6G Port Multipliers (no less than 550MB/s upstream throughput), supporting JBOD and RAID 0/1

- Infiniband VPI QDR x8 or ODR x4 controller (PCIe v.2.0 x16 link or PCIe v.3.0 x8 link to one QDR x8 or ODR x4 Infiniband port plus 40/10GB LAN port), allowing ultra-fast computers interconnect

Otherwise this is a much more powerful and more compact "Super workstation" system. So only key components are described here, all other are generally the same with the "Super workstation" project.

Some pre-considerations

Some judgments in this section are quite subjective, but I think are more or less close to the point.

Number of proccessor cores that make sence to be used in graphics workstation

Generally the efficiency of each processor core added to the system varies in between 90-97% of the efficiency of previously added. The best effect comes when each core runs it's own application. In multi-threaded applications the effect is usually less - often even less than 0.9.
Unlike as in the the server applications, where efficiensy varies in between 0.93-0.97, the workstation applications cannot generate a big number of practically independent subtasks. Generally only a few of subtasks might be more or less independent (I'd name networking, RAM-drive, RAID main thread, graphics main thread, might be a couple more). As these tasks usually are not run simultaneously, the cores added over the 4-th one, would not add more than 95% productivity over the previously added (really even less). That means that with three 8-core processors, the last core adds no more than 36% of it's real efficiensy. An addition of the 4th 8-core processor would add less than 2.4 cores efficiency (compared to about 4 cores of the third one). This means that the optimal number of 8-core processors in the system is about 3 (maximum 4). Adding the fifth 8-cores processor would add less than 1 core power to the system, so more than 4-processor systems make no sence for typical workstation aplications.
Similar consideration limits the useful number of RAID co-processors cores and graphics conductor cores by 6-12 (one-two 6/8-cores co-processors).

Processor-attached and QPI-attached memory

Processor-attached memory is much faster than QPI-attached memory, but adding more memory modules (even ranks) decreeses the effective memory speed.
On the other hand, a huge memory in workstation applications is usually needed for RAM-drive(s), mostly used as temporary/scratch files storage.
So, there is a sence to have a somewhat limited amount of ultra-fast processor-attached memory and big enough pool of slower (and cheaper) QPI-attached memory.
Westmere-EX memory architecture allows up to eight single-rank memory modules (totalling in 32GB of memory) per processor running at maximum speed. That gives 96GB for the 3-processors system and 128GB for the 4-processors system. I think that 48-64GB will be enough in most cases.
With quad-rank 16GB modules and 16 modules per processor (or quad channel memory controller with QPI switch), each processor/controller may support up to 256GB (of slower memory).
The RAM-drive capacity should be at least ten times bigger than main memory volume, so, 1-2TB of QPI-attached memory (four-eight memory controllers with QPI switch) would be quite useful.

Specification

Multi-board dual-case system: a rackmount main system plus a desktop I/O subsystem

The main system includes: - the four-layers "sandwich" motherboard: main (processors) board (processors, processors-attached memory, QPI links), embed I/O board (basic and additional chipset, on-board and rear I/O connectors), slots expansion board (chipset, PCIe/PCI slots), memory board (memory controllers, QPI-attached memory, QPI links);
- the cable QPI attached main Graphics board (two-layers "sandwich": main board (processors, memory, QPI links), daughterboard (chipset, PCIe slots);
- the secondary Graphics board (chipset, PCIe slots) - cable QPI attached to the main graphics board;
- the cable QPI attached main RAID board (two-layers "sandwich": main board (processors, memory, QPI links), daughterboard (chipset, PCIe slots);
- the secondary RAID board (chipset, PCIe slots) - cable QPI attached to the main RAID board;
- the additional memory board (one-two layes "sandwich": main board (memory controllers, QPI-attached memory, QPI links), optional daughterboard (memory controllers, QPI-attached memory, QPI links));
- storage subsystems and constant I/O connections.

The desktop I/O subsystem is connected to the main system by PCIe x16 optical cable and provides all needed at hand connections and removables.

Main system

Chassis

42U high, 36" deep, 19" water-cooled rackmount chassis.
Rear: four 1U PSUs, 7U motherboard conpartment, one 5U and one 4U graphics compartments, one 5U and one 4U RAID compartments, 12U cooling compartment
Front: 4U 5.25" bays and controls comparment (8 x 5.25" bays), five 2U 24 x 2.5" bays hot-swap SSD cages, two 4U 36 x 3.5" bays hot-swap HDD cages, 6 x 3U 120mm fans compartments, one 2U blank.

PSU

4 x 2000W 1U redundant PSU units with power management units.

Motherboard overall configuration

The four-layer "sandwich" board.
Board configuration (top to bottom):
- Slots expansion board (chipset, PCIe/PCI slots)
- Embed I/O board (basic and additional chipset, on-board and rear I/O connectors)
- Main (processors) board (processors, processors-attached memory, QPI links)
- Memory board (memory controllers, QPI-attached memory, QPI links)
The space between the boards should be about 1U
Total "sandwich" dimemtions (with expansion cards) is 16.4" x 28" x 7U,
(Board layout allows to build quite a powerfull 2D workstation w/o adding a Graphics expansion board)

Processors board

The board full size (including the QPI connectors) is 16.4" x 28"
- Four processor sockets, 12 downstream QPI links - two pairs of external QPI connectors, four pairs of internal QPI connectors (two pair on the board face side, two - on the board rear side)
(QPI connectors also provide signalling lanes, nessesary for non-independent modules)
- Thirty two memory slots (eight per processor) - up to 128GB of single-rank memory

Embed I/O daughterboard

16.4" x 14" board. Two upstream QPI connectors, an opening for QPI links from the Processors board to the Slots expansion board, full-width I/O panel, power connector

UEFI

The UEFI should:
- support intallation of up to four different (same QPI speed) processors.
- support memory to HDD shutout option

Basic chipset

- Two daisy-chaned northbridges (the 32 lanes PCIe v.3.0 northbridge and the 48 lanes PCIe v.2.0 northbridge) and the PCIe v.2.0 southbridge connected to the PCIe v.2.0 northbridge.
The PCIe v.3.0 northbridge provides three PCIe v.3.0 x8 links and two PCIe v.2.0 x4 links.
The PCIe v.2.0 northbridge provides one PCIe v.2.0 x16 link, two PCIe v.2.0 x8 links, one PCIe v.2.0 x4 link and two PCIe v.2.0 x1 links.
The PCIe v.2.0 southbridge provides one PCIe v.2.0 (2.5Mb/s) x4 link, 6 x SATA 3G ports (RAID 0/1/10), 8 x limited speed SATA 6G ports ((2 x 4) RAID 0/1/10, allowing RAID 1/0 array over two (up to four drives) RAID 0 arrays), 4 x USB 3.0 ports, 14 x USB 2.0 ports, two LAN ports, (7+1) audio and legasy ports.
- Intel 41110 bridge: PCIe x4 link into PCI bus - using southbridge PCIe x4 link.

On-board network

- Two Infiniband VPI QDR x8 or ODR x4 controllers: provide two pairs of (QDR x8 or ODR x4 80GB InfiniBand port and 40/10GB LAN port) - using two northbridge PCIe v.3.0 x8 links;
- Intel 82580 Gigabit Ethernet controller - providing four RJ45 LAN ports - using northbridge PCIe x4 link;
- Two southbridge Gigabit Ethernet LAN ports;

On-board storage

- PCIe v.3.0 x8 RAID controller - sixteen SAS lines, four internal miniSAS connectors - using northbridge PCIe v.3.0 x8 link;
- Four internal and two expernal (eSATA) SATA 6G ports from the southbridge;
- Six internal and two external (eSATA) SATA II ports from the southbridge;
- jMicron JMB362 controller with two jMicron JM20330 SATA-to-IDE bridges, providing two hot swap IDE ports (2 devices each) - using northbridge PCIe x1 link,

On-board USB/FireWire

- Six external USB 3.0 ports from the southbridge
- 10 external plus 4 internal USB 2.0 ports from the southbridge
- Texas Instruments TSB83AA23 1394b chip: 2 external and 1 internal IEEE 1394b ports - using northbridge PCIe x1 link;

On-board Audio

- Realtec ALC888 codec - provides optical and copper S/DPIF, 6-jacks copper I/O and on-board stereo sound I/O - Intel southbridge audio link;

Super I/O

- Winbond 83627HF legacy ports chip - keyboard and mouse PS/2 ports, one serial port, infrared port header - using southbridge legacy port;

On-board video

- ATI ES1000 video chip - using Intel 41110 PCI bus;

IMPI

Winbond® WPCM450 BMC - IPMI 2.0 with virtual media over LAN and KVM-over-LAN support, provides separate dedicated LAN port

Rear side connections

Two QPI connectors (with needed signalling lanes).

On-board connections

- Four miniSAS connectors (16 SAS lanes)
- Four SATA 6G ports
- Four SATA II ports
- Two IDE connectors
- 2 x 2 port USB 2.0 headers (4 ports)
- IEEE 1394b header
- CD audio in connector
- Front panel audio connector
- 1 x consumer infrared port header

I/O panel connectors:

- PCIe v.2.0 x16 external connector
- Two PCIe v.2.0 x8 external connectors
- Two PCIe v.2.0 x4 external connectors
- Two QDR x8 or ODR x4 InfiniBand ports
- Two 40/10GB LAN ports
- Six RJ45 LAN ports
- One IMPI RJ45 LAN port
- Two powered eSATA 6.0 ports
- Two powered eSATA II ports
- Six USB 3.0 ports
- Ten USB 2.0 ports
- Two IEEE 1394b ports (with 9-to 6 and 9-to-4 pin adapters)
- Optical S/PDIF Out port
- Copper S/PDIF Out port
- 6-jack ports of audio I/O
- PS/2 keyboard and mouse ports
- VGA port
- COM port

Slots expansion daughterboard

16.4" x 11" board. Two tall upstream QPI connectors, 20 expantion slots, power connector

Basic chipset

- Two daisy-chained PCIe v.2.0 48 lanes Intel northbridges, providing two PCIe v.2.0 x16 links, five PCIe v.2.0 x8 links, five PCIe v.2.0 x4 links and four PCIe v.2.0 x1 links.
- Intel 41210 bridge: PCIe x8 link into two PCI-X buses (wired to two PCI-X 133MHz slots) - using PCIe x8 link
- Intel 41110 bridge: PCIe x4 link into PCI bus (wired to four PCI slots) - using PCIe x4 link

Rear side connections

Two "tall" QPI connectors (with needed signalling lanes)

Expansion slots

- PCIe v.2.0 x16 slot
- PCI slot
- PCIe v.2.0 x16 slot
- PCI slot
- Four PCIe v.2.0 x8 slots
- Four PCIe v.2.0 x4 slots
- Four PCIe v.2.0 x1 slots
- Two PCI slots
- Two PCI-X 133MHz slots

Memory daughterboard

16.4" x 28" board
QPI connectors: two top internal upstream pair, four external downstream pairs.
Four quad channel memory controllers (with 6-way QPI switches), 64 memory slots (16 per controller), supporting up to 1TB of memory.
Memory controllers QPI links: one upstream, three on-board interconnect, two downstream.

Graphics main board

16.4" x around 25-27" total size with 16.4" x 13" daughterboard.
Three pairs of external QPI connectors (one upstream pair, two downstream pairs), two pairs of internal downstream QPI connectors (to the daughterboard).
Two processors sockets
Sixteen memory slots (eight per processor), supporting up to 16 memory modules (up to 512GB of memory)

Connectors

- Three pairs of external QPI connectors (one - link to the motherboard, two - to the secondary graphics boards)
- Two pairs of internal QPI connectors (links to the daughterboard)

Daughterboard

16.4" x 12-13" board size.
- Two pairs of internal QPI connectors (links to the main graphics board) at the rear side
- Two daisy-chained 56 lanes Intel northbridges, providing six PCIe v.2.0 x16 links.
- Two daisy-chained 40 lanes Intel northbridges, providing four PCIe v.2.0 x16 links.
- One southbridge with connection to the startup BIOS and two on-board eSATA 6G ports (RAID 1 array with driver OS)
- Ten PCIe x16 double-width expansion slots

Graphics secondary board

16.4" x 12-13" board size.
- One pair of external QPI connectors (links to the main graphics board)
- Six-way QPI switch (one upstream, two dounstream pairs)
- Two pairs of daisy-chained 40 lanes Intel northbridges, providing eight PCIe v.2.0 x16 links.
- Eight PCIe x16 double-width expansion slots

RAID main board

16.4" x around 25-27" total size with 16.4" x 13" daughterboard (the same as the Graphics board)
Three pairs of external QPI connectors (one upstream pair, two downstream pairs), two pairs of internal downstream QPI connectors (to the daughterboard).
Two processors sockets
Sixteen memory slots (eight per processor), supporting up to 16 memory modules (up to 512GB of memory)

Connectors

- Three pairs of external QPI connectors (one - link to the motherboard, two - to the secondary RAID boards)
- Two pairs of internal QPI connectors (links to the daughterboard)

Daughterboard

16.4" x 12-13" board size.
- Two pairs of internal QPI connectors (links to the main RAID board)
- Two pairs of daisy-chained 32 lanes PCIe v.3.0 northbridges, providing 4 x PCIe v.3.0 x8 links, 16 x PCIe v.3.0 x4 links and six PCIe v.2.0 x4 links.
- One southbridge with connection to the startup BIOS and two on-board eSATA 6G ports (RAID 1 array with driver OS)
- Six LSI LSISAS2008 6Gb/s SAS chips (on PCIe x4 links), providing twelve internal miniSAS connections (48 SAS links).
- Four PCIe v.3.0 x8 expansion slots
- Sixteen PCIe v.3.0 x4 expansion slots

Secondary RAID board

16.4" x 12-13" board size.
- Two pairs of external QPI connectors (links to the main RAID board)
- Two pairs of daisy-chained 32 lanes PCIe v.3.0 northbridges, providing 8 x PCIe v.3.0 x8 links, 8 x PCIe v.3.0 x4 links and 8 x PCIe v.2.0 x4 links.
- Six PCIe v.3.0 x4 8-ports RAID controller chips, providing twelve internal miniSAS connections (48 SAS links).
- Eight LSI LSISAS2008 6Gb/s SAS chips (on PCIe x4 links), providing sixteen internal miniSAS connections (64 SAS links).
- Eight PCIe v.3.0 x8 expansion slots
- Two PCIe v.3.0 x4 expansion slots

Additional memory board

16.4" x about 28" board size
QPI connectors: two external upstream pairs, two external downstream pairs, two rear side internal downstream pairs.
Four quad channel memory controllers (with 6-way QPI switches), 64 memory slots (16 per controller), supporting up to 1TB of memory.
Memory controllers QPI links: one upstream, three on-board interconnect, two downstream.
Internal downstream pairs are for the use of additional memory daughterboard - same with the one used with the system main board

Processors:

Processors board:
- Four fastest workstation-class processors, overclocked to 4-4.5GHz (water-cooled)
Graphics board:
- Two fastest graphics co-processors, recognised by the system as graphics controllers (water-cooled)
RAID board:
- Two fastest RAID co-processors (water-cooled)

Memory controllers:

Memory board:
- Four memory controllers (water-cooled)
Additional memory board:
- Four memory controllers (water-cooled)

Memory:

Processor board: 64GB (up to 128GB single-rank (512GB quad-rank) may be installed) - 32 x 2GB DDR3 1333MHz (or faster) 240pin ECC Registered CL7 memory
Memory daughterboard: 512GB (up to 1TB may be installed) - 32 x 8GB DDR3 800MHz 240pin ECC Registered CL7 memory - used as a RAM-drive
Memory additional board: 512GB (up to 1TB may be installed) - 32 x 8GB DDR3 800MHz 240pin ECC Registered CL7 memory - used as a RAM-drive
Graphics co-processor: 16GB (up to 64GB single-rank (256GB quad-rank) may be installed) - 16 x 1GB DDR3 1333MHz (or faster) 240pin ECC Registered CL7 memory/
RAID co-processor: 16GB (up to 64GB single-rank (256GB quad-rank) may be installed) - 16 x 1GB DDR3 1333MHz (or faster) 240pin ECC Registered CL7 memory/

Expansion cards:

Motherboard:
- Up to twenty needed expantion cards
Main graphics board:
- Two GPU cards, each supporting 6-8 miniDisplayPort connectors.
- Eight GPU/GPGPU cards (may be w/o video connectors).
Secondary graphics board:
- Four GPGPU cards (w/o video connectors).
Main RAID board:
Six PCIe v.3.0 x4 8-ports RAID controllers (two internal mini-SAS
connectors each)

Storage

SSD drives used: 60GB SAS 6G SSD with no worse than up to 450GB/s throughput and up to 25000 IOPS.

Storage configuration

- 480GB system storage (up to 3.5GB/s read/write, up to 200 000 IOPS): embed I/O daughterboard RAID controller hardware RAID 10 over (2x8 + 1 cold-spare) 60GB SATA 6G SSD - occupies one 2U 24-drives front compartment.
- Two 512GB RAM-drives (may be automatically copied to the working storage at shutout and emergency)
- 7.3TB (using 96 x 60GB SAS SSDs) working storage (up to 30GB/s read/write, up to 1 800 000 IOPS): RAID subsystem hardware RAID 6 over (11 + 1 hot-spare) RAID cards (and RAID subsystem on-board PCIe v.3.0 SAS 6G controllers) hardware RAID 0 (each over 8 x 60GB SATA 6G SSDs) - occupies four 2U 24-drives front compartments.
- 72TB HDD archive storage (up to 8GB/s read/write, up to 7 000 IOPS): RAID subsystem hardware RAID 6 over (13 + 1 hot-spare) LSISAS2004 hardware RAID 0 (each over eight Seagate Barracuda LP ST31000520AS 1TB HDDs) - occupies two 4U 36-drives front compartments.

Front I/O

- DVD drive,
- 5.25" front connections hub: USB 3.0 to four USB 3.0 (using VIA VL810), USB 2.0 to four USB 2.0 (using NEC µPD720114), IEEE 1394b to two IEEE 1394b (using TI TSB81BA3), SATA II to three SATA II (using Silicon Image SiI3726)
- up to four other 5.25" (or 3.5") needed front I/O devices

External expansion

Desktop subsystem - fiber optic connection using PCIe x8 external connector.

Expandability

A fully confugured system still have a space for 64GB of 1333MHz RAM and 1TB (2TB with Additional memory daughterboard) of 800MHz RAM.
Four GPGPU cards may be added (plus eight more with additional Secondary graphics board).
Twenty four (twelve PCIe v.3.0 x8 and twelve PCIe v.3.0 x4) RAID cards (connected to external storage) may be added.
Two spare PCIe v.2.0 x8 external links and two spare PCIe v.2.0 x4 external links allow to direct-attach up to four expantion subsystems (up to 72 expansion slots) with not-too-fast I/O devices.
Two fastest Infiniband and two fastest LAN connectors allow to connect it to ultra-fast servers (including application servers). Four 1GB LAN ports allow lot of network options,
Four spare QPI pairs (four more may be added with Additional memory daughterboard) allows to add any other expansion subsystem (including two-four additional processors). Adding a four-to-twenty QPI switch (four six-way QPI switches) and additional four-processors modules will allow to transform the system into a supercomputer.

Cooling

Approximate heat dissipation:
8 overclocked processors - 250W x 8 = 2000W
8 memory controllers - 100W x 8 = 800W
16 GPU processors - 250 x 16 = 4000W
That will need an external outdoor cooling system with internal and external loops and heat exchangers
Draft sample internal loop presented below:
12 x Koolance PMP-450S pumps with Koolance OV-RP450 Nozzle & Reservoir Base (each serves up to three cooling points)
12 x Peltier chillers (two Koolance HD-60 HDD coolers with ~ 400-600W Peltier chip between them - installed between pumps outlets and hot water reservoir inlets - chills water to a bit lower than ambient temperature)
Cold and hot water reservoirs (common for all pumps)
6 x Koolance HXP-193 Plate Heat Exchangers (connection to external cooling system)
6 x Koolance HX-CU1320V 4-fans radiators (cools water in the case of external cooling fail - up to 8KW heat dissipation)
Needed control subsystem
External 8-10KW cooling system (1-1½" tubing) with parallel distribution to six Plate Heat Exchangers

Desktop subsystem

Case

12 x 5.25" bays full tower case with 500W Power Supply.
Support for ATX form-factor with 13 expansion slots motherboard.

Backplane

13-slots ATX form-factor expansion board

Basic chipset

- PCIe v.2.0 x16 upstream connector
- PLX PEX 8648 switch - one PCIe x16 into three PCIe x8 and eight PCIe x1 links;
- PLX PEX 8624 switch - one PCIe x8 into four PCIe x4 links;
- Intel 41210 bridge: PCIe x8 link into two133MHz PCI-X buses (wired to two PCI-X slots)
- Intel 41210 bridge: PCIe x4 link into two PCI buses (wired to three PCI slots and three on-board PCI controllers)

Storage

- LSISAS2116 controller: sixteen SATA 6G ports (four eSATA, two used by SATA-to-IDE bridges, ten internal) - using PCIe x8 link;
- Two jMicron JM20330 SATA-to-IDE bridges, providing two hot swap IDE ports (2 devices each)

USB/FireWire

- Two NEC µPD720200 USB 3.0 2-ports Host Controllers - three external, one internal USB 3.0 ports - using two PCIe x1 links;
- Three NEC µPD720101 USB 2.0 5-ports Host Controller - six external, four internal USB 2.0 ports - using PCI bus;
- Texas Instruments TSB83AA23 1394b chip: 2 external and 1 internal IEEE 1394b ports - using PCIe x1 link;

On-board connectors

- Ten SATA 6G ports
- Two IDE ports
- One USB 3.0 connector
- Three USB 2.0 connectors (six ports)
- One FireWire connector

External connectors

- PCIe v.2.0 x16 upstream connector
- Four eSATA ports
- Three USB 3.0 ports
- Nine USB 2.0 ports
- Two IEEE 1394b port

Expansion slots

- Three PCIe x4 slots
- Five PCIe x1 slots
- Two PCI-X 133MHz slots
- Three PCI slots

Front I/O devices and connections

- 5.25" front connections hub: USB 3.0 to four USB 3.0 (using VIA VL810), USB 2.0 to four USB 2.0 (using NEC µPD720114), IEEE 1394b to two IEEE 1394b (using TI TSB81BA3), SATA II to three SATA II (using Silicon Image SiI3726)
- BR DVD RW, DVD RW and CD RW drives
- IDE and SATA 3.5" hot-swap HDD mobile racks (with 2.5" and 1.8" converters)
- Audio card front plate
- All formats flash cards reader
- PCIe-to-ExpressCard Reader front adapter
- PCI-to-PCMCIA Reader front adapter
- Other needed front devices

Add-on cards and rear expansion connections

- Audio card slot part
- PCIe-to-ExpressCard Reader slot part
- PCI-to-PCMCIA Card Reader slot part
- Other needed cards and connectors

External I/O devices

Monitors

Six 2560x1600 monitors

Other I/O devices

Needed set of I/O devices