Main -> Super workstation

Super workstation

Last updated 03.06.2010

I'd like to describe the system build around Wish list board, but not sure they'll be built.

So, this is an evolution of the "May be built" system - the illustration of what might be build (if price is not an option).
It's a bit crazy system - I do not think anybody would need all the possibilities at a time (and in the single system), but it's just a dream, anyway.

The main part of the system is built inside a single "superchassis" with several rackmount slide-ins.
To isolate nosy and hot main system part from the operator, the operator working environment set aside from the main system (up to 150m) and connected to it by fiber-optics lines.
Two "deskside" boxes support all possibly needed connection. A miniature desktop box holds only very few most used devices and connector - to provide desktop space for a huge 6-monitors video subsystem.

All not off-the-shelf items used here are described in system specification - really somewhat similar devices already exist - Lucid Adventure 2500 (provides only x8 connections for 4 cards), One Stop Systems OSS-PCIe-2U-ENCL-EXP-4-1 PCIe x16 expansion box (I do not like its design and crazily high pricing) and One Stop Systems switching PCIe x4/x8 HBAs. So, if needed (and financed) , the system may be easily built within 3-4 months.

Current pricing for PCIe expansion boards are crazily high because the vendors are trying to get 1000% profit selling one board a month. With decent pricing they could sell thousands of boards (with at least no less resulting profit).

System overall description

The system consists of the main workstation, several auxilary/satellites systems, the system server and a number of stand-along I/O devices/subsystems.
The main workstation is built around a 40U high, 36"deep server rack structure with two separate "deskside" and one desktop I/O.
The deskside part and monitors are connected to the main workstation rack by fiber-optics cables, a desktop box is connected to deskside part by USB cable.

Main system specification

System case

A structure build around Rackmount Solutions 19" RSP-702036 Server Rack (40U high, 36"deep).
Double-leaf side doors for easy access to the internals.
Front and rear slide-ins - open inside and with minimal nessesary sides height built-to-order sub-chassis.
Unused front and rear openings are covered by blank plates.
The approach allows easy system upgrade, use of less number of power supplies and more compact overall design.

Power supply

Three 1U rear slide-in with Sure Star TC-1800RHN4 1800W 3+1 redundant PSU and power distribution unit each
One PSU serves as a main one (connected to power switch, two other are connected to it by triple PSU 24-Pin Adapter Cable (to supply "power good" signal)

Motherboard slide-in

Chassis:

4U rear slide-in: 16.4" wide backpanel with standard-size I/O panel and 13 slots openings

Motherboard:

Supermicro X8DTG-QF - the best workstation motherboard to-day, though not w/o some drawbacks (first of all - artifisially limited number of USB ports).

Processors:

2 x water-cooled Intel Xeon W5680 overclocked to 3.8-4.0GHz

Memory:

192GB: 12 x 16GB DDR3 1333MHz 240pin ECC Registered CL9 Hynix HMT42GR7AMR4C-H9 - providing 48GB of working memory plus 144GB RAM-drive

Expansion cards:

Cyclone PCIe-426 PCIe x16 v.2.0 host bus adapter - connected to the GPU/GPGPU slide-in.
2 x Cyclone PCIe-426 PCIe x16 v.2.0 host bus adapters - connected to the RAID expansion slide-in.
Cyclone PCIe-426 PCIe x16 v.2.0 host bus adapters - connected to the Network expansion slide-in.
Two Dolphin DXH510 PCIe x8 v.1.1 copper / fiber optics HBAs installed into built-to-order switching PCIe v.2.0 x4 to PCIe x8 (v.1.1 throughput) adapter risers - fiber optics link to the deskside expansion boxes (using northbridges PCIe v.2.0 x4 slots).
One Stop SystemsOSS-PCIe-HIB2-x4-H host bus adapters - connected to the I/O expansion slide-in (using southbridge PCIe x4 slot).
SIBA SD-VIA-5U 4-port USB 2.0 PCI card
StarTech PCI3UV22F 3 external, 1 internal ports USB 2.0, 2 external, 1 internal ports FireWire (IEEE-1394a) PCI card

GPU/GPGPU slide-in

Chassis:

4U rear slide-in: backpanel for 16.4" wide board with 3.6" wide I/O panel and 16 slots openings.

Backplane

Built-to-order GPU/GPGPU PCIe x16 expantion backplane (16.4" wide backplane with one PCIe x16 connector and eight double-width PCIe x16 slots) - connected to the first motherboard's Cyclone PCIe-426.
Lucid Hydra 200 LT22102 chip (PCIe x16 into two PCIe x16).
Two Lucid Hydra 200 LT24102 chips (PCIe x16 into two PCIe x16) - using Lucid Hydra 200 LT22102 links.
Four PLX PEX 8647 switch (PCIe x16 into two PCIe x16) - using Lucid Hydra 200 LT24102 links.
8 dual-wide PCIe x16 slots.

Graphics/GPGPU cards:

- Four XFX 4GB Radeon HD 5970 Black Edition- factory overclocked with 6 mini DisplayPort connectors (with professional firmware and drivers)
- Two GeForce GTX 480 (with modified Tesla C2050 firmware and drivers)
- Two free slots
The main graphics card is connected to the desktop box by six Atlona AT-F13044L-10 mini DisplayPort Fiber Optics cables.

RAID cards slide-in

Chassis:

4U rear slide-in: 16.4" wide backpanel with standard I/O panel and 13 slots openings, internal 20 slots middlepanel (about 10" from the backpanel)

Backplane

Built-to-order dual PCIe x16 to 6 x x8 expantion backplane ( ATX form-factor, two PCIe x16 upstream cable connectors, two PLX PEX 8648 switches (each providing 3 x PCIe v.2.0 x8 slots) - connected to two of the motherboard's PCIe x16 HBAs

Expansion cards:

6 x LSI MegaRAID SAS 9261-8i PCIe x8 SAS 6G RAID controllers - each set of 3 controllers uses one of the motherboard x16 links, each controller is connected to two SAS Expanders.
12 x built-to-order LSISAS2x36-based SAS 36-Port Expander cards (1 x internal miniSAS upstream connector, 2 external, 6 internal miniSAS downstream connectors) - installed at the middlepanel.
Six external dual miniSAS expansion brackets (connected to the SAS expanders external connectors).

Network slide-in

Chassis:

4U rear slide-in: ATX form-factor backpanel

Backplane

Built-to-order ATX form-factor expantion backplane, PCIe x16 upstream cable connectors, PLX PEX 8648 switch, providing 2 x PCIe v.2.0 x8 slots, 2 x PCIe x4 slots and 3 x PCIe x1 slots - connected to the fourth motherboard's PCIe x16 HBA

Expansion cards:

Two PCIe x8 Mellanox MHZH29-XTR - 40GB QSFP Infiniband connector and SFP+ 10GB LAN connector each
Two PCIe x4 Intel E1G44ET - four RJ-45 1GB LAN ports each.
Three spare PCIe x1 slots

I/O expansion slide-in

Chassis:

4U rear slide-in: backpanel for 16.4" wide board with 3.6" wide I/O panel and 16 slots openings

Backplane

Built-to-order expantion backplane (ATX-wide backplane with PCIe x4 connector, four PCI-X 66MHz slots and six PCIe x1 slots) - connected to the motherboard's PCIe v.1.1 x4 HBA
PLX PEX 8614 switch (PCIe x4 into one PCIe x4 link and six PCIe x1 slots)
Intel 41110 bridge (PCIe x4 link into four PCI-X 66MHz slots

Expansion cards:

AVerTV Satellite Trinity satellite TV tuner (PCIe x1)
AVerMedia AVerTV Duo Hybrid PCI-E II terrestial TV tuner (PCIe x1)
Four spare PCI-X 66MHz slots
Four spare PCIe x1 slots

Flash SSD slide-ins

Chassis:

Four 2U front slide-ins (8U in total) with 24-drives pass-through 2.5" hot-swap bays backplanes, supporting up to 96 x 2.5"drives.

Storage

96 drives 3.8TB software RAID 5 array over (5 + 1 hot spare) hardware RAID 0 arrays (up to 1 000 000 IOPS (4KB), up to 11,5GB/s read, up to 7,2GB/s write)
Each hardware RAID 0 array is driven by a LSI MegaRAID SAS 9260-8i card with two SAS expanders and consists of 16 x OCZ Vertex EX 60GB SSD drives, connected 8/8 to SAS expanders

Working HDD slide-ins (front)

Chassis:

Four 4U front slide-ins (16U in total) with 24-drives pass-through 3.5" hot-swap bays backplanes, supporting up to 96 x 3.5"drives.

Storage

96 drives 19TB software RAID 8 array over (5 + 1 hot spare) hardware RAID 0 arrays (up to 25 000 IOPS (4KB), up to 11,5GB/s read, up to 7,2GB/s write)
Each hardware RAID 0 array is driven by a LSI MegaRAID SAS 9260-8i card with two SAS expanders and consists of 16 x Seagate Cheetah 15K.7 ST3300657SS 300GB HDD drives, connected 8/8 to SAS expanders (same with SSD)

Removable drives / Front connections slide-in

Chassis:

4U front slide-in chassis with 8 x 5.25" bays

Removable drives/Front connections:

LG GH22LS30 DVD-RW with LightScribe
SIIG JU-H62B22-S2 4-Port USB2.0, 2-port FireWire bay hub

Deskside I/O subsystem

Really this couple of somewhat set-aside desktop boxes provides all needed front-side and handy rear I/O connectivity.
Two USB links from it allow desktop box to provide minimally needed connections to serve as a mock-up "computer".

Cases

Two LIAN LI PC-A77F full tower cases with 400W PSUs - each modified for installation of Dolphin DXE410 expantion board with two 3-slots risers as additional backplanes
Provide 24 x 5.25" bays, seventeen available PCIe slots, three PCI-X slots and three PCI slots

Backplanes

Two Dolphin DXE410 Mini-ATX expansion boards - PCIe v.1.1 x8 fiber optic connection - 4 x (PCIe x8/x4 plus PCIe x0/x4) slot pairs
Two Advantech AIMB-R4301-03A1E - PCIe x4 to three PCIe x1 expansion card
One Advantech AIMB-R430P-03A1E - PCIe x4 to three PCI expansion card
One Supermicro CSE-RR2UE-AX - PCIe x8 to three PCI-X expansion card
All Advantech/Supermicro cards are connected to PCIe slots by flexible cable riser and installed as backplanes

Add-on cards

Three PCIe x4 flexible cable risers (links to Advantech "backplanes")
One PCIe x8 flexible cable riser (links to Supermicro "backplane")
Startech PEX2EC35 PCI Express to ExpressCard Adapter card with front card loader (PCIe x1)
DigiGear PCD-RP220CS PCI Dual Front Load PCMCIA PC Cardbus Card Reader (PCI)
Creative's Sound Blaster PCI Express X-Fi Titanium Fatal1ty Champion Series sound card (PCIe x1)
Grass Valley FIRECODER Blu Blu-ray Encoder card (PCIe x1)
Grass Valley EDIUS SP-SDI video NLE subsystem (one PCIe x1, one PCI, one any free slot)
Grass Valley HDSTORM Plus video NLE card (PCIe x1)
Grass Valley FireCoder real-time MPEG encoder card (PCIe x1)
TC Electronic PowerCore Express audio engine (PCIe x1)
UAD-2 QUAD Omni audio engine (PCIe x1)
LSI SAS3081E-R HBA - 8 internal SAS/SATA ports (PCIe x8)
BYTECC BT-PESAPA PCIe to 2 x eSATA II (RAID-capable) + PATA Card (PCIe x1)
Asus U3S6 USB 3.0 / SATA 6G combo card - two external USB 3.0 connectors, two internal SATA 6G connectors (PCIe x4)
SYBA SD-PEX50011USB/FireWire combo card - 2 x 1394b + 2 x USB external ports, 1 x 1394b + 3 x USB internal ports (PCIe x1)
Two SYBA SD-PEX-VT4E4IUSB/FireWire combo card - 2 x 1394a + 2 x USB external ports, 1 x 1394a + 3 x USB internal ports (PCIe x1)
SYBA SD-89300205 4 x External, 1x Internal USB 2.0 ports

Rear expansion connections

Okgear OK302 PCI 2 x Female Molex + 2 x Male 3-pin Fan Bracket - lets supply power for external devices

Front I/O devices and connections

Internal USB hubs:
Four SYBA SD-PCI-UHUB4 USB hubs - installed internally, serving top USB connectors, BTO front panel USB connectors and other front USB-connected devices
Top plates: eight USB ports, two eSATA ports

Installed in 3.5" bays with 3.5" to 5.25" adapters:
Startech PEX2EC35 PCI Express to ExpressCard Adapter front bay (with USB port connected to SYBA SD-PCI-UHUB4)
DigiGear PCD-RP220CS Dual Front Load PCMCIA PC Cardbus Card Reader
Icy Dock MB4448AR-B - with built-in USB-to-IDE converter - for 2.5" IDE drives
Addonics MSUDDUS Micro SATA UDD - for 1.8" SATA drives
Icy Dock MB448SR-B - for 2.5" SATA drives
SABRENT CRW-UINB 68-in-1 USB 2.0 Internal Card Reader

Installed in 5.25" bays:
Creative's Sound Blaster PCI Express X-Fi Titanium Fatal1ty Champion Series front I/O box
Grass Valley HDSTORM Plus video NLE front bay
Intel AXXUSBFLOPPY internal USB floppy drive
IOMagic I523252 CD-RW (using BYTECC BT-PESAPA PATA port) - seem to be the only CD-RW drive awailable now
LG GGW-H20L Blu-Ray DVD-RW + HD reader (connected to LSI SAS3081E-R SAS port)
LG GH22LS30 DVD-RW with LightScribe (connected to LSI SAS3081E-R SAS port)
Icy Dock MB123AK-B with Koutech IO-UIC120 USB-to-IDE converter - for 3.5" IDE drives
Icy Dock MB123SK-1B - for 3.5" SATA drives
Built-to-order front plate - four USB, one FireWire 1394a, one FireWire 1394b, one eSATA, one Power eSATA

Desktop box

No that this box is really needed - the system may be nicely built withuot it, but just imagine a newcomer surprise seing a small ITX case "powering" the superpowerful system.

Case

APEX MI-110 Mini-ITX tower computer case with 250W power supply

Case internals

- Power Switch Board (same as on Chenbro Universal Expander Kit)
- Built-to-order I/O panel with six mini DisplayPort to DVI convertors (six vertical DVI connectors, two "columns" of mini DisplayPort receptacles (female connectors) - 3 connectors in a "column" ), six USB connectors (with Type A internal connectors).
- Dual USB PCI bracket (with idividual female pin headers)
- Two internally installed SYBA SD-PCI-UHUB4 USB hubs (providing 8 USB connections)
- SIIG JU-H42B22-S2 USB hub - installed into front 3.5" bay
- LG GH22LS30 DVD-RW with LightScribe connected to USB using Nippon Labs AD-USB-SATA USB to SATA HDD Bridge Adapter - installed into front 5.25" bay

This mock-up "computer" imitates direct to box display connections and provides some at-hand USB connectivity (keyboad, mouse, some other devices)

External I/O devices

KVM

Software Infiniband "KVM" - Scalable Graphics Remote Desktop over Infiniband - used to control satellites and the server from the Workstation

Monitors, keyboards, mice

6 x NEC Display Solutions NEC LCD2690WUXi2-BK (lower row - four vertical monitors, upper row - two horisontsl monitors
Microsoft Keyboard 4000
Wacom Intuos4 Large with full pens set

Scanners

Epson Perfection V750-M Pro - for film scan
Plustek OpticPro A320 - for paper scan
Contex HD3650 - for large format paper scan

Other I/O

Logitech Z-5500 - connected to audio card optical port
Logitech ClearChat PC Wireless - connected to desktop box front USB port
cirago BTA-3190 USB 2.0 Bluetooth Dongle - connected to desktop box USB port
Logitech QuickCam Pro 9000 - connected to desktop box rear USB port
Nexcopy USB200PC USB Duplicator - connected to desktop box rear USB port

Additional servers / satellites / other hardware

Claster satellites

Two similar systems

System barebone

2U 6026TT-HDIBQRF dual board server with 1400W PSU (2 x X8DTT-HIBQF+ motherboards - with on-board Mellanox ConnectX QDR Infiniband 40Gbps conections)

Processors:

Two pairs of Intel Xeon X5550

Memory:

2 x 48GB: 4 x (3 x 4GB) DDR3 1066MHz 240pin ECC Registered CL7 Crucial CT3KIT51272BB1067

Expansion cards:

2 x 2GB Radeon HD 5970 (with professional firmware and drivers)

SSD drives

System / Working:
2 x 120GB OCZSSD2-1CLS120G 3.5" SSD drives in RAID 1

Removable drives/Front connections:

USB DVD drive (when needed)

Archive/multimedia server

Archive server, main LAN server, ADSL gateway and guest LAN gateway, print server and - connected to workstation/clasters by switched x4 40GB Infiniband link and to LAN switch by 10GB link.

Case

9U Chenbro RM91250 - supports 50 x 3.5" hot-swap drives.

Motherboard:

Supermicro X8DTH-6F

Processors:

2 x Intel Xeon E5520

Memory:

48GB: 4 x (3 x 4GB) Hynix HMT151R7AFP8C DDR3 1333MHz 240pin ECC Registered CL9 memory modules

Expansion cards:

Two Mellanox MHZH29-XTR Infiniband 4x 40Gb/s plus 10GB Ethernet host bus adapters - used as dual x4 IB connection to the InfiniBand switch and dual 10GB connection to the LAN switch
>LSI MegaRAID SAS 9260-8i SAS 6G RAID controller with two Chenbro CK13601 SAS 36-Port Expander Cards

SSD/HDD drives

System: 2 x internal 60GB OCZ Vertex EX in RAID 1 - connected to the motherboard SATA ports
Archive storage
20TB 22-drives RAID 6 array (up to 2.4GB/s, ~2000 IOPS)
(22 + 2 hot spare) x Seagate Barracuda 7200.12 ST31000528AS 1TB HDD drives
Multimedia storage
32TB - 4 x 8TB RAID 5 arrays (~400MB/s)
22 (4x5 + 2 hot spare) x WD WD20EADS 2TB HDD drives

Removable drives/Front connections:

Slim rear DVD drive

System racks, UPS, cooling and interconnects

System rack

Rackmount Solutions 19" RSP-702036 Server Racks (40U high, 36"deep) with Pagoda style top with fans.
This rack hosts all auxilary chassis

UPS

9U APC Symmetra APC Symmetra LX 12kVA with additional battery module
5U APC Symmetra LX Extended Run battery

Main system water cooling

3U Koolance ERM-2K3U
4U Koolance ERM-3K4U5 might be added for GPU cooling

Rack-mounted system interconnects

1U Mellanox MTS3600 switch (48 x 40GB InfiniBand ports) - connects system components
1U NETGEAR GSM7352S switch (48 x 1GB ports) with two AX741 10GB adapters - connects archive/multimedia server to other home computers

Free-stand system interconnects

Tp-Link TL-WR941ND Wireless N Router - connects connects I/O expansion server to guest wireless computers (server 1GB LAN port)
ZyXEL P-662HW-D ADSL 2+ 4 Port Router with Firewall - connects I/O expansion server to Internet via ADSL (server 1GB LAN ports)

Other I/O devices

Connected to the server

Printers

HP Color LaserJet 4700 (Q7491A) with 544 MB memory and one additional 500 pages tray - everyday printer
Epson Stylus Pro 4880 - high quality / tabloid printer
HP Designjet Z3200ps 1118mm (Q6721A) - large format printer
Roland MDX-20 Scanner / Milling Machine (203.2 mm (X) x 152.4 mm (Y) x 60.5 mm (Z) operation area) - metal/plastic models
Z Corporation Spectrum Z510 (10" (X) x 14" (Y) x 8" (Z) operation area) - printed color models

Other I/O

4U rack-mount Primera Bravo XR Disc Publisher
4U rack-mount Primera Bravo XRP-Blu Blue-Ray Disc Publisher