Main -> System expanding -> What one true PCIe x16 v. 2.0 link may support
What one true PCIe x16 v. 2.0 link may support
True PCIe thoughput
According to Intel "An Introduction to the Intel QuickPath Interconnect" (page 20), the real data tranfer speed of single PCIe x1 v.1.1 bus is fluctuating from 170 to 198MB/s , PCIe x1 v.2.0 - from 340 to 395MB/s.
The additional bus/protocol connection data overhead eats up about 3-10% more from this data transfer speed.
So, the maximum real data transfer rates are about as follows:
PCIe v.1.1:
x1: 150-190MB/s
x4: 600-760MB/s
x8: 1200-1520MB/s
x16: 2400-3040MB/s
PCIe v.2.0:
x1: 300-380MB/s
x4: 1200-1520MB/s
x1: 2400-3040MB/s
x1: 4800-6080MB/s
Graphics cards needs
nVIDIA lies (and other vendors unwillingness to disclose them or even supporting these lies - like AMD, for example) made us think that PCIe x16 v. 2.0 link is a must for supporting graphics cards.
Well, to the extent it is - nVIDIA cards needs x16 link with PCIe v. 2.0 power option
But this does not mean that we need full PCIe x16 v. 2.0 bandwidth for graphics cards.
Really graphics cards work at very slow speed - no more than 1/2 of the PCIe v. 1.1 speed (less than 1/4 for nVIDIA cards) - hence is the need for a wide channel. They also use the (not widely advertised) ability of PCIe v. 2.0 to work w/o parity checking (so-called Graphics PCIe mode) - this allows wider data bus, helping to raise the productivity of slow graphics cards interface.
It is not widely known, but AMD cards (at least latest ones) can nicely run on PCIe x8 v. 2.0 w/o any productivity degradation. AMD followed nVIDIA in throwing away parity checking when possible, so, might be their cards will not work at full in PCIe x8 v. 1.1 slots (but will not loose more than 10-15% of productivity anyway). And even PCIe x4 v. 1.1 bus is enough for the latest graphics card working as SLI/CrossFire "slaves"
This mean that four independent graphics cards working in 3D each need no more than a quarter of the PCIe x16 v. 2.0 bandwidth. Even less is needed for in SLI/CrossFire mode.
What one PCIe x16 v. 2.0 link may support
The rest 3/4th of PCIe x16 v. 2.0 bandwidth may be nicely used for other purposes (well - not on the motherboards with nVIDIA chipsets - they can not support more than 1.6GB raw throughput even with overclocking (no more than 1.2GB without) - most really no more than 1.0GB (PCIe x4 v. 1.1 at full speed L)
There seem to be no PCIe v. 2.0 RAID cards (there was no need - 16 HDD drives in RAID 0 cannot fully utilize the bandwidth, only newest SSDs may push the need). The only other cards that really need PCIe x8 v. 2.0 connection are the x4 QDR 2-ports Infiniband cards (rear birds in usual environment), so
up to three more PCIe x8 cards (RAID/LAN/whatever else) may nicely join four graphics cards on one PCIe x16 v. 2.0.
Please mind that real (practical) RAID card productivity can not exeed 1.5GB/s - see previous paragraph.
Even in this configuration one true PCIe x16 v. 2.0 link may support 4 graphics cards and 3 RAID / 10GB LAN / other PCIe x8 v. 1.1 cards w/o any problems (really even more, as not all of them work simultaneously) - that will need 11 slots position to install (no ATX/eATX motherboard may allow that!)
Nehalem platform possibilities
Intel x58 and 5500 chips have two PCIe x16 v. 2.0 and one PCIe x4 v. 2.0, so in addition to above mentioned, may add controllers using at least 40 more PCIe 1.1 lanes (at least four PCIe x8 slots, much more with x4, x1, PCI-X and PCI connectors)
x58 and 5500 chips may talk to processor at up to 12.8GB/s each-way speed (or about 11.4GB/s of raw data), all peripherial interfaces have total 19GB/s speed - seems not all the connectivity bandwidth may be used - but are all the connections active at a time? Of cause not (see the southbridge - only 1 GB/s connection to the north bridge and more than 5GB/s in devices connections). Also the PCIe is duplex connection (while some devises "read", others "write", while QPI is two one-way buses), so all those lanes may be nicely used.
Nehalem EP planform may support more than one 5500 chip - so at least two more times connections - the question is - what form-factor motherboard may implement all them?