Understanding InfiniBand to PCI Impedance
 Published on Thursday, 11 March 2010 06:27
 Written by Douglas Eadline
 Hits: 6080
From the why didn't I think of that department
Recently, Richard Walsh put together a table with both PCI and InfiniBand (IB) specifications. He published the spreadsheet on the Beowulf Mailing List. Cluster Monkey contacted Richard and asked permission to reproduce the table in HTML so that it was more accessible. We broke it in half resulting in two tables so it would fit on the page, but the information is the same.
While the information is not new, collecting it in these tables was a great idea. The issue is the potential "bandwidth impedance" mismatch between and IB HCA's (Host Channel Adapters or cards) and the PCI bus. (Note: In this sense the impedance mismatch is not considered electrical, but bandwidth.) As an example, the current fastest usable IB speed is QDR (Quad Data Rate, 4.0 GigaBytes/sec) this must be matched with PCI 2.0 (also called Gen2) 8x interface (4.0 GigaBytes/sec) for maximum performance. Most new QDR IB cards support up to 8x PCI Gen2 speeds and will autonegotiate to lower speeds if the PCI bus is slower resulting in lower performance.
The following table is the current and future PCI Express transfer modes. The list of abbreviations and terms used in both tables are as follows (Giga=10^9).
 Lanes  the number of PCI lanes, A lane is composed of a transmit and a receive pair of differential lines. Each lane is composed of 4 wires or signal paths.
 Wires/Pairs  number of wires used for the connection, similar to Lanes.
 LVDS Signal Rate  Low Voltage Differential Signal Rate is the speed at which signals change on the wire.
 Transfer Rate  the number of Transfers/second GT/sec = Giga Transfers/second over the wire. Similar to Signal Rate.
 Raw Data Rate  the maximum speed at which data can move through the bus or card. Reported in Gigabits/second.
 Effective Data Rate  the real or effective rate of data transfer when the signaling overhead is removed. This is the maximum that an application can achieve and is often called "wire speed." Reported in Gigabits/second.
 Bandwidth  the same as the Effective Data Rate, but in bytes/second
PCI Express  
# Lanes  Transfer Rate  Raw Data Rate (Per Direction) 
Effective Data Rate (Per Direction) 
Bandwidth Data Rate (Per Direction) 

1.0 Gen1  20% drop (8B/10B loss) 

1x  1  2.5 GT/sec  2.5 Gbits/sec  2.0 Gbits/sec  250 Mbytes/sec 
4x  4  2.5 GT/sec  10.0 Gbits/sec  8.0 Gbits/sec  1.0 Gbytes/sec 
8x  8  2.5 GT/sec  20.0 Gbits/sec  16.0 Gbits/sec  2.0 Gbytes/sec 
16x  16  2.5 GT/sec  40.0 Gbits/sec  32.0 Gbits/sec  4.0 Gbytes/sec 
32x  32  2.5 GT/sec  80.0 Gbits/sec  64.0 Gbits/sec  8.0 Gbytes/sec 
2.0 Gen2  20% drop (8B/10B loss) 

1x  1  5.0 GT/sec  5.0 Gbits/sec  4.0 Gbits/sec  500 Mbytes/sec 
4x  4  5.0 GT/sec  20.0 Gbits/sec  16.0 Gbits/sec  2.0 Gbytes/sec 
8x  8  5.0 GT/sec  40.0 Gbits/sec  32.0 Gbits/sec  4.0 Gbytes/sec 
16x  16  5.0 GT/sec  80.0 Gbits/sec  64.0 Gbits/sec  8.0 Gbytes/sec 
32x  32  5.0 GT/sec  160.0 Gbits/sec  128.0 Gbits/sec  16.0 Gbytes/sec 
3.0 Gen3  1.5% drop (128B/130B loss) 

1x  1  8.0 GT/sec  8.0 Gbits/sec  8.0 Gbits/sec  1.0 Gbytes/sec 
4x  4  8.0 GT/sec  32.0 Gbits/sec  32.0 Gbits/sec  4.0 Gbytes/sec 
8x  8  8.0 GT/sec  64.0 Gbits/sec  64.0 Gbits/sec  8.0 Gbytes/sec 
16x  16  8.0 GT/sec  128.0 Gbits/sec  128.0 Gbits/sec  16.0 Gbytes/sec 
32x  32  8.0 GT/sec  256.0 Gbits/sec  256.0 Gbits/sec  32.0 Gbytes/sec 
The following table is the current and future InfiniBand transfer modes.
InfiniBand  
# Wires  # Pairs  LVDS Signal Rate 
Raw Data Rate (Per Direction) 
Effective  Bandwidth Data Rate (Per Direction) 

SDR (Single Data Rate)  20% drop (8B/10B loss) 

1X  4  2  2.5 GHz  2.5 Gbits/sec  2.0 Gbits/sec  250 Mbytes/sec 
4X  16  8  2.5 GHz  10.0 Gbits/sec  8.0 Gbits/sec  1.0 Gbytes/sec 
12x  48  24  2.5 GHz  30.0 Gbits/sec  24.0 Gbits/sec  3.0 Gbytes/sec 
DDR (Double Data Rate)  20% drop (8B/10B loss) 

1X  4  2  5.0 GHz  5.0 Gbits/sec  4.0 Gbits/sec  500 Mbytes/sec 
4X  16  8  5.0 GHz  20.0 Gbits/sec  16.0 Gbits/sec  2.0 Gbytes/sec 
12x  48  24  5.0 GHz  60.0 Gbits/sec  48.0 Gbits/sec  6.0 Gbytes/sec 
QDR (Quad Data Rate)  20% drop (8B/10B loss) 

1X  4  2  10.0 GHz  10.0 Gbits/sec  8.0 Gbits/sec  1.0 Gbytes/sec 
4X  16  8  10.0 GHz  40.0 Gbits/sec  32.0 Gbits/sec  4.0 Gbytes/sec 
12x  48  24  10.0 GHz  120.0 Gbits/sec  96.0 Gbits/sec  12.0 Gbytes/sec 
EDR (Eight Data Rate)  3% drop (64B/66B loss) 

1X  4  2  25.78 GHz  25.78 Gbits/sec  25.0 Gbits/sec  3.125 Gbytes/sec 
4X  16  8  25.78 GHz  103.12 Gbits/sec  100.0 Gbits/sec  12.50 Gbytes/sec 
12x  48  24  25.78 GHz  309.36 Gbits/sec  300.0 Gbits/sec  37.50 Gbytes/sec 
Be sure to double check all the specifications for your IB HCAs and servers to make sure you have a good match.