From the why didn't I think of that department

Recently, Richard Walsh put together a table with both PCI and InfiniBand (IB) specifications. He published the spreadsheet on the Beowulf Mailing List. Cluster Monkey contacted Richard and asked permission to reproduce the table in HTML so that it was more accessible. We broke it in half resulting in two tables so it would fit on the page, but the information is the same.

While the information is not new, collecting it in these tables was a great idea. The issue is the potential "bandwidth impedance" mismatch between and IB HCA's (Host Channel Adapters or cards) and the PCI bus. (Note: In this sense the impedance mismatch is not considered electrical, but bandwidth.) As an example, the current fastest usable IB speed is QDR (Quad Data Rate, 4.0 GigaBytes/sec) this must be matched with PCI 2.0 (also called Gen2) 8x interface (4.0 GigaBytes/sec) for maximum performance. Most new QDR IB cards support up to 8x PCI Gen2 speeds and will auto-negotiate to lower speeds if the PCI bus is slower resulting in lower performance.

The following table is the current and future PCI Express transfer modes. The list of abbreviations and terms used in both tables are as follows (Giga=10^9).

• Lanes - the number of PCI lanes, A lane is composed of a transmit and a receive pair of differential lines. Each lane is composed of 4 wires or signal paths.
• Wires/Pairs - number of wires used for the connection, similar to Lanes.
• LVDS Signal Rate - Low Voltage Differential Signal Rate is the speed at which signals change on the wire.
• Transfer Rate - the number of Transfers/second GT/sec = Giga Transfers/second over the wire. Similar to Signal Rate.
• Raw Data Rate - the maximum speed at which data can move through the bus or card. Reported in Gigabits/second.
• Effective Data Rate - the real or effective rate of data transfer when the signaling overhead is removed. This is the maximum that an application can achieve and is often called "wire speed." Reported in Gigabits/second.
• Bandwidth - the same as the Effective Data Rate, but in bytes/second

The following table is the current and future InfiniBand transfer modes.

Be sure to double check all the specifications for your IB HCAs and servers to make sure you have a good match.