In all High Performance and Supercomputer systems, the HPC interconnect is of paramount importance. In contrast to server farms or cloud computing, the interconnect provides fast, low-latency and high throughput communication between all compute nodes.

MPI programs use the interconnect to utilize many compute nodes in parallel for large and complex calculations. By means of the interconnect, the MPI tasks convey data, states and intermediary results among each other.

In the Lichtenberg cluster, the interconnect is also used for accessing the storage/file systems.

As HPC clusters can consist of several parts or expansion stages, these distinct group of compute nodes are often called an “island”. This in particular holds true if the HPC interconnect is “fully meshed” only within such an island (see down below).

The HPC interconnect's top level is defined by the Infiniband HDR of our storage system (Home, Projects, Scratch). Being the “core” of the interconnect fabric, it manages the whole system based on HDR = „High Data Rate“ 100 = 100 GBit/s (formerly EDR).

Infiniband HDR: the compute and login nodes of this cluster stage (“island”) i02 feature Mellanox ConnectX cards of the “HDR100” standard:
“High Data Rate 100” = 100 GBit/s.

While HDR in general offers up to 200 GBit/s, the nodes of this stage have cards with only 100 GBit/s – just the infiniband switches in between them communicate with “HDR 200”.

The internal IB topology of this cluster stage (“island”) is 1:1 non-blocking: any given (group of) compute nodes can communicate completely unharmed by the network traffic of any other (group of) compute node.

In other words, nowhere in the IB network of this stage (island) is the number of interconnect links less than “one per compute node”.

Infiniband HDR: the compute and login nodes of this cluster stage (“island”) i01 feature Mellanox ConnectX cards of the “HDR100” standard:
“High Data Rate” 100 = 100 GBit/s.

While HDR in general offers up to 200 GBit/s, the nodes of this stage have cards with only 100 GBit/s – just the infiniband switches in between them communicate with HDR 200.

The internal IB topology of this cluster stage (“island”) is 1:1 non-blocking: any given (group of) compute nodes can communicate completely unharmed by the network traffic of any other (group of) compute node.

In other words, nowhere in the IB network of this stage (island) is the number of interconnect links less than “one per compute node”.

Further details on the Lichtenberg-HPC components can be found in our Hardware Overview.