Switching

The cluster's proposed switch topology is as shown below:

Nemo Cluster Switching Topology

XFig image source file

This is a network design that is oversubscribed at one or two locations, depending upon the choice of the core switch on the right. While it would be possible to design a network with no oversubscription, this would increase costs by approximately $120k, and we believe that this would not increase the performance of the cluster on typical analylsis codes.

This design differs from the original proposal. The design in the original proposal was fully meshed and not oversubscribed. However we believe that based on observation of current analysis codes, it makes more sense to have a network design that is oversubscribed and to spend the savings on additional nodes.

The network is oversubscribed at the edges by a factor of two. The edge switches are SMC 8505T (5 port) or 8508T (8 port) unmanaged layer two switches. These are based on a Broadcom chip set, and cost approximately $62 and $89 each, respectively. These switches are non-blocking at all packet sizes and can handle jumbo frames up to 9kB in size.

There are a handful of options for the core network switch. These are:

We have tested the edge switches and on-board Broadcom NICs both for overall performance and for jumbo frame support. There are two choices of Linux driver for this NIC. NEED BCM RESULTS.



The following netperf tests were performed using netperf-2.4.0, available at netperf's home page. For each test, I ran netserv on one machine (with the command netserver -v 4 -4 -d, where -v 4 turned up verbosity, -4 specifies TCP/IP v4.0, and -d would return extra debug info), and then ran the following netperf commands from the remote machine. In the tests that measure bidirectional speeds I ran a netserver and netperf on each machine.



SMC 8505T Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8505T switch, and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput:
-Average CPU Usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8505T switch, and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8505T switch, and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte



SMC 8508T Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8508T switch, and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput:
-Average CPU Usage



This
graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8508T switch, and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage



This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via 8508T switch, and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte


D-Link DGS-108 Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via DGS-108 switch, and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput:
-Average CPU Usage



This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via DGS-108 switch, and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage



This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via DGS-108 switch, and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte



SMCGS5 Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS5 switch, and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput:
-Average CPU Usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS5 switch, and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS5 switch, and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte



SMCGS8 Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS8 switch, and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput
-Average CPU Usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS8 switch, and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via SMCGS8 switch, and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte




Crossover Cable Benchmarking
This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via a crossover cable and running netperf. The bottom plot shows avg cpu % of the two machines while the top plot shows total throughput
-Average CPU Usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via a crossover cable and running netperf. The bottom plots shows both machines cpu usage while the top plot shows total throughput.
-Individual control and slave CPU usage


This graph shows total throughput of two machines, both Supermicro H8DAR-T motherboards, TCP settings were not optimized, attached via a crossover cable and running netperf. The bottom (blue) plot shows both machines cpu usage while the top (red) plot shows total throughput and finally the middle (violet) plot is instructions per byte.
-Instruction per byte