The installation process continues...
The shelving units in our computer room, with about 3/4 of the machines
installed. There are 13 on each of the bottom two shelves, then 10
on the third shelf (leaving room for the networking switches) and 12 machines
on the top shelf. Total weight of all the machines is about 1500
lbs. Notice the "safety-belt" straps that have been added to the
ends to prevent any machines from toppling off the end of the shelving.
The square bar angling down and to the right in the top right of the photo
is actually a bit of hollow plastic drainpipe, that carries an extension
cord, two monitor cables, two keyboard cables, and two mouse cables from
the equipment rack on the left to the console table on the right.
During the installation process -- here about 36 of the machines are
on the shelves. The networking switch is located right in the middle of
the third shelf.
The back of the shelving unit -- the bottom 24 machines are operational,
the top 24 are just in the process of being installed. Notice the
plastic rain-gutters (white on the inside) being used as cable trays to
support the weight of the CAT-5 cabling. Just behind these are 20-amp
power strips (two per shelf) which provide a convenient access to power
from anywhere on the shelving. Unfortunately our machines won't boot
without keyboards attached, so the keyboards are stored on the top of each
machine, and are left plugged in. [If anyone knows where to buy "keyboard
eliminators" or "keyboard emulators" for IBM PS/2 style keyboards,
please email Bruce Allen.]
Close-up of the back of node 19. The key hanging from a twist-tie
is the key for opening the case. Notice the unused RJ-45 and thin
ethernet connectors just above the loops of the power cord. These
are the original (supplied with the machine) 10BaseT ethernet ports.
The ethernet board which is actually being used is hidden behind the loops
of the power cord, and is a 100BaseT card. The systems are cooled
internally by two variable-speed fans. One, located at the bottom
front of the case, pulls in air through the lower part of the grill.
The other fan, located internally in the power supply, exhausts through
the grill above the orange tag on the power cord.
Close-up view of the switches. The top switch is ready for the
installation of the second twenty-four machines - the bottom switch is
already connected. Sitting on top of the switches is our "temperature
monitoring system"- a Radio Shack indoor/outdoor thermometer with
a min/max memory. The switch has green "connected" lights that provide
instant visual feedback about which machines are up and networked and which
are not. Here (during the installation process) the only three nodes
that are operating are the Master (N001) and slaves 23 and 24 (N023 and
N024).
Bala seated at the terminal in the machine room. We normally use
the Beowulf over the net, but it's convenient to have a terminal
in the room for installation and when something goes wrong. The panel
on the right is an (out-of-service) power distribution panel for
some of the Physics labs. 100 amps at 5 volts, anyone?
Wensheng Hua working on the power wiring. Initially the electrician put
the outlets for our circuits on the ground, which was a real mistake.
We have now had the wiring moved 10 feet up the wall, so that we can
walk all the way around the computer shelving with no obstructions on
the floor, since the wiring is now overhead.
The power panel breaker box for the Beowulf system. The large black
box hanging off the bottom right hand corner of the panel box is a surge
protector that protects all the circuits within the box. As shown there
are 3 breakers on the left (60 amps each) for the three-phase power
coming into the box. On the right are 6 20-amp breakers for the
computer power-strips. Eventually we will add 2 additional 20-amp
circuits so that we will have a total of eight, one for each of the
eight power strips. In our experiments, we were able to put 10
computers on a single 20-amp circuit without triggering the circuit
breaker, so powering only 6 machines per 20-amp circuit should be
plenty of margin. [Actual measurements of the current consumption
show a total of about 60 amps for all 48 machines, so about 1.2 amps
or 145 watts/machine. Thus eight 20-amp circuits give us about a 2.5
safety margin.] Heat dissipation in the room may be a problem
in the future -- there are about 7200 Watts of heat (about a 2-ton air
conditioning load) to dissipate. Fortunately the HVAC people were
able to crank up the supply of chilled air to the room to ~ 500 CFM,
and this keeps the temperature in the mid 70s. We'll probably add
cheap temperature-logging to the master at some point and log
the temperature one per minute at a few different locations in the room).
Wensheng Hua at back of the equipment rack.
R. Balasubramanian taking a break in front of the machine rack.
Bala seated at the terminal in the machine room. Notice the switch to
the right which allows the monitor/keyboard/mouse to be connected to two
different computers. Position # 1 is permanently connected to the master
(node 1) machine, and Position # 2 is connected to a set of three long
cables which can be plugged into any machine on the rack. It turned out
that this particular switch was a piece of junk (one could not properly
boot a machine connected to it: the AlphaBIOS keyboard check fails)
and has been replaced with a Belkin one that does allow proper booting.
This shows the cabling for the "Position # 2" option coming down
the back of the rack. The cables are long enough to plug into any
machine.
The cabling for the "Position # 2" option.