Building Serious Clusters or Your Cluster Needs a Happy Home

Remember these words; location, location, location. Clusters need space, power, cooling, and network access. So far, this column has focused on getting you started on cluster computing, where by "started" I mean started at the very beginning with the basic ideas of parallel computing. If you've been reading along, you'll recall that we began by describing the very simplest of cluster designs, the "I already have a Network Of Workstations, presto-change-o and it's a NOW cluster" design. We then put this simple cluster to work with a threaded Perl task distribution script, learned about Amdahl's Law, and most recently how learned to write a "real" parallel application using PVM.

Hopefully you are convinced at this point that clusters can be put to work for you to speed up your parallelizable high performance computational task. You're ready to write your grant proposal, to go to the board for funds, to pull money out of your own darned pocket if necessary but one way or another, build yourself a spanking new cluster and put it to work. The only problem is that you don't quite know what you need. What does one need to think about and plan for to build a cluster of any given scale, anyway?

The next few columns will be for you. In this one, we will examine cluster infrastructure -- once one departs from the NOW model clusters need a place to live, power (possibly quite a bit of power), cooling, (possibly quite a bit of cooling), controlled access and security, and hot and cold running networks. A happy cluster room will also have certain amenities -- a jacket (it gets COLD in there, or should), a work bench, a refrigerator well-stocked with beer and jolt cola. You may not get all of these, but it can't hurt to try.

Housing Your Nodes

Let us suppose that you are preparing to build a real cluster. You have to make several decisions right away. The first one is its size. How many nodes will your new cluster have? If the answer is four, your cluster can go on your desk and you can skip the rest of this article. However, answers like 16, 64, 1024 require careful consideration of how you build your cluster and where you build your cluster. (Don't ask me why, but cluster builders tend to think in powers of two. Imagine that.)

If your cluster is going to be small and will remain small (where small is less than or equal to, say, 32 nodes), it can be built out of mid-tower systems stacked up on ordinary steel shelving from your friendly local hardware store. One good sized heavy duty shelf will typically hold roughly 16 nodes (up to 32 processors in a dual CPU configuration), so a couple or three units will hold your cluster. Its "footprint" (occupied floor space per shelf) will be something like two meters square, allowing for room to get around the shelf and access to the front and back. Shelf mounted clusters are the cheapest you can build from the hardware point of view and very common for starter/trainer clusters, hobbyist clusters, small special purpose clusters. They do waste space, though, and at some scale space becomes more expensive than hardware.

If your cluster is going to be larger than eight nodes but with no clear upper bound on size in sight -- 16 nodes this year, but maybe 32 more next year and who knows beyond that -- then you should almost certainly consider rack mounted nodes. Rack mounted nodes cost a bit more, and racks cost more than cheap shelving, but consider their footprint: A rack is a bit over 20" wide including the base and supports, and as deep as the node cases you buy, typically around 30". You can put TWO racks side by side in two square meters. with access to the front and back and room to walk around.

A rack is measured in "U", where a U is 1.75". Racks come in various heights with 40 to 45U being typical for a cluster environment. Nowadays it is easy to buy dual CPU nodes in 1U cases. One rack can therefore hold 32 to 40 nodes, 64 to 80 CPUs (depending on how much room you reserve for patch panels, network switches, power regulation devices). Two together therefore can hold as many as 160 CPUs in two meters square, where shelving holds only 32.

There is a third alternative for people with severe space and power/cooling requirements: Bladed clusters. Blades are basically single board computers that snap into a common rack mounted backplane that might be 6-8U in height. Blades permit you to double or quadruple CPU density relative to ordinary rack mounts, but often do so at the expense of CPU speed and are always considerably more expensive per CPU.

Physical Space

Once you've decided how you're going to house and shelve your nodes, you need a place to put them. This decision is not as simple as it sounds; the space needs to have certain properties and to provide certain resources. Here is a short summary:

• Space. You need to be able to physically locate all your shelves or racks in such a way that you can easily access them to add new or remove broken nodes for service, get to the back to cable them, get to the front to turn them on. Plan on at least a meter between shelves or racks front to back. You also need to worry about delivery of cool air to the fronts and removal of warm air from the rears (see section on cooling).
• Floor support. A loaded rack can weigh as much as a half a ton, all concentrated on less than a square meter of floor space. Nothing can ruin your day like having a half-ton of very expensive equipment break through a floor and fall onto the head of a colleague in the office below. Ruin your life, even. Get a professional architect or structural engineer to inspect your proposed space if there is ANY DOUBT about the ability of the floor to sustain the weight of the proposed cluster.
• Raised floor vs solid. This option is a related question. Some server/cluster rooms are built with raised floors so that all cabling and cooling can be delivered from underneath. This is good in that it results in a very neat and tidy room; it is bad in that it is a pain to GET to all the cabling and cooling for modification or maintenance and raised floors need to be carefully built to be structurally sound. Two post racks generally require solid floors as the racks have to be affixed to the floor with bolts to withstand a certain amount of torque. I use solid floors (and cheaper two post racks to some extent) but am neutral on the issue.
• Power. I could write a book on power alone. Or give it its own section, below. The short answer (for now) is that one needs very roughly 100 watts of power per CPU with most modern node designs, and should wire for twice this at least to accommodate future power needs and a degree of spare capacity.
• Cooling. Again, worthy of its own section. Every watt that goes into your space has to go out again, or the temperature goes up. Computers HATE being hot -- they break when they get hot. In a cluster they can break a hundred thousand dollars' worth at a time.
• Network. A cluster REQUIRES networking to function. Every node will therefore require at least an Ethernet connection (switched 100BT or 1000BT) and high end clusters might require fiber or specialty cabling for high end networks such as Myrinet or Serial Channel Interconnect (SCI). Wiring trays, patch panels, room for switching devices all need to be accommodated in your plan.
• Security and Access. A cluster can cost quite a lot of money and is composed of components that are attractive to thieves. The cluster room therefore needs to be physically secured with locks and possibly alarms and monitors. However, a cluster also breaks and needs maintenance, so it has to be accessible to administrators and possibly users.
• Physical Environment. A cluster room often needs monitoring facilities that can keep an eye on things like sound levels, temperature and humidity in an automated fashion. If cooling fails, a densely packed cluster room can get hot enough to cause massive system failures rather quickly.
• Amenities. Things that you very likely want to have handy in your cluster room include: Storage for spare parts, systems, tools. A well-equipped work bench where nodes can be assembled and disassembled for repair. Extra cabling of all sorts. A jacket (your cluster room SHOULD be too cold for comfort). Headphones and a source of music both for diversion and to keep the roar of the air conditioner out of your ears. I was just kidding about the refrigerator and beer. Wistfully kidding, but kidding. [Raised floor panel AC vents make great beverage chillers. -- Ed.]

Power

Electrical power is a cluster's natural food, and they need a lot of it. In fact, they need more of it than you might expect or estimate, so it is wise to have a solid surplus -- more circuits than you will likely need. Providing power to nodes is also not as simple or obvious as it might appear -- there is more to good electrical power in a cluster room than just having a bunch of circuits.

It is beyond the scope of this column to teach you everything you need to know about electrical wiring to ensure that your cluster is well-fed and happy. However, I can direct you at various useful resources where you can learn more. Let me start with two or three bits of very basic advice:

• The room should be wired by professional, certified electricians, ideally ones with direct experience in wiring computer rooms.
• The room wiring layout should be designed by a professional e.g. architect with experience in computer rooms as well.
• The wiring should be done according to all applicable codes.

These rules are just because faulty wiring can kill people and start fires. You can also HOPE that your professionals know enough not to make horrible mistakes in wiring a computer room.

However, from my own direct experience, this is often not the case. Wiring a computer room differs from ordinary commercial or residential wiring in a number of important ways. Let's review some of those differences, so you can "keep an eye on" the folks who design and install your wiring to make sure that they do NOT "work you over" by installing wiring in ways that will prove to be inadequate -- and then charge you again to come in and make it right.

Note that you may not understand all of the rules below, but your electrical contractor should.

• Keep all wiring runs back to the distribution panel short, and be sure that the circuits have properly arranged ground and neutral connections. Computers do not like ground loops (significant voltages between the neutral line and local ground). One of the references below is devoted just to the correct rules for grounding circuits used in information technology applications.
• When wiring power poles with multiple phases, do not share a neutral wire. Each circuit should have its own neutral wire. This requirement is needed because switching power supplies only draw current in the middle third of each half-phase, so that currents on a shared neutral wire of three balanced phases add instead of cancel. The power factor of switching power supplies is not particularly close to unity and they generate significant harmonics at multiples of the basic line frequency, e.g. 180 Hz
• Similarly be aware that peak currents will be higher than expected based on wattage of the load alone, resulting in more neutral wire heating and lowered nominal circuit capacity. Design for a healthy margin of capacity on each circuit.
• Consider installing a thermal cut-off so that if the room temperature ever exceeds (say) 90F, all power to the nodes shuts down. Better shut down than burned out.
• Consider using a Harmonic Distortion Correcting room transformer, especially for new construction. Such a transformer protects your main building supply transformer and dynamically corrects for the distorting properties of switching power supplies. They can save money in several ways (by protecting equipment and making equipment run more efficiently).
• Depending on your budget and the cost of downtime, consider further power conditioning, surge protection, and uninterruptible power. Be sure uninterruptible power is wired according to codes and marked as such so that fire department personnel can readily shut it down in the event of a fire.

There are guidelines for wiring computer spaces available online, as you might expect, but they aren't horribly easy to find. See the Resources sidebar for the links. Remember, you are your own best advocate here. The beowulf mailing list abounds with stories of people who trusted their electricians to do a proper job of wiring a new computer space, only to learn the hard way that the electricians did it wrong.

Cooling

Computers "like" to breath in air for cooling that is less than 20C (68F). In fact, they really like to breath in ambient air for cooling that is less than 15C/60F, and don't mind if the air is down as cool as 50F/10C as long as it is fairly dry so condensation isn't a problem.

Power used by the cluster is released into the cluster room as heat. All of it. All of the time. The amount of heat can be quite staggering. A loaded rack with 80 CPUs can easily draw 8000 to 10000 watts. That is like running eight to ten room-capacity space heaters all at once in a single room -- the full power capacity of five to seven 20 ampere circuits (probably spread out on eight or more circuits).

If you release this heat into a small room with well-insulated walls, the temperature in that room will go up -- fast! If you release this heat into a large room with poorly insulated walls, the temperature of that room will still go up fast (just not as fast). To prevent the temperature in the room from zipping right on past the "happy zone" below 68F/20C into the "about to cause several hundred thousand dollars worth of damage zone" around 104F/40C (which might take as little as ten or twenty minutes depending on the room size and just how much power is going into it) one has to remove all of power you send in as fast as the computers spit it out as heat.

This task requires air conditioning and cooling capacity -- lots of it. The capacity computation is just like your power computation. You need enough air conditioning capacity to remove all of your power capacity, with a fairly hefty margin of surplus capacity so you can not only keep up with the heat, you can get ahead and keep the space cool.

This isn't all, however. Air conditioning is again a complex issue. You don't want air cold enough to create a block of ice in one end of the room and a pocket of trapped air in the other end of the room (where all of the systems are) hot enough to roast an egg or fry a system. This condition can easily happen if your cold air distribution and warm air returns aren't sensibly arranged. You also don't want your racks arranged so warm exhaust air from one row is used as intake cooling air by the second row and so forth -- each computer cools its CPUs relative to ambient air at the intake side, and feeding it already warm air can cause it to overheat.

This situation again suggests that you have your cluster room air conditioning designed and installed professionally, with special care taken to ensure adequate capacity and proper delivery of cool air where you need it and removal of warm air where you don't. This requirement may well involve extensive ducting either overhead, around the walls, or up through a raised floor if you go that way.

A few warnings: Be sure to guard against condensation -- in a humid environment, the room and ductwork will be cold enough to drip water down into expensive machines if the ductwork isn't appropriately wrapped. Also, having some sort of thermal monitoring and alarm system is very definitely called for. You don't want to discover that your A/C failed by discovering a room full of expensive baked hardware the next day (see also the suggestion for a thermal kill switch on your power above). Finally, beware the tendency in many institutions to turn off chillers and air conditioners in the wintertime. After all, it's cold outside!

Not in the cluster room. In there it is hot hot hot, and if the chiller goes off it will fry in no time, in the middle of winter. Be sure to make arrangements for the cluster room air conditioning system to be online 24x7 all year long!

Conclusion

This initial article is hardly exhaustive. There are lots of elements of cluster room design that we only touched upon, such as locks and security, monitoring, amenities, wiring trays and so forth. There are also style issues that we've avoided -- why it is a good idea to keep things neat and tidy, with cables tied up and in suitable channels, even if it costs a bit more that way. Hopefully we'll address some of these issues in future columns when we get down into the nitty-gritty of building serious clusters.

Hope to see you there!

Sidebar: Resources

Computer Room Wiring Three Phase Power Source Overloading ITIC Grounding Guidelines (pdf) Mirus International: Harmonics and Harmonic Mitigating Transformers The Mirus International FAQ is very informative on the subject of harmonic distortion by switching power supplies and power for computer rooms in general.

This article was originally published in ClusterWorld Magazine. It has been updated and formatted for the web. If you want to read more about HPC clusters and Linux, you may wish to visit Linux Magazine.

Robert Brown, Ph.D, is has written extensively about Linux clusters. You can find his work and much more on his home page