Getting Serious: Cluster Infrastructure

Published on Tuesday, 16 May 2006 04:45
Written by Robert G. Brown
Hits: 12135

Building Serious Clusters or Your Cluster Needs a Happy Home

Remember these words; location, location, location. Clusters need space, power, cooling, and network access. So far, this column has focused on getting you started on cluster computing, where by "started" I mean started at the very beginning with the basic ideas of parallel computing. If you've been reading along, you'll recall that we began by describing the very simplest of cluster designs, the "I already have a Network Of Workstations, presto-change-o and it's a NOW cluster" design. We then put this simple cluster to work with a threaded Perl task distribution script, learned about Amdahl's Law, and most recently how learned to write a "real" parallel application using PVM.

Hopefully you are convinced at this point that clusters can be put to work for you to speed up your parallelizable high performance computational task. You're ready to write your grant proposal, to go to the board for funds, to pull money out of your own darned pocket if necessary but one way or another, build yourself a spanking new cluster and put it to work. The only problem is that you don't quite know what you need. What does one need to think about and plan for to build a cluster of any given scale, anyway?

The next few columns will be for you. In this one, we will examine cluster infrastructure -- once one departs from the NOW model clusters need a place to live, power (possibly quite a bit of power), cooling, (possibly quite a bit of cooling), controlled access and security, and hot and cold running networks. A happy cluster room will also have certain amenities -- a jacket (it gets COLD in there, or should), a work bench, a refrigerator well-stocked with beer and jolt cola. You may not get all of these, but it can't hurt to try.

Housing Your Nodes

Let us suppose that you are preparing to build a real cluster. You have to make several decisions right away. The first one is its size. How many nodes will your new cluster have? If the answer is four, your cluster can go on your desk and you can skip the rest of this article. However, answers like 16, 64, 1024 require careful consideration of how you build your cluster and where you build your cluster. (Don't ask me why, but cluster builders tend to think in powers of two. Imagine that.)

If your cluster is going to be small and will remain small (where small is less than or equal to, say, 32 nodes), it can be built out of mid-tower systems stacked up on ordinary steel shelving from your friendly local hardware store. One good sized heavy duty shelf will typically hold roughly 16 nodes (up to 32 processors in a dual CPU configuration), so a couple or three units will hold your cluster. Its "footprint" (occupied floor space per shelf) will be something like two meters square, allowing for room to get around the shelf and access to the front and back. Shelf mounted clusters are the cheapest you can build from the hardware point of view and very common for starter/trainer clusters, hobbyist clusters, small special purpose clusters. They do waste space, though, and at some scale space becomes more expensive than hardware. {mosgoogle right}

If your cluster is going to be larger than eight nodes but with no clear upper bound on size in sight -- 16 nodes this year, but maybe 32 more next year and who knows beyond that -- then you should almost certainly consider rack mounted nodes. Rack mounted nodes cost a bit more, and racks cost more than cheap shelving, but consider their footprint: A rack is a bit over 20" wide including the base and supports, and as deep as the node cases you buy, typically around 30". You can put TWO racks side by side in two square meters. with access to the front and back and room to walk around.

A rack is measured in "U", where a U is 1.75". Racks come in various heights with 40 to 45U being typical for a cluster environment. Nowadays it is easy to buy dual CPU nodes in 1U cases. One rack can therefore hold 32 to 40 nodes, 64 to 80 CPUs (depending on how much room you reserve for patch panels, network switches, power regulation devices). Two together therefore can hold as many as 160 CPUs in two meters square, where shelving holds only 32.

There is a third alternative for people with severe space and power/cooling requirements: Bladed clusters. Blades are basically single board computers that snap into a common rack mounted backplane that might be 6-8U in height. Blades permit you to double or quadruple CPU density relative to ordinary rack mounts, but often do so at the expense of CPU speed and are always considerably more expensive per CPU.

Physical Space

Once you've decided how you're going to house and shelve your nodes, you need a place to put them. This decision is not as simple as it sounds; the space needs to have certain properties and to provide certain resources. Here is a short summary:

Power

Electrical power is a cluster's natural food, and they need a lot of it. In fact, they need more of it than you might expect or estimate, so it is wise to have a solid surplus -- more circuits than you will likely need. Providing power to nodes is also not as simple or obvious as it might appear -- there is more to good electrical power in a cluster room than just having a bunch of circuits.

It is beyond the scope of this column to teach you everything you need to know about electrical wiring to ensure that your cluster is well-fed and happy. However, I can direct you at various useful resources where you can learn more. Let me start with two or three bits of very basic advice:

These rules are just because faulty wiring can kill people and start fires. You can also HOPE that your professionals know enough not to make horrible mistakes in wiring a computer room.

However, from my own direct experience, this is often not the case. Wiring a computer room differs from ordinary commercial or residential wiring in a number of important ways. Let's review some of those differences, so you can "keep an eye on" the folks who design and install your wiring to make sure that they do NOT "work you over" by installing wiring in ways that will prove to be inadequate -- and then charge you again to come in and make it right.

Note that you may not understand all of the rules below, but your electrical contractor should.

There are guidelines for wiring computer spaces available online, as you might expect, but they aren't horribly easy to find. See the Resources sidebar for the links. Remember, you are your own best advocate here. The beowulf mailing list abounds with stories of people who trusted their electricians to do a proper job of wiring a new computer space, only to learn the hard way that the electricians did it wrong.

Cooling

Computers "like" to breath in air for cooling that is less than 20C (68F). In fact, they really like to breath in ambient air for cooling that is less than 15C/60F, and don't mind if the air is down as cool as 50F/10C as long as it is fairly dry so condensation isn't a problem.

Power used by the cluster is released into the cluster room as heat. All of it. All of the time. The amount of heat can be quite staggering. A loaded rack with 80 CPUs can easily draw 8000 to 10000 watts. That is like running eight to ten room-capacity space heaters all at once in a single room -- the full power capacity of five to seven 20 ampere circuits (probably spread out on eight or more circuits).

If you release this heat into a small room with well-insulated walls, the temperature in that room will go up -- fast! If you release this heat into a large room with poorly insulated walls, the temperature of that room will still go up fast (just not as fast). To prevent the temperature in the room from zipping right on past the "happy zone" below 68F/20C into the "about to cause several hundred thousand dollars worth of damage zone" around 104F/40C (which might take as little as ten or twenty minutes depending on the room size and just how much power is going into it) one has to remove all of power you send in as fast as the computers spit it out as heat.

This task requires air conditioning and cooling capacity -- lots of it. The capacity computation is just like your power computation. You need enough air conditioning capacity to remove all of your power capacity, with a fairly hefty margin of surplus capacity so you can not only keep up with the heat, you can get ahead and keep the space cool.

This isn't all, however. Air conditioning is again a complex issue. You don't want air cold enough to create a block of ice in one end of the room and a pocket of trapped air in the other end of the room (where all of the systems are) hot enough to roast an egg or fry a system. This condition can easily happen if your cold air distribution and warm air returns aren't sensibly arranged. You also don't want your racks arranged so warm exhaust air from one row is used as intake cooling air by the second row and so forth -- each computer cools its CPUs relative to ambient air at the intake side, and feeding it already warm air can cause it to overheat.

This situation again suggests that you have your cluster room air conditioning designed and installed professionally, with special care taken to ensure adequate capacity and proper delivery of cool air where you need it and removal of warm air where you don't. This requirement may well involve extensive ducting either overhead, around the walls, or up through a raised floor if you go that way.

A few warnings: Be sure to guard against condensation -- in a humid environment, the room and ductwork will be cold enough to drip water down into expensive machines if the ductwork isn't appropriately wrapped. Also, having some sort of thermal monitoring and alarm system is very definitely called for. You don't want to discover that your A/C failed by discovering a room full of expensive baked hardware the next day (see also the suggestion for a thermal kill switch on your power above). Finally, beware the tendency in many institutions to turn off chillers and air conditioners in the wintertime. After all, it's cold outside!

Not in the cluster room. In there it is hot hot hot, and if the chiller goes off it will fry in no time, in the middle of winter. Be sure to make arrangements for the cluster room air conditioning system to be online 24x7 all year long!

Conclusion

This initial article is hardly exhaustive. There are lots of elements of cluster room design that we only touched upon, such as locks and security, monitoring, amenities, wiring trays and so forth. There are also style issues that we've avoided -- why it is a good idea to keep things neat and tidy, with cables tied up and in suitable channels, even if it costs a bit more that way. Hopefully we'll address some of these issues in future columns when we get down into the nitty-gritty of building serious clusters.

Hope to see you there! {mosgoogle right}

Sidebar: Resources

Computer Room Wiring

Three Phase Power Source Overloading

ITIC Grounding Guidelines (pdf)

Mirus International: Harmonics and Harmonic Mitigating Transformers

The Mirus International FAQ is very informative on the subject of harmonic distortion by switching power supplies and power for computer rooms in general.

This article was originally published in ClusterWorld Magazine. It has been updated and formatted for the web. If you want to read more about HPC clusters and Linux, you may wish to visit Linux Magazine.

Robert Brown, Ph.D, is has written extensively about Linux clusters. You can find his work and much more on his home page

Unfortunately you have Javascript disabled, please enable Javascript in order to experience the comments correctly