Being the good cluster-er that he is, Jim promptly
wrote
to the list with some good comments. His opening comments were,
Sure it's possible.
Your problems are going to be power, cooling, and structures (assuming
you're not in an environment where people care about electrical codes, RF
interference, etc.)
He then went to explain each of these in a little detail with some
warnings (e.g. watch for grounding loops).
Jim's comments led Robert to kick into "Watch Out There" mode and
offer
some warnings about creating dangerous conditions (he didn't mention
anything about running with scissors though). But he did make one
comment that I've seen in the past several times,
Yeah, I think that Jim's observation that you should think carefully
about the diminishing returns of building a free-form caseless cluster is
very apropos -- you'll save a bit of money on space and cases -- maybe
-- at the expense of more hands on work building the cluster and at the
risk of having to resolve problems with shielding and so on.
He did offer some very good suggestions though. He ended with this
comment.
If you go anywhere beyond this, I'd
REALLY recommend that you only proceed if you completely understand
electricity and electrical wiring and know what a ground loop IS and so
on.
A fitting comment if I've ever read one. This should be posted in
every electrical section in Home Depot and Lowe's. But then again
the
Darwin Awards just
wouldn't be the same.
On a more serious note, Josip Loncaric wrote that it's possible to find cheap cases for about $20 and these can save you some work, but not necessarily shelf space.
Marh Hann
echoed
these comments by explaining the lure of inexpensive hardware to make
your cluster. He gave an interesting example of a 1,500 CPU cluster
where you allow some money for the CPUs, motherboard, chassis, power
supply, memory, and found that the sum was about 20% the cost of the
real thing. (this is tempting isn't it?) He did mention that Google
doesn't use cases. Rather they have bare motherboards on trays, perhaps
much like Fernando wants to do. Mark finished his comments
with the following.
In summary, subtracting the chassis sounds smart, but really only makes
sense if you follow through with the rest - cheap motherboard, cheap cpu,
minimal cpu, minimal network, cheap labor, workload that is embarrassingly
parallel, and not long-running...
In short, you get what you pay for. (I've been burned on cheap memory
several times in the past - never again).
H. Vidal then made a quick
post
that I think is interesting.
What's remarkable to consider is that one of the very largest
(if not the largest?) data cluster systems in the world is a bare
motherboard system, strapped together with lots of simple
screws and Velcro.
That's Google, in case you did not know. I was shocked to see
this when I saw a presentation recently by one of the Google
guns here in NYC (actually, the inventor of Froogle).
He showed us pix of a bunch of nodes essentially
sitting on some insulating material, screwed to a simple
frame-style chassis with careful consideration of grounding
and power. His point was to emphasize that google considers
lots of very cheap, very simple nodes key to their growth, and cases
are 'right out' when you go to this scale (he would not share the
exact N of nodes with us, but alluded to something on the order of
100K, at that time, and this is *always* growing).
I had heard about this in 2005. I think it's fairly common
knowledge now. But it's still very interesting.
After a brief discussion about Google, Jim Lux
came
back with some interesting back of the envelope calculations. He
was interested in the amount of time it takes to drill holes in a
piece of sheet metal or aluminum as a base plate for a motherboard.
Assuming that you could do about 12 plates at the same time, he
ended up with an estimate that it takes about 30 minutes to
drill and screw in a single motherboard. If you guess about
$10/hour in labor costs plus the price of materials, and that
cheap $20 case looks pretty attractive. Jim then finished with a
true "Tool Time" suggestion.
There IS a faster way, for a bare system approach. Use double sided sticky
foam tape. Plenty strong, it will last 2 or 3 years.
Then Doug Eadline
weighed
in and strongly recommended using regular case. He mentioned our
Kronos Project to build the fastest system we could for $2,500. At that
time, we found a well engineered small for about $40. Now you can
find them for $30 or less. Doug followed up these comments with
a slightly philosophical comment.
One of things I have learned when building clusters is to take advantage
of mass produced anything (mostly hardware). Looking inside a diskless
node, I often get the urge to build a better enclosure, but then realize
that the cost and time to fuss around with everything is not worth it. As
a hobby, sure, it might be fun, but my interest is software, a "good
enough" solution that costs much less in both time and money always seems
to win the day for me. YMMV
I think this is well said (although I can think of situations
where a custom case is warranted). But, the siren song of commodity
pricing is very hard to resist.
Right after Doug, Andrew Piskorski wrote that his favorite custom packaging scheme du jour was cookie sheets! He just uses basic cookie sheets and mounts the systems to the sheets. He said that there are ready made racks for these sheets. It's a long posting with lots of details, but he talked about how many micro-ATX motherboards he could get in a single rack (up to about 78) and how the density was more efficient than using standard cases. Since my wife's family is in the doughnut business, I think this is a great idea! This is really taking advantage of commodity components.
By the way, Andrew posted some links to bare bones motherboard systems. All of the links are still active. My favorite is the zBox.
These types of discussions are always fun. You get to see the creative side of various people come out and the contrasts are always fun as well (I still love the cookie sheet idea). Some of the ideas are worthwhile I think, but in many cases, it may be more effective to just use micro-ATX cases with micro-ATX boards.
Dr. Jeff Layton hopes to someday have a 20 TB file system in his home computer (donations gladly accepted) so he can store all of the postings to all of the mailing lists he monitors. He can sometimes be found lounging at a nearby Fry's, dreaming of hardware and drinking coffee (but never during working hours).