- Published on Monday, 21 May 2007 03:54
- Written by Jeff Layton
- Hits: 6439
Being the good cluster-er that he is, Jim promptly
to the list with some good comments. His opening comments were,
Sure it's possible. Your problems are going to be power, cooling, and structures (assuming you're not in an environment where people care about electrical codes, RF interference, etc.)
He then went to explain each of these in a little detail with some warnings (e.g. watch for grounding loops).
Jim's comments led Robert to kick into "Watch Out There" mode and
some warnings about creating dangerous conditions (he didn't mention
anything about running with scissors though). But he did make one
comment that I've seen in the past several times,
Yeah, I think that Jim's observation that you should think carefully about the diminishing returns of building a free-form caseless cluster is very apropos -- you'll save a bit of money on space and cases -- maybe -- at the expense of more hands on work building the cluster and at the risk of having to resolve problems with shielding and so on.
He did offer some very good suggestions though. He ended with this comment.
If you go anywhere beyond this, I'd REALLY recommend that you only proceed if you completely understand electricity and electrical wiring and know what a ground loop IS and so on.
A fitting comment if I've ever read one. This should be posted in every electrical section in Home Depot and Lowe's. But then again the Darwin Awards just wouldn't be the same.
On a more serious note, Josip Loncaric wrote that it's possible to find cheap cases for about $20 and these can save you some work, but not necessarily shelf space.
these comments by explaining the lure of inexpensive hardware to make
your cluster. He gave an interesting example of a 1,500 CPU cluster
where you allow some money for the CPUs, motherboard, chassis, power
supply, memory, and found that the sum was about 20% the cost of the
real thing. (this is tempting isn't it?) He did mention that Google
doesn't use cases. Rather they have bare motherboards on trays, perhaps
much like Fernando wants to do. Mark finished his comments
with the following.
In summary, subtracting the chassis sounds smart, but really only makes sense if you follow through with the rest - cheap motherboard, cheap cpu, minimal cpu, minimal network, cheap labor, workload that is embarrassingly parallel, and not long-running...
In short, you get what you pay for. (I've been burned on cheap memory several times in the past - never again).
H. Vidal then made a quick
that I think is interesting.
What's remarkable to consider is that one of the very largest (if not the largest?) data cluster systems in the world is a bare motherboard system, strapped together with lots of simple screws and Velcro.
That's Google, in case you did not know. I was shocked to see this when I saw a presentation recently by one of the Google guns here in NYC (actually, the inventor of Froogle). He showed us pix of a bunch of nodes essentially sitting on some insulating material, screwed to a simple frame-style chassis with careful consideration of grounding and power. His point was to emphasize that google considers lots of very cheap, very simple nodes key to their growth, and cases are 'right out' when you go to this scale (he would not share the exact N of nodes with us, but alluded to something on the order of 100K, at that time, and this is *always* growing).
I had heard about this in 2005. I think it's fairly common knowledge now. But it's still very interesting.
After a brief discussion about Google, Jim Lux
back with some interesting back of the envelope calculations. He
was interested in the amount of time it takes to drill holes in a
piece of sheet metal or aluminum as a base plate for a motherboard.
Assuming that you could do about 12 plates at the same time, he
ended up with an estimate that it takes about 30 minutes to
drill and screw in a single motherboard. If you guess about
$10/hour in labor costs plus the price of materials, and that
cheap $20 case looks pretty attractive. Jim then finished with a
true "Tool Time" suggestion.
There IS a faster way, for a bare system approach. Use double sided sticky foam tape. Plenty strong, it will last 2 or 3 years.
Then Doug Eadline
in and strongly recommended using regular case. He mentioned our
Kronos Project to build the fastest system we could for $2,500. At that
time, we found a well engineered small for about $40. Now you can
find them for $30 or less. Doug followed up these comments with
a slightly philosophical comment.
One of things I have learned when building clusters is to take advantage of mass produced anything (mostly hardware). Looking inside a diskless node, I often get the urge to build a better enclosure, but then realize that the cost and time to fuss around with everything is not worth it. As a hobby, sure, it might be fun, but my interest is software, a "good enough" solution that costs much less in both time and money always seems to win the day for me. YMMV
I think this is well said (although I can think of situations where a custom case is warranted). But, the siren song of commodity pricing is very hard to resist.
Right after Doug, Andrew Piskorski wrote that his favorite custom packaging scheme du jour was cookie sheets! He just uses basic cookie sheets and mounts the systems to the sheets. He said that there are ready made racks for these sheets. It's a long posting with lots of details, but he talked about how many micro-ATX motherboards he could get in a single rack (up to about 78) and how the density was more efficient than using standard cases. Since my wife's family is in the doughnut business, I think this is a great idea! This is really taking advantage of commodity components.
These types of discussions are always fun. You get to see the creative side of various people come out and the contrasts are always fun as well (I still love the cookie sheet idea). Some of the ideas are worthwhile I think, but in many cases, it may be more effective to just use micro-ATX cases with micro-ATX boards.
Dr. Jeff Layton hopes to someday have a 20 TB file system in his home computer (donations gladly accepted) so he can store all of the postings to all of the mailing lists he monitors. He can sometimes be found lounging at a nearby Fry's, dreaming of hardware and drinking coffee (but never during working hours).
- << Prev Page
- - Next Page