Jeff takes a look at what type of cluster $2500 will buy with today's technology. Back in 2005, 2.5x10E5 pennies bought eight Semprons and 14.5 GFLOPS. How many GFLOPS has two years brought us?
It's always fun to play what-if. What if I looked like Brad Pitt? What if I had lots of money? What if I lost a bunch of weight? What if I looked like Brad Pitt, lost of a bunch of weight, and had lots of money? What if I could spend that money on a new home cluster and build it myself? While worthy of an episode of Jerry Springer I think the last what-if is more appropriate for Cluster Monkey. [Men who spend their kids lunch money on Linux clusters on the next Jerry Springer Show - Ed.]
If you recall, our protagonists, Jeff and Doug, had built an 8 node cluster named Kronos for about $2,500. That was almost 18 months ago. What could we do now with $2,500? While we haven't built anything at this time, let's have a little fun and take a look at what we could build. (and what we might be building real soon now).
What could I do today?
A while ago (about 18 months), Doug Eadline and I wrote a series of articles about building a simple cluster for about $2,500. We chose to name the cluster Kronos for no real particular reason, but basically because we like old Sci-Fi movies that the younger generation can't appreciate. The first article talked about the "parts" list and the construction of the cluster. The second article talked about the software for the cluster. The third article talked about installing some MPI libraries and a job scheduler (SGE). In the final article we talked about some optimization we did to improve our HPL scores. Then Doug followed up with an article about trying to further improve the HPL scores.
The cluster, while a great source of cluster horsepower for us, is getting a bit long in tooth. Now we have dual-core and quad-core CPUs, memory has dropped in price, GigE has dropped in price, and the software is more mature. So I thought I would take a few moments and look at what we could do if we chose to build "Son of Kronos." While I don't have the funds to build such a beast, I think it's worthwhile to see what a good home built cluster would cost these days. (I do have a proposal in to the house management. It is still under consideration.)
Before I launch into showing you the prices let me start with some guidelines that I used in the pricing.
- I'm using AMD Athlon 64 X2 processors. Why? No particular reason. I'm sure I'll get lots of email saying "Intel Core 2 Duo is faster." While that may be true, I just chose to look at AMD for this article (and I have no desire to do a AMD vs. Intel pricing comparison. But if someone wants to fund me to build both systems and then run benchmarks on them I'll be glad to do that).
- I chose to use a disk-less booting system (Warewulf or Percus). Also, I decided not to put any disks in the compute nodes and put all of the storage in the master node using NFS.
- I decided to go with two GigE networks that are jumbo packet capable. One is for computational traffic and the second is for booting the nodes and for storage traffic.
- I wanted to have a decent amount of memory per core so I'm trying to get at least 1 GB of memory per core (2GB per node in the case of dual-core).
- I'm putting a couple of hard drives in the master node to provide storage for the cluster. You can run it as RAID-1 to get a little resiliency in the system or as RAID-0 to get some speed. It's up to you.
- To make my life easier, I chose to just use Newegg for the pricing. While I'm not advertising for them I have found them to have great prices, a wide selection, and good customer service.
- Lastly, the system will have at least one head node and one compute node. But I'll explore how many nodes I can get for various amounts of money.
So with these ground rules, let's start by looking at the head node.
Head Node
The table below lists the items I chose for my head node. I have included links to newegg for the items. The prices are current as of 1/25/2007 but are likely to change. Plus don't be surprised if some of the items disappear from the newegg site (welcome to the wild world of commodity parts!). In this initial configuration, I've chosen to use the 2.2 GHz AMD Athlon 64 X2 4200+ CPU. I thought it had a good price/performance so I decided to use it. Later on, I'll look at other CPUs and the affect on the total pricing.
| Head Node (one) | |
| Motherboard: ASRock ALiveNF6G-DVI | $73.12 |
| Processor: AMD Athlon 64 X2 4200+ | $169.00 |
| Memory: 2GB (2x1GB) DDR2-667 | $178.99 |
| OS Hard Drive: Seagate 7200.9 80 GB | $49.93 |
| Data HDD (two of these): Seagate 7200.10 320GB SATA 3.0 | $189.98 |
| Computation GigE NIC (2 of these): Intel PWLA8391GT PCI PRO/1000 GT | $34.98 |
| DVD (Dual-layer): LG 18X DVD+RW | $35.63 |
| Case: RAIDMAX Scorpio 668WP w/ 420W PS | $55.98 |
| Total: | $822.59 |
I looked at various options for the parts and selected them for various reasons. I chose a motherboard with built-in graphics so I didn't have to buy a separate graphics card. Plus the motherboard has a built-in GigE NIC that I can use for booting the nodes and for the storage network. I also added an Intel GigE NIC for the computational traffic (I plan on using GAMMA and it works best with Intel NICs). The second GigE NIC is to connect the head node to the outside world. I chose Seagate hard drives since I've had good luck with them and the performance of the new ones (7200.10) is particularly good. I also put in a DVD burner, because you are likely to need something to load the OS on the machine and you can always use to archive data from the head node. Also notice that I put the OS on a separate disk. I did this so it's easier to RAID the data hard drives.
The hardest decision I had to make was on the case. There were some cheaper cases, but they looked to be more difficult to work with. I chose a case that had at least one 120mm fan in the front and one in the back. My desktop case has 120mm fans in it and they are deadly quiet. Also, I wanted a case that had slots for hard drives that are sideways. That is you can pop off the side of the case and just slide the hard drives in without having to take off the front of the case. The case I chose is larger than the compute node case (in Kronos we used the same case for all of the nodes) and it doesn't look the same, but I value convenience, price, cooling, and the skin on my knuckles more than aesthetics.
Compute Node
Next I want to define the compute nodes. Remember I made these nodes disk-less so I don't need any hard drives in them. To make things easier I will use the same motherboard, CPU, and memory that I chose for the head node. I also put the same Intel NIC in this node for the computational network.
| Compute Node (one) | |
| Motherboard: ASRock ALiveNF6G-DVI | $73.12 |
| Processor: AMD Athlon 64 X2 4200+ | $169.00 |
| Memory: 2GB (2x1GB) DDR2-667 | $178.99 |
| Computation GigE NIC: Intel PWLA8391GT PCI PRO/1000 GT | $34.98 |
| Case: LinkWorld 617-C.88-P4 MicroATX Mini-Towerw/ 300W PS | $55.98 |
| Total: (per node) | $512.07 |
Notice that the case I chose is a Micro-ATX case. I chose to use a Micro-ATX motherboard since I don't need a lot of components on the motherboard and it saves a little bit of money. The case is pretty small: 15" x 14" x 5.3"(HxDxW). It's maybe a little bit bigger than the compute cases we used in Kronos, but that's fine.
Networking
| Networking/Cables | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Computational Switch: SMC SMCGS8 8-port jumbo packet capable switch | $62.32 |
| Booting/Storage Switch: SMC SMCGS8 8-port jumbo packet capable switch | $62.32 |
| Network Cables (estimated) | $32.00 |
| Total: | $156.64 |
I chose 8-port GigE switches for two reasons. They are not much more expensive than 5-port switches but they are much cheaper than 16-port switches. Also, they allow room to grow the system configurations to a total of 8 nodes.
Configurations
Now let's start looking at pricing for various configurations. Let's start with the basic of a single head node and a single compute node (not much fun there).
| Final Total for 2 Nodes (1 compute node + 1 head node) | |
| Head Node: | $822.59 |
| Compute Nodes (1): | $512.07 |
| Networking: | $156.64 |
| Grand Total: | $1,491.03 |
Just to recap, the system has 4 cores (if you count the head node), and a total of 4 GB of memory. This price doesn't look too swift though. It's about $364.08 per core. Plus we have all of these unused GigE ports hanging around. But the configuration does give you an idea of what you could do as a starting point.
Let's see what we can get for about $2,500 (the original price for Kronos).
| Final Total for 4 Nodes (3 compute nodes + 1 head node) | |
| Head Node: | $822.59 |
| Compute Nodes (1): | $1,526.21 |
| Networking: | $156.64 |
| Grand Total: | $2,505.44 |
So you for about $2,500, which is what we originally spent on Kronos, results in 8 cores and 8 GB of memory. So this is about $310.06 per core (recall that the 2 node system was about $364.08 per core). Here's a quick comparison of the this system versus Kronos:
- CPUs: Current: 8x2.2GHz Athlon 64 cores (64-bit), Kronos: AMD Sempron 2500+ (32-bit)
- Memory: Current: 8GB total (2GB per node), Kronos: 2.3 GB total (512MB head node, 256MB per compute node)
- Interconnect: Current: 2xGigE, Kronos: GigE, FastE
So you can now get a machine with 8x2.2GHz (17.6 GHz total) 64-bit cores with 512KB of L2 cache per core versus 8x1.7GHz (13.6 GHz total) 32-bit cores with 128KB of L2 cache per core. From another perspective you get almost four times as much memory and a better storage network.
Let's keep going and examine a 6 node system (1 head node and 5 compute nodes).
| Final Total for 6 Nodes (5 nodes + head node) | |
| Head Node: | $822.59 |
| Compute Nodes (5): | $2,560.35 |
| Networking: | $156.64 |
| Grand Total | $3,539.58 |
So for the configuration you get 12 cores and 12GB of memory. This works out to be about $292.05 per core.
Finally, let's go all the way and have 8 total nodes: 1 head node and 7 compute nodes. This will use all of the GigE ports on the system. So unless we upgrade the network switches, this is the largest machine in terms of the number of nodes.
| Final Total for 8 Nodes (7 nodes + head node) | |
| Head Node: | $822.59 |
| Compute Nodes (7): | $3,584.49 |
| Networking: | $156.64 |
| Grand Total | $4,563.72 |
So for under $4,500 you get 16 cores and 16GB of memory. Not bad, not bad at all. This works out to about $285.23 per core.
Dollars per Jigawatt
One of the fun measures that people like to look at is dollars per GFLOPS (Giga Floating Point Operations per second). I'm going to use the theoretical peak performance of the CPUs, since I don't have the machine built and can't test it :). I'm using 2.2GHz CPUs which have a theoretical peak performance of 4.4 GFLOPS per core. So let's do some math and determine the $/GFLOPS
| Number of Cores | Total GFLOPS |
Cost | $/GLOPS |
| (2 nodes) = 4 cores | 17.6 | $1,491.03 | $84.72 |
| (4 nodes) = 8 cores | 35.2 | $2,505.44 | $71.18 |
| (6 nodes) = 12 cores | 52.8 | $3.539.58 | $67.04 |
| (8 nodes) = 16 cores | 70.4 | $4,563.72 | $64.83 |
| Kronos = 8 cores | 28.10 | $2,490 | $88.61 |
There are several of ways to look at the results. For the same amount of money (about $2,500) we are getting about 35.2 GFLOPS (theorteical peak) instead of 28.1 GFLOPS. Also the $/GLOPS is now $71.18 instead of $88.61 (not a bad improvement in 18 months). For the larger systems, the $/GLOPS is even more exciting. We're fast approaching the $50/GFLOPS mark. Of course, this is still theoretical GFLOPS and more meaningful numbers would be for real HPL performance.
Optional Configurations
In this section I'll look at pricing with optional things such as more memory, faster CPUs, etc. Everyone love speed so I'll look at increasing the clock speed of the CPUs to 2.6 GHz. But there are also times when you need more memory to run larger jobs, so I'll look at those numbers as well. Finally, I'll look at using 16-port GigE switches to go even larger (don't let you significant other see that pile of PCs in the basement).
Faster CPUsLet's increase the CPU speed to 2.6GHz. The price of the CPU increases to $279.00. Here is the cost breakdown on an 8 node system.
| Final Total for 8 Nodes (7 nodes + head node) | |
| Head Node: | $932.59 |
| Compute Nodes (7): | $4,354.49 |
| Networking: | $156.64 |
| Grand Total | $5,443.72 |
This is an increase of $880. The theoretical performance increases to about 5.2 GFLOPS per core (83.2 GLOPS total). So the cost per GFLOPS is $65.87 which is a bit larger than that for 2.4 GHz cores. But still, for a bit over $5,000 you can get 16 cores running at 2.6 GHz!
More Memory Please!
In some cases codes need more memory per node or you may want to run more than one job at a time on the cluster. So, let's look at putting 4GB in each node. I'm using 2x2GB DIMMs that are DDR2-667. I'm also going to stick with 2.4 GHz CPUs to save a little bit of money.
| Final Total for 8 Nodes (7 nodes + head node) | |
| Head Node: | $1,078.58 |
| Compute Nodes (7): | $6,320.67 |
| Networking: | $156.64 |
| Grand Total | $7,555.89 |
Notice that the price for this configuration is much greater than 2GB per node. You have 32GB of total memory, but the cost has jumped to about $107.33 per GFLOPS.
Is that a 16-port Switch in Your Pocket?
There may be a few people out there who would like to make slightly larger clusters. But we're limited to a total of 8 nodes because of switch sizes. So I want to look at increasing the total size of the cluster to 16 nodes. For the cost table below I will use the node prices from the first section and then update the Network section.
| Final Total for 16 Nodes (15 nodes + head node) | |
| Head Node: | $822.59 |
| Compute Nodes (7): | $7,681.05 |
| Networking: | $371.98 |
| Grand Total | $8,875.62 |
Actually this cost isn't too bad for a total of 32 cores and 32 GB of memory. That's about $277.36 per core. This is a little bit better than the 8 node system. This also works out to $63.04 per GFLOPS which is pretty good (but close to the 8 node configuration).
What have we learned?
Besides being a fun little "what-if" exercise what have we learned? I will tell what I've learned. First, typing all of these silly HTML tables is a pain. But more importantly, I've learned that in 18 months technology has really changed. For $2,500 we have gone from eight 32-bit cores at 1.7 GHz to eight 64-bit cores at 2.4 GHz, from 2.3GB of total memory to 8 GB of total memory, from one GigE network to two, and we doubled our disk space. That's a pretty nifty increase if you ask me.
So if I were to build a system for $2,500 it would be the 4-node, dual-core AMD Athlon X2 2.2 GHz (4200+) with 2GB of memory per node and two GigE networks. that I discussed above. But then again, I might ask for an increase in budget for the 8 node system (about $5K). The cool thing is that AMD will have quad-core chips out later this year that they claim are drop-in replacements for the current dual-core chips (i.e. no motherboard swap will be needed). Depending upon the price, the upgrade later this year might be worth it. But then again, Intel has quad-core chips out now. Maybe I'll look at an Intel system now....
Jeff Layton is one of the monkeys at clustermonkey.net. He enjoys pining away for ideal home cluster setups that he can build. When he dies he wants to be buried underneath the Playboy Mansion volleyball court (face-up - but his wife threatens to bury him face down if he doesn't behave).