Im a grad student preparing plans for my prof for a new 50-100 node duallie cluster in our lab. Currently, we've got a 16 node 1900 MP cluster, each in a mini chassis. Its LOUD. CRAZY loud.
Yes, there is a possibility that we can put the new cluster in another building, but that sets up administration and accessibility issues. For the sake of this discussion, lets assume this cluster isnt going to be in a machine room. But we would ALSO like to keep the labspace holding the cluster as useable as possible: Unfortunately, with 200 cpu fans and 400 case fans whirlling, we'll all be deaf as Kiss. So heres my problem:
I would like to get your input on ways to QUIETLY cool 100 rackmounted duallie boxes. Money is not really an issue; Im just askin for ideas.
In particular, I've been trying to see if it is possible to centrally watercool the cluster, by running all the waterstreams through one big-a$$ exchanger. Has anyone got any experience with this? One of the problems would be the pump: 100 half inch tubes moving water off CPUs is a lot of water per second. It would have to be a hardcore pump.
There are also failure issues: if the pump dies, we dont want to lose 100 machines. Is there a way to get a pump to start up if water pressure drops? Are there waterblocks that can dissipate enough heat with no water that the CPU doesnt die before auto-shutoff?
A) how would you watercool rackmounted 200 opterons or 200 MPs? And would it even be that quiet? I think it could be made quiet with central cooling (i.e. giant exchanger, 1 giant pump)
or
B) how would you cool it otherwise?
Constraint: No downclocking/volting
Silent Cluster computing
Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee
I'm thinking that the water pump will be pretty loud. Using a ~1/2 hp industrial pump which is only able to do about a few gpm, it's very loud.
I assume these are 1U or maybe 2U cases?
You could move the power supply outside the case and get low noise power supplies.
Perhaps some kind of 80 mm fan solution would be doable also. That would reduce the noise from those very small fans that are probably on the CPU cooler.
Another way is to tray mount the computers so everything is exposed and have the racks with vented front and rear doors and flow the air from front to back. This way you can use larger, more effective, and less whiny fans.
I assume these are 1U or maybe 2U cases?
You could move the power supply outside the case and get low noise power supplies.
Perhaps some kind of 80 mm fan solution would be doable also. That would reduce the noise from those very small fans that are probably on the CPU cooler.
Another way is to tray mount the computers so everything is exposed and have the racks with vented front and rear doors and flow the air from front to back. This way you can use larger, more effective, and less whiny fans.
yeah we're lookin at 1U or 2U.
What sort of thing can I get to move the powersupply off the cases? Is there like a monster power supply that can be bought to power 10, 20, even 50 machines?
Whats the difference between rack mounting and tray mounting? Is it essentially the same, except no front and backs? I could imagine having a fan panel of 120s on either side just moving air over the trays, but Im concerned about the hot spots from the CPUs:
when you have the mobos stacked on top of each other, they line up exactly, with the CPUs right under/over each other, at very small proximity. Sure, moving air accross that cavitiy will help, but will that be enough?
What sort of thing can I get to move the powersupply off the cases? Is there like a monster power supply that can be bought to power 10, 20, even 50 machines?
Whats the difference between rack mounting and tray mounting? Is it essentially the same, except no front and backs? I could imagine having a fan panel of 120s on either side just moving air over the trays, but Im concerned about the hot spots from the CPUs:
when you have the mobos stacked on top of each other, they line up exactly, with the CPUs right under/over each other, at very small proximity. Sure, moving air accross that cavitiy will help, but will that be enough?
I don't think there is a power supply like that, well at least I'm not sure it would be practical to try something like that.
But if you get some normal ATX units you will have larger quieter fans and temperature speed control on them.
A tray would just be a shelf with the bottom and sides only.
If you use 2U then there is probably enough space between them, and you can also fit in heatsinks with 80mm fans.
If you are concerned then you could stagger the motherboards. One of the left, then on the tray above it it could be on the right, etc...
What other equipment has to be connected to each computer?
But if you get some normal ATX units you will have larger quieter fans and temperature speed control on them.
A tray would just be a shelf with the bottom and sides only.
If you use 2U then there is probably enough space between them, and you can also fit in heatsinks with 80mm fans.
If you are concerned then you could stagger the motherboards. One of the left, then on the tray above it it could be on the right, etc...
What other equipment has to be connected to each computer?
For watercooling, it might be possible to have the flow running under gravity through the blades. The system I'm thinking of would have 2 reservoirs, above and below the system.
The water would flow from the top reservoir to the bottom one under gravity, via the CPUs. You could then pump it back up again, using multiple smaller pumps. This would probably be quite a bit quieter and reduces the risk of catastrophic failure. It should be quite possible to fit some kind of water level alarm in the top reservoir such that if the water level starts dropping below a certain level (i.e. pump failure) either a spare pump kicks in or the system turns off. This would also give you a certain amount of time between pump failure and the cooling stopping in which to shut down the system - not a sudden failure as you might get if relying on a single pump and no reservoir.
To cool the water, rather than cooling all of it I would suggest extracting a proportion of the water in each reservoir, running it through a large heat exchanger and returning it to the same reservoir. This would produce reduced cooling, but would probably be good enough.
You're looking at a what, 20kW heat load to dissipate? That's going to produce some major heating so if you can it would be a good idea to mount the radiators outside - things are going to get rather warm otherwise, and maybe give heating problems if you're got bad ventilation inside.
To dump the heat, I'm not exactly sure what sort of radiator would be needed, but I suspect something like 4 car radiators could probably cope with the heat. If they're mounted outside this would also get the fans out of the way, cutting the noise further.
The water would flow from the top reservoir to the bottom one under gravity, via the CPUs. You could then pump it back up again, using multiple smaller pumps. This would probably be quite a bit quieter and reduces the risk of catastrophic failure. It should be quite possible to fit some kind of water level alarm in the top reservoir such that if the water level starts dropping below a certain level (i.e. pump failure) either a spare pump kicks in or the system turns off. This would also give you a certain amount of time between pump failure and the cooling stopping in which to shut down the system - not a sudden failure as you might get if relying on a single pump and no reservoir.
To cool the water, rather than cooling all of it I would suggest extracting a proportion of the water in each reservoir, running it through a large heat exchanger and returning it to the same reservoir. This would produce reduced cooling, but would probably be good enough.
You're looking at a what, 20kW heat load to dissipate? That's going to produce some major heating so if you can it would be a good idea to mount the radiators outside - things are going to get rather warm otherwise, and maybe give heating problems if you're got bad ventilation inside.
To dump the heat, I'm not exactly sure what sort of radiator would be needed, but I suspect something like 4 car radiators could probably cope with the heat. If they're mounted outside this would also get the fans out of the way, cutting the noise further.
Just got an idea by combining a thread or two of the forums:) The new VIA ITX-form factor is bound to be available in a not to distant futeture. Those are just 12*12 cm, so starting with one 120mm-fan in one end, a sealed "tube" and a 120mm fan in the other end and some Zalman HS on the CPU should minimize fan-needs and could be rather silent, depending on the number of puters/tube...
As I said, I am just thinking loud here. Not sure it can be done:) But it is a thought.
As I said, I am just thinking loud here. Not sure it can be done:) But it is a thought.
-
herosformula
- Posts: 231
- Joined: Wed Apr 16, 2003 11:40 am
- Location: seattle, wa
well, here is the link to VIA:s presentation of the board. However, the board is just released, but if things go as they did with their mini-itx form factor there should be products on this form, but with intel-cpu:s at some point in a not to distant future... That is the time I meant in my previous post:)
Edit: Forgott the link:)
http://www.viaembedded.com/product/epia ... oardId=221
Edit: Forgott the link:)
http://www.viaembedded.com/product/epia ... oardId=221