I need a 4000 (four thousand) watts PSU

comomolo · Post by **comomolo** » Tue May 25, 2004 7:36 pm

Yes, you read right:

I need a 4000 Watt PSU, because I'll be feeding 40 Opterons at once (in 10 Quad-CPU motherboards).

It's an experimental cluster, but I can't use 10 regular PSUs.

I don't know if there's even such thing out there. It might be a good idea to ask someone to custom make that monster for me, but I wouldn't even know where to start (big companies would probably charge too much).

Of course I'm asking this here because I can't use something making as much noise as an helicopter. I need it to be reasonably quiet... (I know, I know I'm asking much)

Any ideas?

CM

PS: Suggestions for a 2000 Watt unit are welcome too. I'll be trying a 10x2 CPUs design soon.

mpteach · Post by **mpteach** » Tue May 25, 2004 8:06 pm

What voltage will the Powersupply run on?

In the United states average wall plugs are usually 120V x 15A = 1800W assuming a PF of 1. Large appliances run on 240V lines with much higher amp ratings.

ChucuSCAD · Post by **ChucuSCAD** » Tue May 25, 2004 9:12 pm

We use gigantic powersupplies like that in our theatre projects. You are going to need to consult an electrical engineer on the required wiring, consult the facilities manager of the building you plan on putting this thing into to see if the infastructure is there, and possibly even your local power company for this type of application.

Now of course this is being applied to extreamly sophisticated dimmer panels and specialty theatrical lighting what I am used to. This equipment is EXTREAMLY LOUD and EXTREAMLY HOT often requiring special HVAC considerations.

I severly doubt that a supply is available off the shelf with little modification to do what you want. And making one will probably not be feasable.

If you are running 10 quad motherboards why not a series of powersupplies connected via a simple on off devices so that they will all power up at the same time?

chucuSCAD

Post by **Rusty075** » Tue May 25, 2004 9:17 pm

Quad Opteron motherboard= $1500 ea. times 10
4-way Opterons=$650 ea. times 40

Total= >$40000.00

What exactly are you planning?

Call me cynical, but you're planning on powering that kind of hardware based on the advice of random people in a forum?

Don't get me wrong...it sounds like fun...but this may be one of those times to call a professional.

1398342003 · Post by **1398342003** » Tue May 25, 2004 10:24 pm

Don't forget Ram, Opterons require registered memory. That's another $500 per GB, or $1000 per CPU. For 40 CPUs that's about $40,000 for each CPU to have 2 GBs. The total is now $80,000. Then you have to consider the harddrives. You will likely want SCSI for your high performance machine/s, though you may choose ATA. The cheapest ATA drive I have seen is about $80 for a Seagate 40GB, making $800 the least you could spend on storage, The total is now $80,800. Add 650W redundant PSUs at $1000 each and you have a $90,800 computer/s. You want it to be more silent? Add 40 SLK7000A-CUs and you must add $3000 for a total of $93,800. Even more silent? Then you water cool. 40WBs ($60each, $2,400total), a good size heatercore ($20) for each 4 CPU machine($200), a pump for each machine ($100-->$1,000)

This gives a total very near 100,000, can you afford a $94,400 computer, and can you afford to get a powersupply that might cost $10,000-$20,000?

------

Out of curiosity, why can't you use several PSUs. Why must it be one?

Trip · Post by **Trip** » Wed May 26, 2004 12:11 am

Try 2CPU.com or an overclocking forum for info. on that PSU.

want to fold for SPCR?

If you watercool, post pictures!

comomolo · Post by **comomolo** » Wed May 26, 2004 6:09 am

OK guys, I can't hide it completely from you, but I can't disclose it either... (not yet at least).

Your numbers are right. We're building a VERY EXPENSIVE computer.

There are good reasons why we can't use that many regular PSUs, but I believe I can manage to use 4 x 1000 Watt, or 2 x 2000 Watt IF THEY'RE SMALL ENOUGH (how small I can't tell exactly now, but think of something like twice as much a regular ATX PSU, more or less). Anything like that off the shelf?

Cooling is being made by a professional and will cover the PSU as well as the CPUs. Be sure that if our project succeeds I'll do my best for Mike to get one of our monsters for review.

Regarding why am I asking for advice in this forum, well, what can I say? I'm currently selling the quietest workstations in my country and almost everything I've learned about silent cooling has come from this forum. I don't see this as a hobbyists place.

Keep suggesting things. I'm listening closely.

Regards,

CM

mpteach · Post by **mpteach** » Wed May 26, 2004 6:35 am

Is the sytem using air cooling or water cooling? I could imagine a 1000 watt psu twice the size of a normal psu, but I dont think one that small could be quiet with air coooling. For instance, tape 2 500watt Psus together and you get a power supply of the right size but w/ hideous noise.

1398342003 · Post by **1398342003** » Wed May 26, 2004 7:00 am

My recommendation is that you get eight 500W PSUs and use the information on the ProCooling Forums to make one. Here's the thread:

http://forums.procooling.com/vbb/showth ... eadid=4369

That would be your best bet for making something cheap. (The thread is 9 pages long though.)

comomolo · Post by **comomolo** » Wed May 26, 2004 8:04 am

We'll be using some sort of water cooling.

Thanks for the Procooling link.

CM

DynamicalSystem · Post by **DynamicalSystem** » Wed May 26, 2004 8:30 am

If the solution is going into an ordinary comms room, then you might do well to look at clusters of something like these.

Alternatively see if your target comms room is equipped to deliver 48vDC. This may offer a substantially cheaper option.

Post by **MikeC** » Wed May 26, 2004 8:41 am

comomolo wrote:Cooling is being made by a professional and will cover the PSU as well as the CPUs. Be sure that if our project succeeds I'll do my best for Mike to get one of our monsters for review.

Regarding why am I asking for advice in this forum, well, what can I say? I'm currently selling the quietest workstations in my country and almost everything I've learned about silent cooling has come from this forum. I don't see this as a hobbyists place.

Keep suggesting things. I'm listening closely.

Regards,

CM

Well, after such kind words, how can I not chime in?

I have always noted that there seems to be a lot of tech folks for PC companies floating around here from time to time; your comment confirms this yet again.

Anyway, I think your best approach is to approach a real PSU designer/manufacturer such as Seasonic, Fortron-Source, Seventeam, etc... They're the ones who have the knowhow and experience to handle a project like yours. I would guess that the ones who have been around the longest probably have experience with something similar to what you're wanting to build. You'd want them to be working with your cooling pro so that they end results dovetail well together.

Sure, the volume may be small, but a project like this could be in their interest for the PR / kudos value. Depends partly on how you pitch it. If you need contacts in those big PSU companies, I might be able to oblige.

(email me on that if you're interested)

alglove · Post by **alglove** » Wed May 26, 2004 8:09 pm

mpteach wrote:What voltage will the Powersupply run on?

In the United states average wall plugs are usually 120V x 15A = 1800W assuming a PF of 1. Large appliances run on 240V lines with much higher amp ratings.

208V, in the case of our office server room. The advantage, though, is that they can get away with *lower* amp ratings to deliver the same power, since the voltage is higher.
208V x 9A = 1800W (roughly)

kesv · Post by **kesv** » Thu May 27, 2004 2:43 am

comomolo wrote: There are good reasons why we can't use that many regular PSUs, but I believe I can manage to use 4 x 1000 Watt, or 2 x 2000 Watt IF THEY'RE SMALL ENOUGH (how small I can't tell exactly now, but think of something like twice as much a regular ATX PSU, more or less). Anything like that off the shelf?

If you really want off the shelf, you probably need an EPS PSU. EPS are for server usage and I think they are slightly bigger than ATX, but not much. In fact quad-opteron boards probably have an EPS connector.

I know Enermax makes a 660W EPS psu and I think I have seen an 800W model somewhere, just can't remember where.

And take what I said with a grain of salt. I have nowhere near enough money to get this kind of hardware myself.

kesv · Post by **kesv** » Thu May 27, 2004 3:21 am

Actually, after doing a bit of research. Everyone interested in these kinds of powersystems should read www.ssiforum.org
They are responsible for the EPS standard and also for MPS, DPS, TPS among others.

In fact I'm thinking that DPS is the way to go if you need a 4000w powersupply. DPS stands for Distributed Power Supply.

Still anything that creates this much power is *not* going to be quiet. These kinds of systems are put into machine rooms so quietness is not such an issue anyway.

Goat_guy · Post by **Goat_guy** » Thu May 27, 2004 3:22 pm

Still anything that creates this much power is *not* going to be quiet. These kinds of systems are put into machine rooms so quietness is not such an issue anyway.

....with huge air conditioning units outside to cool the room......and lets not forget about the necessary air circulation inside the room to move the cooled air around!

greeef · Post by **greeef** » Thu May 27, 2004 3:54 pm

i wouldnt worry about powerdrain from the wall, most home circuits happily run devices with those kind of wattages (3000W electric kettles!)

I would think a project like this would need to be designed from the ground up, big, high quality toroidal transformers and the like. I can't see why it should have to be noisy if you're prepared to sacrifice a little space.
I have a very mild electronic tech background (guitar amps) and i know that power and wattage come from copper wrapped round iron and nowt else.

griff

mpteach · Post by **mpteach** » Thu May 27, 2004 5:00 pm

I think MIkeC's advice is right on the money.

Goat_guy, comomolo said that this monster computer is going to be customly professionally water cooled. Obviously something thats watercooled can be quiet. @ 4000 watts this thing will probably be directly conneceted into the hvac system. So it can be quiet

greef Anything over 1800W requires a 220v line. Those are usually installed for large applience such as dryers, stoves, air conditioners, supercomputes etc. Regular 110v wont do becasue P=IV

greeef · Post by **greeef** » Thu May 27, 2004 5:09 pm

mpteach wrote: greef Anything over 1800W requires a 220v line. Those are usually installed for large applience such as dryers, stoves, air conditioners, supercomputes etc. Regular 110v wont do becasue P=IV

i know, moot point cos the guy is in spain. In europe all our mains are that high.

NOTE i may be entirely wrong.

mpteach · Post by **mpteach** » Thu May 27, 2004 6:15 pm

The voltages are high but what Amperages are your outlets rated for?

If thier rated for lower amps youd save money but have less wattage ratiing.

Tobias · Post by **Tobias** » Thu May 27, 2004 11:17 pm

Depends... In sweden, if you have alot of appliances running on the same line there is no problem to have that line upgraded to 20A, which would equate 4400W which would be enough power. This is for hoeusers. A company, I believe, would have even less problems gettings highamp-lines, unless, of course, they don't allready have them:)

ruprag · Post by **ruprag** » Fri May 28, 2004 2:44 am

Well in Iceland we use 220v (actually 230V I think) and
the connections for the stoves are all 25A and it is quite common to have 16-20A rated lines for appliances like washingmachines etc..

kesv · Post by **kesv** » Fri May 28, 2004 3:59 am

mpteach wrote:I think MIkeC's advice is right on the money.

Goat_guy, comomolo said that this monster computer is going to be customly professionally water cooled. Obviously something thats watercooled can be quiet.

Good point. Watercooling will make it easier to keep cool. A custom powersupply would certainly be one possibility, but it will be significantly more costly than using something off the shelf. Since this is already an expensive setup that might not be such a big deal, but you would also become dependent on one supplier.

If I understood the specs correctly DPS bricks have temperature controlled fans and draw air from front to back. They are also contructed so that they won't transfer heat through the sides. This is most likely so they won't heat other system components. If you were to provide cool air at the intake and draw the hot air out of the system at the back in a separated air circuit this could be reasonably quiet.

Still we have to remember that you would need atleast 4 1200w bricks to get over 4000w of power. The manufacturers quote efficiency ratings of 83% and over for DPS PSUs. This would mean about 600W of heat at full load. The heat has to go somewhere. Those bricks have 2 fans each so it's 8 fans. No matter how you put it 8 fans do create noise, even if they are low noise models. To quiet them you would need to isolate the whole powerbrick setup somehow.

I have to say this is an interesting setup for the power requirements alone. I would certainly be interested in how it is solved in the end.

comomolo · Post by **comomolo** » Fri May 28, 2004 7:15 am

I'm overwhelmed. Thanks.

Our system is being designed to avoid the need of an air conditioned room. That's why noise is an issue. I know we can't have a silent system, but the goal is to end up with a system as quiet as possible. Please forgive me for not giving you guys more details about the target of our systems and other design details, but legal issues prevent me from doing so at this moment. I'll be happy to let you know about the whole thing as soon as these issues are cleared.

I've found that IBM uses 1800 W power supplies for their Blade Center. You can buy one of those for around US$1000. Anyone has any details on this unit? I might buy one just to have a look a it, but I'm interested in your thoughts.

Thanks again. I'm reading carefully and I hope I'll be able to give something back to this community ASAP.

Regards

CM

Stigma · Post by **Stigma** » Thu Jun 17, 2004 5:23 am

There are many 1000 or 2000watts PSU's, but none of them will deliver what you are accustomed to PC PSU's giving (both 12V, 5V and 3,3 all in one) There may be some specially-built PSUs for powering supercomputers, and similar projects by the big computer companies, but I expect those would cost an insane amount of money.

Heck I even have a 1000W PSU right here that I use for powering my peltiers. Got it cheap of ebay (slightly used rugged industial equipment). However it only delivers 24V. Im sure you could find something that gave you 12V without problems, but then you would still need sources for 3,3 and 5V, wich are just as important. (3,3V drives most of the elecronics in a PC).

Therefore, I would rather suggest you buy some super-powered "normal" PSUs. I think you can get some 550 and 600watt models these days.

Here is a tip for you if your looking for a way to cool it witohut having a special room for keepingthe ambient temp down, and completely noiseless:
http://babelfish.altavista.com/babelfis ... l&lp=de_en
Note: its machine translated from german by babelfish and thus might sound fuky in some sentences.

8 of those could do the job, and completely noiseless too.. put a large radiator on the outside of the building or something similar, and youl have NO noise, and extremely little heat being leaked into the room by the PSUs.

Example of suitable radiator:
http://www.microplex.no/rubweb/varer.asp?ARTNR=AC31041

Of course, you could easily run water directly from the tap to cool the PSU. Thats what im doing on my new system im building. Of course, I dont pay for using water where I live, and you may need to take some precautions against condencation ect, but I would consider it for your project. In industries, all big power-hugry machines that need cooling are usually water cooled. Why? its effective as hell.

Hope some of these suggestions helped.

starsky · Post by **starsky** » Thu Jun 17, 2004 9:12 pm

Forgetting the PSU for the moment...

You are definitely going to need some SERIOUS electrical work to get what you need. No Rack will handle that much throughput (Amps)

For a start you are going to need 3 phase power, and and probably multiple runs to get the Amps you need.

You probably need to get one of these custom built if you are trying to provide all that power to 1 board... . Mind you , I can't imagine how you are going to get all those opteron's on one board. So lets assume you have multiple clustered computers... then you are just buying normal PSU's... If you are building a supercomputer and you have a single board for all these, yeah follow MikeC's advice.

Rod M · Post by **Rod M** » Wed Jun 30, 2004 7:14 pm

I think I know what he is trying to build,

I used to work on Digital/Compaq/HP GS series (Alpha processor) systems, these things are monsters and are basically one system with up to 64 processors mounted in 1-2 42RU cab's, apart from being stupidly powerful they are also need the IT budget of a small third world country to buy,

by putting 10x quad opteron boards fully decked out in a custom made 42RU case with all the trimmings, it would cost a fraction that of a similar GS1280, I do believe that they use three phase power as well,

I recommend you look at the specs of the GS1280 to give you a idea of how the power issue is resolved,
http://h18000.www1.hp.com/products/quic ... 21_na.HTML