Was 3.7TB - then 2.7TB, now 19.1TB "quiet" server

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Wed Dec 19, 2007 10:10 am

Or his samba version... mine is working just fine, easily saturating gigabit :)

trxman
Posts: 145
Joined: Wed Dec 15, 2004 5:45 pm
Location: system

Post by trxman » Wed Dec 19, 2007 1:17 pm

Wibla wrote:Or his samba version... mine is working just fine, easily saturating gigabit :)
samba version is:

Code: Select all

net-fs/samba-3.0.28
on both server and client.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Wed Dec 19, 2007 2:04 pm

Code: Select all

$ aptitude show samba
Package: samba
State: installed
Automatically installed: no
Version: 3.0.24-6etch9

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Apr 17, 2008 1:43 am

A few tweaks later, and my performance is as follows:

Code: Select all

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Oberon           4G           119643  19 70150  11           417908  28 103.5   0
Oberon,4G,,,119643,19,70150,11,,,417908,28,103.5,0,,,,,,,,,,,,,

Settings:

Oberon:/mnt/raid# blockdev --getra /dev/sda; cat /sys/block/sda/queue/max_sectors_kb; cat /sys/block/sda/queue/nr_requests; cat /sys/block/sda/queue/scheduler
65536
64
256
noop anticipatory [deadline] cfq

Also swapped out the Noctua 120mm fan in the top 4in3 module for a Scythe S-FLEX 120mm / 1200rpm unit, hdd temps are abit more reasonable now:

Image

toki_c
Patron of SPCR
Posts: 64
Joined: Tue Feb 19, 2008 2:27 pm
Location: Earth

Post by toki_c » Thu Apr 17, 2008 8:08 am

P4D 3.2... 10x500GB 7200rpm HDDs... you sure are not on the green side of conputing, right?

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Apr 17, 2008 8:11 am

Its a (prescott) Celeron 3.2, not my first choice, but the damn mobo doesnt like C2D's (old version).

And, to be honest, it doesnt really use THAT much power, and more than half of the year the power spent is going towards heating up the house...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Tue Dec 16, 2008 12:55 pm

Time to upgrade this server soon, depending on the paycheck i get on the 22nd..

Upgrade path:
Tagan 480W -> Tagan 480W + Corsair CX400
Asus P5WDG2-WS -> Asus P5E WS PRO
Celeron 3.2GHz -> E5200
2GB ram -> 4-6GB ram
10x500GB -> 5x1.5TB (maybe 4, not sure yet)
30GB WD -> 80-120GB Seagate 2.5" SATA (system drive)

Keeping the 3ware controller, and considering windows server 2003/2008 instead of debian linux, or vmware of some sort.

I'm not 100% certain about how good the seagate 1.5TB drives are, so this server will be considered a beta test box for a while.

This setup should give me 5.44TB or 4.08TB usable in raid5, and is expandable to 13.6TB usable space (11x1.5TB + hotspare).

Nick Geraedts
SPCR Reviewer
Posts: 561
Joined: Tue May 30, 2006 8:22 pm
Location: Vancouver, BC

Post by Nick Geraedts » Tue Dec 16, 2008 1:09 pm

Wait.... which 3ware controller are you going to use? You had problems before with that motherboard and it's PCIe-PCIX adaptor chip (see here)...

Also, be aware that while Server 2008 is awesome (so many cool features!), you won't be able to get support for your 9550SE card or use Hyper-V with the E5200 (if you want to use virtualization). The 9650SE models are supported with the latest drivers - Link to KB article.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Tue Dec 16, 2008 1:16 pm

According to Asus the problems should be solved, but I'm intending to test it in a normal pci slot aswell, as these are 66MHz on the newer mobos, which gives a max speed of 266MB/s, and that is actually enough.

I was planning to keep the 9500S, according to the KB it should work, just not be supported officially, which doesnt matter much to me, the "it works"-part is the key.

Edit: Might also get a Supermicro SATA/SAS controller and combine that with onboard ports, and then run mdraid in linux, or something on WHS... choices choices...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat Aug 15, 2009 7:12 pm

That paycheck never worked out as it should, and my car had to be fixed, so i was broke for the last part of december + half of january. Then I decided to leave it alone for the time being...

But not anymore.

Two drives in the RAID5 array decided to go bonkers on the same time, leaving me with a degraded array, so no choice but dismantle the server to get the drives out, and rebuild it.

Must say the filters on the CM stacker really works, but still the insides of this server looks like hell...

Image
ducttape, works wonders to keep the air going where it should, atleast in theory...

Image
Very bad choice to install the psu like that, too bad it didnt fit above the mobo.

Image
Dust...

Image
Moar dust!

Image
These worked as advertised, very happy about that.. next stop for the filters are some heavy duty cleaning, then reassembly.

Image
With absolutely NO cable managment it was pretty easy to get the old psu out, didnt even have to remove the back panel...

Image
6 drives off one molex, probably not the smartest thing in the world...

Edit:

Current status is: new PSU in, all power laid out and ready, with spare cables for more drives, cabled up for 10 SATA + 1 IDE, with provision for 4-6 more SATA if needed.

Drive cages removed, defective drives removed from drive cages, probably gonna reorganize the drives and remove the dust from each drive and cages at the same time. Have vacuumed the fans to get the worst dust off, might wipe them with moist cloth or something tomorrow to remove the rest...

Plans for the next couple of months is testing the 3ware as a pure SATA controller, with the remaining 8 500GB drives in mdraid6 on debian linux, I have good hopes that it will be faster than the old HW raid5, but in any case it'll only be used for playing around and an extra backup of important stuff, in the improbable case that monster fails.

Come september/october and the tax money I'm planning on getting new 1.5TB Samsung drives and an UPS so I can actually upgrade capacity overall, aiming for 18-20TB formatted, with monster being 9.1TB of that.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Tue Aug 18, 2009 1:05 pm

Box is almost ready for boot.. just need to clean out the cpu heatsink, looking at replacing it for a xigmatek aswell, and probably swapping out the stone-age era 30GB for a 2.5" SATA.

Seems I've forgotten the old tricks from wiring up monster tho, meh. Not gonna waste more time on it.

Image

Image
Velcro tape to anchor down the system drive, also gives some vibration dampening.. (tho not alot)

Image

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Fri Sep 11, 2009 3:51 pm

Komplett.no has a weekend sale, with 2TB Seagate drives for 999 NOK, about USD $168 .. ordered 6, which is going into this server... stay tuned :mrgreen:

(I just hope i get them, when i was finished ordering, the price was gone from the product info, and no stock info was shown.. meh)

That should bring my total storage up to about 19TB, if i run them in raid6, or 20.9TB if i run them in raid5.. time will show.

frenchie
Friend of SPCR
Posts: 1346
Joined: Tue Jul 08, 2008 4:53 am
Location: CT

Post by frenchie » Fri Sep 11, 2009 11:49 pm

I'm jaelous... I always wanted to download the internet but it won't fit on my 1TB drive.... :lol:
Have you ever thought of renting all that space to google or one of those companies ? They could be interested !!!

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Sat Sep 12, 2009 4:16 am

They have their own storage solutions, stuff that makes me jealous :(

Almost regretting i didnt order 8 now :mrgreen:

yamahaSHO
Posts: 426
Joined: Sun Mar 07, 2004 4:38 am
Location: Bellvue, Nebraska
Contact:

Post by yamahaSHO » Tue Sep 29, 2009 7:54 am

I'm sorry if I missed it, but do you know the power consumption?

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Tue Sep 29, 2009 11:04 am

Dont have any hard numbers, I keep forgetting to buy a kill-a-watt.

I'd guesstimate around 120-150W idle, 220W load.

The 2TB deal didnt go off, komplett = assholes.

Planning 8x1.5TB when the tax returns arrive :)

yamahaSHO
Posts: 426
Joined: Sun Mar 07, 2004 4:38 am
Location: Bellvue, Nebraska
Contact:

Post by yamahaSHO » Tue Sep 29, 2009 11:53 am

Please keep us posted. I just built a server and although I currently only have one drive in it (have more ready to go in), it is only pulling about 60w at idle.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Wed Sep 30, 2009 3:32 am

Will do :)

Got a thread for your computers? eager to see what your server looks like

yamahaSHO
Posts: 426
Joined: Sun Mar 07, 2004 4:38 am
Location: Bellvue, Nebraska
Contact:

Post by yamahaSHO » Wed Sep 30, 2009 6:08 am

I don't have any picture of the server yet since I haven't installed the last 4 HDD's, however, in the link in my sig, I used the case and motherboard from the P180B build. I also just rebuilt/upgraded my NSK-2480 for like the 3rd time since I've owned the case. I'll take some pics and make a post and link you from here.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Oct 22, 2009 7:15 am

Just ordered 16 x 1.5TB Seagate LP drives destined for this computer... :mrgreen:

FartingBob
Patron of SPCR
Posts: 744
Joined: Tue Mar 04, 2008 4:05 am
Location: London
Contact:

Post by FartingBob » Thu Oct 22, 2009 8:30 am

Ok seriously i know im a packrat, but even when with my increasingly large HD collection and 50Mb internet speed it would take me a good couple of years to fill 24TB. How full are your current HDD's?

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Oct 22, 2009 10:29 am

monster is currently 99% full, and oberon (this box) is only used for backups atm.

This is a long term setup, with 19TB formatted (RAID6) it should last atleast 1.5-2 years (even when moving 8-9TB of data from monster to this server), long enough to replace the drives in monster with the current "cheapest per TB" drive out when the 19TB's are full

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Wed Oct 28, 2009 2:44 pm

Upgrade finally done...

Image

Image

Image

Code: Select all

/dev/md0               20T  5.5M   20T   1% /mnt/raid
The Seagate LP drives are not noisier than the T166 they replaced, atleast that is the first impression. The noisiest thing in the box is still the WD 30GB.

Fayd
Posts: 379
Joined: Thu May 10, 2007 2:19 pm
Location: San Diego

Post by Fayd » Wed Oct 28, 2009 8:44 pm

all that powered on an HX-520. amazing.

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Oct 29, 2009 12:19 am

The drives pull 5.5W idle (90W total), 6.8W average (110W total) and max current usage at boot is 2A per drive, so 24W per drive or 380W total. I'd probably get into trouble if I tried to run 20 drives, but 16 is just fine.
I'm planning on replacing the 3ware with two supermicro pci-x 8port controllers, as its getting a tad slow.

I also had to use a 64bit kernel to get the filesystem running, Linux 32bit has a limit on 16TB...

mark19891989
Posts: 187
Joined: Tue May 26, 2009 10:37 am
Location: UK

Post by mark19891989 » Thu Oct 29, 2009 9:56 am

Wibla wrote:Upgrade finally done...

Image

Image

Image

Code: Select all

/dev/md0               20T  5.5M   20T   1% /mnt/raid
The Seagate LP drives are not noisier than the T166 they replaced, atleast that is the first impression. The noisiest thing in the box is still the WD 30GB.
WOW! i love that much space, my file server has 3tb which is enough for me, but this setup puts mine to shame.

what are you storing on this thing?

veseliburek
Posts: 7
Joined: Fri Jun 05, 2009 12:55 pm
Location: Serbia

Post by veseliburek » Thu Oct 29, 2009 12:58 pm

ever considered undervolting your CPU, buying a large heatsink and running it fanless?
and dude, you have all that money for 19 terabytes while you use a 30gb western digital that doesn't even use a SATA interface and it's noisy like hell...

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Thu Oct 29, 2009 1:41 pm

I'm not doing anything to compromise the stability of the cpu, as I'm running swraid I cant afford any oddities. The cpu fan isnt really audible outside the case either.

The WD 30GB is on the shortlist of stuff to fix, and im contemplating getting a Xigmatek 1283 for the cpu, as I did on the 9.1TB box. IDE vs SATA isnt really a big issue, I'm planning on putting in a Samsung 320GB IDE drive to replace the WD 30GB, just to get the noise level down, and because I need to reinstall the server to 64bit debian anyhow.

I wasnt planning on putting alot (more) cash into this setup, it was basically "toss the 500GB'ers, put in something larger, leave it at that"...

@mark: The world's largest collection of pron, obviously (or maybe not :P)

Backups is taking up more and more space, truth be told, I'm guessing atleast 8TB of the total capacity of this server will be backups from the old before I even start putting in new stuff on it...

Some performance numbers:

Code: Select all

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
Oberon           4G           74787  16 68462  19           521149  53 113.3   0
I think I'm being bitten by 64K chunk size from linux md, and the fact that the 3ware isnt strong as a pure SATA controller any which way you turn it.. will need some tweaking to break the 100MB/s barrier, first off will probably be a raid rebuild to 128K chunk size... And maybe a couple of supermicro sas/sata cards? hmhm, tempting! (ebay <3 )

Wibla
Friend of SPCR
Posts: 779
Joined: Sun Jun 03, 2007 12:03 am
Location: Norway

Post by Wibla » Mon Nov 02, 2009 4:23 pm

Replaced WD 30GB PATA with Samsung 320GB PATA for much improved performance on system stuff, and alot less noise... Went from Debian Etch with Lenny-backports 2.6.30-amd64 kernel to Debian Lenny with stock 2.6.26-2-amd64 kernel, and to my surprise performance increased ALOT.

Code: Select all

Version 1.03d       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
oberon.lan.wibla 4G           118752  34 110140  38           491916  59 284.5   1
This is after I enabled write-back cache on the harddrives, and tweaked readahead and stripe_cache abit.. Trying to do the same on the old kernel netted me the result in the previous post, not exactly impressive.

End result: server is (almost) "ready" for production, noise levels are definently tolerable, and im happy :mrgreen:

misterd
Patron of SPCR
Posts: 14
Joined: Thu Jul 26, 2007 5:41 pm
Location: Charlottetown, Canada

Post by misterd » Tue Nov 03, 2009 6:18 am

Wibla: i'm in love! *drool* What kind of silly Norweigian connection are you sucking down the internet with?

Post Reply