ServeMi, 23TB Server

Show off your quiet rig.

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Post Reply
Abula
Posts: 3662
Joined: Tue Nov 02, 2004 12:22 pm
Location: Guatemala

ServeMi, 23TB Server

Post by Abula » Sun Dec 11, 2011 4:26 pm

This is by far the hardest build i ever done, a lot of issues happen before i could finish it, more than 2 months all weekends working on it. Starting with Asus P8B WS mobo with restarting issues, bought ne PSU, new memory and still no fix, so i return it and got a Supermicro mobo, the first day everything was perfect, second day OS hdd started clicking and no boot, swaped the Hitachi 5k500B for intel x25m 120gb that i had left over (wasn't planing on using it but with hdd prices today....i just grabbed what i had in hand), everything was perfect until i started noticing a loud intermittent noise from the fans, seems a lot of Supermicro mobos have issues controlling PWM fans, it drops them down to 0 and then to full 1600rpm, its an endless loop and very noisy and disturbing, i tried Speedfan and no luck... so i swapped all fans to 3pins and finally everything is good.

This is mostly a storage server for my local network, i had such a good experience with WHSv1 and Acer AH-342 that i decided to build my own as the space wasnt enough for all the media storage for the home network. I had an antec 1200 left over form my past gaming pc, so decided to get some hdd cages to fit 16x hitachi 5k3000, but with the flooding i had to settle for a combination samsung 1.5tb that i had left from the previous server. Here is the list of hardware currently in use,

MOBO: SUPERMICRO MBD-X9SCM-F-O LGA 1155 Intel C204 Micro ATX Intel Xeon E3 Server Motherboard
MEMORY: Kingston 4GB (2 x 2GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1333 Server Memory Model KVR1333D3E9SK2/4G
CPU: Intel Xeon E3-1230 Sandy Bridge 3.2GHz 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 80W Quad-Core Server Processor
CPU Cooler: Thermalright AXP-140 RT Low Profile HTPC Heatsink
CPU Fan: Noiseblocker NB-BlackSilentPro PK-1 140mm x 25mm Ultra Quiet Fan - 700 RPM - 9 dBA
Case fans: 7x Noiseblocker NB-Multiframe M12-S1 120mmx25mm Ultra Silent Fan - 750 RPM - below 6 dBA
OS drive: Intel X25-M SSDSA2MH120G2K5 2.5" 120GB SATA II MLC Internal Solid State Drive (SSD)
HDD: 8x HITACHI Deskstar 5K3000 HDS5C3020ALA632 (0F12117) 2TB 32MB Cache SATA 6.0Gb/s 3.5" Internal Hard Drive
HDD: 6x SAMSUNG EcoGreen F2 HD154UI 1.5TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive
HBA: 2x Intel RAID Controller Card SATA/SAS PCI-E x8 8internal ports (SASUC8I)
Sata cables: 4x 3ware CBL-SFF8087OCF-05M 1 Unit of .5M Multi-lane Internal (SFF-8087) Serial ATA Breakout Cable
Sata power: 4x Silverstone Four-in-one SATA Power Connectors with Power Stabilizing Capacitors Model CP06
PSU: Antec CP-850 850W Continuous Power CPX SLI Certified CrossFire Ready 80 PLUS Certified Modular Active PFC
CASE: Antec Twelve Hundred Black Steel ATX Full Tower Computer Case
HDD Cages: 4x COOLER MASTER STB-3T4-E3-GP 4-in-3 Device Module Hardisk Cage
OS: Windows Home Server v1

Here some pics of the build,

Image Image Image
Image Image Image Image

After adding all hdds, here is the storage pool,

Image

This are the idle and load (prime95) temps and power consumptions (with the flash the screen appear white and no good daylight there, in case you cant see was 123W idle and 197W load)

Image Image
Image Image

Here is a test of transfering a large file from the server to my desktop, and a test while watching a movie over the network,

Image Image

Here is HDtune of the Hitachi 5k3000 ran inside the server,

Image


Comments,
I had a lot of fun doing the server, started planning a low powered server, but with the trouble i had with the initial parts and wanting to do transcoding on the fly, and maybe even move to a newer OS with ZFS, so i went with better cpu/mobo than i initially planned, but really been a great experience with supermicro and xeon. The SSD is an overkill for this build, specially once i added the SASUC8I, the server takes a long time to boot, 3min+, the ssd doesn't make much of a difference, but i had it laying around and my hitachi 5k500b died while building, and didn't want to spend more so i just used the x25m that i had, still you do feel it once inside windows, like on my desktop everything loads instantly, but for the server doesn't make much of a difference. Im pretty happy with the transfer rate results, im able to sustain close to 100 MB/s over my home gigabit network (really depends on other factors, some files do not transfer that that rate). With what i experience, the only change i would do is going for a server case, while the antec 1200 works and probably maintains the hdds cooler as each cage has its own fan and directly blow air through... its very hard to work with, specially the Cable management, i really tried to hide as much as i could, but once i was dealing with all the sata cables and the PSU cables, there isn't enough room on the 1200 to router all in the back, maybe with some rerouting.... but still its not ideal.

The hdds are pretty quiet, a little noisy on startup... but hey its 14 none suspended mechanical 3.5 hdds, once it boots to WHS the drives are very quiet, even transfering i can bearly notice them. Im also happy with the fans, although its a little weird, the M12 S1 are suppose to be 750rpm fans, but the mobo reads them as 850rpm, still they are very quiet, and the PK1 is also, while not silent, im very pleased with the temps and noise.

Future,
This was suppose to be an 30TB server, but with flooding and today hdds pricing, was impossible to get new drives, so i had to reuse the samsung from my old server, and didnt have 16 drives, but ended with 14, so once the hdds prices go back to normal, im planning on increasing the storage, but i might just leave it as it is, and move to another OS that supports 2tb+ hdds. Im also looking for software that can transcode on the fly for mobile stuff like ipads, iphones, etc.

If you have any questions or suggestions or comments, pls post them.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Tue Dec 13, 2011 2:04 am

What are you using such a large server for?

Kidding aside, did you measure the power consumption without drives, with and without the controller cards?
There's a general lack of data on power consumption so it would be interesting to know how much the mobo+CPU draws as well as how much the cards draw. Obviously this would only be helpful if we knew the PSU efficiency at low loads (I didn't look it up).

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Tue Dec 13, 2011 10:29 am

You're really running that many drives without any form of redundancy?

Have fun when a drive fails.

Abula
Posts: 3662
Joined: Tue Nov 02, 2004 12:22 pm
Location: Guatemala

Re: ServeMi, 23TB Server

Post by Abula » Tue Dec 13, 2011 12:58 pm

HFat wrote:What are you using such a large server for?

Kidding aside, did you measure the power consumption without drives, with and without the controller cards?
There's a general lack of data on power consumption so it would be interesting to know how much the mobo+CPU draws as well as how much the cards draw. Obviously this would only be helpful if we knew the PSU efficiency at low loads (I didn't look it up).
Sorry didnt measure while it was without HBAs or hdds, i did measure with 11hhds and was idling around 112W, and now with 14 is around 123W, the difference in consumption makes me think that the drives are not spinning down, but im trying to confirm and see how to make them spin down when not in use.

The HBAs i read they consume around 10W each, but again didnt test for my self. Maybe in the future when i clean it i might do just a BIOS no HBA or HDD loading, but for now i dont want to mess with it, starting building the HTPC now :)
Monkeh16 wrote:You're really running that many drives without any form of redundancy?

Have fun when a drive fails.
Don't worry this is mostly storing movies from my legally owned BR collection, all is easily replaceable, its just time. Now since you are thinking about a drive failing, yes it will happen, its just a matter of time, but this is one of the reasons i went with WHS and not a server with a raid array. If a drive fails i only lose what was on that drive, WHS doesn't allocate information spread around, once it starts using a hdd it will continue using it until its full and then move on to next drive and so on, so if a drive fails here i will only re do all my ISOs that were located on the the failing hdd. Now if the OS drive fails, its fine also, all hdds are readable from any PC with windows installed, its not an raid array where its not readable in other PC or a crazy OS / Format that will be useless if the raid fails. Either way, its a matter of choice, but as you can see its not as bad as you might think.

Image

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Tue Dec 13, 2011 1:14 pm

Abula wrote:If a drive fails i only lose what was on that drive, WHS doesn't allocate information spread around, once it starts using a hdd it will continue using it until its full and then move on to next drive and so on, so if a drive fails here i will only re do all my ISOs that were located on the the failing hdd.
If I lose a drive in my RAID arrays I just swap the drive out. Nothing else happens.
Now if the OS drive fails, its fine also, all hdds are readable from any PC with windows installed, its not an raid array where its not readable in other PC or a crazy OS / Format that will be useless if the raid fails.
RAID arrays aren't locked to one machine, and there's nothing 'crazy' about not using Windows.. All my arrays (there are five or so in here) are readable on any machine with enough (anywhere from two to eight) SATA ports.
Either way, its a matter of choice, but as you can see its not as bad as you might think.
A choice which guarantees you'll have to spend several days restoring data at least several times in the future. :/

Abula
Posts: 3662
Joined: Tue Nov 02, 2004 12:22 pm
Location: Guatemala

Re: ServeMi, 23TB Server

Post by Abula » Tue Dec 13, 2011 1:47 pm

Monkeh16 wrote:If I lose a drive in my RAID arrays I just swap the drive out. Nothing else happens.
You could lose another drive while rebuilding the array, if your parity is not more than than the amount of drive that failed, then what.... I seen it happen, hope you never have to deal with it.
Monkeh16 wrote:A choice which guarantees you'll have to spend several days restoring data at least several times in the future. :/
Yes, but raid itself is not a backup as you make it seem, its just uptime, arrays can fail completely, to me my backup is on my BR, if one of my drive fails, and i want to watch it, ill just pull it from shelf, but personally i dont find raid attractive, if more drives than your parity fail or fail while rebuilding... you are really screwed, in my case all drives are completely independent, what im trying to say, is that i prefer to lose small and have a downtime on partial storage than the risk of a total failure.

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Tue Dec 13, 2011 2:24 pm

I never suggested it was a backup. Merely a timesaving device for when your drives go kaput. Which they will.

It seems insane to me to plan to use time to restore all that data when you have the money and physical space to remove the requirement to do so in 95% of cases.

Then again, it's also insane to rely on Windows for any form of storage.

I'll leave you to it. I hope one day to have that sort of spare money and time.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Wed Dec 14, 2011 4:51 am

You don't need Windows to read NTFS.
Abula wrote:
Monkeh16 wrote:If I lose a drive in my RAID arrays I just swap the drive out. Nothing else happens.
You could lose another drive while rebuilding the array, if your parity is not more than than the amount of drive that failed, then what.... I seen it happen, hope you never have to deal with it.
And people have lost RAID>1 arrays after failure of a single drive. It's not supposed to happen but it does. RAID is not a replacement for backups.

Right now drives are expensive but if I was in Abula's situation, I would back up to external hard drives once prices get more reasonable. But if I wasn't willing to pay for that, I would probably go with RAID5. It's not that risky compared to having no redundancy.
In the meantime, I would backup to the empty drives. Why not use those empty drives? Even if you're unwilling to disconnect them from the server, you can still copy the files so that each one is on at least two drives.

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Wed Dec 14, 2011 5:21 am

HFat wrote:You don't need Windows to read NTFS.
Nope. But he's using Windows to write it, and that's the first step along a long road of corrupted pain.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Wed Dec 14, 2011 6:15 am

I lost data to NTFS, but not when running bare-metal on such hardware.
The data on the X-25M is unsafe but I would expect those hard drives and controllers to behave reasonably.
Compared to the risk of drive failure, what do you figure are the chances of NTFS issues in this case?

The one thing I'd be wary about is how robust this multi-drive partition thing would be in case a drive failed. Maybe I misread the snapshot but I don't think I ever used anything like that on Windows.

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Wed Dec 14, 2011 6:28 am

HFat wrote:Compared to the risk of drive failure, what do you figure are the chances of NTFS issues in this case?
I've had a lot more corrupt files thanks to Windows than I've had drive failures, I can tell you that much. And I go through a lot of drives.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Wed Dec 14, 2011 6:39 am

Same here but I have no real Windows server.
Why would you get corruption on that hardware? I'm not saying you're wrong but Microsoft is a serious business and it's not rocket science to get this right if the hardware is adequate.

Abula
Posts: 3662
Joined: Tue Nov 02, 2004 12:22 pm
Location: Guatemala

Re: ServeMi, 23TB Server

Post by Abula » Wed Dec 14, 2011 7:19 am

To what i read, not experience it directly yet, WHS did have some issues with corruption, specially with files related to multiple users, when they modified files over the network directly on the server. HDDs have errors every second while writing data, they have algorithms that do correct it, but not everything is corrected by them, thus there will always be a chance for silent corruption no matter what, there are filesystems like ZFS that have multiple ways to correct or minimize this, but nothing is fully certain always. Raid 5 doesn't protect you from silent corruption, if a file goes bad, goes bad the array and the parity, thats why true backups should always be external, but this is why ZFS seems so appealing to me, just dont have the time yet to really learn it, but this is the plan in time to move the server to it.

@Monkeh16, you seem to know your ways around server and filesystems, so maybe you could tell us more about how and why it happens under windows, or under what scenarios is most likely to happen?

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Wed Dec 14, 2011 7:24 am

HFat wrote:Same here but I have no real Windows server.
Why would you get corruption on that hardware? I'm not saying you're wrong but Microsoft is a serious business and it's not rocket science to get this right if the hardware is adequate.
On any hardware. I've experienced random corruption on Windows on multiple machines using totally different hardware. Serious business or not, the architecture of their OS is a bad joke.
Abula wrote:@Monkeh16, you seem to know your ways around server and filesystems, so maybe you could tell us more about how and why it happens under windows, or under what scenarios is most likely to happen?
How, I don't know. As for scenarios, copying large files or large numbers of small files, in my experience, has a high risk of corruption on Windows (XP or Vista7. I don't use Windows Server.). I have never experienced random corruption on other platforms, and I've dealt with far, far larger amounts of data elsewhere.

Abula
Posts: 3662
Joined: Tue Nov 02, 2004 12:22 pm
Location: Guatemala

Re: ServeMi, 23TB Server

Post by Abula » Wed Dec 14, 2011 7:31 am

Thanks for the info.

BTW what OS/Filesystem do you run?

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Wed Dec 14, 2011 7:35 am

Abula wrote:Thanks for the info.

BTW what OS/Filesystem do you run?
Several. My storage is a fairly normal Linux box using ext4. Simplicity works.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Wed Dec 14, 2011 7:56 am

ext4 on Linux is also my preference even if I reckon modern versions of Windows are decent.
Abula wrote:HDDs have errors every second while writing data, they have algorithms that do correct it, but not everything is corrected by them, thus there will always be a chance for silent corruption no matter what, there are filesystems like ZFS that have multiple ways to correct or minimize this, but nothing is fully certain always.
That's not really the point. Schemes like ZFS protect you against storage corruption in general. So far, I've narrowed down corruption to controllers but never to drives (unless you count bad sectors).
edit: I mean never to *hard* drives of course (other types of drive are not always reliable)

But you don't need ZFS for ISOs. Just use hashes to detect corruption (a filesharing app will do this for you if you have no better tools) or Parchive if you want to be able to recover from slight corruption (probably overkill in your case since you expect to restore from optical drives). This has been pointed out many time on SPCR.
Abula wrote:under what scenarios is most likely to happen?
In my experience the top causes (excluding network badness, malware, user errors and the like) are crappy controllers and VMs without I/O passthrough (especially bad if you don't have ECC RAM).

JLee
Posts: 31
Joined: Mon May 26, 2003 2:39 pm
Location: Vancouver

Re: ServeMi, 23TB Server

Post by JLee » Wed Dec 14, 2011 9:58 pm

ohnoes the linux fanboys are telling everyone that windows will corrupt your data!1!!11 save the data!! won't ANYONE PLZZZZZzzz thknk of the data!

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Thu Dec 15, 2011 3:53 am

JLee wrote:ohnoes the linux fanboys are telling everyone that windows will corrupt your data!1!!11 save the data!! won't ANYONE PLZZZZZzzz thknk of the data!
And as usual the desperately ignorant come to proclaim us fanboys.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Re: ServeMi, 23TB Server

Post by HFat » Thu Dec 15, 2011 4:02 am

HFat wrote:I'm not saying you're wrong but Microsoft is a serious business and it's not rocket science to get this right if the hardware is adequate.
Am I a fanboy because I didn't say Mokeh16 was wrong before hearing her arguments? Or because I suggested that Windows doesn't work well on inadequate hardware? I wonder.

Monkeh16
Posts: 507
Joined: Sun May 04, 2008 2:57 pm
Location: England

Re: ServeMi, 23TB Server

Post by Monkeh16 » Thu Dec 15, 2011 6:21 am

HFat wrote:
HFat wrote:I'm not saying you're wrong but Microsoft is a serious business and it's not rocket science to get this right if the hardware is adequate.
Am I a fanboy because I didn't say Mokeh16 was wrong before hearing her arguments? Or because I suggested that Windows doesn't work well on inadequate hardware? I wonder.
I'm quite male, thank you.

Post Reply