RAID5-like solutions without striping - a safer way?

Silencing hard drives, optical drives and other storage devices

Moderators: NeilBlanchard, Ralf Hutter, sthayashi, Lawrence Lee

Post Reply
shunx
Posts: 341
Joined: Sun Oct 06, 2002 1:20 pm
Location: Vancouver

RAID5-like solutions without striping - a safer way?

Post by shunx » Sat Aug 01, 2009 12:47 pm

While RAID5 and similar solutions such as Drobo provide redundancy without wasting space from mirroring as in RAID1 or WHS, the problem is that multiple drive failures will destroy the entire array. I also had experience where a RAID5 array in Linux was apparently taken out after the OS drive failed, but that might be a different issue.

One solution is to add a parity drive without striping the data drives, leaving each drive in a file system that can be read individually in case of drive faillure. This adds redundancy to all data without additional risk; if multiple drives fail, only those drives lose data, no worse than a scenario of single drives. The sole downside I can think of is slower performance from parity calculation and a lack of striping.

Implementations of this concept include UnRaid and FlexRaid, from what I've read. UnRaid is a commercial product and supports one parity drive. FlexRaid appears to be free, supports multiple parities, but doesn't have a long history yet. Anyone used one of these or knows of others?

nutball
*Lifetime Patron*
Posts: 1304
Joined: Thu Apr 10, 2003 7:16 am
Location: en.gb.uk

Post by nutball » Sun Aug 02, 2009 7:50 am

Maybe I'm missing something, but don't you risk losing more data if you don't stripe and concentrate all parity on one disc (which is RAID3, is it not?)

Have you considered RAID6?

shunx
Posts: 341
Joined: Sun Oct 06, 2002 1:20 pm
Location: Vancouver

Post by shunx » Tue Aug 04, 2009 3:26 am

Striping is only used for performance gain and you don't gain any safety from it, as far as I know. If the array fails to reconstruct part of a striped sequence, all of the striped sequence would be lost. In contrast, any loss in an unstriped array would be limited to data in the failed drives, unless there is a corruption that goes through every disk in the set.

RAID5 with a dedicated parity disk is like RAID3 or 4, which offers the same level of protection but with slower writes. All regular RAID levels except RAID1 use striping for data. A premise of RAID5 is that it would be rare for more than one drive to fail at a given time, but I believe this happens more often than one might imagine. In my case only one drive had failed, yet mdadm, the Linux RAID manager, stopped recognizing the array after the failure. I tried reconstructing the filesystem but what I retrieved in the end was mostly garbage.

All data in a RAID6 array will be lost if 3 drives fail at once for any reason, which to me appears slightly better than RAID5 but is not fundamentally so. While I can't prove that the use of data striping was the reason I was unable to recover my data, it seems likely that leaving the data on each disk independent of each other would increase the probability of data integrity.

What this indicates is that while RAID2/3/4/5/6 may offer protection against a one- or two-drive failure, all of them may also increase risk in the way of forcing every bit of data to be dependent on many other bits in the array for survival.

By the way, note to anyone trying out RAID5: do NOT install the OS on the same drive as the RAID array.

dhanson865
Posts: 2198
Joined: Thu Feb 10, 2005 11:20 am
Location: TN, USA

Post by dhanson865 » Tue Aug 04, 2009 5:32 am

Just keep it simple and do RAID 1 if you want redundancy.

No parity checksums but then you have more full drives in redundant roles.

If you want to read some anti RAID 5 thoughts there are plenty quoted at viewtopic.php?p=388987. along with some thoughts about other RAID levels.
.
Please put a country in your profile if you haven't already.
This site is international but I'll assume you are in the US if you don't tell me otherwise.
RAID levels thread http://www.silentpcreview.com/forums/viewtopic.php?p=388987

shunx
Posts: 341
Joined: Sun Oct 06, 2002 1:20 pm
Location: Vancouver

Post by shunx » Tue Aug 04, 2009 12:37 pm

RAID1's wasted space is 50%, which is practical mostly when you don't have a lot of files. If you have terabytes of data, the parity drives quickly add up. Higher RAID levels are desirable in that they lower wasted space to the order of 1/n drives, yet they are undesirable in the additional problems they create, making them a double-edged sword. Unstriping a RAID4/5/6 type of system retains the same redundancy with no real additional risk compared to a scenario of individual disks. That is ideal for a home setting where maximum performance is not crucial as long as it's usable, such as for a HTPC.

The concept is new and unexplored by most, leading to a lack of available options. UnRAID seems to work fairly well for its users based on postings. You have to buy a license that's tied to a physical USB key. FlexRAID currently creates parity only intermittently, though the author promises a version similar to UnRAID that's long in coming. Some other solutions may be in development from what I've read.

dhanson865
Posts: 2198
Joined: Thu Feb 10, 2005 11:20 am
Location: TN, USA

Post by dhanson865 » Wed Aug 05, 2009 4:48 am

The only upside to an SPCR regular is that unRAID allows drives to spin down independently so noise would drop when the array is idle.

The huge down side to unRAID is that the parity disk is a single point of failure. If that specific disk dies you now have a JBOD instead of a RAID. They compare it to RAID4 without striping but I'd call it JBOD + a parity disk.

The only two RAID levels that interest me are RAID 1 and RAID 10. I'll pay the space penalty as I'm almost always more concerned about performance than I am about space and I'm definitely more concerned about reliability than I am about space.

The only thing that changes the game significantly is ZFS or similar where the file system structure doesn't care what drives it is spanned across so you can or it can mix raid levels as needed. But for small companies and home users this is overkill. I'll stick with simple RAID 1 or RAID 10 for now.
.
Please put a country in your profile if you haven't already.
This site is international but I'll assume you are in the US if you don't tell me otherwise.
RAID levels thread http://www.silentpcreview.com/forums/viewtopic.php?p=388987

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Wed Aug 05, 2009 9:28 am

dhanson865 wrote:The huge down side to unRAID is that the parity disk is a single point of failure. If that specific disk dies you now have a JBOD instead of a RAID. They compare it to RAID4 without striping but I'd call it JBOD + a parity disk.
If you can loose the parity disk and it reverts to a JBOD without loosing any data, it isn't a single point of failure :wink:


Also, the only functional difference between RAID 4 (dedicated parity) and RAID 5 (distributed parity) is performance, not redundancy... the parity block can be calculated no matter which stripe is missing, or which drive it lies on.

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Wed Aug 05, 2009 9:58 am

Twiek wrote:If you can loose the parity disk and it reverts to a JBOD without loosing any data, it isn't a single point of failure :wink:
Agreed. Drives in an unRAID array are self-contained, utilize ReiserFS (at least the version I use) and are individually accessible in virtually all Linux-based systems. Iirc, I was able to view the drives' contents using an Ubuntu live cd. I believe there might also be ReiserFS drivers available for Windows. That's one of the reasons I chose to go with unRAID. Even if the developer stops work on the software or my flash drive gets fried, I still have access to my data since each drive has its own file system.

Really, the only downside to unRAID is performance (or lack thereof) and the somewhat limited hardware support. If you have a large array of archive/mostly static data (e.g. media server), unRAID works pretty well.

Sure, RAID1 or RAID10 would be nice, but it's not really feasible if you need multi-terabyte storage with a small budget. I currently have a 12TB unRAID server (8x1.5TB data, 1x1.5TB parity). Assuming current hard drive prices, unRAID would cost $1080, RAID1 - $1920, RAID10 - $3840. Those prices are just for hard drives, without supporting hardware. It's much more expensive to house 16 or even 32 drives compared to just 9.

Terje
Posts: 86
Joined: Wed Sep 08, 2004 4:50 am

Post by Terje » Wed Aug 05, 2009 10:25 am

shunx wrote:SA premise of RAID5 is that it would be rare for more than one drive to fail at a given time, but I believe this happens more often than one might imagine.
The main problem is probably that most people think "disk stops working" when they think failures, but you just need a few blocks failing on another disk and you are in trouble. A lot of raid system get trouble recovering from that and even if they can recover, failing disk blocks might put the drive itself into a state were it refuses to work so you need a way to "skip" bad areas of the drive, and that is something I have yet to see support for in raid systems for recovery purposes.

Thats were raid6 really shows its strengths.

I don't see how any of the other parity based methods are any different here in terms of redundancy be it that you do block level parity, filesystem level parity or anything else. If you are lucky, whatever device you have will be able to at least recover all the data that has working blocks, more often than not, you are not that lucky and the entire recovery just craps out and you loose it all.

You need replication or mirroring (raid1) of data to be relatively safe when stuff like this happens.

In the end, I have seen more data lost on buggy raid software and hardware, corrupted filesystems or human mistakes than I have due to multi disk failures in raid 5 or other setups. Which raid is the most failure safe considering failure rates on drives etc. etc. are interesting theoretical numeric games that definitely are very good to understand, but in the end, they are rarely my top concern when configuring servers.

At home, I use raid 1 for storage of important data and raid 5 and 0 where I need capacity.

I got a spare SATA and power cable laying behind the cover in one of the 5 1/4" bays on my server and I hook up a 1TB drive there and dump a backup every now and then which I store at my parents house.

I got 2 SSDs in a raid 0 on my desktop windows 7 PC. I don't normally store important data there, but I happened to have a spare 1TB SATA drive around when I built it and I just set up the built in backup software in win 7 to do regular backups of the SSD to the 1TB Sata drives.

Darned simple to set up, the software is as basic as backup software can be, but its actually doing the job surprisingly well.

It does not protect me completely against my own stupidity or a crazy virus trying to delete all my data, but its good enough since I normally got the important stuff on the server anyway and its a good fail save when you run crazy stuff like raid 0.

On my imac, I got an old 500GB USB drive and time machine. Just flip the drive on every now and then before I go to bed.

Saved my ass 2 months ago when the imac disk stopped working.

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Wed Aug 05, 2009 11:54 am

Terje wrote:The main problem is probably that most people think "disk stops working" when they think failures, but you just need a few blocks failing on another disk and you are in trouble. A lot of raid system get trouble recovering from that and even if they can recover, failing disk blocks might put the drive itself into a state were it refuses to work so you need a way to "skip" bad areas of the drive, and that is something I have yet to see support for in raid systems for recovery purposes.

Thats were raid6 really shows its strengths.

I don't see how any of the other parity based methods are any different here in terms of redundancy be it that you do block level parity, filesystem level parity or anything else. If you are lucky, whatever device you have will be able to at least recover all the data that has working blocks, more often than not, you are not that lucky and the entire recovery just craps out and you loose it all.
You're thinking in terms of traditional RAID systems while the OP is researching less conventional data storage solutions. What you've said is true but they mostly apply only to striped systems. FlexRAID and unRAID work differently.

As I mentioned before, all drives in unRAID work independently. It just has an additional layer of protection due to the parity drive. Similar to RAID5, it can sustain loss of a single-drive without losing any data. However, if two or more drives fail, then only data on those drives are lost. The data on the rest of the drives remain intact. Bad blocks on hard drive? You only lose data on those blocks. Heck, if your parity is intact, unRAID will flag those blocks as bad and recover the data elsewhere on the hard drive.

Would I keep my one and only copy of irreplaceable data on unRAID? Heck, no! I've got copies of important documents, family photos and videos, etc on all computers at home (thank you, Dropbox). However, unRAID does work very nicely as a semi-protected storage for Blu-Ray, HD-DVD and DVD rips without the pitfalls inherent in RAID5.

Unfortunately, I can't really speak much about FlexRAID other than it being "snapshot RAID".

As you mentioned, though, people should use the appropriate level of protection for their data. Another thing to keep in mind, RAID is not designed for backup, it's for uptime. Even RAID1 is not completely safe. You should always keep back-ups of important data preferrably with at least one off-site. If a fire hits your home, RAID1 or RAID10 isn't going to do you any good.

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Wed Aug 05, 2009 12:14 pm

Terje wrote:The main problem is probably that most people think "disk stops working" when they think failures, but you just need a few blocks failing on another disk and you are in trouble. A lot of raid system get trouble recovering from that and even if they can recover, failing disk blocks might put the drive itself into a state were it refuses to work so you need a way to "skip" bad areas of the drive, and that is something I have yet to see support for in raid systems for recovery purposes.

Thats were raid6 really shows its strengths.
Exactly. RAID 5 can't rebuild an array if it encounters an unrecoverable read error (after replacing a bad drive). This has become a problem in recent years because the unrecoverable read error rate hasn't gotten any better as drives have gotten bigger, and while the error rate used to be two or three orders of magnitude smaller than the size of the drives, they're not anymore... One is now likely to encounter an error during a rebuild on the larger arrays many SPCR followers have. RAID 6 does provide redundancy against simultaneous drive failure, but it's mostly used (at least in enterprise) to keep URE's at bay.

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Wed Aug 05, 2009 12:17 pm

ilovejedd wrote:As you mentioned, though, people should use the appropriate level of protection for their data. Another thing to keep in mind, RAID is not designed for backup, it's for uptime. Even RAID1 is not completely safe. You should always keep back-ups of important data preferrably with at least one off-site. If a fire hits your home, RAID1 or RAID10 isn't going to do you any good.
Absolutely. One of the fundamental rules in IT is that "RAID != Backup"

I'm personally starting to lean towards non-standard "softer" RAIDs like unRAID or FlexRAID that the OP talked about for home/HTPC storage... it just fits the usage better.
Last edited by Twiek on Wed Aug 05, 2009 12:24 pm, edited 2 times in total.

dhanson865
Posts: 2198
Joined: Thu Feb 10, 2005 11:20 am
Location: TN, USA

Post by dhanson865 » Wed Aug 05, 2009 12:23 pm

Twiek wrote:
dhanson865 wrote:The huge down side to unRAID is that the parity disk is a single point of failure. If that specific disk dies you now have a JBOD instead of a RAID. They compare it to RAID4 without striping but I'd call it JBOD + a parity disk.
If you can loose the parity disk and it reverts to a JBOD without loosing any data, it isn't a single point of failure :wink:
Touche!

unRAID claims to handle a dropped drive scenario better than RAID 5/6 do. I've often wondered why traditional RAID controllers don't just turn the raid array into a read only array when the drives are marginal instead of dropping the array entirely no matter how small the errors are.

If I have a RAID 1/10/single drive boot partition and RAID 5 data array I'd like to have the choice to chuck the data or retrieve it once 2 drives "fail". All the controllers I've seen act like the array doesn't exist at all if 2 drives lose integrity. I'd be happy if the read only array were a controller boot option for recovery instead of on the fly automatic behaviour but I haven't even seen that.
.
Please put a country in your profile if you haven't already.
This site is international but I'll assume you are in the US if you don't tell me otherwise.
RAID levels thread http://www.silentpcreview.com/forums/viewtopic.php?p=388987

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Wed Aug 05, 2009 12:51 pm

dhanson865 wrote:unRAID claims to handle a dropped drive scenario better than RAID 5/6 do.
It should and it does. Assuming you have 3 individual hard drives installed on your desktop (let's say C:, D: and E: in Windows) all of them configured as Basic Disk formatted as NTFS. C:, as per Windows default, contains the operating system, etc and the other two contain plain data.

If E: fails and the BIOS wouldn't even recognize it anymore, does C: or D: care? Not really. You'd still be able to boot into Windows and everything and access all the data stored in C: and D:. You just lose all data on E:. Same thing with unRAID, except, assuming you have a working parity drive, you'll be able to rebuild/recover the content of E:.

Aside from the parity drive, the other (data) drives don't know and don't care about each other's existence.

fyleow
Posts: 175
Joined: Sun Sep 11, 2005 4:23 pm
Location: California, USA

Post by fyleow » Wed Aug 05, 2009 2:59 pm

ZFS can reduce the risk of encountering an unrecoverable read error during a critical time such as rebuilding when compared with more traditional RAID5 implementations.

-ZFS stores multiple copies of metadata even on a single drive so if corruption happens to metadata it can be recovered (Had this happen to me on a boot drive actually).

-ZFS can be set to scrub the data regularly which will do a checksum of all the data on the drive. Read errors are likely to be caught in this way.

-ZFS raid is aware of the underlying filesystem. On a rebuild it does not need to rebuild every bit on the drive, only the space that is being used. This means faster rebuilds and no unnecessary data read.

http://www.mail-archive.com/zfs-discuss ... 17368.html

The solution is quite simple actually. Take regular backups. You don't need redundancy on the backups and using JBOD it's not that expensive.

Terje
Posts: 86
Joined: Wed Sep 08, 2004 4:50 am

Post by Terje » Wed Aug 05, 2009 5:54 pm

ilovejedd wrote:You're thinking in terms of traditional RAID systems while the OP is researching less conventional data storage solutions. What you've said is true but they mostly apply only to striped systems. FlexRAID and unRAID work differently.

As I mentioned before, all drives in unRAID work independently. It just has an additional layer of protection due to the parity drive. Similar to RAID5, it can sustain loss of a single-drive without losing any data. However, if two or more drives fail, then only data on those drives are lost. The data on the rest of the drives remain intact. Bad blocks on hard drive? You only lose data on those blocks. Heck, if your parity is intact, unRAID will flag those blocks as bad and recover the data elsewhere on the hard drive.
Yes, you are right, unRaid can potentially be better in the sense that it would be easier to recover parts of the data.

The probability of loosing data is equally high though and since you don't really know what you will be loosing when the accident is there, I don't really see the big difference in terms of data protection unless you only use it for different types of scratch data (movie/tv recording storage might be a good usage as you suggest).

Terje
Posts: 86
Joined: Wed Sep 08, 2004 4:50 am

Post by Terje » Wed Aug 05, 2009 6:30 pm

Twiek wrote: Exactly. RAID 5 can't rebuild an array if it encounters an unrecoverable read error (after replacing a bad drive). This has become a problem in recent years because the unrecoverable read error rate hasn't gotten any better as drives have gotten bigger, and while the error rate used to be two or three orders of magnitude smaller than the size of the drives, they're not anymore... One is now likely to encounter an error during a rebuild on the larger arrays many SPCR followers have. RAID 6 does provide redundancy against simultaneous drive failure, but it's mostly used (at least in enterprise) to keep URE's at bay.
While I know the theory behind this the recent focus on URE's and raid 5 very well, I have to admit that in real life, I don't see any massive increase of these errors from what I did 17 years ago when I started working with large system environments.

If all of this were reality and not just play with the MTBF like numbers, you should see people loose data all the time on their 1TB drives (not just in raid setup) and there would probably be a massive uprising on the way about 1TB drives in general.

I frankly see no more problems with 1TB drives in terms of URE's than I did with with the first 20MB drive I bough back in the 80s or my first floppy. I guarantee you. That equipment had much worse performance in terms of UREs :)

Yes, I know the vendors numbers have not really improved in terms of URE's vs. number of bytes per drive, I have read the "famous" Zdnet that started all the talk about URE's as well. That article smells of sponsoring from alternative storage vendors.

Nothing wrong in that of course. The more highlighting of potential problems you get, the more vendors will improve.

In my very personal opinion, it was much more common back in the 80's and 90's that drives gradually "degraded" getting more and more block errors before they failed.

Drives today, in my experience, is much more likely to just die (which, strangely enough, is probably an advantage in this context).

I don't know why, never tried to figure that out, but its highly likely that it is a result of the mechanical parts and heads getting smaller and less robust as well as operation limits on the mechanics has become much stricter as data is stored more and more densely. 20 years ago, there might have been some "room" for wear and tear, today it either works perfectly or it does not work at all.

Just a theory, but I suspect its at least a bit truth in it :)

The biggest difference is probably that there is a hell of a lot more raid 5's out there based on "commodity" drives now than before so you are much more likely to experience problems like this today than 20 years ago in general.

Who on earth could afford enough disk drives for a raid 5 for private use 20 years ago anyway :)

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Thu Aug 06, 2009 4:20 am

Terje wrote:While I know the theory behind this the recent focus on URE's and raid 5 very well, I have to admit that in real life, I don't see any massive increase of these errors from what I did 17 years ago when I started working with large system environments.

If all of this were reality and not just play with the MTBF like numbers, you should see people loose data all the time on their 1TB drives (not just in raid setup) and there would probably be a massive uprising on the way about 1TB drives in general.

I frankly see no more problems with 1TB drives in terms of URE's than I did with with the first 20MB drive I bough back in the 80s or my first floppy. I guarantee you. That equipment had much worse performance in terms of UREs :)
It's not the 1+TB drives themselves that cause the issue, it's the fact that we're now building huge, 10+TB arrays out of them, adding another order of magnitude to the size. And in normal, non-RAID use, URE's don't cause major problems like they do in RAID 5's... so "normal" users of large drives aren't going to encounter show-stopping URE's.

I'll admit that I've never encountered a problem with URE's on the 500-1000 GB SAS RAID 5's that I work on most of the time, but that's not exactly the situation we're discusssing :wink:

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Thu Aug 06, 2009 4:27 am

fyleow wrote:The solution is quite simple actually. Take regular backups. You don't need redundancy on the backups and using JBOD it's not that expensive.
It's still far more expensive than one additional drive for parity, and often not really an option on the huge arrays many of the users here have or plan to build. I mean, how do you backup a 10TB JBOD? Another 10TB JBOD? Talk about expensive...

nutball
*Lifetime Patron*
Posts: 1304
Joined: Thu Apr 10, 2003 7:16 am
Location: en.gb.uk

Post by nutball » Thu Aug 06, 2009 10:27 am

Twiek wrote:
fyleow wrote:The solution is quite simple actually. Take regular backups. You don't need redundancy on the backups and using JBOD it's not that expensive.
It's still far more expensive than one additional drive for parity, and often not really an option on the huge arrays many of the users here have or plan to build. I mean, how do you backup a 10TB JBOD? Another 10TB JBOD? Talk about expensive...
Isn't that called RAID1?

And yeah it's expensive but as was said above if that's too expensive then your data isn't worth what you think it's worth.

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Thu Aug 06, 2009 11:05 am

nutball wrote:Isn't that called RAID1.
No, it's not the same. RAID1 is done in real-time and if you do something stupid or get hit by a virus, both drives are affected. He did mention JBOD. It could be in one case, or it could be a bunch of external USB hard drives. Heck, you can even mix and match. Either way, if you've got 10TB worth of data, then a full backup is going to be expensive. Maybe unreasonably so depending on what the data is.
nutball wrote:And yeah it's expensive but as was said above if that's too expensive then your data isn't worth what you think it's worth.
Sure, the data isn't worth much, but the time and effort spent to build the data collection might be as seems to be the case with the OP. Looks to me, he plans on using it as an HTPC/media storage, in which case, unRAID or similar solution is ideal. The data isn't absolutely critical, but it would still be a major pain to re-rip all those DVDs in the event of drive failure. With unRAID, you're making a compromise between the level of protection and cost.

By the way, I believe there are plans to add an extra parity drive to unRAID - that's if it's not in the beta yet. Haven't checked for updates in a long time since my unRAID server just works.

As for FlexRAID, as far as I know, you can assign as much parity as you want on a set of data. Parity calculations are based on files/data and not hard drive blocks.

Twiek
Posts: 6
Joined: Fri May 01, 2009 12:36 pm
Location: Kentucky

Post by Twiek » Thu Aug 06, 2009 11:15 am

nutball wrote:
Twiek wrote:
fyleow wrote:The solution is quite simple actually. Take regular backups. You don't need redundancy on the backups and using JBOD it's not that expensive.
It's still far more expensive than one additional drive for parity, and often not really an option on the huge arrays many of the users here have or plan to build. I mean, how do you backup a 10TB JBOD? Another 10TB JBOD? Talk about expensive...
Isn't that called RAID1?

And yeah it's expensive but as was said above if that's too expensive then your data isn't worth what you think it's worth.
If you can say to yourself that's too expensive, then your data is worth exactly what you think it's worth.

I think that's the whole point the OP was making... some things simply aren't worth spending $1000 to back up... but that doesn't mean they're worthless.

The whole concept of "go RAID 1 or go home" is just ridiculous. There's definitely a middle ground between worthless and critical data, and striping methods like RAID 5/6 or unRAID/FlexRAID/etc are usually just what large array users are looking for... Say you have 6TB worth of movie ISOs ripped to a JBOD. Copyright arguments aside, that takes a lot of time and effort to do. To me, it's worth spending an extra $80 for some fault tolerance, as it would prevent me from having to re-rip a whole bunch of discs should a drive fail. However, it certainly isn't worth paying several hundred dollars for a second JBOD to keep a complete copy. To me, for that use, they're functionally the same.

I don't think many people are keeping 5+ TB of super critical data on these arrays, and as such, parity does exactly what they're looking for: fault tolerance. Why spend all that money when a single drive will accomplish what they want?

Edit: Looks like ilovejedd beat me to the punch :)

fyleow
Posts: 175
Joined: Sun Sep 11, 2005 4:23 pm
Location: California, USA

Post by fyleow » Thu Aug 06, 2009 12:23 pm

Twiek wrote:
fyleow wrote:The solution is quite simple actually. Take regular backups. You don't need redundancy on the backups and using JBOD it's not that expensive.
It's still far more expensive than one additional drive for parity, and often not really an option on the huge arrays many of the users here have or plan to build. I mean, how do you backup a 10TB JBOD? Another 10TB JBOD? Talk about expensive...
The setup you're talking about is different from what I suggested. I'm suggesting using a raidz1 (which is like a RAID5 and tolerates one failed drive) and then using a JBOD in addition to that to backup data that is important.

If all 10 TB is critical irreplaceable data then yes, build a 10 TB JBOD to back that up. If it's replaceable then keep it on the RAID5 without the additional backup.

For my own personal setup I have a 3TB array but I only backup about 100 GB. Only 100 GB is critical data so I use an external drive for that.
Isn't that called RAID1?
No that is not called RAID1. A backup on a separate system with JBOD can be put in a different physical location as a safeguard against natural disasters like fires. If you're just using an external hard drive for the backup you can power if off and store it safely away etc. It also won't write the changes immediately so an accidental deletion or corruption won't take out your data. You can also use an rdiff-backup strategy to keep snapshots of your data at different times to rollback. JBOD also allows you to use different sized drives which works well because most people can just throw in drives they have lying around to build it.

If someone wants additional safeguards without spending too much then just step up to the next level of raidz2 (RAID6 for others) which uses 2 parity drives. The cost isn't much more and the chances of failure during a rebuild is reduced.

Another nice thing about ZFS is it allows you to take snapshots instantly so it's trivial to protect against accidental deletions (at the cost of not being able to reclaim the space of deleted items immediately of course)

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Thu Aug 06, 2009 12:45 pm

fyleow wrote:Another nice thing about ZFS is it allows you to take snapshots instantly so it's trivial to protect against accidental deletions (at the cost of not being able to reclaim the space of deleted items immediately of course)
Out of curiosity, is ZFS still Solaris only?

It was one of the solutions I was considering when I was researching how I should store my media, but the almost "for-dummies" simplicity of unRAID won me over.

Terje
Posts: 86
Joined: Wed Sep 08, 2004 4:50 am

Post by Terje » Wed Aug 12, 2009 5:05 am

ilovejedd wrote:
fyleow wrote:Another nice thing about ZFS is it allows you to take snapshots instantly so it's trivial to protect against accidental deletions (at the cost of not being able to reclaim the space of deleted items immediately of course)
Out of curiosity, is ZFS still Solaris only?
No, its not, but its license is currently not compatible with GPL, so it cannot be integrated with the linux kernel.

There is a port for FUSE (user space filesystems which does not require any kernel changes) but I think most people see that one more as a bit of a curiosity than for real use due to performance issues with FUSE

Terje
Posts: 86
Joined: Wed Sep 08, 2004 4:50 am

Post by Terje » Wed Aug 12, 2009 6:49 am

Twiek wrote: It's not the 1+TB drives themselves that cause the issue, it's the fact that we're now building huge, 10+TB arrays out of them, adding another order of magnitude to the size. And in normal, non-RAID use, URE's don't cause major problems like they do in RAID 5's... so "normal" users of large drives aren't going to encounter show-stopping URE's.

I'll admit that I've never encountered a problem with URE's on the 500-1000 GB SAS RAID 5's that I work on most of the time, but that's not exactly the situation we're discusssing :wink:
URE - Unrecoverable Read Error.
That is Unrecoverable as in data lost

Yes, it has potentially worse effects in a 10TB raid 5 than in a single drive, but I am willing to bet that there are much more than 10x people with single drives than there are people with 10TB raids out there.

There should be a heck of a lot of people with single drives around by now which are loosing or getting corrupt data at random place, but I see no evidence of that. Anyway, the data used for that article seems to be based on some dated info.

For instance
http://www.wdc.com/en/library/sata/2879-701229.pdf
The entire WDC GP drive series for instance, all <10^15 for URE.
(while the zdnet article claims that the norms is 10^14 and not improving).

A quick search on the net shows that maxtor diamondmax sata drives had 10^15 URE rates back in 2003 already (although I don't really care, maxtor SATA drives in 2003 was endlessly more likely to fail than a 10^14 URE seagate barracuda).

(note, all these are "max" numbers. Not average or min number. This is worst case).

Interesting enough, seagate seems to have removed URE type numbers from their latest desktop drive spec sheets.

Regardless, I have been responsible for datacenter setups before with hundreds of servers in raid 5. I don't run datacenters anymore, but I work with customers that has hundreds of servers with raid 5.

And actually.. I am one of the weirdos with more than 10GB of disk at home....

Looking aside from the occasional bad disk model/series, I see no indication whatsoever that disks fail more today than they did 5, 10 or 15 years ago. I am willing to bet that we had a lot more disk failures 15 years ago vs. the number of disks in use, but maybe I am getting old and my memory is getting bad (which should really have me talking about the great old days I think....)

I personally think that these numbers mostly haven't changed in the document literature because disc vendors want to be as much on the safe side as they can in case the shit hits the fan and they sell a disk series that has problems and they get sued. If the numbers aren't useful in marketing there is never any point in promising anything more than you need.

We can only hope that we get more updated and competitive numbers from disk vendors as a result of this. That would be a win for everybody.

For myself, I am no more worried about my raid 5 data today than I was 5 years ago. I don't trust parity based raid for the important data unless I have a backup setup that compensates for the risk. All experience I got shows that it is much more likely that the raid software or filesystem will mess itself up that multi disk failures.

Important data with somewhat relaxed backup strategy (which you usually have at home) is always raid 1 for me.

Jay_S
*Lifetime Patron*
Posts: 713
Joined: Fri Feb 10, 2006 2:50 pm
Location: Milwaukee, WI

Post by Jay_S » Wed Aug 12, 2009 1:52 pm

FreeNAS (www.freenas.org) versions 0.7x (currently in RC stage) feature kernel-level ZFS. The ZFS version in FreeBSD is more advanced than the one in FreeNAS (FreeNAS is always a few steps behind the parent OS).

My biggest problem with ZFS - and the reason I went with unRAID for my media server - is expansion. Growing ZFS pools is non-trivial, to say the least.

I love the ZFS feature set though, especially the checksumming and snapshotting for bit-death and corruption protection/prevention.
it's the only jib I got, baby

ilovejedd
Posts: 676
Joined: Tue Oct 21, 2008 4:14 pm
Location: in the depths of hell

Post by ilovejedd » Wed Aug 12, 2009 2:49 pm

Terje wrote:And actually.. I am one of the weirdos with more than 10GB of disk at home....
I don't think it's that rare anymore.

...unless of course, your only computer is one those early EEE PC netbooks with 7" screens. :P
Jay_S wrote:My biggest problem with ZFS - and the reason I went with unRAID for my media server - is expansion. Growing ZFS pools is non-trivial, to say the least.
I hear 'ya. That's another reason I went with unRAID. I've read a lot of horror stories regarding RAID-5 expansion and I really don't have the dough to plunk on a decent RAID-6 capable card.

The unRAID approach to storage expansion is much less heart-attack inducing.

HFat
Posts: 1753
Joined: Thu Jul 03, 2008 4:27 am
Location: Switzerland

Post by HFat » Thu Aug 13, 2009 10:24 am

Parity isn't magic.
Though RAID1 is the gold standard, you're not taking too many chances with two or three data drives and a parity drive. But 8 data drives per parity drive and no backup? n+1 drives means your risk is x to the power of n. I don't know how low you figure x is...
This proprietary system might be better than RAID4 for some purposes but that doesn't mean one should be blind to the risk of scaling n up.

The most obvious solution to lack of dinero + heaps of worthless data is deleting.

Post Reply