options for backup/syncing

mcbry · Post by **mcbry** » Fri Mar 20, 2009 5:53 am

Sry, I know this isn't really an SPCR topic, but I have seen some opinions expressed here on this topic. I just can't find them now...

I recently built my first desktop mostly following component advice I got here. I have 2 WD10EADS 1 TB harddrives. I bought two thinking to use one for backups. I now have the first drive with XP pro and separate partitions for personal files and media storage. the second drive is installed but still blank.

So, I'd like some advice concerning possible configurations and/or software (especially if its free!!) for facilitating the process, and other concerns I should consider in deciding how best to set up the backup /sync process between the two drives.

Also, I'd like to be able to sync files easily with my laptop.

I know this is a very newb question, so thanks for your patience and any advice or links to information would be greeatly appreciated.

Bry

InfyMcGirk · Post by **InfyMcGirk** » Fri Mar 20, 2009 6:49 am

If you're comfortable with command line tools, look into using 'robocopy' (included with XP). It's got plenty of powerful options if you read the docs allowing you to set up lists of folders to synchronise and that sort of thing.

Alternatively, if you prefer something prettier and friendlier looking, try Microsoft's 'SyncToy' (I believe it's a freely downloadable powertoy for XP).

If neither of these options sound attractive, you could try the built-in 'Backup' facility at Start > Programs > Accessories > System Tools > Backup.

I've never tried the last option but I've used the first two with good results. You could then set up scheduled tasks to automate your backups for convenient times, once you had the basic process set up.

If you want to survive a disk crash rather than just keep your data, you'll need to use something which can backup your entire disk including boot sector and that sort of thing. Something like Symantec Ghost or Acronis True Image is probably what's needed.

There are undoubtedly many other free and commercial tools for backups, of course!

One last option... if your disks are identical and your motherboard has built-in support for RAID1, you could set up a mirror and have all your data automatically written to both disks all the time. This would cost you some performance but mean that if either disk does down, you can still function as usual until you replace it (assuming the second disk doesn't fail in the mean time).

AZBrandon · Post by **AZBrandon** » Fri Mar 20, 2009 7:40 am

There's tons of ways to back up files, synctoy is probably one of the more popular windows ones, but the drawback is that you still don't have a way to recover a whole system. Entire disk failures are rare, but do happen. As was said, if you want a "bare metal restore" you need to be doing whole-disk backup images. I use Acronis True Image 11's boot CD capability. This way you boot up off the CD so the OS is totally idle and Acronis is able to create a backup image of the OS while it's static. I actually have two external drives and once a week I take one to my deposit box at the bank so even if my house burns down or thieves break in and steal everything, I still have a full backup of my PC.

Rebellious · Post by **Rebellious** » Fri Mar 20, 2009 4:47 pm

MS robocopy is powerful stuff and it's free. Mine runs at night with windows' scheduler (I have no idea if my ubuntu has a scheduler). My 2nd disk is ready to use any time so there's nothing to restore in case your 1st disk fails, in fact I boot it whenever I want to try out new software or experiment with something. But robocopy won't backup the registry so you need to do that separately, plus a minor registry tweak to make windows on 2nd think it's booting C. You don't need a "mirrored" disk as long as all the files are there, and you don't have to do anything to the boot sector, other than format the 2nd disk. See my post here: viewtopic.php?p=445542&highlight=#445542

K.Murx · Post by **K.Murx** » Fri Mar 20, 2009 4:57 pm

Rebellious wrote:(I have no idea if my ubuntu has a scheduler).

Go to a command line and type "man cron".
If you consider it not very user friendly, use Synaptic or apt-get to install gnome-schedule.
Tools to back stuff up are rsync (non-incremental) and rdiff-backup (incremental). Both are command-line-tools, but I am pretty sure there is a GUI around somewhere. rsync should be preinstalled, but you might need to get rdiff-backup.
For the usage, see the manpages ("man rsync" or "man rdiff-backup")

Rebellious · Post by **Rebellious** » Fri Mar 20, 2009 6:21 pm

Command line? Linux uses forward slashes

Do you realize what / does to a \/\/ \ /\/ D 0 \/\/ S user?

mcbry · Post by **mcbry** » Mon Mar 23, 2009 3:25 pm

Thanks for the input, all. That gives me some direction, at least, to focus my energy. A lot of this is just going over my head still, so it's good to have some place to start.

The Robocopy option sounds very interesting, InfyMcGirk and Rebellius. I'll look into that a little more closely. I don't really know much about using command lines. I've played around a little with Ubuntu, but not enough to make that learning curve actually start to curve. Anyway, I'm not afraid to learn new things.

Would you mind if I PM you Rebellious if I get stuck in my reading?

Thanks again all for your input.

Rebellious · Post by **Rebellious** » Mon Mar 23, 2009 5:33 pm

PM is fine, or just post here or in my other thread, there's always somebody else searching the web for the same info. Practice a bit with robocopy, it will do 99% of the work for you, and download regback from petri.

yensteel · Post by **yensteel** » Mon Mar 23, 2009 6:50 pm

I prefer mirrored backups, but I don't like raid 1. I use ICEMirror to do it instead. With a bit of scripting and windows scheduler you can have it automatically sync once every day, every hour, or every minute invisibly.

Also, The checking speed is really fast. it takes about a couple of seconds to check my entire hard disk for any changes, then overwrite or copy files at normal transfer speeds.

The only gripe I have with it is that it doesn't have volume shadow copy. Maybe it shouldn't. Also, it's only for windows.

http://www.ice-graphics.com/ICEMirror/IndexE.html

cb95014 · Post by **cb95014** » Tue Mar 24, 2009 11:10 pm

SyncBack is an outstanding rsync program that is available in free and licensed versions. I use the licensed SE version, but used the free version for months before upgrading. Even the freeware version is quite robust.

See: http://www.2brightsparks.com/downloads.html#freeware (scroll to the bottom) for the freeware version.

See: http://www.2brightsparks.com/syncback/s ... tures.html for the licensed SE version.

See: http://www.2brightsparks.com/syncback/compare.html for a feature comparison.

FWIW, consider putting the second drive in a separate enclosure, at least USB or eSATA attached, but preferably on the network in a different room. The key to good backups is completely separate copies, and RAID is definitely NOT a backup. The nice thing about NAS-attached backups is that they are completely coherent and do not need to be "restored."

These "2brightsparks" guys know how to write great code. For example, check out their FindOnClick tool - amazingly fast, and no index required.

Rebellious · Post by **Rebellious** » Wed Mar 25, 2009 4:20 am

cb95014 wrote:...
FWIW, consider putting the second drive in a separate enclosure, at least USB or eSATA attached, but preferably on the network in a different room. The key to good backups is completely separate copies, and RAID is definitely NOT a backup. The nice thing about NAS-attached backups is that they are completely coherent and do not need to be "restored."

...

I very much agree. My backup (referenced above) is a bootable disk and there is no need to restore anything.

MS never wanted to make it easy for people to make working copies of their disk, even though that is in their customers' best interest. Even if you had mobo failure and transferred the backup disk to new hardware, the OS is perfectly capable of re-detecting the new system and reconfiguring itself, but they won't do it.

Contrast that with Linux, my USB flash drive with Ubuntu will attempt to boot in ANY computer you insert it in, i.e. ideally your data is completely portable. We're not quite there yet, but that's the right way to do it, if we agree that the users' data is most valuable and that the OS is replaceable/disposable.

blackworx · Post by **blackworx** » Wed Mar 25, 2009 5:38 am

For whole drive imaging I use UBCD4Win + DriveImageXML. There are better solutions (like Acronis) but this one is completely free, plus UBCD4Win comes with an excellent range of system tools.

Building a UBCD4Win disc is not trivial, as many of the tools it contains tend to cause false positives with antivirus/security software, but the guide on the author's site is comprehensive and there are many other 3rd party guides both for UBCD and DriveImageXML.

The most important tip when using DriveImageXML itself is to make sure that, when you create a backup image, you completely defragment the source drive and make its partition not much bigger than that required for the drive contents - even if only temporarily. This is because, when restoring images, DriveImageXML can only restore to a same size or larger partition. The image file for a huge, mainly empty partition will not take up much extra room over and above the actual data, but you will be limited as to how and where you may restore that image.

Nadrek · Post by **Nadrek** » Thu Mar 26, 2009 7:07 pm

I can't address syncing with a laptop; I manually move files around and I don't put sensitive files on a laptop.

For backup/archive, I will give the short lecture:
Backups are done to improve the ability to recover to a (recent) known good configuration after a catastrophic event - drive dies, OS overwrites the drive entirely, accidental erasure of critical files (data, OS, program, whatever), OS/program/driver/kernel update goes bad, fire, flood, earthquake, theft, etc.

Archives are done to be able to see information from a certain point in time. Archives are best done to write-once media, and are common in situations where SEC regulations govern (i.e. show me how much money that trade was for 3 years ago... no, not what someone might have altered on your disk since then, pull the archive). They're beyond the scope of this discussion except to make the point that they are not backups (though they can sometimes be used as such)... and backups are not archives.

RAID is done to reduce one contributing factor in catastrophic events - the dead /erroneously reading hard drive, or to increase speed (IO/s per second, IOPS, or MB/s, throughput). RAID is not and cannot be a backup - it doesn't prevent catastrophic events, and also, all drives in a RAID are typically subject to the same environment (heat, power fluctuations, vibration, workload, etc.), and can easily die.

Buy drives from different vendors at somewhat different times by a couple months - reduce the chance that they're all from the same lot (remember the IBM Deathstars) and give yourself a greater chance of noticing drive batches that die fast (early 7200.11's) before too many drives die at once.

Your simplest backup solution is Acronis True Image (which I use in boot CD mode) or another drive imaging software, as others have mentioned. Either clone the disk directly to the other disk, or dump compressed copies on the other disk. This will only protect you against single-drive issues - the OS can still wipe both of them, and anything hitting the entire box (lightning) fries your primary and your backup.

Get your backup drive out of the box.

Note that if you can only put one backup on your backup drive, then every time you're in the middle of doing a backup, you have no backup - a single drive failure during a backup means you're dead in the water.

Therefore, you need to get a second drive.

A corollary is backup speed - note that Acronis does support differential drive images, which can be nice. Acronis is cheap - ~$50ish.

I see you also have a laptop - back that up too.

Another free, more traditional enterprise like solution is Bacula (Google it), which I intend to start using after my current round of "swap the failing hard drive/data from a failing server" is finished. This has agents (clients) that run on a variety of operating systems, and can theoretically do system backups (that restore on bare metal like a drive image would, but with more work), and does handle differential and incremental backups, multiple backup media, and so on.

#1: Back things up at all.

If nothing else, Start, Control Panel, Scheduled Tasks, and go to a batch file that copies the most important file from laptop to desktop and/or vice versa.

Beware of putting sensitive data on the laptop unless the drive is encrypted.

Encrypted drives require sector by sector backup if you use offline utilities (drive image off a CD).

No, it's not simple... start small.

Most important of all: Pick something that you can and will continue to do regularly.

AZBrandon · Post by **AZBrandon** » Thu Mar 26, 2009 10:51 pm

Nadrek wrote:Archives are done to be able to see information from a certain point in time. Archives are best done to write-once media, and are common in situations where SEC regulations govern (i.e. show me how much money that trade was for 3 years ago... no, not what someone might have altered on your disk since then, pull the archive). They're beyond the scope of this discussion except to make the point that they are not backups (though they can sometimes be used as such)... and backups are not archives.

Of the 4 major companies I've worked for, only one of them used WORM for their 7-year archives. The rest just used the regular TSM/NetBackup system to disk storage units (like VTL's) or magnetic tape. Of the 4, only that one company even seemed to take their backups seriously at all, the rest it was just considered an afterthought, like "Oh, well maybe we'll put some money in backups, but not too much."

Nadrek wrote:Another free, more traditional enterprise like solution is Bacula (Google it), which I intend to start using after my current round of "swap the failing hard drive/data from a failing server" is finished. This has agents (clients) that run on a variety of operating systems, and can theoretically do system backups (that restore on bare metal like a drive image would, but with more work), and does handle differential and incremental backups, multiple backup media, and so on.

I was hoping to see that Bacula was like TSM or NetBackup, but it doesn't even mention either of those in the FAQ. I'm a little puzzled by that, since those are the most widely used products for enterprise backups. I'll have to take a look at Bacula though for my nightlies. I only do boot CD Acronis backups once a week, and for the rest of the week I rely on their (horrible) scheduler to push backups to a USB attached drive. Their boot CD backups are great, their live OS backups are not so great, and their scheduler just plain stinks. I find it skips backups at least once a week.

Nadrek wrote:Most important of all: Pick something that you can and will continue to do regularly.

That's why I love scheduled backups. I hate doing my weeklies and admit to skipping sometimes, especially if I didn't do any major content creation that week. I'd love a more reliable way for doing LAN backups though, especially something like a home version of TSM, which is kind of an "incremental backups forever" product, if you've ever used it. Great for remote offices and places where you can bite the bullet and take a long period of time for the initial full backup and it's just incrementals after that. TSM isn't perfect (at least the last time I used it, several years ago) but it was great for preserving back end resources and bandwidth. It seemed less robust than NetBackup, but still easier on resources.

matt_garman · Post by **matt_garman** » Fri Mar 27, 2009 5:32 am

One idea I'll throw out that I didn't see in my quick skimming of this thread: get a third drive (a smaller/faster one, like a VelociRaptor or SSD) to use as your system drive, and use the two big drives in a RAID1 (mirrored) configuration. Your motherboard might support RAID1, or maybe Windows can do it in software (but I don't know, I'm a Linux guy).

Anyway, that's how I recently re-configured my system: VelociRaptor for the OS and when I need fast disk I/O, and two 5400 RPM (one Samsung, one WD) 1 TB drives in RAID1 for practically everything else.

I have a second tier of storage for true backup as well...

Nadrek wrote:RAID is done to reduce one contributing factor in catastrophic events - the dead /erroneously reading hard drive, or to increase speed (IO/s per second, IOPS, or MB/s, throughput). RAID is not and cannot be a backup - it doesn't prevent catastrophic events, and also, all drives in a RAID are typically subject to the same environment (heat, power fluctuations, vibration, workload, etc.), and can easily die.

That is true, and needs to be repeated often, because I get the impression that a lot of people equate RAID with backups. It also doesn't prevent against what (I'm guessing is) the most common catastrophic event---user error!

On the other hand, I think in some cases, it's appropriate to use as a "quasi" backup: e.g., say you rip all your movies and music to your system. Since you own the original backup media, you already have a true backup, but ripping (and encoding, if you do that) takes a lot of time and effort. But perhaps you can't afford to do a proper backup, given the sheer volume of data. In that case, something like RAID5 or RAID6 is a nice compromise, I think.

Anyway, while I use RAID1 as a pseudo-backup, I also have a real backup: a second system with a bunch of disks. A lot of people just use external drives; conceptually, that's all I'm doing, just on a bigger scale. I got myself a Norco RPC-4020, and put a couple drives in there. As my data set grows, I can add or swap drives as needed. That case is dead silent... when powered off.

It's quite loud actually, in stock form, but I only turn it on to do backups.

So getting back to the original poster: perhaps that is another suggestion, instead of getting a third drive, just take your second drive and put it in an external enclosure, or a second computer. Like Nadrek said, true backup needs to be physically separate from the source. Unfortunately, true backups are also boring, potentially expensive, and require discipline and time. And, ironically, the best backups are the ones you never need!

Rebellious · Post by **Rebellious** » Fri Mar 27, 2009 4:49 pm

Enterprise backups? OK, I have a question too then:

You have 2 networked computers at home. You run a script on Windows' scheduler on #1 and do a backup onto #2. But sometimes #2 is in S3 suspend mode. How do you wake up #2 from #1? How do you put #2 back to sleep after the backup is done?

//

options for backup/syncing

options for backup/syncing

ditto on robocopy....

Re: ditto on robocopy....

Try SyncBack

Re: Try SyncBack

Q