Windows Backup Woes

I’m finding myself in the awkward position of considering the purchase of a Symantec product…

Recently, I decided to reconfigure our backups at work. This wasn’t a bad idea, as our current system of plugging USB drives into the server based on the day of the week was…not the best plan. Windows Server doesn’t particularly care for random device removal and the drives probably don’t quite like it either. Also, the drives were stored in our server room, which is shared with the upstairs neighbors and I didn’t like going in there every day (or having someone else go in if I was gone).

To rectify this, I decided that a networked storage location was in order. Unfortunately, I didn’t have another server lying around. Fortunately, I did have a little netbook sitting on a shelf that no one was using and a pile of USB drives that I could connect to it.

A little while later, the drives were shared and I was testing out my backup settings. The problem was, Windows Backup was slow. Like 18 hours for 10 GB slow. Since I wanted a daily backup and not an “about every three days depending on luck and the phase of the moon” backup, this really wasn’t going to work.

My next plan was robocopy scripting. Now, robocopy is a good program for file backup but it has some limitations (like any program does…). The two limitations that affect my situation are:

robocopy cannot copy “in use” files. That is, any file that the system is currently working on or has been locked in some other way is unavailable for a copy.
robocopy can’t efficiently copy large files (10+GB). This is primarily due to robocopy using buffered I/O calls that significantly slow its copying for files that would require lots of buffering.

So…robocopy on its own was out, but I didn’t want to abandon it yet…

Enter diskshadow and xcopy.

Diskshadow is the command line interface to Windows Volume Shadow Services. This tool allows you to make a copy of those “in use” and locked files so you can actually access the file properly for backup. Diskshadow isn’t terribly easy to use, but definitely works well and covers my first issue with robocopy.

Xcopy is an old (and supposedly deprecated) Microsoft tool that robocopy supposedly replaces. The catch is, xcopy can do unbuffered I/O. This takes care of my second problem. I’ll simply use xcopy to transfer the mailbox database and use diskshadow and robocopy for everything else.

I was feeling pretty good about this. Unfortunately, my first test taught me otherwise. Attempting to copy the mailbox database with xcopy caused the server to hard lock (likely to to an overuse of memory in xcopy that wasn’t properly handled). Well, I’m not terribly interested in continuing to test this tool, so it looked like I’d have to keep searching for something that can handle my backup requirements.

At this point, I tried out Cobian Backup. It seemed to actually be able to copy everything without locking up the server or taking forever and a day. But it’s scheduling didn’t actually work. Regardless of how or when I scheduled backup times, Cobian happily sat, running, on the server and did nothing until I manually started the backup (at which point it rather handily copied everything). Low and behold, Cobian had a cli interface that could get me around the scheduling problems. A little bit of tweaking later, it was actually running at scheduled times and successfully backing up the data from my server.

So…with a bit of clever scripting and some persistence, I managed to create a network backup of my server for free. But I’m not entirely comfortable with the tools I’m currently using. Cobian works, but it doesn’t have very good logging or verification tools. Also, with a large Exchange database like we have, I really want something that can actually hook into Exchange and back everything up, which my current plan definitely doesn’t do.

In pursuit of that, I hoped on reddit and started asking questions: http://www.reddit.com/r/sysadmin/comments/nwejy/looking_for_a_windows_server_backup_solution_for/

So far I’ve gotten some interesting replies, but after poking around at the suggestions, Symantec’s BackupExec is looking like it may be my best bet…

Edit: So eseutil is a tool for repairing Exchange databases. It can also do unbuffered I/O copying (like XCopy). I didn’t try it when I was experimenting with built-in tools because XCopy burned me so bad. Turns out it works tremendously well (2 hours for 40 GB) and doesn’t hard lock the server. Hurray! I get to use robocopy, diskshadow, and eseutil to back everything up!

So…here’s the completed script (with some directories removed):

@ECHO OFF


SET SQLCMDDIR=
SET SAGEDBBACKUPDIR=
SET CMDLOCATION=
SET BACKUPTARGET=
SET BACKUPDRIVE=
SET LOGLOCATION=
::Number of days to store Sage backups
SET NUMDAYS=30

::Remove old finance backups created by Sage
FORFILES /P %SAGEDBBACKUPDIR% /M *.NPS /D -%NUMDAYS% /C "CMD /C DEL /Q @FILE" 2>NUL
::Remove old SQL dumps
DEL /S /Q "D:\SQL Backup\*"
::Clean up VSS metadata
DEL /S /Q %LOGLOCATION%\diskshadow.cab

::Mount remote directory
IF NOT EXIST %BACKUPDRIVE\. NET USE %BACKUPDRIVE% %BACKUPTARGET%

ECHO Executing SQL commands
::The log directive for SQLCMD (-o) appears to fail rather spectacularly if you try to use a variable
::so hard-coded paths it is!
%SQLCMDDIR%\SQLCMD.EXE -S -i %CMDLOCATION%\SQL\script1.sql -o "D:\Logs\Backup\Script1Backup.log"
%SQLCMDDIR%\SQLCMD.EXE -S -i %CMDLOCATION%\SQL\script2.sql -o "D:\Logs\Backup\script2Backup.log"
%SQLCMDDIR%\SQLCMD.EXE -S -i %CMDLOCATION%\SQL\script3.sql -o "D:\Logs\Backup\script3Backup.log"
%SQLCMDDIR%\SQLCMD.EXE -S -i %CMDLOCATION%\SQL\script4.sql -o "D:\Logs\Backup\script4Backup.log"

ROBOCOPY "D:\SQL Backup" %BACKUPDRIVE%\SQLBackup /R:2 /MIR /NP /LOG:%LOGLOCATION%\SQLBackup.log
ROBOCOPY "%SAGEDBBACKUPDIR% %BACKUPDRIVE%\SageBackup /R:2 /MIR /NP /LOG:%LOGLOCATION%\SageBackup.log
ROBOCOPY "" %BACKUPDRIVE%\MIPShare /R:2 /MIR /NP /LOG+:%LOGLOCATION%\SageBackup.log

::Remove remote copy of Exchange Database
DEL /Q /S %BACKUPDRIVE%\ExchangeDatabase.edb > %LOGLOCATION%\ExchangeBackup.log
::Create VSS copy of D drive to copy running exchange
DISKSHADOW /S %CMDLOCATION%\diskshadowstart.dsh >> %LOGLOCATION%\ExchangeBackup.log
ROBOCOPY "H:\Exchange" %BACKUPDRIVE%\ExchangeBackup /R:2 /MIR /NP /LOG+:%LOGLOCATION%\ExchangeBackup.log /XF "Mailbox Database.edb"
::Copy Exchange database
ECHO. >> %LOGLOCATION%\ExchangeBackup.log
ECHO Copying Exchange Database... >> %LOGLOCATION%\ExchangeBackup.log
ECHO %DATE% - %TIME% >> %LOGLOCATION%\ExchangeBackup.log
ECHO. >> %LOGLOCATION%\ExchangeBackup.log
ESEUTIL /Y "H:\Exchange\Mailbox\First Storage Group\Mailbox Database.edb" /D%BACKUPDRIVE%\ExchangeDatabase.edb
ECHO. >> %LOGLOCATION%\ExchangeBackup.log
ECHO Completed Exchange Database copy... >> %LOGLOCATION%\ExchangeBackup.log
ECHO %DATE% - %TIME% >> %LOGLOCATION%\ExchangeBackup.log
ECHO. >> %LOGLOCATION%\ExchangeBackup.log

::Clean up VSS copies
DISKSHADOW /S %CMDLOCATION%\diskshadowend.dsh >> %LOGLOCATION%\ExchangeBackup.log

ROBOCOPY "D:\Scripts" %BACKUPDRIVE%\ScriptsBackup /R:2 /MIR /NP /LOG:%LOGLOCATION%\ScriptBackup.log
::We don't want to remove old files in this copy
ROBOCOPY "D:\WebsiteBackup" %BACKUPDRIVE%\WebsiteBackup /R:2 /E /NP /LOG:%LOGLOCATION%\WebsiteBackup.log

EXIT