Geeks With Blogs
Lou Vega a thump to the brain

So I’m reading the blog post from Yow-Hann Lee about this guy who worked for the Alaska Department of Revenue who managed to slick a hard drive containing $38 Billion in data…oh and he slicked the backup hard drive too…nice touch.


What gets me even more than that he slicked the drive(s) – is the part of the article where they talk about the backup tapes being unreadable...what is it with organizations that decide “OK we’ve *got* to have a disaster recovery plan” but then they don’t think it through all the way and actually *TEST* the backups they are making to ensure they will do the job when the time comes.


The reason I started out by asking if this was a “government thing” was because two different government organizations I’ve worked for did the same – had a great backup plan, disaster recovery plan and even a continuity of operations plan (all of which I helped write) but then when I actually recommended we test the plan I was met with a bunch of resistance…”Oh we don’t have time and budget to do that right now…”. Fortunately with the two organizations I was with they didn’t ever need to do a “real world” recovery…but another government organization we worked with called me in a hurry one day when they found themselves in a similar situation as the Alaska Dept. of Revenue – they had their *only* domain controller crash (the secondary one had crashed long ago and they never got around to fixing it) and they found their most recent backup was 1) over three weeks old and 2) completely useless because the media was bad. ***Side note – I was able to fix things for them using some pretty neat and rather unorthodox tricks but I’ll blog about that another time***


In my IT career I’ve only ever been in one organization that actually had the proper planning and tested their plans and that was a credit union – which I suppose makes perfect sense in that with them time down is quite literally money lost. In fact we had regular test restores on local standby hardware to ensure that our media was readable and about twice a year we did full disaster recovery testing where we tested as though everything had been wiped out by a hurricane (after all we do live in Charleston, SC) and we needed to continue operations. With all that testing it was not a worry when someone called up occasionally to say “I lost some data can you restore from tape?” because we *knew* the tapes were good.


I’ve talked to a lot of folks who work outside of government organizations as well who look at me funny when I ask if they’ve ever tested a tape restore….So back to my question – why is it that it seems nobody (outside the financial industry anyway) actually tests their backups? How many horror stories is it going to take to realize it’s a worthwhile investment?


OK I’m stepping off my soap box now….

Posted on Friday, March 23, 2007 7:00 PM Blowing off steam | Back to top

Comments on this post: Is it just a government thing or what?

# re: Is it just a government thing or what?
Requesting Gravatar...
Unfortunately, that mentality is not restricted to government. In fact, while in the Navy we tested like that all the time. We even shutdown the reactors for the ship...all at the same time (called "Dead in the Water" or DIW). Yes, it's dangerous. Yes, it's challenging. Yes, it stops pretty much everything else on the ship. But we had to be able to recover from it.

I guess I'm now even more scared that the military does better planning and testing than most companies...and does so in a much more complicated environment.

Nobody (except us of course :) )cares about quality anymore. I wonder if ever they did.
Left by Codesailor on Mar 23, 2007 7:46 PM

# re: Is it just a government thing or what?
Requesting Gravatar...
Hmm - maybe the afloat folks in the Navy do things better then - the one organization I'm talking about that I went to "rescue" was an ashore group ;)
Left by Lou on Mar 23, 2007 8:06 PM

Your comment:
 (will show your gravatar)

Copyright © Lou Vega | Powered by: