Sunday, January 7, 2018

Paranoia is a Life Skill! Backups are your Friend!

In some fields, paranoia is considered unhealthy, but when dealing with software and computers, paranoia is definitely a life skill worth having.  Here are a few of the things that helped reinforce this for me:

One of the people I used to work with had a saying: 
If you take just one backup, it will have errors and be unreadable. If you take just two backups, you will have errors on both that will make them unreadable. If you take three backups, all three will be good!
Experience tells me these words are true just often enough to be worth believing!

When testing a program that does a series of updates, make a copy of the file and test against that copy. Then verify results.  If possible, do all development work in an isolated backup account / directory / whatever...  Don't do it in production.

Before running a test of a program that does a big update, make sure you are in the right place.  This was a real conversation that I was party to:
User phones in:  We're getting data errors in xxxxx entry program!
Us, checking: Hold on a second... Huhhhhhh???!!! The master file is empty!
We put the phone down, walk over to the development manager's office.
Us: Hey {dev manager's name}, what are you doing right now?
{dev manager}: I'm setting up the test account.
Us: Did you just clear the master file?
{dev manager}: Yes, I just did, right now.  Why? Do you need the test account for something?
Us: Can you check what account you are in?
{dev manager}:  @&&&#@@@!!!!
Fortunately, we had more than three day's worth of  backups, so all were good, and we got the data back. Unfortunately, staff had to reenter the morning's data that was entered since the backup was taken.

Microdata Reality systems had reports that would come out of the backup.  In those early days, we tried hard to train all our customers to check for a number at the end of the report. That was the number of "Group Format Errors" that the backup had encountered.  We trained them to call is in an immediate panic if that number was not zero, as it meant that their file system was corrupted!  There were stories of users who ignored these until their systems actually crashed. At that point, there were no backups that were fully usable. It was a mess!

As a result of all of this, I developed a healthy paranoia, and I got in the habit of hauling around a 9-track tape or two, and I would backup my own account. It didn't matter if the customer was doing a backup. I'd back my own account up.  I've had customers lose my account after I had done a week of work on it.  They thought they'd have to pay me to re-do the work, but I had my daily backup, so we only lost an hour's work.  The customer's respect was well earned!

I found a video of someone loading an old 9-track tape drive. It will start right where he loads the old one.  For you who have never had to do this, I could do it in my sleep!

Over time, the backups changed. You had 4mm, 8mm, and other tape formats. They were faster than 9-track, and held way more, so it was a good thing, but there were some downsides.  Not all 4mm or 8mm tape drives were compatible with each other. Probably the worst problem was that the report telling you of data corruption didn't exist in a lot of the newer operating systems that I worked with.  Paranoia had to take a new form.

Once a week, you would take a dummy record, somewhere near the end of your backup, rename it, and restore it from the backup tape, just to make sure the backup was good.  If you had done this for a while with consistent success, then you might drop back to once a month.

I no longer backup my systems to tape.  Some customers still do.

Nowadays, I have a product called Acronis.  It backs up my PC, my wife's PC, and my Mac, each to their own 2TB USB drives every night.  I'd occasionally swap one of these drives and send it offsite to a family member, but that got to be a lot of work.  My PC had over 250 GB of data. Backup over the internet, when I had 1MBPS DSL upload speed was simply not practical. It would take weeks to upload. Recently I upgraded to Telus Fibre Optic.  Now, all 3 computers back up over the internet, once a week!  In an emergency I could restore over the internet in a couple days.  I no longer have to get a family member to keep a backup offsite for me.

Every once in a while, I rename a file and restore from one of my backups, just to make sure it's all working!  Yes, Paranoia is definitely a useful life skill!

Note: Paranoia extends beyond backups - security, firewalls, password vaults, cloud solutions, and more.  Those will be for another day!

No comments:

Post a Comment