PermaLink How to explain to non-Domino people how to back up a Domino server.03/31/2008 01:56 PM
As I've mentioned in earlier posts, I don't fully control my production Domino servers.  Unfortunately, the people who do... don't seem to understand that backing up Domino is often quite a bit different than anything else they know.  If they listened to me, here's what I'd tell them:

My production beasts are large V2000-series Suns. On them, we're running good ole Domino 8.00, and it's very happy and healthy.

One of my contractors has been looking at the server logs and has found a lot of instances where "they" (them what runs the hardware and OS for me) are downing and recycling the machines at all sorts of weird hours. Since they went into production in 1999, these machines and their predecessors have been rebooted every Monday morning at 4am local time whether they needed it or not. Since the original servers were Windows NT 4.0, they definitely needed it. If you're running Domino on NTor its children, and you let that sucker run for more than a week without a reboot, you're a fool and a daredevil.

But this guy noticed that they're also being downed over the weekend at various times, times they never told us about. The boxes aren't crashing, they're being downed. We have no real idea why. And they're both downing at about the same time, meaning the site is completely unavailable during that period. Since few users would be on in the middle of a Saturday night, I don't worry that much about it on its face, but I am wondering if they don't have a wrong idea or two about how to back up Domino.

These boxes are clustered, but they are not identical by any means. One of the two machines has primary responsibility for running a bunch of scheduled agents that shuffle content around and clean things up, and the other often has test and development code on it the other does not have.

They're apparently downing Domino, then backing up the entire machine to tape, then restarting them. There's absolutely no reason to do this, and here's why:

If you kill a Sun -- if it goes supernova and eats itself -- you have no alternative but to reinstall the operating system anyway. So, there's no point in backing up the OS and all its minutiae, any more than you need to with a clean install of Windows Server. Restoring a working Sun or Windows server from tape is damn near impossible without mostly reinstalling the entire OS and patches anyway.

Domino is a snap to reinstall, so there isn't a lot of point in backing up the Domino code.

Since the servers are clustered, and since anything unique exists on at least one other machine around here (the test/dev code, the special agents and the raw data they eat all exist elsewhere) there even isn't a lot of point in backing up both machines. Just back up the stuff that's in common, then the unique stuff. Not complete backups of both machines... most of that you'll just throw away or not restore anyway.

Tape is still, after all these years, the least reliable backup and retrieval method, and one of the slowest. I find it much faster to reinstall the OS and Domino than to even wait around for someone to try and find the most recent tape, and even then usually we find there's some defect in the tape anyway. Therefore, I find that tape backup exists mostly to make know-nothing executives think all the data's safe.

The tape backup mechanism they have is completely Domino-stupid, so it can't back up the server while Domino is running. Yes, there are tools out there and have been for years that are Domino-smart, but they didn't buy any of those because they never bothered asking me what would be best.

How I'd Do It:

If I was president of the fruit juice companies, here's how I'd back up the Domino servers I have:

  1. Take down the #2 server in the cluster.
  2. Copy all the important (i.e., non-stock-Lotus-never-used) NSFs and NTFs to an external drive of some sort.
  3. Bring the #2 server back online
  4. Take down the #1 server
  5. Copy all the important stuff that was not backed up in Step 2 to the same external drive.
  6. Bring #1 back up.

Alternative method:
  1. Arrange the clusters so that both servers are, in fact, identical
  2. Down the #2 server
  3. Copy off the important stuff
  4. Bring #2 back up
  5. Never take down #1 or back it up (you don't need to).

Both methods ensure that the system is always available. Both methods minimize the time spent down, since you're not backing up the OS and crap that is better reinstalled from DVDs or another Domino server anyway. Neither method really resembles what they're doing now.

But again, I don't control my servers.

Quick Summary

Tape backup sucks. Bit-image backups are meaningless and dangerous. Operating systems and application services are better reinstalled from scratch in the event of failure. Downing servers in 2008 is dumb.

By the way, I'm reasonably available to come and run your operating environment. It'll be better, trust me.

This page has been accessed 313 times. .
Blabber :v

1. Jim Casale03/31/2008 05:54:53 PM

I agree with Dennis. TSM and Tivoli Data Protection for Domino is the next best thing to Domino itself. Luckily, I do the backups and it is all automated except for putting tapes in and taking tapes out. Yeah yeah, I know tape is not reliable. When you use TSM it is. Data goes to disk first, and then to tape without me thinking about it. Best of all, I can get any document back no matter when it was deleted using transaction log backups. Heck, I can even get documents you deleted 10 minutes ago even though the backup ran last night. I do agree it is useless to backup the OS (even Windows). It if went up in smoke it would be faster to install Windows from scratch and then restore the Domino data.

2. Dennis van Remortel03/31/2008 02:32:14 PM
Homepage: http://http:/

If you do it properly, get yourself TSM

3. Turtle03/31/2008 02:32:04 PM

Yeah, OK, you win. You can have the rest of my soda.

4. Steve Davis03/31/2008 02:22:45 PM

We have 7 Domino servers and back all of them up to another Domino server using replication. OS level scripts add any new databases to the backup server and copy non-nsf files (notes.ini,, etc). The data on the backup server is then copied to another location weekly and monthly (a separate copy is saved for Jan, Feb, March, etc).

Other stuff to waste your time:
Weightless Dog
My YouTube videos
My Head Talking
Today's Poll
Recent Entries
The BlogRoll
No calendar found.
Monthly Archive
Lotus Domino ND8 RSS News Feed RSS Comments Feed RSS Validator Blog Admin Lotus Geek OpenNTF BlogSphere
Say hi