HA Network Attached Storage (NAS)

Happy Halloween!

So, I know I promised this towards the beginning of the month, but it’s proven to be trickier than I initially expected (and I expected it to be tricky!)

I’ve been doing a lot of research and a lot of trial and error and I’m pretty sure I know what it takes to configure this correctly, but I’m waiting on some new servers before I set this up for real.

Since we mount our shares using NFS, (see NFS for HA Lighttpd) there’s a little more involved than just replicating the volumes between two NAS servers and using a load balancer to distribute traffic between them.

Here is a great reference that really helped me get a good understanding on what’s involved in HA NFS: http://www.linux-ha.org/HaNFS (this site also has a lot of good HA info for Linux admins)

First Attempt:

First, I attempted to create an Active – Active, multi-master configuration between our existing NAS servers (two Linksys NSS6000’s that I will call NAS1 and NAS2).

Unfortunately, the NSS6000 is not configured to do synchronization between NAS servers and since you have very little access to the OS, it’s not really a good option to try to hack it to enable synchronization.

To get around this, I setup Unison on a third server (SRV1) that NFS mounts the volumes from NAS1 and NAS2 and handles the synchronization.  At this point, I realized that an Active – Active setup would not be possible using this configuration (as Unison must be scheduled using cron and can’t detect file/directory changes as they occur).

So I settled for an Active – Passive config and setup SRV1 to synchronize the volumes between NAS1 and NAS2, set NAS1 to be the primary server, set NAS2 as the secondary server, and used our load balancer to handle fail over.

Unison was handling the file synchronization just fine (though with considerable network traffic) and the multi-master relationship was solid, but we had a couple instances where NAS1 locked up and fail over to NAS2 was unsuccessful.

Dang!  Back to the drawing board….

Bottom Line:

If you check out the link concerning HA NFS, I’m sure you’ll find a bunch of reasons why this config was not successful.  HA NAS with Linux servers really should be using Heartbeat to facilitate the complexities that HA NAS requires (especially during fail over).

However, rather than beating my head against the wall trying to make a HA config work with servers that weren’t designed for HA, I decided to purchase new NAS servers that were designed for HA NAS.

I’ll be demoing the new NAS servers next week to make sure they can achieve an Active – Passive configuration with fail over (as the vendor literature says it can).

Cross your fingers! 

I’m hoping this is the answer to this problem.