HA Network Attached Storage (NAS)

Happy Halloween!

So, I know I promised this towards the beginning of the month, but it’s proven to be trickier than I initially expected (and I expected it to be tricky!)

I’ve been doing a lot of research and a lot of trial and error and I’m pretty sure I know what it takes to configure this correctly, but I’m waiting on some new servers before I set this up for real.

Since we mount our shares using NFS, (see NFS for HA Lighttpd) there’s a little more involved than just replicating the volumes between two NAS servers and using a load balancer to distribute traffic between them.

Here is a great reference that really helped me get a good understanding on what’s involved in HA NFS: http://www.linux-ha.org/HaNFS (this site also has a lot of good HA info for Linux admins)

First Attempt:

First, I attempted to create an Active – Active, multi-master configuration between our existing NAS servers (two Linksys NSS6000’s that I will call NAS1 and NAS2).

Unfortunately, the NSS6000 is not configured to do synchronization between NAS servers and since you have very little access to the OS, it’s not really a good option to try to hack it to enable synchronization.

To get around this, I setup Unison on a third server (SRV1) that NFS mounts the volumes from NAS1 and NAS2 and handles the synchronization.  At this point, I realized that an Active – Active setup would not be possible using this configuration (as Unison must be scheduled using cron and can’t detect file/directory changes as they occur).

So I settled for an Active – Passive config and setup SRV1 to synchronize the volumes between NAS1 and NAS2, set NAS1 to be the primary server, set NAS2 as the secondary server, and used our load balancer to handle fail over.

Unison was handling the file synchronization just fine (though with considerable network traffic) and the multi-master relationship was solid, but we had a couple instances where NAS1 locked up and fail over to NAS2 was unsuccessful.

Dang!  Back to the drawing board….

Bottom Line:

If you check out the link concerning HA NFS, I’m sure you’ll find a bunch of reasons why this config was not successful.  HA NAS with Linux servers really should be using Heartbeat to facilitate the complexities that HA NAS requires (especially during fail over).

However, rather than beating my head against the wall trying to make a HA config work with servers that weren’t designed for HA, I decided to purchase new NAS servers that were designed for HA NAS.

I’ll be demoing the new NAS servers next week to make sure they can achieve an Active – Passive configuration with fail over (as the vendor literature says it can).

Cross your fingers! 

I’m hoping this is the answer to this problem.

Javascript: Copy Text to Clipboard

Update: this method no longer works.  For details: http://cthayer.wordpress.com/2009/02/26/javascript-copy-text-to-clipboard-not-anymore/

So, today we’re thinking that our users could benefit from some areas of our site auto-copying text to their clipboards so that they can just click and paste instead of copy, click, and paste.

IE has the handy window.clipboardData object which allows javascript direct access to the clipboard.  However, Mozilla/Firefox disables this functionality by default because of security concerns.  So, while the IE method works in Safari and Opera, it fails in Mozilla/Firefox and therefore we needed another way.

Turns out that while javascript is not allowed to access the clipboard, flash is.  So, by using flash and a little ajax magic you can create a copyToClipboard function in javascript that works in all browsers that support flash and ajax.

This code is posted on the web in many places and in this case it came from here:

function copyToClipboard(text) {
	var flashId = 'flashId-HKxmj5';

	/* Replace this with your clipboard.swf location */
	var clipboardSWF = 'http://appengine.bravo9.com/copy-into-clipboard/clipboard.swf';

	if (!document.getElementById(flashId)) {
		var div = document.createElement('div');
		div.id = flashId;
		document.body.appendChild(div);
	}

	document.getElementById(flashId).innerHTML = '';
	var content = '<embed src="' +
	clipboardSWF +
	'" FlashVars="clipboard=' + encodeURIComponent(text) +
	'" width="0" height="0" type="application/x-shockwave-flash"></embed>';

	document.getElementById(flashId).innerHTML = content;
}

NFS for HA Lighttpd

Currently, we have lighttpd deployed in a high availability (HA) configuration.  In order to provide our users with a consistent experience, regardless of which web server they are connected to, we use NAS and NFS to provide identical content to each web server.

The diagram below should make our setup a little clearer:

Configuration Diagram

Last week, we encountered some problems where requests for our website would hang.  Lighttpd had nothing to say about this problem and restarting it did nothing.  After about 10-20 minutes, the problem would resolve itself and everything would go back to business as usual.

Personally, this type of problem really frustrates me, so I made it a mission to figure out what was going wrong and fix it. (not that I had much of a choice, this is a production setup)
 
I had some initial hunches to check first:

1. some sort of short term DOS attack
2. a traffic spike
3. lighttpd and php (running in fcgi mode) running out of available file descriptors

Our monitoring software showed no signs of abnormal traffic behavior during the times when the problem was occurring.  So that ruled out the first two possible causes.

The last possible cause is more difficult to track down.  Lighttpd’s own documentation says that the error messages regarding a shortage of file descriptors may not be written to the error log and may only show up in test cases.  Well doesn’t that just take the cake?!   I checked anyway, but found nothing of interest in lighttpd’s error logs.

Next step?  Calculate the number of file descriptors currently being used by lighttpd and php under normal load and compare them to the maximums defined by lighttpd’s configuration file and the Operating System.

It turns out that lighttpd runs pretty light (pun intended) in terms of file descriptors.   PHP, on the other hand, uses (at least) 7 file descriptors per child!   Since we’re running 4 php processes that each have 128 children, I decided to increase the OS file descriptor limit from 1024 per process to 32768 per process.   Also, just to be safe, I increased lighttpd’s server.max-fds configuration option to 16384.

Unfortunately, this didn’t solve the problem.  We had another incident the day after I made the above changes.

Not to be deterred, I went digging in the syslog (/var/log/messages on Redhat variants) and found errors regarding lockd and our NFS mounts that corresponded to the times of the incidents.   Aha!  Now we’re on to something.  I added the nolock option to the mounting options in fstab and remounted the directories.

This seems to have solved the problem.  It’s been a couple days (with constantly increasing traffic) and we’ve had no more incidents.  Bottom line?  When using NFS in a HA configuration you should consider mounting the shares with the nolock option.

The next step is to setup HA NAS.  I’ll post again when we’ve accomplished this task.

Sendmail: Smart Host Authentication by Sender

Sendmail is one of those applications that can do just about anything (as long as you know the secret handshake that gets you access to detailed information about it’s configuration).

I’ve been using it for about 1 year to allow my company’s website to send email to our users.  To avoid all the headaches involved in making sure our emails don’t get rejected as spam, we use googlemail as a smart host since they host our email accounts for our domain.

This has been working great, but recently we ran into a limitation that would be a deal breaker if it couldn’t be solved.

The Problem:

1. When using googlemail as your smart host, you must use authentication and Google rewrites the from address of all your emails to the user’s email address that you use to authenticate.  Thus, if you authenticate with webmaster@example.com’s credentials, all email from your server will seem to come from webmaster@example.com.  Even if you specify a different sender!
2. When sendmail is configured to use a smart host that requires authentication (like googlemail), it chooses the authentication credentials based on the host name or IP address of the smart host.  So, unless you use multiple smart hosts, sendmail will only use 1 set of authentication credentials when connecting to the smart host.
3. We are now hosting 2 websites on the same server with completely different domain names that each need to send email from different addresses that belong to their respective domains.

When you combine all three parts, the problem becomes clear:

How do I send email from info@yyy.com when sendmail will always use info@zzz.com’s credentials for smart host authentication, thus making Google rewrite the from address so that all email from my server appears to come from info@zzz.com?

The Solution:

To me, the best solution would be for sendmail to choose smart host authentication credentials based on the sender address of each email.

For example:

1. Sendmail receives an email message to deliver from info@yyy.com
2. Instead of looking up “smtp.googlemail.com” in authinfo, it looks up “info@yyy.com”

Seems simple enough, but, after hours of searching the web, I found that sendmail is just plain not configured to do this.

However, everything it needs to be able to do this is already available, it just doesn’t use it.  So I decided to make sendmail use it by editing the sendmail.cf file.

I know, I know, I’m supposed to use sendmail.mc, but darn it all if I wasn’t able to get this to work using the macro file.

So, without further ado, here’s what I did to make sendmail behave the way I wanted it to.

The How To:

Assumptions:

I’m assuming that you already have sendmail configured to use a smart host that requires authentication.  It is beyond the scope of this document to cover that.

Instructions:

1) Edit the authinfo section of sendmail.cf so that it looks like this:

######################################
### authinfo: lookup authinfo in the access map
###
###     Parameters:
###             $0: {f}
###             $1: {server_name}
###             $2: {server_addr}
######################################
Sauthinfo
R$*        $: <$(authinfo AuthInfo:$&{f} $: ? $)>
R<?>      $: <$(authinfo AuthInfo:$&{server_name} $: ? $)>
R<?>      $: <$(authinfo AuthInfo:$&{server_addr} $: ? $)>
R<?>      $: <$(authinfo AuthInfo: $: ? $)>
R<?>      $@ no                      no authinfo available
R<$*>    $# $1

The key here is the $&{f} macro.  In sendmail, it stands for the sender’s address from the envelope of the email.

The above code tells sendmail to look for entries in the authinfo file that match the sender’s email address.  If no match is found, then check for entries that match the smart host’s name/IP.

In effect, this preserves sendmail’s original behavior as the fall back if no match is found in the authinfo file for the sender’s address.

2) Add sender credentials to the authinfo file (its location is defined by the authinfo feature in sendmail.mc):

AuthInfo:info@ex.com “U:info@ex.com” “P:xx” “M:PLAIN”
AuthInfo:info@yyy.com “U:info@yyy.com” “P:xx” “M:PLAIN”
AuthInfo:info@zzz.com “U:info@zzz.com” “P:xx” “M:PLAIN”
AuthInfo:smtp.sh.com “U:info@ex.com” “P:xx” “M:PLAIN”
AuthInfo: “U:info@ex.com” “P:xx” “M:PLAIN”

In the example above, the first 3 entries are credentials for different senders that use the server to send email (they take advantage of the changes we made in step 1).  The last 2 entries are the original smart host authentication configuration which ensures that sendmail will still be able to connect to the smart host and send the email using a default set of credentials, even if the sender’s address does not match any of the other entries.

3) Remake the authinfo database using the “makemap” command.

4) Make sure to comment out any lines in your init script for sendmail that recompile sendmail.cf using the sendmail.mc script.  The location of this file on Redhat variants is: /etc/init.d/sendmail

(if you don’t do this, your changes to sendmail.cf will be lost when you restart sendmail!)

5) Restart sendmail.

You should now be able to control which user’s credentials are used to authenticate with the smart host on a case by case basis by changing the sender’s address on the email.

This is specified using the -f flag when calling sendmail on the command line (like PHP’s mail() function does).

Final Thoughts:

While this gets the job done, it would be nice to be able to incorporate the changes in step 1 into the sendmail.mc file.  This would make it much easier to maintain since anyone who runs the “make” command on the /etc/mail directory will destroy the changes we made to sendmail.cf in step 1.

Hello Word!

Hi everyone.

In case the title is confusing, this is my web blog. 

There’s not much here now, but here’s a little teaser……

My next post will be a technical article on how to hack sendmail so that it will choose smart host authentication credentials based on the sender’s email address rather than the host name of the smart host.

This issue is what inspired me to start a blog.  I searched all over the web and couldn’t find anything related to this anywhere so I wanted to record it for my own reference and to share it with others who might be looking for the same information.

See you soon!