June 30, 2009

Cut Your Exchange Backup Window in Half

Going all the way back to the Exchange 5.5 days, I've preferred doing disk-to-disk-to-tape Exchange backups. I'll use NTBackup for the disk-to-disk part, and the regular file backup agent that comes with whatever backup software we happen to be using for the to-tape part. This means eschewing the Exchange backup agents that most vendors provide, which in my mind is a big plus. Opinions will certainly vary, but using this method provides cost savings, fast restore times, extra protection and redundancy, and the reassurance that comes with using the native Exchange backup application. If you need more convincing, there are a few white papers from Microsoft describing this same setup as the Exchange backup method used by their in-house staff.

A client that is currently using this configuration has just surpassed the 200 GB mark for their combined mailbox store size, and the nightly full backups were taking a little longer than 6 hours to complete. This large backup window was preventing the nightly database maintenance tasks from completing, so a new strategy was in order. While thinking through some possibilities, I remembered reading about some registry tweaks that could improve the NTBackup performance when backing up to disk. After a little research, I made the changes, and the results were almost unbelievable: the Exchange backup job that had previously taken more than 6 hours to complete now finished in just under 2 1/2 hours!

While taking advantage of this dramatic speed boost only requires three registry changes and an additional command line parameter, there is a big bummer at first glance: the NTBackup registry keys that need to be changed reside in the HKEY_CURRENT_USER hive. This really cramps my style as I always configure the scheduled task that kicks off the NTBackup job as the NT AUTHORITY\SYSTEM account with a blank password. If you work in an environment with strict password change policies, even for system accounts, you know the pain of having to maintain passwords in scheduled tasks and scripts. Life is so much easier if it can just be avoided. But since the system account doesn't execute NTBackup interactively, the registry keys don't get created, and I assumed this meant there was no way to have the application check for the configuration tweaks.

But thankfully I was wrong, and it's a pretty simple process to manually create the necessary keys in the right spot:

  • First of all, you need to actually complete a backup job once to get the registry entries all set up, so as a regular administrator on the Exchange server, launch NTBackup, select a single temp file somewhere to backup, let the job run to completion, and then just delete the temporary backup set.

  • Launch regedit, and drill down to HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine

  • You should already see the values we're about to change, if not, something didn't get created properly, so try a manual NTBackup job again. If the keys are present, make the following changes:

    • Change Logical Disk Buffer Size from 32 to 64

    • Change Max Buffer Size from 512 to 1024

    • Change Max Num Tape Buffers from 9 to 16

  • After making the changes, select the Backup Engine key from the left pane, and right click and select Export. Save it as a .reg file, and make sure Selected branch at the bottom of the Export window is set to HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine

  • Now we'll locate the system account's registry settings, with regedit still open, browse to HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup. The S-1-5-18 is the standard identifier for the system account, and unless you've scheduled NTBackup to run as NT AUTHORITY\SYSTEM before, the key will most likely be empty.

  • We need to schedule a job to run as NT AUTHORITY\SYSTEM to create the default keys, so launch NTBackup in advanced mode, select the Schedule Jobs tab, and set up a temp job to just back up any text file and schedule it to run in a couple of minutes from now. When prompted for the credentials that should be used for the job, you'll need to change the user account to NT AUTHORITY\SYSTEM with a blank password, several times. In fact, it still won't save it as the account to use, so after saving the scheduled job, open the task from the Scheduled Tasks panel and change the user account to NT AUTHORITY\SYSTEM with a blank password again.

  • After the job runs, you should see the following registry keys have been created under HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup; Backup Engine, Backup Utility, Display, and Log Files. But if you drill into Backup Engine, you'll see it didn't create the keys we modified a few steps ago.

  • To easily create the keys, just edit the .reg file we exported earlier in Notepad. Change the line [HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine] to [HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup\Backup Engine], and save the file.

  • Now right click the .reg file, and select Merge. You should find the registry settings have been created for the system account, and NTBackup will now use the much speedier settings even when running as the system account.

There's another performance mod we need to make to give the backup even more boost. Since Windows Server 2003 Service Pack 1, NTBackup has been equipped with a secret and offensively named /fu switch, for 'file unbuffered' mode. To bolt this on, just edit the Scheduled Task for the NTBackup job, and add the /fu switch after the /hc:off parameter. When you're done, the Run: text box of the Scheduled Task will look something like this:


  C:\WINDOWS\system32\ntbackup.exe backup "@C:\Documents and Settings\
    Administrator\Local Settings\Application Data\Microsoft\Windows NT\
    NTBackup\data\Exchange_Daily.bks" /n "exchange_Backup.bkf created 
    6/30/2009 at 6:06 PM" /d "Set created 6/30/2009 at 6:06 PM" /v:no 
    /r:no /rs:no /hc:off /fu /m normal /j "Exchange_Daily" /l:s /f 
    "E:\exchange_ backups\exchange_Backup.bkf"


...read more

June 10, 2009

Configure a Vyatta Cluster for Redundant Virtual Firewalls

If you missed the Protect the Service Console Network With a Virtual Firewall project, we looked at how to use a Vyatta firewall to protect the ESX Service Console network and restrict SSH and VI or vSphere Client access to only a few specific workstations. Vyatta offers an impressive network operating system that can be run from a live CD, permanently installed on physical or virtual hardware, or downloaded as a virtual appliance. It comes with some high end features like stateful packet inspection, site-to-site VPN, OSPF, and BGP. There's a completely free edition with unrestricted access to all the features, but it can also be purchased with support offerings.

If you followed along with the original post, you may have noticed a potential pitfall: what if the ESX server hosting the Vyatta virtual machine goes down? You may have HA enabled, but what if it takes several minutes for the VM to boot all the way up on another host? Even a couple of minutes with no SSH or VM console access during a crisis would feel like an eternity.

Amazingly, the Vyatta operating system also includes clustering, and it's very simple to configure. To set up a cluster, we'll need the following:

  • Two Vyatta VC5 virtual machines, preferably with very similar configurations, see Protect the Service Console Network With a Virtual Firewall for a quick setup tutorial

  • Both Vyatta VMs will need two virtual NICs, each with its own real IP address: one in the Service Console network, and one in a LAN network

  • The clustered Vyatta VMs will host two virtual IPs: one will be the default gateway address configured on every ESX host (172.20.1.254 in our case), and the second will be a LAN address you specify as the route to the Service Console network on the Layer 3 device routing between your LAN subnets. In our case, we have a very simplified setup and the Vyatta's LAN facing interface is the default gateway for the LAN (10.1.1.254)

There's some good documentation on setting up a cluster in the High Availability Reference Guide for VC5 available for download at the Vyatta website.

Once you've got the two Vyatta VMs up and running on different ESX hosts, this is how the cluster configuration will look on the primary Vyatta firewall. We're using a conservative dead-interval of ten seconds, meaning a failover will only occur if keepalives are missed for that long, and keepalives are being sent out every two seconds over the eth0 (Console Network) interface.

The service commands define the virtual IPs the cluster will bring up on the secondary if the primary stops responding:

cluster {
    dead-interval 10000
    group sc-cluster {
        auto-failback true
        primary sc-firewall-pri
        secondary sc-firewall-sec
        service 10.1.1.254/24/eth1
        service 172.20.1.254/24/eth0
    }
    interface eth0
    keepalive-interval 2000
    pre-shared-secret ****************
}

interfaces {
    ethernet eth0 {
        address 172.20.1.252/24
        hw-id 00:50:56:9c:3b:0b
    }
    ethernet eth1 {
        address 10.1.1.252/24
        hw-id 00:50:56:9c:04:a5
    }
    loopback lo {
    }
}


And here's the cluster configuration on the secondary firewall. The actual cluster commands are identical, only the real IPs assigned to the interfaces are different:

cluster {
    dead-interval 10000
    group sc-cluster {
        auto-failback true
        primary sc-firewall-pri
        secondary sc-firewall-sec
        service 10.1.1.254/24/eth1
        service 172.20.1.254/24/eth0
    }
    interface eth0
    keepalive-interval 2000
    pre-shared-secret ****************
}

interfaces {
    ethernet eth0 {
        address 172.20.1.253/24
        hw-id 00:50:56:9c:3c:e0
    }
    ethernet eth1 {
        address 10.1.1.253/24
        hw-id 00:50:56:9c:38:04
    }
    loopback lo {
    }
}

As you can see, it's pretty simple to set up, but one annoyance with the cluster feature is that you have to create the same firewall rules on each device, there's no functionality for syncing up the configurations.

It wasn't too hard to write a quick and dirty little shell script to copy just the firewall configuration from the primary to the secondary, however, so you only need to maintain the rules on the primary, and then remember to run the script after saving the changes. If you like, you can set up public key authentication for SSH access from the primary to the secondary like we did in DIY ESX Server Health Monitoring - Part 2, but it's not necessary, the script will prompt you for the password during the SSH connection attempt.


#!/bin/bash

# Vyatta cluster firewall sync script
# by Robert Patton - 2009
#
# Copies firewall rules from primary to secondary
# and applies them to the appropriate interfaces.
#
# Deletes existing firewall rules on secondary and
# removes any firewall sets on interfaces, so make
# sure this is only run from the primary.
#
# Replace the SECONDARY value with the hostname or IP
# of the secondary device in the cluster.

SECONDARY="sc-firewall-sec"

TEMPFWRULES=$(mktemp TEMPFWRULES.XXXXXXXX)
TEMPINTCMDS=$(mktemp TEMPINTCMDS.XXXXXXXX)
TEMPSETCMDS=$(mktemp TEMPSETCMDS.XXXXXXXX)

# Match just the firewall section from the boot config file
awk '/^firewall {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPFWRULES

# Match the interface section, we filter for firewall set statements later
awk '/^interfaces {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPINTCMDS

# Create a script to run on the secondary with the firewall set commands
# The vyatta-config-gen-sets.pl script creates set commands from the config
cat > $TEMPSETCMDS <<'EOF1'
configure
# First remove any firewalls from interfaces
for int in $(show interfaces ethernet | \
awk '/eth[0-9]/ {print $1}'); \
do delete interfaces ethernet $int firewall; \
done
# Now delete all firewalls
for fwall in $(show firewall name | \
awk '/^ \w* {$/ {print $1}')
do delete firewall name $fwall; \
done
EOF1

cat >> $TEMPSETCMDS <<EOF2
# Create firewalls found on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPFWRULES)
# Apply firewalls to interfaces as defined on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPINTCMDS | grep firewall)
commit
save
exit
exit
EOF2

# Force a tty for the ssh connection - Vyatta environment variables
# and special shell are only set up during an interactive login
cat $TEMPSETCMDS | ssh -tt $SECONDARY

rm -f $TEMPFWRULES $TEMPSETCMDS $TEMPINTCMDS



...read more

June 5, 2009

Another ESX Server HTTP File Trick

While reading some docs on the vSphere CLI, I came across this note:

...you can browse datastore contents and host files using a Web browser. Connect to the following location:

http://ESX_host_IP_Address/host
http://ESX_host_IP_Address/folder

You can view datacenter and datastore directories from this root URL...


I knew you could browse the datastores this way as there is a link from the main welcome page, but the /host URL is news to me. Log in with the root account, and it brings up a page titled Configuration files with links to view a bunch of important - you guessed it - configuration files.

Not too terribly interesting, but I can already see myself hitting this URL just to get the vSphere license key or double check that the proper entries are in an ESX server's hosts file.

...read more