tag:blogger.com,1999:blog-16562493319727106482024-03-21T19:01:35.278-07:00Virtual FoundryRobert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.comBlogger24110tag:blogger.com,1999:blog-1656249331972710648.post-77601613219893843872010-01-06T16:23:00.000-08:002010-03-30T12:02:52.951-07:00Build a Windows 7 PE boot CD with defragIf you deploy virtual machines from templates using the standard vSphere tools, you know how valuable it is to get your templates as small as possible. A handy trick I learned a while back is to use vCenter Converter to import a virtual machine that I'm working up to be a template, and choose to resize the disk to the minimum size. The process is pretty simple: configure the system as though it were going to be deployed as an image, remove every temp file and hotfix uninstall folder, run sysprep and power down, boot up from an .iso of a rescue boot disk with a defrag utility, delete the pagefile.sys file, run a defrag, power down again, and use vCenter Converter to import the powered off virtual machine as a new machine and shrink the virtual hard disk during the process. Using this procedure, I've gotten a Windows XP template down from a 5.27 GB minimum size in vCenter Converter, to 3.08 GB. That's an almost 42% reduction in size, and is a huge time saver when deploying new systems.<br><br>
In the past I have used a customized BartPE boot disk for this, but I wanted a more lightweight solution that booted up faster and only had the tools needed for the specific job. Windows PE fit the bill, but doesn't include defrag.exe by default. There are some tutorials out there on how to get defrag working in the Vista version of Windows PE, but I don't have any Vista installs around, and would like to keep it that way. So I dug around for a bit, and was able to get defrag working in the Windows 7 version of PE<span class="fullpost">.<br><br>
<span style="font-weight: bold;">Windows AIK</span><br>
First thing, you need to download a copy of the Windows Automated Installation Kit (just google Windows AIK). There are three versions of the AIK now, and the latest one supports Windows 7, so make sure you download the Windows 7 version.<br><br>
<ul>
<li>Insert the AIK DVD you created and open startcd.exe if the main menu screen doesn't open automatically. Choose Windows AIK Setup from the menu.<br><br>
<li>It's your standard install process, click through the defaults and when it's finished you'll find a Microsoft Windows AIK folder has been created in the Start menu.<br><br>
<li>Open the <span style="font-weight: bold;">Deployment Tools Command Prompt</span> from the <span style="font-weight: bold;">Microsoft Windows AIK</span> menu, which will bring up a command prompt with the AIK executables added to the PATH environment variable.<br><br>
<li>In the command prompt, type the following, using C:\PEBuild or another folder name for your build folder. The command expects to create the build folder, and will fail if the folder already exists:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
copype.cmd x86 c:\PEBuild
</code></pre></div>
</td></tr></table><br>
The copype.cmd script copies the files and directories needed to build a PE boot disk for a specific architecture, which is x86 here. Note that the script also changes the working directory to the path we specified for the build files. Make sure you stay in this path for the following commands.<br><br>
<li>Now we'll use the ImageX tool to mount the default Windows PE image that ships with AIK in read/write mode on the local filesystem. By mounting the image, we'll be able to add some files to it:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
imagex.exe /mountrw winpe.wim 1 mount
</code></pre></div>
</td></tr></table><br>
<li>This next command copies the ImageX deployment tools to the PE boot disk, allowing you to capture an image of a hard disk in ImageX format. This isn't really relevant to this project, but we may as well add them in case we want to use them later:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
xcopy "C:\Program Files\Windows AIK\Tools\x86\*.*" mount\ /s
</code></pre></div>
</td></tr></table><br>
<li>Here comes the customization part, and we'll need two files from a running Windows 7 workstation. The commands we've used so far will produce a 32-bit Windows PE image, so be sure the Windows 7 system you copy the files from is also 32-bit. My first attempt at this was on my 64-bit workstation, and when I tried to use the defrag utility from within Windows PE, I got a message that it wasn't compatible with the version of Windows I was using.<br><br>
From the 32-bit Windows 7 workstation, copy:<br><br>
<span style="font-weight: bold;">C:\Windows\System32\Defrag.exe</span><br>
to<br>
<span style="font-weight: bold;">C:\PEBuild\mount\Windows\System32</span><br><br>
- and -<br><br>
<span style="font-weight: bold;">C:\Windows\System32\en-US\Defrag.exe.mui</span><br>
to<br>
<span style="font-weight: bold;">C:\PEBuild\mount\Windows\System32\en-US</span><br><br>
<li>Now unmount the build image with another ImageX command, but make sure you don't have any Windows Explorer windows or the command prompt open to the mount folder first:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
imagex.exe /unmount mount /commit
</code></pre></div>
</td></tr></table><br>
<li>If you type a <code>dir</code> command, you'll see the modified winpe.wim image file in the C:\PEBuild folder. We need to copy this to a specific folder and file name for the next few AIK commands to work:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
copy /y winpe.wim ISO\sources\boot.wim
</code></pre></div>
</td></tr></table><br>
<li>This command will create the PE .iso file in the c:\PEBuild folder, naming it pe-defrag.iso, so modify for your build path and desired file name. This is a long command, so make sure you get the whole line if copying it by double clicking in the grey box:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
oscdimg.exe -n -b"C:\Program Files\Windows AIK\Tools\PETools\x86\boot\etfsboot.com" C:\PEBuild\ISO C:\PEBuild\pe-defrag.iso
</code></pre></div>
</td></tr></table><br>
</ul>
<span style="font-weight: bold;">Boot it</span><br>
Now you can either copy the .iso file over to an ESX server and VMFS LUN, or use the local vSphere client to attach the .iso to a VM. Boot up a test VM with the PE .iso. After a minute or so, you'll find the VM has booted up to a command prompt, with a grey, Vista'ish background. Pretty boring, but a command prompt is all we need for the next few steps.<br><br>
<ul>
<li>In the command prompt, change the working directory to the (hopefully) offline mounted NTFS volume that holds the operating system files for the template:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
c:
</code></pre></div>
</td></tr></table><br>
<li>Since the pagefile.sys file is hidden, enter a dir command that shows hidden files:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
dir /a:h
</code></pre></div>
</td></tr></table><br>
<li>Remove the pagefile.sys file, and notice that the del command also needs to have hidden files specified:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
del /a:h pagefile.sys
</code></pre></div>
</td></tr></table><br>
<li>Now defrag the volume, forcing it to defrag all files (-w) and print verbose output (-v):<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
defrag c: -w -v
</code></pre></div>
</td></tr></table><br>
<li>You can exit PE with an <code>exit</code> command, but that reboots the VM, which has ruined a few sysprepped images when I forgot to interrupt the boot. So now I use the <code>shutdown</code> command to force it to shut down:
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
C:\WINDOWS\system32\shutdown /s
</code></pre></div>
</td></tr></table><br>
The shutdown command can take a few seconds to actually power down the VM, so be patient.<br><br>
</ul>
<span style="font-weight: bold;">Seal, shrink, deploy</span><br>
You'll want to run the defrag on a template that's been sealed up with sysprep. After deploying one of these shrunken templates, you'll need to edit the virtual hard disk before the first power on and expand it to a reasonable size. Windows will automatically expand the operating system partition during the first boot if you include the command <code>ExtendOemPartition=1</code> in the <code>[Unattended]</code> section of the sysprep.inf file for a Windows XP/2003 image, or <code>ExtendOSPartition=true</code> in the Unattend.xml file for a Vista/Windows 7 image.<br><br>
If you deploy templates in VMware Workstation, you'll find that you aren't allowed to just simply edit the virtual hard disk and expand it. For deployments in
Workstation, you'll need to use the <code>C:\Program Files\VMware\VMware Workstation\vmware-vdiskmanager.exe</code> utility to expand the .vmdk before the
first boot.<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.comtag:blogger.com,1999:blog-1656249331972710648.post-50409834547919934342009-07-23T18:30:00.001-07:002013-07-09T15:50:26.479-07:00Virtualize Cisco Routers with GNS3<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuvEKB__zcYrEJpapOepXjHJ7tLCDUILgJc1osdI72T5ncySsFtYF6SZZ84qH9TNwcaq_iEQKxRxugkdhadT_8CxfviusuxZ8IUDOxAmJwWM37IN2a20S7T6CTGhwBs1PhuSK7ZV8_kw/s1600-h/gns3.png"><img src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjAuvEKB__zcYrEJpapOepXjHJ7tLCDUILgJc1osdI72T5ncySsFtYF6SZZ84qH9TNwcaq_iEQKxRxugkdhadT_8CxfviusuxZ8IUDOxAmJwWM37IN2a20S7T6CTGhwBs1PhuSK7ZV8_kw/s400/gns3.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5361833184089772930" /></a><br>
This title is a little misleading, I figure I get to do that at least once. Yes, GNS3, <a href="http://www.gns3.net/">http://www.gns3.net</a>, emulates the hardware of several Cisco router platforms, and it will boot real IOS images. It's also free, easy to install, and there are abundant video and written tutorials. But you're not going to use it in your production environment.<br><br>
So what's it good for? Well, there's the obvious; if you are studying for a Cisco certification test, GNS3 is like a dream come true. Though there have been Cisco emulation packages around for a while, they are usually expensive, and only provide a small fraction of the IOS feature set.<br><br>
But even if you're not studying for an exam, or have no interest in Cisco IOS, GNS3 is a fantastic tool for testing network and service architecture designs. When used in combination with VMware Workstation, connecting your virtual machines up with a GNS3 topology takes about three clicks<span class="fullpost">. In the simple diagram above, I've connected two VMs across a virtual WAN link, utilizing two Cisco 3640 routers and OSPF as the dynamic routing protocol. The computer shapes are really cloud objects that I've customized the shape for. When you configure a cloud object, the <span style="font-weight: bold;">vmnet</span> virtual networks appear in the drop down list of available NIO Ethernet objects. To connect a virtual machine to a GNS3 virtual router, just add the virtual network the VM is plugged into in the cloud configuration window, and then create a Fast Ethernet connection to a router.<br><br>
The mind boggles at the thought of all the projects you can test with this setup. Imagine being able to test a change to the Active Directory infrastructure between multiple WAN sites, or verify client connections after changing from a static routing configuration to OSPF. A few months back I completely modeled a client's network infrastructure using GNS3, and proved how they could reduce their routing tables by about 10,000 entries by configuring EIGRP route summarization on a single link.<br><br>
If you are running a recent version of Ubuntu on your demo laptop, installing GNS3 is as simple as <code>sudo apt-get install gns3</code>. The install is pretty painless on XP as well, and possible on Vista (though I've read of folks having a lot of issues), but GNS3 runs a lot slower and you won't be able to run as many router instances with XP or Vista.<br><br>
My current demo laptop, a Core2 Duo T9600 running Ubuntu 9.04 64-bit, VMware Workstation 6.5 and GNS3 is the coolest thing since sliced bread. And I would know, because I eat a lot of sandwiches.<br><br>
<span style="font-style:italic;">Oh that's bad..... but it's true, I do eat a lot of sandwiches</span><br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.comtag:blogger.com,1999:blog-1656249331972710648.post-19191551395752071862009-07-12T19:27:00.000-07:002009-07-12T19:31:14.369-07:00Know Your HistoryThe UNIX/Linux shells evolved on a planet where saving every keystroke and millisecond of time was absolutely essential to survival. As a result, they're chock full of shortcuts, many of them with overlapping functionality, letting the user choose the method that works best for them.<br><br>
The shell command history is a prime example. Even a newborn knows to use the up and down arrow keys to recall commands, and most toddlers are piping <code>history</code> into <code>less</code> to perform manual searches for complicated commands they don't feel like recreating. But if you ever have the chance to stare over the shoulder of a grey-bearded shell guru, you'll see that the true masters use several different techniques to pull up commands in the most efficient way. The following report was compiled from the sage advice given by these mysterious wizards<span class="fullpost">.<br><br>
<span style="font-weight: bold;">Bang!</span><br>
We'll start with the classic C Shell history syntax. You've no doubt used an exclamation point to re-execute a command from the history list, like <code>!42</code>, or have used the double exclamation to execute the previous command, <code>!!</code>. You can get pretty fancy with the available event designators, for instance, <code>!-3</code> will execute the command three commands ago. And <code>!echo</code> will execute the most recently run command that began with echo.<br><br>
This syntax has always scared me, as I could see myself executing a dangerous command without realizing it, especially if I'm in a hurry. But there is a safer way to use it, just include the <code>:p</code> modifier, which displays the command, but doesn't actually execute it. For instance, if the last command executed was <code>echo Hello</code>, typing <code>!!:p</code> would preview the command without running it, so you'd see <code>echo Hello</code>, instead of <code>Hello</code>. This allows you to make sure it's the command you intended to recall, and you can simply press the up arrow to execute it.<br><br>
The <code>!</code> syntax can be really useful in a grep-the-history-file workflow. Say I grep for the last time I issued an esxcfg-firewall command:<br>
<pre><code>
history | grep esxcfg
Output:
203 esxcfg-firewall --openPort 9090,tcp,in,SimpleHTTP
!203:p
Output:
esxcfg-firewall --openPort 9090,tcp,in,SimpleHTTP
</code></pre><br>
Now if I hit the up arrow, the command is put on the prompt, and I can edit it to add a different port, etc.<br><br>
If you'd like to delve deeper, google <span style="font-style:italic;">c shell command history</span><br><br>
<span style="font-weight: bold;">Fix Command</span><br>
Another method for recalling commands from the history list is the bash built-in Fix Command, invoked with <code>fc</code>. To get the skinny on <code>fc</code>, bring up the man page for the built-in commands, with <code>man builtins</code>.
Invoking fc with the <code>-l</code> option will print the last 16 commands. You can also specify a range of history commands to display, like:<br>
<pre><code>
fc -l 208 234
</code></pre><br>
The <code>fc</code> command can come in real handy when you are trying to recall and edit a whole series of commands. For instance, say you remember adding an alias to your .bashrc file, and you want to add an additional alias using the same series of commands to make sure it was configured properly. You recall using a <code>cp</code> command first to backup the .bashrc file, and specifying 'cp' as a search string after <code>fc -l</code> will print the history list beginning with the last occurrence of the command that matches the search string:<br>
<pre><code>
fc -l cp
110 cp .bashrc backup_bashrc
111 echo "alias lax='ls -lax'" >> ~/.bashrc
112 cat .bashrc
113 . .bashrc
114 lax
</code></pre><br>
To create an <code>la</code> alias, invoke <code>fc</code> with a range of history events to copy them all into the default editor, which is set to vi in the ESX console:<br>
<pre><code>
fc 110 114
</code></pre><br>
Running the above command will copy the history events 110 through 114 into a vi editor session, where they can be modified to create the alias for <code>la</code>. Typing <code>ZZ</code> in vi will exit the editor and execute all the commands in the buffer: backing up .bashrc, creating the alias, cat'ing the file to see the change, restarting bash, and finally testing the new <code>la</code> alias. I don't use <code>fc</code> very often, but for scenarios like this it is a great tool to be familiar with.<br><br>
<span style="font-weight: bold;">vi Command Editing</span><br>
The next two methods for using the command history rely on functionality provided by the GNU Readline library, a standard set of C functions available to any application, including bash. There are two editing and key-binding modes, emacs and vi. The default mode is emacs, and you can see what mode your shell is in now by typing:<br>
<pre><code>
set | grep SHELLOPTS
</code></pre><br>
If you're in the default emacs mode, you'll see emacs in the colon-separated list of options. To change this to vi mode, type:<br>
<pre><code>
set -o vi
</code></pre><br>
Now if you check the <code>SHELLOPTS</code> variable, you'll find emacs has been replaced with vi. I'm a vi kind of guy, so I always add the set option to my .bashrc file, with a command like:<br>
<pre><code>
echo "set -o vi" >> ~/.bashrc
</code></pre><br>
And source .bashrc again to get the changes:<br>
<pre><code>
. ~/.bashrc
</code></pre><br>
We'll look at how to use the Readline library in vi editing mode with an easy example. I've just grepped the default word file that comes with the service console for every word that begins with 'a' (this word file is not present in ESX 4.0). I'll do the same for the letters 'b' through 'f', and if I execute <code>history</code>, this is the output:<br>
<pre><code>
[admin@esx02 admin]$ history
1 grep ^a /usr/share/dict/words
2 grep ^b /usr/share/dict/words
3 grep ^c /usr/share/dict/words
4 grep ^d /usr/share/dict/words
5 grep ^e /usr/share/dict/words
6 grep ^f /usr/share/dict/words
7 history
</code></pre><br>
Back at the shell prompt, we can search through the history list for the last instance of the grep command. Press <code>Esc</code> to enter vi command editing mode, and then use a forward slash to search, just like in a normal instance of vi:<br>
<pre><code>
/grep
</code></pre><br>
After hitting return, we should find <code>grep ^f /usr/share/dict/words</code> has been placed on the command line. Pressing <code>n</code> will iterate through each grep command until the first instance is found -- the grep command for words starting with 'a'. Continuing to press <code>n</code> will do nothing now, as we've reached the end of the matches for grep. However, pressing <code>N</code> will now iterate through the grep matches in reverse, working its way to the most recent grep command. This is handy and easy to remember, <code>n</code> or <code>N</code> for next, forward or backward through the history list.<br><br>
Of course, the whole point of accessing the history list with this method is to easily edit the commands before executing them. If we wanted to modify the grep command to find all words starting with 'fox', we can just press <code>Esc</code> at a bash prompt to enter vi command editing mode, type <code>/grep ^f</code> to find the history entry where we searched for words starting with f, press <code>i</code> to enter vi insert mode, edit the command: <code>grep ^fox /usr/share/dict/words</code>, and press return to execute it. If we want to get every word that starts with ox, we just perform another search, <code>/grep ^fox</code>, move the cursor over the 'f', and hit <code>x</code> -- the vi delete character command -- to remove it.<br><br>
This example is beyond goofy, but if you play around with this method, you'll find it to be very powerful and a huge timesaver.<br><br>
<span style="font-weight: bold;">Ctrl-r</span><br>
If you press <code>Ctrl-r</code> in your terminal, you'll get a curious looking prompt:<br>
<pre><code>
(reverse-i-search)`':
</code></pre><br>
If you start typing a part of the command you wish to search through the history for, the command will appear after the prompt:<br>
<pre><code>
(reverse-i-search)`grep': grep ^f /usr/share/dict/words
</code></pre><br>
This is the most recent grep found while searching back through the history list. If you press <code>Ctrl-r</code> again, you'll find the one before that:<br>
<pre><code>
(reverse-i-search)`grep': grep ^e /usr/share/dict/words
</code></pre><br>
If you hit return, the command will execute. If you hit the left or right arrow keys, the command will be left on the prompt so you can edit it first.<br><br>
The <code>reverse-i-search</code> prompt is a bash built-in named <span style="font-style:italic;">reverse-search-history</span>. You can see that by default it is bound to the <code>Ctrl-r</code> key by issuing a <code>bind -P</code> command. If you look through the list, you should also see <span style="font-style:italic;">forward-search-history</span> has been bound to <code>Ctrl-s</code>. The <span style="font-style:italic;">forward-search-history</span> command is a useful companion because if you play with <code>Ctrl-r</code>, you'll notice that as it iterates through the history list, it remembers where it left off. So if you reverse search all the way to the start of the history file, you then have to cycle back around, even though the command you want was just one command more recent.<br><br>
But there is a big problem with <span style="font-style:italic;">forward-search-history</span>, it's bound to <code>Ctrl-s</code> by default, which is also bound to the <code>stty</code> stop output command, and <code>stty</code> will intercept it before anything has a chance to see it (if you already pressed <code>Ctrl-s</code> and your terminal appears dead, just type <code>Ctrl-q</code> to bring it back to life).<br><br>
You can view the key bindings used by stty with this command:<br>
<pre><code>
stty -a
</code></pre><br>
But these two commands are too handy to let a little key binding issue get in the way, and we can easily add custom key bindings to our .bashrc file:<br>
<pre><code>
bind '"\C-f": forward-search-history'
bind '"\C-b": reverse-search-history'
</code></pre><br>
Just edit the ~/.bashrc file with the editor you prefer, add those two lines in, and re-source with: <code>. ~/.bashrc</code><br><br>
Now <code>Ctrl-b</code> searches back through the history file, and <code>Ctrl-f</code> searches forward, and you can toggle back and forth between them and the search term will remain. Nice!<br><br>
<span style="font-weight: bold;">Share your secrets</span><br>
Hopefully you found a new trick to play with, and if you have your own history workflow, please leave a comment with the details.<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com0tag:blogger.com,1999:blog-1656249331972710648.post-79139170138893990802009-06-30T15:26:00.000-07:002009-06-30T15:28:46.976-07:00Cut Your Exchange Backup Window in HalfGoing all the way back to the Exchange 5.5 days, I've preferred doing disk-to-disk-to-tape Exchange backups. I'll use NTBackup for the disk-to-disk part, and the regular file backup agent that comes with whatever backup software we happen to be using for the to-tape part. This means eschewing the Exchange backup agents that most vendors provide, which in my mind is a big plus. Opinions will certainly vary, but using this method provides cost savings, fast restore times, extra protection and redundancy, and the reassurance that comes with using the native Exchange backup application. If you need more convincing, there are a few white papers from Microsoft describing this same setup as the Exchange backup method used by their in-house staff.<br><br>
A client that is currently using this configuration has just surpassed the 200 GB mark for their combined mailbox store size, and the nightly full backups were taking a little longer than 6 hours to complete. This large backup window was preventing the nightly database maintenance tasks from completing, so a new strategy was in order. While thinking through some possibilities, I remembered reading about some registry tweaks that could improve the NTBackup performance when backing up to disk. After a little research, I made the changes, and the results were almost unbelievable: the Exchange backup job that had previously taken more than 6 hours to complete now finished in just under 2 1/2 hours!<span class="fullpost"><br><br>
While taking advantage of this dramatic speed boost only requires three registry changes and an additional command line parameter, there is a big bummer at first glance: the NTBackup registry keys that need to be changed reside in the <code>HKEY_CURRENT_USER</code> hive. This really cramps my style as I always configure the scheduled task that kicks off the NTBackup job as the <code>NT AUTHORITY\SYSTEM</code> account with a blank password. If you work in an environment with strict password change policies, even for system accounts, you know the pain of having to maintain passwords in scheduled tasks and scripts. Life is so much easier if it can just be avoided. But since the system account doesn't execute NTBackup interactively, the registry keys don't get created, and I assumed this meant there was no way to have the application check for the configuration tweaks.<br><br>
But thankfully I was wrong, and it's a pretty simple process to manually create the necessary keys in the right spot:<br><br>
<ul>
<li>First of all, you need to actually complete a backup job once to get the registry entries all set up, so as a regular administrator on the Exchange server, launch NTBackup, select a single temp file somewhere to backup, let the job run to completion, and then just delete the temporary backup set.<br><br>
<li>Launch <span style="font-weight: bold;">regedit</span>, and drill down to <code>HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine</code><br><br>
<li>You should already see the values we're about to change, if not, something didn't get created properly, so try a manual NTBackup job again. If the keys are present, make the following changes:<br><br>
<ul>
<li>Change <span style="font-weight: bold;">Logical Disk Buffer Size</span> from 32 to 64<br><br>
<li>Change <span style="font-weight: bold;">Max Buffer Size</span> from 512 to 1024<br><br>
<li>Change <span style="font-weight: bold;">Max Num Tape Buffers</span> from 9 to 16<br><br>
</ul>
<li>After making the changes, select the Backup Engine key from the left pane, and right click and select <span style="font-weight: bold;">Export</span>. Save it as a .reg file, and make sure <span style="font-weight: bold;">Selected branch</span> at the bottom of the Export window is set to <code>HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine</code><br><br>
<li>Now we'll locate the system account's registry settings, with regedit still open, browse to <code>HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup</code>. The <code>S-1-5-18</code> is the standard identifier for the system account, and unless you've scheduled NTBackup to run as <code>NT AUTHORITY\SYSTEM</code> before, the key will most likely be empty.<br><br>
<li>We need to schedule a job to run as <code>NT AUTHORITY\SYSTEM</code> to create the default keys, so launch NTBackup in advanced mode, select the <span style="font-weight: bold;">Schedule Jobs</span> tab, and set up a temp job to just back up any text file and schedule it to run in a couple of minutes from now. When prompted for the credentials that should be used for the job, you'll need to change the user account to <code>NT AUTHORITY\SYSTEM</code> with a blank password, several times. In fact, it still won't save it as the account to use, so after saving the scheduled job, open the task from the <span style="font-weight: bold;">Scheduled Tasks</span> panel and change the user account to <code>NT AUTHORITY\SYSTEM</code> with a blank password again.<br><br>
<li>After the job runs, you should see the following registry keys have been created under <code>HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup</code>; Backup Engine, Backup Utility, Display, and Log Files. But if you drill into Backup Engine, you'll see it didn't create the keys we modified a few steps ago.<br><br>
<li>To easily create the keys, just edit the .reg file we exported earlier in Notepad. Change the line <code>[HKEY_CURRENT_USER\Software\Microsoft\Ntbackup\Backup Engine]</code> to <code>[HKEY_USERS\S-1-5-18\Software\Microsoft\Ntbackup\Backup Engine]</code>, and save the file.<br><br>
<li>Now right click the .reg file, and select <span style="font-weight: bold;">Merge</span>. You should find the registry settings have been created for the system account, and NTBackup will now use the much speedier settings even when running as the system account.<br><br>
</ul>
There's another performance mod we need to make to give the backup even more boost. Since Windows Server 2003 Service Pack 1, NTBackup has been equipped with a secret and offensively named <code>/fu</code> switch, for 'file unbuffered' mode. To bolt this on, just edit the <span style="font-weight: bold;">Scheduled Task</span> for the NTBackup job, and add the <code>/fu</code> switch after the <code>/hc:off</code> parameter. When you're done, the <span style="font-weight: bold;">Run:</span> text box of the Scheduled Task will look something like this:<br><br>
<pre><code>
C:\WINDOWS\system32\ntbackup.exe backup "@C:\Documents and Settings\
Administrator\Local Settings\Application Data\Microsoft\Windows NT\
NTBackup\data\Exchange_Daily.bks" /n "exchange_Backup.bkf created
6/30/2009 at 6:06 PM" /d "Set created 6/30/2009 at 6:06 PM" /v:no
/r:no /rs:no /hc:off /fu /m normal /j "Exchange_Daily" /l:s /f
"E:\exchange_ backups\exchange_Backup.bkf"
</code></pre><br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com11tag:blogger.com,1999:blog-1656249331972710648.post-91013228523627731222009-06-10T19:11:00.000-07:002009-06-10T19:26:36.990-07:00Configure a Vyatta Cluster for Redundant Virtual Firewalls<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFsj-AmP-MeBHErC922TOwWwpKe55tMDgq0MFWhP8_cZ4Xjru7BqmwqEPkKnynnjsGT7svaBvs4kR16zn5DtLmX_YUZa6ddlEQV1u6bAJztwwsHZCd2zUpdE3VQuosuk-Lr4ftHZiWKx8/s1600-h/vyatta-cluster.png"><img style="float:left; margin:20pt 20pt 20pt 20pt;cursor:pointer; cursor:hand;width: 400px; height: 315px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFsj-AmP-MeBHErC922TOwWwpKe55tMDgq0MFWhP8_cZ4Xjru7BqmwqEPkKnynnjsGT7svaBvs4kR16zn5DtLmX_YUZa6ddlEQV1u6bAJztwwsHZCd2zUpdE3VQuosuk-Lr4ftHZiWKx8/s400/vyatta-cluster.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5345887532902103378" /></a>
If you missed the <a href="http://virtualfoundry.blogspot.com/2009/04/protect-console-virtual-firewall.html">Protect the Service Console Network With a Virtual Firewall</a> project, we looked at how to use a Vyatta firewall to protect the ESX Service Console network and restrict SSH and VI or vSphere Client access to only a few specific workstations. Vyatta offers an impressive network operating system that can be run from a live CD, permanently installed on physical or virtual hardware, or downloaded as a virtual appliance. It comes with some high end features like stateful packet inspection, site-to-site VPN, OSPF, and BGP. There's a completely free edition with unrestricted access to all the features, but it can also be purchased with support offerings.<br><br>
If you followed along with the original post, you may have noticed a potential pitfall: what if the ESX server hosting the Vyatta virtual machine goes down? You may have HA enabled, but what if it takes several minutes for the VM to boot all the way up on another host? Even a couple of minutes with no SSH or VM console access during a crisis would feel like an eternity.<br><br>
Amazingly, the Vyatta operating system also includes clustering, and it's very simple to configure. To set up a cluster, we'll need the following<span class="fullpost">:<br><br>
<ul>
<li>Two Vyatta VC5 virtual machines, preferably with very similar configurations, see <a href="http://virtualfoundry.blogspot.com/2009/04/protect-console-virtual-firewall.html">Protect the Service Console Network With a Virtual Firewall</a> for a quick setup tutorial<br><br>
<li>Both Vyatta VMs will need two virtual NICs, each with its own real IP address: one in the Service Console network, and one in a LAN network<br><br>
<li>The clustered Vyatta VMs will host two virtual IPs: one will be the default gateway address configured on every ESX host (172.20.1.254 in our case), and the second will be a LAN address you specify as the route to the Service Console network on the Layer 3 device routing between your LAN subnets. In our case, we have a very simplified setup and the Vyatta's LAN facing interface is the default gateway for the LAN (10.1.1.254)
</ul><br>
There's some good documentation on setting up a cluster in the High Availability Reference Guide for VC5 available for download at the <a href="http://www.vyatta.com">Vyatta</a> website.<br><br>
Once you've got the two Vyatta VMs up and running on different ESX hosts, this is how the cluster configuration will look on the primary Vyatta firewall. We're using a conservative dead-interval of ten seconds, meaning a failover will only occur if keepalives are missed for that long, and keepalives are being sent out every two seconds over the eth0 (Console Network) interface.<br><br>
The <code>service</code> commands define the virtual IPs the cluster will bring up on the secondary if the primary stops responding:<br>
<pre><code>
cluster {
dead-interval 10000
group sc-cluster {
auto-failback true
primary sc-firewall-pri
secondary sc-firewall-sec
service 10.1.1.254/24/eth1
service 172.20.1.254/24/eth0
}
interface eth0
keepalive-interval 2000
pre-shared-secret ****************
}
interfaces {
ethernet eth0 {
address 172.20.1.252/24
hw-id 00:50:56:9c:3b:0b
}
ethernet eth1 {
address 10.1.1.252/24
hw-id 00:50:56:9c:04:a5
}
loopback lo {
}
}
</code></pre><br><br>
And here's the cluster configuration on the secondary firewall. The actual cluster commands are identical, only the real IPs assigned to the interfaces are different:<br>
<pre><code>
cluster {
dead-interval 10000
group sc-cluster {
auto-failback true
primary sc-firewall-pri
secondary sc-firewall-sec
service 10.1.1.254/24/eth1
service 172.20.1.254/24/eth0
}
interface eth0
keepalive-interval 2000
pre-shared-secret ****************
}
interfaces {
ethernet eth0 {
address 172.20.1.253/24
hw-id 00:50:56:9c:3c:e0
}
ethernet eth1 {
address 10.1.1.253/24
hw-id 00:50:56:9c:38:04
}
loopback lo {
}
}
</code></pre><br>
As you can see, it's pretty simple to set up, but one annoyance with the cluster feature is that you have to create the same firewall rules on each device, there's no functionality for syncing up the configurations.<br><br>
It wasn't too hard to write a quick and dirty little shell script to copy just the firewall configuration from the primary to the secondary, however, so you only need to maintain the rules on the primary, and then remember to run the script after saving the changes. If you like, you can set up public key authentication for SSH access from the primary to the secondary like we did in <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-two.html">DIY ESX Server Health Monitoring - Part 2</a>, but it's not necessary, the script will prompt you for the password during the SSH connection attempt.<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
#!/bin/bash
# Vyatta cluster firewall sync script
# by Robert Patton - 2009
#
# Copies firewall rules from primary to secondary
# and applies them to the appropriate interfaces.
#
# Deletes existing firewall rules on secondary and
# removes any firewall sets on interfaces, so make
# sure this is only run from the primary.
#
# Replace the SECONDARY value with the hostname or IP
# of the secondary device in the cluster.
SECONDARY="sc-firewall-sec"
TEMPFWRULES=$(mktemp TEMPFWRULES.XXXXXXXX)
TEMPINTCMDS=$(mktemp TEMPINTCMDS.XXXXXXXX)
TEMPSETCMDS=$(mktemp TEMPSETCMDS.XXXXXXXX)
# Match just the firewall section from the boot config file
awk '/^firewall {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPFWRULES
# Match the interface section, we filter for firewall set statements later
awk '/^interfaces {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPINTCMDS
# Create a script to run on the secondary with the firewall set commands
# The vyatta-config-gen-sets.pl script creates set commands from the config
cat > $TEMPSETCMDS <<'EOF1'
configure
# First remove any firewalls from interfaces
for int in $(show interfaces ethernet | \
awk '/eth[0-9]/ {print $1}'); \
do delete interfaces ethernet $int firewall; \
done
# Now delete all firewalls
for fwall in $(show firewall name | \
awk '/^ \w* {$/ {print $1}')
do delete firewall name $fwall; \
done
EOF1
cat >> $TEMPSETCMDS <<EOF2
# Create firewalls found on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPFWRULES)
# Apply firewalls to interfaces as defined on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPINTCMDS | grep firewall)
commit
save
exit
exit
EOF2
# Force a tty for the ssh connection - Vyatta environment variables
# and special shell are only set up during an interactive login
cat $TEMPSETCMDS | ssh -tt $SECONDARY
rm -f $TEMPFWRULES $TEMPSETCMDS $TEMPINTCMDS
</code></pre>
</div>
</td></tr></table><br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com3tag:blogger.com,1999:blog-1656249331972710648.post-177785955706607302009-06-05T16:18:00.000-07:002009-06-05T16:36:49.100-07:00Another ESX Server HTTP File TrickWhile reading some docs on the vSphere CLI, I came across this note:<br><br>
<span style="font-style:italic;">
...you can browse datastore contents and host files using a Web browser. Connect to the following location:<br><br>
http://ESX_host_IP_Address/host<br>
http://ESX_host_IP_Address/folder<br><br>
You can view datacenter and datastore directories from this root URL...
</span>
<br><br>
I knew you could browse the datastores this way as there is a link from the main welcome page, but the <code>/host</code> URL is news to me. Log in with the root account, and it brings up a page titled <span style="font-style:italic;">Configuration files</span> with links to view a bunch of important - you guessed it - configuration files<span class="fullpost">.<br><br>
Not too terribly interesting, but I can already see myself hitting this URL just to get the vSphere license key or double check that the proper entries are in an ESX server's <code>hosts</code> file.<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com0tag:blogger.com,1999:blog-1656249331972710648.post-42320648754030912992009-05-29T11:04:00.000-07:002009-05-29T11:09:39.171-07:00Instantly Serve an ESX Directory via HTTP<br>
<code>python -m SimpleHTTPServer 9090</code>
<br><br>
Before you type that in, understand that it's not going to work unless you've made some ill-advised changes to the Service Console firewall. Also, make sure you fully grasp the security risk you're about to take. This command will start up a simple web server on TCP port 9090 in the current working directory, allowing anyone to browse the files and subdirectories from a web browser under the security context of the user that executed the command. In other words, if you execute this as the root user, in the root directory, any file in the Service Console can be downloaded from a web browser.<br><br>
This one-liner is extremely dangerous, but it is also extremely handy, and if used correctly in a properly designed environment, the potential risks can be managed. I use this all the time in my test lab to get output files from scripts by simply cd'ing to the script directory, running the above command, and pointing a web browser to <span style="font-weight: bold;">http://IP_OF_ESX:9090</span> from the vCenter server<span class="fullpost">.<br><br>
<span style="font-weight: bold;">How to make it safer:</span><br>
<ul>
<li>The ESX Service Console network should be completely isolated from the LAN, and only vCenter servers and specific administrative workstations are allowed access<br><br>
<li>The Python command should be executed while the working directory is a folder created just for this purpose, and only contains the specific files you want to share and no subdirectories<br><br>
<li>The command should only be executed by a non-root user and the web server torn down as soon as the files have been downloaded by issuing a <code>Ctrl-C</code><br><br>
<li>The root user must open a specific port in the firewall prior to using the command; for example, to open TCP port 9090:<br><br>
<code>esxcfg-firewall --openPort 9090,tcp,in,SimpleHTTP</code><br><br>
<li>The port should then be closed immediately after the needed files have been downloaded; for example, to close down the previous command:<br><br>
<code>esxcfg-firewall --closePort 9090,tcp,in</code>
</ul><br><br>
This also works in ESX 3.5, but the version of Python in the Service Console lacks the <code>-m</code> option, so the path to <code>SimpleHTTPServer.py</code> must be specified:
<br><br>
<code># ESX 3.5<br>
python /usr/lib/python2.2/SimpleHTTPServer.py 9090</code>
<br><br>
Might be too dangerous for production, so consider the risks carefully. But for testing, it can be really handy.<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com0tag:blogger.com,1999:blog-1656249331972710648.post-88876835298585282732009-05-26T13:23:00.000-07:002009-05-26T14:00:15.633-07:00VM Security in vSphere - Same Ol' Situation (S.O.S.)Over the weekend, I had a chance to test out the directives for locking down the virtual machine security issues discussed in <a href="http://virtualfoundry.blogspot.com/2009/04/hardening-vmx-file.html">Hardening the VMX File</a> with vSphere / ESX 4.0. Unfortunately, all of the security issues are still present in the GA release of vSphere, including non-privileged users having the ability to disconnect virtual NICs and change the time synchronization behavior<span class="fullpost">.<br><br>
I can't imagine why this situation still persists through version 4.0 of VMware's enterprise virtualization platform. Are there customers who prefer non-privileged user accounts retain this ability? And if so, couldn't we disable this functionality by default, and require .vmx directives to enable it?<br><br>
Yes, it is easy to change the default settings, and any sysadmin worth his or her salary will make the changes and audit their environment for compliance. That's a tired argument, however, and better "out of the box" security should be a goal for any product. Anybody remember Windows 2000?<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com0tag:blogger.com,1999:blog-1656249331972710648.post-21959477604911584402009-05-25T19:01:00.000-07:002009-05-25T19:03:40.507-07:00DIY ESX Server Health Monitoring - Part 4If you're just catching this series on creating an ESX health report, in <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-one.html">Part 1</a>, <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-two.html">Part 2</a>, and <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-three.html">Part 3</a> we set up everything we need to schedule the daily health check and send the results in a HTML formatted email. Running the health check once a day is probably not sufficient if you want to be on top of developing issues, however, and if you have a lot of ESX hosts, reading through a long list of performance statistics may be unreasonable. So to wrap this project up, we'll look at setting up a second <code>cron</code> job that will only send out an alert message when an ESX host exceeds a specified threshold.<br><br>
Due to the simple design of the health report scripts, to set up this functionality we only need to modify a few lines from the <code>run-esx-report.sh</code> script<span class="fullpost">:<br>
<ul>
<li>The first change is in the loop where we SSH into each ESX host and run the esx-report.sh script. We'll simply change the append redirection symbols, <code>>></code>, to the create or truncate symbol, <code>></code>, this way we're creating a new report output file for each host, rather than a combined report. To be extra sure the temp file is truncated each time through the loop, we'll use the noclobber override option as well, so the <code>>></code> symbols become <code>>|</code><br><br>
<li>Next, we <code>grep</code> for the word WARNING in the output file, and wrap the rest of the script in an <code>if</code> statement so the email is only sent out if the <code>grep</code> command returns true<br><br>
<li>And finally, we'll just change the subject of the email message<br><br>
</ul>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
###############################################################################
#
# run-esx-threshold.sh
#
###############################################################################
#
# To create the run-esx-threshold.sh script in the ~/esx-report directory,
# copy this entire code segment into your shell.
# If you'd rather copy just the script, select everything between the
# SCRIPTCREATOR limit strings.
#
# putty will ignore all the tabs, making the copied script quite ugly
#
###############################################################################
# If the ~/esx-report directory exists, cd to it so the script is created there
[ -d ~/esx-report ] && cd ~/esx-report
cat > ./run-esx-threshold.sh <<'SCRIPTCREATOR'
#! /bin/bash
PATH="/bin:/usr/bin"
if [ -z $1 ]; then
echo "No ESX hosts specified, exiting"
exit 1
fi
if ! pgrep ssh-agent >/dev/null; then
echo "The ssh-agent process does not appear to be running, exiting"
exit 1
fi
RUNDIR=$(dirname "$(which "$0")")
source "${HOME}/.ssh-agent" >/dev/null || exit 1
THISHOST=$(hostname | cut -d . -f 1)
TEMPTEXT=$(mktemp "${RUNDIR}/temptext.XXXXXXXXXX")
TEMPHTML=$(mktemp "${RUNDIR}/temphtml.XXXXXXXXXX")
for host in $@; do
if [ $(echo $host | cut -d . -f 1) = $THISHOST ]; then
"${RUNDIR}/esx-report.sh" >| "$TEMPTEXT"
else
ssh -q $host "$(cat "${RUNDIR}/esx-report.sh")" >| "$TEMPTEXT" || \
printf "WARNING: SSH connection to $host failed\n\n\n\n" >| "$TEMPTEXT"
fi
if grep WARNING "$TEMPTEXT" >/dev/null; then
cat >| "$TEMPHTML" <<-'HEADEREOF'
	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	<html>
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
	<style type="text/css">
	body { font-family: monospace; font-size: 12px }
	pre { font-family: monospace; font-size: 12px }
	</style>
	</head>
	<body>
	<pre>
	HEADEREOF
cat "$TEMPTEXT" | \
sed -e 's/>/\&#62/g' \
-e 's/WARNING:.*/<span style="color: red">&<\/span>/' >> "$TEMPHTML"
cat >> "$TEMPHTML" <<-'FOOTEREOF'
	</pre>
	</body>
	</html>
	FOOTEREOF
"${RUNDIR}/html-mailer.pl" -f esx-report@yourdomain.dom \
-r administrator@yourdomain.dom \
-s "Alert on $host" \
-m exchange.yourdomain.com \
-b "$TEMPHTML"
fi
done
rm -f "$TEMPTEXT"; rm -f "$TEMPHTML"
SCRIPTCREATOR
chmod 0700 ./run-esx-threshold.sh
###############################################################################
</code></pre>
</div>
</td></tr></table><br><br>
<span style="font-weight: bold;">Don't spam yourself</span><br>
When considering how often you want to run the threshold check script, keep one shortcoming of this method in mind: if a parameter continues to exceed its threshold, the script will continue to email you every time it runs. If you set this up to run every five minutes, and head out into the woods over a holiday weekend, you're going to get a thousand alert messages before you get a chance to resolve the issue.<br><br>
For our purposes, once every 30 minutes will suffice, and so we'll add another <code>cron</code> job by issuing a <code>crontab -e</code> command as the non-root user, press <code>i</code> to enter insert mode, and below the line containing the 7:10 AM ESX server health report job, we'll add:<br>
<pre><code>
0,30 * * * * ${HOME}/esx-report/run-esx-threshold.sh ESX LIST >/dev/null 2>&1
</code></pre><br>
Press <code>Esc</code>, then <code>:wq</code> to write the crontab and exit <code>vi</code>, and we're done!<br><br>
If you do want to run the threshold check every five minutes, instead of specifying a list like 0,5,10,15, etc., use the range of minutes followed by a forward slash and interval, like:
<pre><code>
0-59/5 * * * * ${HOME}/esx-report/run-esx-threshold.sh ESX LIST >/dev/null 2>&1
</code></pre><br>
<span style="font-weight: bold;">Tweak the thresholds</span><br>
You'll definitely want to play with the threshold settings from the <code>esx-report.sh</code> script in <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-one.html">Part 1</a>. The threshold is the third parameter supplied to the <code>scale</code> function, and I've highlighted it below for the memory usage check:
<pre><code>
printf " Memory Usage:\n"
(free | awk '/^Mem:/ {print $3, $2, <span style="color: red;">"100"</span>, $1}
/^Swap:/ {print $3, $2, <span style="color: red;">"1"</span>, $1}') | \
while read line; do scale $line; done
</code></pre><br>
That does it for the DIY ESX Server Health Monitoring project, I hope you'll find this information easy to customize for your own environment. If you add new performance checks or enhancements, feel free to describe the changes in a comment.<br><br>
<span style="font-weight: bold;">Install it</span><br>
If you'd like to set the whole thing up, just copy and paste each code segment with a light blue background into a <code>putty</code> session. To install:
<ul>
<li>Create the <code>esx-report.sh</code> script from <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-one.html">Part 1</a> as the non-root user. Copying the entire code segment in the light blue box into a <code>putty</code> window will create the <code>esx-report</code> folder under the home folder of the user that executes it.<br><br>
<li>From <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-two.html">Part 2</a>, execute the <code>ssh-keygen</code> command as the non-root user. Then run the <code>esxcfg-firewall</code> command as root to open an outbound port for SSH. Create the two remaining scripts; <code>copykey.sh</code>, and <code>start-ssh-agent.sh</code> as the non-root user.<br>
Use <code>copykey.sh</code> to distribute the public key file, then launch ~/<code>start-ssh-agent.sh</code> to load the private key into memory, both as the non-root user. Make test <code>ssh</code> connections to each ESX host you need to run the report on, but make sure you source the <code>.ssh-agent</code> file first so the variables are exported to your shell, <code>source ~/.ssh-agent</code><br><br>
<li>Now create the <code>html-mailer.pl</code> script from <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-three.html">Part 3</a> as the non-root user. As root, run the <code>esxcfg-firewall</code> command to open outbound SMTP in the firewall. Change users back to the non-root user, and create the <code>run-esx-report.sh</code> script and change the email settings for your environment.<br><br>
<li>Create the <code>run-esx-threshold.sh</code> script from this post as the non-root user and change the email settings.<br><br>
<li>Set up the <code>cron</code> jobs for the daily health report and the threshold check. Customize the whole thing any way you see fit.<br><br>
</ul>
<span style="font-weight: bold;">*</span> A couple of tips:
<ul>
<li>Try to schedule the daily health check and threshold checks so they don't run at the same time. The jobs will run fine simultaneously, but the usage numbers could be inflated.<br><br>
<li>Configure reverse DNS records for your ESX hosts on the DNS servers they point to or you'll see long pauses during SSH connection attempts as the server times out attempting to resolve the connecting client's hostname from its IP.
</ul><br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com0tag:blogger.com,1999:blog-1656249331972710648.post-22897308858152132022009-05-19T18:55:00.001-07:002009-06-18T12:16:33.623-07:00DIY ESX Server Health Monitoring - Part 3<span style="font-weight: bold;">Updated: June 18, 2009</span><br>
<span style="font-style:italic;">Added a semicolon to run-esx-report.sh that was left out and responsible for some ugly HTML formatting.</span><br><br>
With the secure SSH access problem solved in <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-two.html">Part 2</a>, we'll move on to getting the data in the proper format and emailing it from the ESX Service Console. As you probably know, the Linux distribution installed with ESX 3.5 lacks <code>sendmail</code> or an equivalent command, but we can roll our own from a perl script.<br><br>
<span style="font-weight: bold;">The perl mailer script</span><br>
We need to import two perl modules for the script, and both are included by default in the Service Console. <code>Getopt::Std</code> provides a simple way to get command line options, and <code>Net::SMTP</code> will interface with an Exchange or SMTP server accessible from the console network<span class="fullpost">:<br>
<pre><code>
use Getopt::Std;
use Net::SMTP;
</code></pre><br>
This <code>getopt</code> call is all that's necessary to declare the command line options (-f, -r, -s, etc.), and it will automatically populate a set of corresponding variables named <code>opt_*</code>. We'll do a quick check to make sure all the command line options were specified, and if not display the usage message:<br>
<pre><code>
getopt ('frsmb');
unless ($opt_f && $opt_r && $opt_s && $opt_m && $opt_b) {
print_usage();
exit 1;
}
</code></pre><br>
Next we'll create a filehandle named <code>BODY</code>, opening the file specified on the command line. After reading in each line to the variable <code>body_data</code>, we'll close the handle:<br>
<pre><code>
open(BODY, $opt_b) || error("Could not open file $opt_b.");
my @body_data=<BODY>;
close(BODY);
</code></pre><br>
The <code>Net::SMTP</code> module is pretty straightforward. To generate a HTML formatted email, we just need to specify the MIME version, the content type as HTML, and the character encoding as ISO 8859-1. If you would rather send the message as plain text, just remove those two lines:<br>
<pre><code>
my $smtp = Net::SMTP->new($opt_m) ||
error("SMTP connection to $opt_m failed.");
$smtp->mail($opt_f);
$smtp->to($opt_r);
$smtp->data();
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-Type: text/html; charset=iso-8859-1\n");
$smtp->datasend("To: $opt_r\n");
$smtp->datasend("From: $opt_f\n");
$smtp->datasend("Subject: $opt_s\n");
foreach $line (@body_data)
{
$smtp->datasend("$line");
}
$smtp->dataend();
$smtp->quit;
</code></pre><br>
Here's the complete <code>html-mailer.pl</code> script:<br><br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
###############################################################################
#
# html-mailer.pl
#
###############################################################################
#
# To create the html-mailer.pl script in the ~/esx-report directory, copy
# this entire code segment into your shell.
# If you'd rather copy just the script, select everything between the
# SCRIPTCREATOR limit strings.
#
###############################################################################
# If the ~/esx-report directory exists, cd to it so the script is created there
[ -d ~/esx-report ] && cd ~/esx-report
cat > ./html-mailer.pl <<'SCRIPTCREATOR'
#! /usr/bin/perl -w
use strict;
use Getopt::Std;
use Net::SMTP;
# Options:
# $opt_f email address of the sender
# $opt_r recipient email address
# $opt_s message subject, enclose in quotes if spaces
# $opt_m SMTP server FQDN or IP address
# $opt_b HTML formatted file for the message body
our ($opt_f, $opt_r, $opt_s, $opt_m, $opt_b);
getopt ('frsmb');
unless ($opt_f && $opt_r && $opt_s && $opt_m && $opt_b) {
print_usage();
exit 1;
}
open(BODY, $opt_b) || error("Unable to open file $opt_b");
my @body_data=<BODY>;
close(BODY);
my $line;
my $smtp = Net::SMTP->new($opt_m) ||
error("SMTP connection to $opt_m failed");
$smtp->mail($opt_f);
$smtp->to($opt_r);
$smtp->data();
$smtp->datasend("MIME-Version: 1.0\n");
$smtp->datasend("Content-Type: text/html; charset=iso-8859-1\n");
$smtp->datasend("To: $opt_r\n");
$smtp->datasend("From: $opt_f\n");
$smtp->datasend("Subject: $opt_s\n");
foreach $line (@body_data)
{
$smtp->datasend("$line");
}
$smtp->dataend();
$smtp->quit;
sub error {
my $msg = shift;
print STDERR "html-mailer.pl: $msg\n";
exit 1;
}
sub print_usage {
print STDERR <<EOF
html-mailer.pl - HTML Formatted Message Mailer
Usage: html-mailer.pl -f FROM -r RECIP -s SUBJ -m SMTP_HOST -b HTML_FILE
Sends an email to the specified address, filling the message body with the
HTML formatted file specified.
EOF
}
SCRIPTCREATOR
chmod 0700 ./html-mailer.pl
###############################################################################
</code></pre>
</div>
</td></tr></table><br><br>
<span style="font-weight: bold;">Enable outbound SMTP</span><br>
Now that we've got a script that we can send test messages with, we need to enable outbound SMTP through the ESX firewall on the ESX server that will have the script scheduled from a <code>cron</code> job. Just type this command as root to open the port:
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
# Execute as root
esxcfg-firewall --openPort 25,tcp,out,SMTP
</code></pre>
</div>
</td></tr></table><br><br>
If your Exchange or SMTP server is reachable from the Service Console network, execute <code>html-mailer.pl</code> with the appropriate parameters, and specify any old text file:
<pre><code>
./html-mailer.pl -f me@mydomain.dom \
-r me@mydomain.dom \
-s "Test message" \
-m exchange.mydomain.dom \
-b ./testfile.txt
</code></pre><br><br>
<span style="font-weight: bold;">The Service Console can't reach the Exchange server...</span><br>
No worries, as long as you're able to reach the VirtualCenter server, we can install the SMTP service and set it up to forward to the Exchange server. To install SMTP on a Windows 2003 server, do the following:<br><br>
<ul>
<li>Open the <span style="font-weight: bold;">Control Panel</span> > <span style="font-weight: bold;">Add or Remove Programs</span> > <span style="font-weight: bold;">Add/Remove Windows Components</span> > double click <span style="font-weight: bold;">Application Server</span> > and then double click <span style="font-weight: bold;">Internet Information Server (IIS)</span>. Put a check next to <span style="font-weight: bold;">SMTP Service</span> and click <span style="font-weight: bold;">OK</span>, <span style="font-weight: bold;">OK</span>, and <span style="font-weight: bold;">Next</span><br><br>
<li>After the SMTP install is complete, open the <span style="font-weight: bold;">Start Menu</span> > <span style="font-weight: bold;">Programs</span> > <span style="font-weight: bold;">Administrative Tools</span> > <span style="font-weight: bold;">Internet Information Services (IIS) Manager</span>, then right click <span style="font-weight: bold;">Default SMTP Virtual Server</span> and select <span style="font-weight: bold;">Properties</span><br><br>
<li>In the <span style="font-weight: bold;">General</span> tab, drop down the <span style="font-weight: bold;">IP address:</span> to the IP address in the Service Console network, if different from the LAN. This will prevent the SMTP service from popping up on your network security guy's port scans :)<br><br>
<li>In the <span style="font-weight: bold;">Access</span> tab, click the <span style="font-weight: bold;">Connection</span> button, choose the <span style="font-weight: bold;">Only the list below</span> radio button, then click <span style="font-weight: bold;">Add</span> to add the appropriate subnet address and mask to the <span style="font-weight: bold;">Group of computers</span> option, or add each ESX server one at a time<br><br>
<li>In the <span style="font-weight: bold;">Access</span> tab again, click the <span style="font-weight: bold;">Relay</span> button, choose the <span style="font-weight: bold;">Only the list below</span> radio button, then click <span style="font-weight: bold;">Add</span> to add the appropriate subnet address and mask to the <span style="font-weight: bold;">Group of computers</span> option, or add each ESX server one at a time. Uncheck the option <span style="font-weight: bold;">Allow all computers which successfully authenticate to relay</span><br><br>
<li>On the <span style="font-weight: bold;">Delivery</span> tab, click the <span style="font-weight: bold;">Advanced</span> button and add your Exchange server information in the <span style="font-weight: bold;">Smart host:</span> box. By specifying a smart host, the SMTP server will simply forward everything to the Exchange server, letting it make all the decisions about which domains to accept mail for, etc.<br><br>
<li>Now test out the SMTP forwarder by using <code>telnet</code> to initiate a SMTP session from the ESX server that will be sending the messages:<br>
<pre><code>
telnet virtualcenter.lab.local 25
ehlo
mail from:spongebob@lab.local
rcpt to:administrator@lab.local
data
Subject:test
.
quit
</code></pre><br>
</ul>
<span style="font-weight: bold;">One script to rule them all</span><br>
Almost there, so let's recap what we've done so far. In Part 1, we created the health check script that will run on each ESX server and send key performance stats and scaled histograms to the terminal. Then in Part 2, we covered how to distribute public keys so the script can be executed on several ESX servers via SSH. So far in Part 3, we've looked at a perl script that will email the combined script output, and now we need to create a script to tie it all together, and then schedule the script from a <code>cron</code> job.<br><br>
Let's break down the main components of the script. First of all, if <code>ssh-agent</code> isn't running, the script isn't going to get very far, so we'll use <code>pgrep</code> to check for the process and exit if it's not found:<br>
<pre><code>
if ! pgrep ssh-agent >/dev/null; then
echo "The ssh-agent process does not appear to be running, exiting"
exit 1
fi
</code></pre><br>
We need to source <code>.ssh-agent</code>, the file with the <code>ssh-agent</code> PID and socket info set up by the <code>start-ssh-agent.sh</code> script, or exit if it doesn't exist:<br>
<pre><code>
source "${HOME}/.ssh-agent" >/dev/null || exit 1
</code></pre><br>
Since we'll be running everything from an ESX Service Console, and that server is likely to be part of the health check, we should compare the list of ESX hosts to the local hostname so we don't open a SSH connection to the local machine. We use <code>cut</code> here to strip off the domain name so we'll match whether the FQDN or just the bare hostname is specified:<br>
<pre><code>
THISHOST=$(hostname | cut -d . -f 1)
for host in $@; do
if [ $(echo $host | cut -d . -f 1) = $THISHOST ]; then
"${RUNDIR}/esx-report.sh" >> "$TEMPTEXT"
else
ssh -q $host "$(cat "${RUNDIR}/esx-report.sh")" >> "$TEMPTEXT" || \
printf "WARNING: SSH connection to $host failed\n\n\n\n" >> "$TEMPTEXT"
fi
done
</code></pre><br>
After the health check script has looped through the list of ESX hosts, we'll start building the HTML file with the necessary tags. Setting the font size for the <code>pre</code> tag is the secret sauce for getting the email to display perfectly on a BlackBerry:<br>
<pre><code>
cat > "$TEMPHTML" <<-'HEADEREOF'
	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	<html>
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
	<style type="text/css">
	body { font-family: monospace; font-size: 12px }
	pre { font-family: monospace; font-size: 12px }
	</style>
	</head>
	<body>
	<pre>
	HEADEREOF
</code></pre><br>
If you need to use < or > symbols in a HTML document, you have to specify the actual ASCII code of the character, as HTML considers words wrapped in those symbols to be tags. We'll use a <code>sed</code> filter to replace all the >'s with the ASCII equivalent, and add a color tag to any lines with the word WARNING to make it stand out:<br>
<pre><code>
cat "$TEMPTEXT" | \
sed -e 's/>/\&#62/g' \
-e 's/WARNING:.*/<span style="color: red">&<\/span>/' >> "$TEMPHTML"
</code></pre>
Then we'll add the closing tags for everything to the end of the HTML file:<br>
<pre><code>
cat >> "$TEMPHTML" <<-'FOOTEREOF'
	</pre>
	</body>
	</html>
	FOOTEREOF
</code></pre><br>
And finally, we'll execute <code>html-mailer.pl</code> with the appropriate parameters. You'll need to change this section of the script for your environment:<br>
<pre><code>
"${RUNDIR}/html-mailer.pl" -f esx-report@lab.local \
-r administrator@lab.local \
-s "ESX Health Report" \
-m lab-vc \
-b "$TEMPHTML"
</code></pre><br>
Here's the <code>run-esx-report.sh</code> script. Remember to change the email address and mail server parameters for your environment:<br>
<table border="0" width="100%"><tr><td>
<div class="codesegment">
<pre><code>
###############################################################################
#
# run-esx-report.sh
#
###############################################################################
#
# To create the run-esx-report.sh script in the ~/esx-report directory, copy
# this entire code segment into your shell.
# If you'd rather copy just the script, select everything between the
# SCRIPTCREATOR limit strings.
#
# putty will ignore all the tabs, making the copied script quite ugly
#
###############################################################################
# If the ~/esx-report directory exists, cd to it so the script is created there
[ -d ~/esx-report ] && cd ~/esx-report
cat > ./run-esx-report.sh <<'SCRIPTCREATOR'
#! /bin/bash
PATH="/bin:/usr/bin"
if [ -z $1 ]; then
echo "No ESX hosts specified, exiting"
exit 1
fi
if ! pgrep ssh-agent >/dev/null; then
echo "The ssh-agent process does not appear to be running, exiting"
exit 1
fi
RUNDIR=$(dirname "$(which "$0")")
source "${HOME}/.ssh-agent" >/dev/null || exit 1
THISHOST=$(hostname | cut -d . -f 1)
TEMPTEXT=$(mktemp "${RUNDIR}/temptext.XXXXXXXXXX")
TEMPHTML=$(mktemp "${RUNDIR}/temphtml.XXXXXXXXXX")
for host in $@; do
if [ $(echo $host | cut -d . -f 1) = $THISHOST ]; then
"${RUNDIR}/esx-report.sh" >> "$TEMPTEXT"
else
ssh -q $host "$(cat "${RUNDIR}/esx-report.sh")" >> "$TEMPTEXT" || \
printf "WARNING: SSH connection to $host failed\n\n\n\n" >> "$TEMPTEXT"
fi
done
cat > "$TEMPHTML" <<-'HEADEREOF'
	<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
	<html>
	<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
	<style type="text/css">
	body { font-family: monospace; font-size: 12px }
	pre { font-family: monospace; font-size: 12px }
	</style>
	</head>
	<body>
	<pre>
	HEADEREOF
cat "$TEMPTEXT" | \
sed -e 's/>/\&#62;/g' \
-e 's/WARNING:.*/<span style="color: red">&<\/span>/' >> "$TEMPHTML"
cat >> "$TEMPHTML" <<-'FOOTEREOF'
	</pre>
	</body>
	</html>
	FOOTEREOF
"${RUNDIR}/html-mailer.pl" -f esx-report@yourdomain.dom \
-r administrator@yourdomain.dom \
-s "ESX Health Report" \
-m exchange.yourdomain.com \
-b "$TEMPHTML"
rm -f "$TEMPTEXT"; rm -f "$TEMPHTML"
SCRIPTCREATOR
chmod 0700 ./run-esx-report.sh
###############################################################################
</code></pre>
</div>
</td></tr></table><br><br>
<span style="font-weight: bold;">To cron foo, thanks for everything</span><br>
Still with us? One more step, and it's an easy one. We'll add a <code>cron</code> job to run the script at 7:10 AM every morning. Remember to add the job for the user account you distributed SSH keys for.<br><br>
To edit the <code>cron</code> entries for the user, type:<br>
<pre><code>
crontab -e
</code></pre><br>
This starts <code>vi</code> and opens up the user's crontab. To enter insert mode, type <code>i</code><br><br>
Assuming you've set everything up using the code segments used in this series, to add an entry for 7:10 AM, type this line, replacing <span style="font-style:italic;">ESX LIST</span> with a space separated list of ESX hosts:<br>
<pre><code>
10 7 * * * ${HOME}/esx-report/run-esx-report.sh ESX LIST >/dev/null 2>&1
</code></pre><br>
After adding the entry, press the <code>Esc</code> key, and type <code>:wq</code> to write the crontab and quit.<br><br>
If you have a long list of hosts, put them all in a text file, separated by spaces or each on its own line, and use command substitution to feed the list to <code>run-esx-report.sh</code><br>
<pre><code>
run-esx-report.sh $(cat ${HOME}/esx-report/hostlist.txt)
</code></pre><br>
<span style="font-weight: bold;">There's more?!</span><br>
What if we wanted to trigger an email warning if an ESX host exceeds a threshold value? As we'll see in <a href="http://virtualfoundry.blogspot.com/2009/05/diy-esx-monitoring-four.html">Part 4</a>, we can do this easily with a quick modification to the <code>run-esx-report.sh</code> script.<br><br>
</span>Robert Pattonhttp://www.blogger.com/profile/09699729758385238019noreply@blogger.com2