Showing posts with label security. Show all posts
Showing posts with label security. Show all posts

June 10, 2009

Configure a Vyatta Cluster for Redundant Virtual Firewalls

If you missed the Protect the Service Console Network With a Virtual Firewall project, we looked at how to use a Vyatta firewall to protect the ESX Service Console network and restrict SSH and VI or vSphere Client access to only a few specific workstations. Vyatta offers an impressive network operating system that can be run from a live CD, permanently installed on physical or virtual hardware, or downloaded as a virtual appliance. It comes with some high end features like stateful packet inspection, site-to-site VPN, OSPF, and BGP. There's a completely free edition with unrestricted access to all the features, but it can also be purchased with support offerings.

If you followed along with the original post, you may have noticed a potential pitfall: what if the ESX server hosting the Vyatta virtual machine goes down? You may have HA enabled, but what if it takes several minutes for the VM to boot all the way up on another host? Even a couple of minutes with no SSH or VM console access during a crisis would feel like an eternity.

Amazingly, the Vyatta operating system also includes clustering, and it's very simple to configure. To set up a cluster, we'll need the following:

  • Two Vyatta VC5 virtual machines, preferably with very similar configurations, see Protect the Service Console Network With a Virtual Firewall for a quick setup tutorial

  • Both Vyatta VMs will need two virtual NICs, each with its own real IP address: one in the Service Console network, and one in a LAN network

  • The clustered Vyatta VMs will host two virtual IPs: one will be the default gateway address configured on every ESX host (172.20.1.254 in our case), and the second will be a LAN address you specify as the route to the Service Console network on the Layer 3 device routing between your LAN subnets. In our case, we have a very simplified setup and the Vyatta's LAN facing interface is the default gateway for the LAN (10.1.1.254)

There's some good documentation on setting up a cluster in the High Availability Reference Guide for VC5 available for download at the Vyatta website.

Once you've got the two Vyatta VMs up and running on different ESX hosts, this is how the cluster configuration will look on the primary Vyatta firewall. We're using a conservative dead-interval of ten seconds, meaning a failover will only occur if keepalives are missed for that long, and keepalives are being sent out every two seconds over the eth0 (Console Network) interface.

The service commands define the virtual IPs the cluster will bring up on the secondary if the primary stops responding:

cluster {
    dead-interval 10000
    group sc-cluster {
        auto-failback true
        primary sc-firewall-pri
        secondary sc-firewall-sec
        service 10.1.1.254/24/eth1
        service 172.20.1.254/24/eth0
    }
    interface eth0
    keepalive-interval 2000
    pre-shared-secret ****************
}

interfaces {
    ethernet eth0 {
        address 172.20.1.252/24
        hw-id 00:50:56:9c:3b:0b
    }
    ethernet eth1 {
        address 10.1.1.252/24
        hw-id 00:50:56:9c:04:a5
    }
    loopback lo {
    }
}


And here's the cluster configuration on the secondary firewall. The actual cluster commands are identical, only the real IPs assigned to the interfaces are different:

cluster {
    dead-interval 10000
    group sc-cluster {
        auto-failback true
        primary sc-firewall-pri
        secondary sc-firewall-sec
        service 10.1.1.254/24/eth1
        service 172.20.1.254/24/eth0
    }
    interface eth0
    keepalive-interval 2000
    pre-shared-secret ****************
}

interfaces {
    ethernet eth0 {
        address 172.20.1.253/24
        hw-id 00:50:56:9c:3c:e0
    }
    ethernet eth1 {
        address 10.1.1.253/24
        hw-id 00:50:56:9c:38:04
    }
    loopback lo {
    }
}

As you can see, it's pretty simple to set up, but one annoyance with the cluster feature is that you have to create the same firewall rules on each device, there's no functionality for syncing up the configurations.

It wasn't too hard to write a quick and dirty little shell script to copy just the firewall configuration from the primary to the secondary, however, so you only need to maintain the rules on the primary, and then remember to run the script after saving the changes. If you like, you can set up public key authentication for SSH access from the primary to the secondary like we did in DIY ESX Server Health Monitoring - Part 2, but it's not necessary, the script will prompt you for the password during the SSH connection attempt.


#!/bin/bash

# Vyatta cluster firewall sync script
# by Robert Patton - 2009
#
# Copies firewall rules from primary to secondary
# and applies them to the appropriate interfaces.
#
# Deletes existing firewall rules on secondary and
# removes any firewall sets on interfaces, so make
# sure this is only run from the primary.
#
# Replace the SECONDARY value with the hostname or IP
# of the secondary device in the cluster.

SECONDARY="sc-firewall-sec"

TEMPFWRULES=$(mktemp TEMPFWRULES.XXXXXXXX)
TEMPINTCMDS=$(mktemp TEMPINTCMDS.XXXXXXXX)
TEMPSETCMDS=$(mktemp TEMPSETCMDS.XXXXXXXX)

# Match just the firewall section from the boot config file
awk '/^firewall {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPFWRULES

# Match the interface section, we filter for firewall set statements later
awk '/^interfaces {/, /^}/' /opt/vyatta/etc/config/config.boot > $TEMPINTCMDS

# Create a script to run on the secondary with the firewall set commands
# The vyatta-config-gen-sets.pl script creates set commands from the config
cat > $TEMPSETCMDS <<'EOF1'
configure
# First remove any firewalls from interfaces
for int in $(show interfaces ethernet | \
awk '/eth[0-9]/ {print $1}'); \
do delete interfaces ethernet $int firewall; \
done
# Now delete all firewalls
for fwall in $(show firewall name | \
awk '/^ \w* {$/ {print $1}')
do delete firewall name $fwall; \
done
EOF1

cat >> $TEMPSETCMDS <<EOF2
# Create firewalls found on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPFWRULES)
# Apply firewalls to interfaces as defined on primary
$(/opt/vyatta/sbin/vyatta-config-gen-sets.pl $TEMPINTCMDS | grep firewall)
commit
save
exit
exit
EOF2

# Force a tty for the ssh connection - Vyatta environment variables
# and special shell are only set up during an interactive login
cat $TEMPSETCMDS | ssh -tt $SECONDARY

rm -f $TEMPFWRULES $TEMPSETCMDS $TEMPINTCMDS



...read more

May 26, 2009

VM Security in vSphere - Same Ol' Situation (S.O.S.)

Over the weekend, I had a chance to test out the directives for locking down the virtual machine security issues discussed in Hardening the VMX File with vSphere / ESX 4.0. Unfortunately, all of the security issues are still present in the GA release of vSphere, including non-privileged users having the ability to disconnect virtual NICs and change the time synchronization behavior.

I can't imagine why this situation still persists through version 4.0 of VMware's enterprise virtualization platform. Are there customers who prefer non-privileged user accounts retain this ability? And if so, couldn't we disable this functionality by default, and require .vmx directives to enable it?

Yes, it is easy to change the default settings, and any sysadmin worth his or her salary will make the changes and audit their environment for compliance. That's a tired argument, however, and better "out of the box" security should be a goal for any product. Anybody remember Windows 2000?

...read more

May 3, 2009

Secrets of the e1000

Updated: May 23, 2009
Note that as of vSphere - ESX 4.0, when using the new virtual machine wizard, if you select a custom configuration, and virtual machine version 7, the e1000 is now presented as a virtual adapter option for a Windows Guest (along with Flexible, VMXNET 2 and VMXNET 3. Sweet!). The operating system swap around and .vmx file editing hacks detailed below should no longer be necessary if you want to use the e1000 in a VM. For details on the available virtual NIC options, see this KB article.

There is still no equivalent to vlance.noOprom = "true" or vmxnet.noOprom = "true" for the e1000 to directly disable the PXE option ROM. However the solution described below also works with vSphere.


If you haven't used the e1000 virtual NIC before, it's a virtual implementation of the ubiquitous Intel PRO/1000 Ethernet adapter. According to this VMware KB article, the performance of the e1000 device lies somewhere in between the vlance and vmxnet devices, making it the perfect choice for a virtual machine that doesn't have VMware tools installed, and is therefore unable to utilize the advanced vmxnet virtual NIC.

Pesky pixies
I've been thinking all along that the e1000 virtual NIC lacks the option ROM necessary for PXE booting, and so have never bothered to disable it. While doing some testing on the Vyatta virtual machine I set up for the Protect the Service Console Network With a Virtual Firewall project, I noticed the familiar network boot screen after forgetting to connect the Vyatta installation CD for the initial boot.

If you read my post on Hardening the VMX File, you'll remember I discussed a potential exploit using PXE. Since I thought the e1000 was option ROM free, I failed to discuss disabling it if you have no use for PXE in your environment.

I can make at least one excuse for this oversight: there actually is no .vmx directive to disable the option ROM in the e1000. The option ROMs in the vlance and vmxnet adapters can be disabled with one of these directives, vlance.noOprom = "true" or vmxnet.noOprom = "true", but there is no equivalent command for the e1000.

There is an easy workaround for the lack of a disabling command, however. We can set the memory size that the BIOS makes available to the option ROM to zero, effectively preventing it from loading with this .vmx directive:


ethernet0.opromsize = "0"


You'll need to add this directive for each e1000 adapter, ethernet1.opromsize = "0" for example, if you have two of them in a virtual machine. I've gone back to the Hardening the VMX File post and added this as an additional recommended directive if you are using e1000 adapters.

Not so secret
In an attempt to understand how I never noticed that PXE boot was available with the e1000, I searched through the release notes for each update version of ESX 3.5 on the VMware downloads page. I was able to confirm that I am actually crazy, and PXE boot has always been available with the e1000. I also discovered that the e1000 is the default when creating virtual machines with some specific operating systems. As of Update 4, selecting one of the Linux 64-bit options, Netware, Solaris, or the Other (64-bit) guest operating system will cause the New Virtual Machine Wizard to present the e1000 as the default network adapter option. So if you don't feel like manually adding an e1000 adapter by editing the .vmx file for a virtual machine, you could initially set it up as one of those guest operating systems and then change it back after the VM is created.

If you don't mind editing a .vmx file, just add this directive, changing the device name to the specific adapter you wish to change:


ethernet0.virtualDev = "e1000"


...read more

April 29, 2009

Protect the Service Console Network With a Virtual Firewall

Unless you've been living under a rock for a while, you've probably read the recommendations to completely isolate the ESX console network. This sage advice addresses a serious threat in a virtual infrastructure: if a malicious person out in the LAN were able to gain SSH access to an ESX console, they would have an unholy amount of power. With just two commands at a shell prompt, they could snapshot a virtual disk, FTP it off somewhere, and then use an assortment of tools on the completely exposed filesystem from the comfort of their lair. This network isolation best practice could be the single most important security consideration in an environment, and should also be extended to the VMotion and storage networks.

In fact, if security were the primary architecture concern, the ESX service console network and VirtualCenter server would be plugged into their own isolated network switch. The only way to log into VirtualCenter or SSH to an ESX server would be to march down the hall and pull out the monitor and keyboard in the server room. But obviously that isn't feasible for most organizations, so some sort of compromise between security and administrative access must be made.

One option is to install two NICs in the VirtualCenter servers; one in the LAN, and one plugged in to an isolated service console network switch. You could further enhance this by placing a physical firewall between the VirtualCenter server and the LAN, only allowing the specific ports needed for Active Directory authentication and VI client access from specific workstations to the VirtualCenter server.

With this design in place, an administrator would be able to get many things done from the comfort of a LAN workstation. But unfortunately there are a few critical elements that require direct network access to the ESX servers, such as virtual machine console access, physical-to-virtual migrations with Converter Enterprise, and VCB backups.

We could require that each workstation that needs VI Client or SSH access to the service console network have a second NIC and be physically patched into the isolated switch. But that's not very scalable, and a burden in all but the smallest of environments. There's a better option, and it provides a higher level of security: a virtual firewall with vNICs in two different port groups that only permits specific traffic between the console network and the LAN.

There are several good virtual firewall options out there, and many of them are ready-to-use virtual appliances. But for this application, a virtual appliance isn't going to cut it. We want to know exactly how the firewall VM was built, what applications were installed, and what services were enabled. Turns out that the Vyatta Community Edition open source networking software is just about perfect for what we're trying to accomplish here.

Vyatta is a full-featured networking operating system that provides routing, firewalling, VPN, IDS, DHCP, dynamic routing protocols, and pretty much all the features you would only expect from one of the major networking vendors. Think of it like a Linux-based version of Cisco IOS that can be installed on x86 hardware. And in the latest version, VC5, just released in March, they've really upped the ante by providing a web-based GUI.

Give it a spin, it's free!
Head over to the Vyatta.org site and you'll find download links that don't require registration. You'll probably notice that Vyatta also offers VC5 in a VMware Appliance format, but we'll be using the live CD image. Check out the video tutorials at the Vyatta.com site as well if you want to get a feel for what we're about to cover.

Once you've got the VC5 ISO file downloaded, set up a new virtual machine, perform a custom install, and give it Vyatta's minimum recommended settings:

  • 512 MB of RAM

  • 2 GB virtual hard disk

  • Other Linux (32-bit) for the guest operating system

  • Two virtual NICs, one in a port group on the same vSwitch as the Service Console, and one in a port group that uplinks to the LAN

Don't power up the VM just yet. First edit the VM's settings, and remove the floppy as we don't need it, and point the CD-ROM drive to the VC5 ISO.

The version I tested wasn't loading the advanced vmxnet virtual NIC, and instead was defaulting to the pcnet32 device, which is a very basic 10/100 NIC with limited features. So I went ahead and forced the VM to use an Intel PRO/1000 vNIC by editing the .vmx file for the VM and changing the vNIC to the e1000 device:

  • Open a SSH session to the ESX server hosting the virtual router

  • Browse to the datastore folder that contains the VM, and edit the .vmx with vi

  • Add a line for each vNIC:
    
    ethernet0.virtualDev = "e1000"
    
    ethernet1.virtualDev = "e1000"
    


  • And delete these lines to force a regeneration of the MAC addresses:
    
    ethernet0.addressType = "vpx"
    
    ethernet0.generatedAddress = "00:50:56:af:18:93"
    
    ethernet1.addressType = "vpx"
    
    ethernet1.generatedAddress = "00:50:56:bd:27:1e"
    
    


  • Now from vi press Esc and then :wq to save the .vmx file
Boot the VM, and when it arrives at a login prompt, log on to the system as root with the password vyatta

At the prompt enter this command:
install-system


The Vyatta installer will launch, an interactive script that prompts you for some basic information and confirmation during the install. Just choose all of the defaults during the install script, until you reach the prompt This will destroy all data on /dev/xxx, type Yes here to continue.

You'll be prompted to set the passwords for the system accounts, type Yes and enter strong passwords for the root and vyatta accounts.

When the installer finishes, shut the vm down with this command:

shutdown -h now
Disconnect the CD-ROM, check Connect at power on on the vNIC connected to the service console network, and boot the VM back up.

When the system boots up to the vyatta login: prompt, log on as user vyatta with the password you established during the install.

We first need to give the service console facing interface an IP address. To accomplish this, we'll enter configuration mode and assign an IP address, replacing the IP below with one appropriate for the environment. We're going to set ours to the gateway address we've already set on our ESX servers:


configure
set interfaces ethernet eth0 address 172.20.1.254/24
commit



All changes from the Vyatta console begin with the configure command to enter configuration mode, and end with a commit command to apply the changes.

We should save the changes now, so the configuration we made will survive a reboot of the Vyatta virtual machine. To save the configuration to the hard drive of the VM, just enter:


save



If you've set everything up correctly to this point, you should be able to ping that address from an ESX server.

If you type exit, you'll exit configuration mode, and be in 'operational' mode. If you have spent any time with IOS, you should be pretty familiar with this type of modal work flow. In fact, you can only view the Vyatta configuration while in operational mode, with the command show configuration, just like in IOS, which doesn't make the show run command available in configuration mode. Also, just like in IOS, if you hit tab, you should get command completion in the Vyatta CLI.

I've had enough of configuring the Vyatta VM through the VI Client, so let's enable SSH access and the Vyatta web GUI with these commands:


set service ssh
set service https
commit
save



Now you should be able to use putty to log in as the user vyatta via SSH. Let's try out the web console, open a web browser from a machine with access to the service console network to the IP address we set up earlier, in our case it will be https://172.20.1.254

Log in with the vyatta user account and password

If you take a look around the web GUI, it should be pretty obvious that this is not a system you're going to just intuitively figure out. The GUI represents the CLI commands as objects, so without knowing the CLI logic, the GUI really isn't that helpful. To see what I mean, drill down to interfaces > ethernet > eth0, selecting eth0. Check out the address: attribute, and notice that the path we took through the web GUI reflects the command we entered through the CLI: set interfaces ethernet eth0 address 172.20.1.254/24

So let's configure the LAN facing interface through the web GUI. Drill down to interfaces > ethernet > eth1, selecting eth1. In the address: attribute section in the right pane, enter the IP address you want to use, in our case we'll enter 10.1.1.254/24. The other settings are good, so we'll press the Set button.

Just like the CLI, we need to commit the changes before they become active, so press the Commit button from the upper right of the web GUI. If you connect the second vNIC from the VI Client now, you should be able to ping that address from the LAN. If everything is set up correctly, the Vyatta virtual machine should be routing between the service console network and the LAN. If you are not able to ping from both sides, then there's likely a routing or default gateway issue on one or both sides.

Let's configure a few system settings from the web GUI. Select system from the left side, and you'll see we can set the domain-name, gateway-address, host-name, name-server, ntp-server, and time-zone settings.

Start your filtering


In order to follow along with the rest of this tutorial, you'll need a few things in place:

  • An isolated ESX service console network.

  • A good relationship with the network administrator in your environment, or something to blackmail them with, because they'll need to add a static route to the ESX console network to the layer three device routing between the subnets in your LAN. If you're using RIP or OSPF in your network, the Vyatta VM is capable of running one of those dynamic routing protocols, but that's way beyond the scope of this.



In our test environment, we've got a simplified setup: the ESX hosts are using the service console facing interface IP of the Vyatta VM as their default gateway, and the LAN connected Windows VM we're using to connect to the ESX servers is using the LAN facing interface IP of the Vyatta VM as its default gateway.


ESX Hosts <-> Vyatta VM <-> LAN VM
172.20.1.x : 172.20.1.254 eth0 : 10.1.1.254 eth1 : 10.1.1.33


So at this point, we are routing between the networks and able to ping from both sides of the Vyatta VM. We want to restrict traffic in both directions, protecting the console network from the LAN, and also limiting the access the console network has to the LAN, so we'll need to create two firewall rule sets and apply them to the two interfaces.

This first rule set, lan_in, grants SSH and VI Client access to a range of ESX hosts (172.20.1.1 to 172.20.1.5) from host 10.1.1.33, and this is how we would add it using the CLI:


vyatta@vyatta:~$ configure
vyatta@vyatta# set firewall name lan_in rule 10 action accept
vyatta@vyatta# set firewall name lan_in rule 10 description "Allow SSH and VI client access from 10.1.1.33"
vyatta@vyatta# set firewall name lan_in rule 10 destination address 172.20.1.1-172.20.1.5
vyatta@vyatta# set firewall name lan_in rule 10 destination port ssh,https,902-903
vyatta@vyatta# set firewall name lan_in rule 10 protocol tcp
vyatta@vyatta# set firewall name lan_in rule 10 source address 10.1.1.33
vyatta@vyatta# set firewall name lan_in rule 10 state established enable
vyatta@vyatta# set firewall name lan_in rule 10 state new enable
vyatta@vyatta# set firewall name lan_in rule 10 state related enable
vyatta@vyatta# set firewall name lan_in rule 10 state invalid disable
vyatta@vyatta# set interfaces ethernet eth1 firewall in name lan_in
vyatta@vyatta# commit
vyatta@vyatta# save


To set up the matching rule set on the interface in the console network, we just flip the source and destination, and configure it like this from the CLI:


vyatta@vyatta:~$ configure
vyatta@vyatta# set firewall name console_in rule 10 action accept
vyatta@vyatta# set firewall name console_in rule 10 description "SSH and VI client replies to 10.1.1.33"
vyatta@vyatta# set firewall name console_in rule 10 destination address 10.1.1.33
vyatta@vyatta# set firewall name console_in rule 10 protocol tcp
vyatta@vyatta# set firewall name console_in rule 10 source address 172.20.1.1-172.20.1.5
vyatta@vyatta# set firewall name console_in rule 10 source port ssh,https,902-903
vyatta@vyatta# set firewall name console_in rule 10 state established enable
vyatta@vyatta# set firewall name console_in rule 10 state related enable
vyatta@vyatta# set firewall name console_in rule 10 state new disable
vyatta@vyatta# set firewall name console_in rule 10 state invalid disable
vyatta@vyatta# set interfaces ethernet eth0 firewall in name console_in
vyatta@vyatta# commit
vyatta@vyatta# save


Now if you attempt to ping an ESX host from the LAN side it should fail as each firewall rule set has an implicit deny as the last rule. In order to restore the ability to ping, you would need to add a rule to both the lan_in and console_in rule sets.

We'll add another accept rule for an additional administrator's workstation. This time we'll do it using the web GUI, so open a web browser, log in to the web GUI, drill down to firewall > name > lan_in > rule, and select rule. In the text box on the right, type 20 and then click Set

>. From the screen that opens up, we will set up the second rule just like the first. Drop down to accept in the action: box, enter a description like Allow SSH and VI client access from 10.1.1.62, enter tcp for the protocol:, and click the Set button.

From the left pane now, select destination, click the Create button, and type 172.20.1.1-172.20.1.5, our range of ESX server IP addresses, in the address: box. In the port: box, enter ssh,https,902-903 and click the Set button.

*By the way, if you're curious as to how the protocol names map to the numbered ports, just type less /etc/protocols

From the left pane again, select source, click the Create button, in the address: box enter 10.1.1.62 and click the Set button.

And finally, from the left pane, select state, click the Create button, check established:, new:, and related:, and then click the Set button.

We need a corresponding accept rule for the reply traffic back to the LAN workstation, so drill down to firewall > name > console_in > rule, and select rule. In the text box on the right, type 20 and then click the Set button.

Drop down the action: box to accept, enter the description SSH and VI client replies to 10.1.1.62, enter tcp for the protocol:, and click the Set button.

From the left pane, select destination, click the Create button, type 10.1.1.62 in the address: box, and click the Set button.

From the left pane again, select source, click the Create button, and in the address: box, enter 172.20.1.1-172.20.1.5. In the port: box, enter ssh,https,902-903 and click the Set button.

And finally, from the left pane, select state, click the Create button, check established: and related:, then click the Set button.

From the top of the web GUI, click the Commit button, and then the Save button, and you should now have VI Client and SSH access to the isolated ESX service console network from the second workstation.

The last configuration we should make to the virtual firewall is to lock down the Vyatta web GUI and SSH access from the LAN to just the workstations we defined above. To do that we'll create another rule set, but his time apply it as local to the interface rather than in. This will filter packets sent directly to the interface IP address.

From the left side of the web GUI, drill down to firewall > name, and select name. In the text box on the right, type lan_admin and then click the Set button.

Enter a description for the new rule set, like "Restricts administrative access from the LAN"

On the left, select rule. In the text box on the right, type 10 and then click the Set button.

Drop down the action: box to accept, enter the description Allow SSH and web GUI access from 10.1.1.33, enter tcp for the protocol:, and click the Set button.

From the left pane, select destination, click the Create button, type ssh,https in the port: box, and click the Set button.

From the left pane again, select source, click the Create button, in the address: box enter 10.1.1.33 and click the Set button.

And finally, from the left pane, select state, click the Create button, check established:, new:, and related:, and then click the Set button.

On the left side, drill down to interfaces > ethernet > eth1 > firewall > local, selecting local, and click the Create button. In the text box on the right, type lan_admin and then click the Set button.

Save the changes by clicking the Commit button, and then the Save button. You could now repeat those steps, creating a rule 20 under rule set lan_admin for the second administrator's workstation, 10.1.1.62.

Gleaming the cube
We've only touched the surface of the Vyatta networking software, and if you head over to the Vyatta website, you can download a lot of good reference material after a quick registration. If you take the time to read some of the documentation, you'll get a real sense for how dense and powerful the Vyatta operating system is, especially for extending virtual networking. Stay tuned, as we'll be doing some interesting projects with Vyatta in the future.
...read more

April 10, 2009

Hardening the VMX File: Redux

If you missed the first post on Hardening the VMX File, I recommend you give it a read, especially if you are currently not adding any VMware Tools hardening parameters to your .vmx files. It's pretty eye-opening to see what a non-privileged Windows user account is capable of changing in your VMs.

I've received some questions and comments about how the post overlooked the fact that a Citrix / Terminal Server can be really locked down, preventing users from opening control panels, command prompts, etc. This is all very true, and a security savvy AD administrator could probably have used some Group Policy changes to lock me out of everything I was able to do as a Terminal Services user.

After reading the post again, I realized that I gone for the low-hanging fruit (Terminal Services user) and had not done a very good job of making my point. What I should have stressed is that many of the powerful features of VMware Tools are not doing any user validation, allowing any process to make significant changes to the VM's configuration.

The Terminal Server example was easy to illustrate, but in order to log into a Terminal Server, the testuser account needed to be a member of the Users and Remote Desktop Users groups. So to make the point a little better, for this quick demonstration we're going to use an IIS website and an ASP form to execute the VMwareService.exe command remotely to make some changes. We'll also set the authentication method for the site to anonymous access, and use the Internet Guest Account, a restricted account that is only a member of the Guests group.

Demo setup
We'll roll out a fresh Windows Server 2003 Standard VM from a template for this demo. This server will have Service Pack 2 installed, a default instance of IIS 6, and will not be in a domain, just a workgroup. The VM will be hosted on an ESX 3.5u4 server, and will have the latest version of VMware Tools shipping with 3.5u4, which is build 153875.

First we'll install IIS in the VM: open the Control Panel > Add or Remove Programs > Add/Remove Windows Components > double click Application Server > and check Internet Information Server (IIS). You're going to need the Windows Server 2003 CD you installed the server with for the IIS installation.

Tip - During the IIS install, you'll be prompted to insert a CD with the Service Pack 2 files, don't bother, the file the installer is looking for, convlog.exe, is actually cached in C:\WINDOWS\ServicePackFiles\i386, just browse for it.

Once the IIS install is complete, open the Start Menu > Programs > Administrative Tools > Internet Information Services (IIS) Manager

We're going to create a new site:
  • Right click Web Sites > New > Web Site

  • Give the website a description of test

  • Drop down the IP address selector to the IP address of the server

  • For the website path, select C:\Inetpub, and click Make New Folder to create a folder in C:\Inetpub named test

  • Select the test directory and continue, leaving Allow anonymous access to this Web site checked

  • Check the Read and Run scripts permissions

We have to enable Active Server Pages for our test, so in IIS Manager, select Web Service Extensions, right click Active Server Pages, and choose Allow

Stop the Default Web Site by right clicking it and choosing Stop, just to make sure it doesn't interfere with our test

Open the C:\Inetpub\test folder, right click in it and create a new text document, naming it get.asp

Open C:\Inetpub\test\get.asp with Notepad, and paste in this code segment:

<html>
<body>
    
  <form method="GET" action="post.asp">
    Command <input type="text" name="MyCommand" size="80"/>
    <input type="submit" />
  </form>

</body>
</html>

Close and save C:\Inetpub\test\get.asp

Right click in C:\Inetpub\test again and create another new text file, naming it post.asp

Open C:\Inetpub\test\post.asp with Notepad, and paste in this code segment:

<html>
<body>
  <%
    Dim cmd, oShell, oCmd, strRes
    cmd = Request.QueryString("MyCommand")

    set oShell = CreateObject("WScript.Shell") 
    set oCmd = oShell.Exec(cmd) 
    strRes = oCmd.StdOut.Readall() 
    set oCmd = nothing
    set oShell = nothing 
 
    response.write "<pre>" & strRes & "</pre>"
  %>
</body>
</html>

Close and save C:\Inetpub\test\post.asp

This may be an overly simple setup for the website, but the purpose of this exercise is not to prove some problem with IIS 6 or ASP, we just want a website running as the Internet Guest Account so we can attempt to run VMwareService.exe as a restricted account with no rights to even log into the server.

Back in IIS Manager, right click the test website and choose Properties, and the Documents tab

Remove the existing documents from the list, and then click Add, type get.asp, and click Apply and OK to save the changes

Now from another system able to reach the test server over the network, open a browser to the IP address of the test server. If you set everything up correctly, you should see a simple web page with a text box labeled Command and a button labeled Submit Query

Since our test website can't execute files outside of the C:\Inetpub\test folder, we need to copy an executable into the folder to confirm things are working. Let's copy C:\WINDOWS\system32\ping.exe to C:\Inetpub\test

Now on the remote machine with the web page open, paste this into the Command text box: C:\Inetpub\test\ping.exe localhost

If you've set everything up correctly, you should get a page with output like this:

Pinging lab-test [127.0.0.1] with 32 bytes of data:

Reply from 127.0.0.1: bytes=32 timeəms TTL=128
Reply from 127.0.0.1: bytes=32 time=6ms TTL=128
Reply from 127.0.0.1: bytes=32 timeəms TTL=128
Reply from 127.0.0.1: bytes=32 timeəms TTL=128

Ping statistics for 127.0.0.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 6ms, Average = 1ms

Now let's copy C:\Program Files\VMware\VMware Tools\VMwareService.exe to the C:\Inetpub\test folder

Back in the Command text box of our website, put this in to see if we can store a variable:

C:\Inetpub\test\VMwareService.exe -cmd "info-set guestinfo.testvar1 This is our first test variable"

We should get this response back. Hmmm, wonder if it stored it:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"

Let's try and retrieve it, in the Command text box, put this in:

C:\Inetpub\test\VMwareService.exe -cmd "info-get guestinfo.testvar1"

Well, we get that warning again... Ah but look, our variable!

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
This is our first test variable

Let's see what this command does:

C:\Inetpub\test\VMwareService.exe -cmd "vmx.capability.edit_devices"

That returned a 1

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
1

Let's see what it reports if we apply isolation.device.edit.disable = "true" to the .vmx and reboot:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
Unknown command

Interesting, it shut down the ability to even query the parameter if it is set. Let's see what this command does:

C:\Inetpub\test\VMwareService.exe -cmd "vmx.set_option broadcastIP 0 1"

Ah, must need to try it with the bits swapped:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
Invalid old value

C:\Inetpub\test\VMwareService.exe -cmd "vmx.set_option broadcastIP 1 0"

Well, that adds tools.broadcastIP = "TRUE" to the .vmx, but I'm not sure what that does.

Let's try changing the time synchronization as the Internet Guest Account, by placing this command in the website:

C:\Inetpub\test\VMwareService.exe -cmd "vmx.set_option synctime 0 1"

This is the response:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
Invalid old value

Oh, we must have already had it enabled, let's try:

C:\Inetpub\test\VMwareService.exe -cmd "vmx.set_option synctime 1 0"

Just the warning this time:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"

Let's check the VMware Tools applet and see... it changed! We changed the time synchronization behavior of our virtual machine from a remote system as the Internet Guest Account!

Backdoor blues
This brings us to something we haven't explored yet, if we uninstall VMware Tools in a VM, have we closed the backdoor? Let's examine what the backdoor is exactly, and a comment from the Open Virtual Machine Tools source code sums it up perfectly:

/*
 * backdoor.c --
 *
 *    First layer of the internal communication channel between guest
 *    applications and vmware
 *
 *    This is the backdoor. By using special ports of the virtual I/O space,
 *    and the virtual CPU registers, a guest application can send a
 *    synchroneous basic request to vmware, and vmware can reply to it.
 */

If you are really interested, you can read ./lib/backdoor/backdoorGcc32.c and find the inline assembly code for sending messages to the hypervisor and reading the responses. The backdoor is always there, but we need a client in the VM to communicate with it. So while removing VMware Tools might leave the VM unable to use the backdoor, we could always just compile our own tool and copy it over. Or in the worst case, some malware application could have backdoor communication functions built in! And as we've shown, the backdoor is not doing any user privilege checking, it's using special processor instructions embedded in the calls to the virtual cpu, so that malware could be nasty.

However, we can close the backdoor with a single .vmx directive:

monitor_control.restrict_backdoor = "true".

Let's see what happens if we try a command through the backdoor now:

Warning: Unable to open "C:\Documents and Settings\All Users\Application Data\VMware\VMware Tools\tools.conf"
Log: The VMwareService application must be run in a Virtual Machine.
Log: Backtrace:
Log: ----Backtrace using dbghelp.dll----
Log: Module path: C:\Inetpub\test\VMwareService.exe
Log: Module directory: C:\Inetpub\test\

 -- output omitted --

Wow, looks like the VMwareService application doesn't even think it's running in a VM any longer.

Wait, are you suggesting we just shut down the backdoor in our VMs? No, that's what the of list of .vmx directives is for, we can select which portions of the backdoor to restrict.

So this list covers every possible malicious thing someone could attempt? I can't say that, but this list covers what VMware recommends in the VI3 Security Hardening white paper as well as several other published ESX security related guidelines.

"I don't like you because your dangerous..."
I hope this has illustrated how important it is to apply these .vmx file hardening parameters to your virtual machines, whether they have VMware Tools installed or not installed. It doesn't matter if the potential attacker is using a Citrix account, or just a cracked website, any process with any level of privileges is able to make significant changes to the configuration of your VMs. I'll list the recommended parameters we've explored in these two posts again, but check out Hardening the VMX File to see in detail what each parameter does:

Recommended VMX Directives:

isolation.device.connectable.disable = "true"
isolation.device.edit.disable = "true"
isolation.tools.setOption.disable = "true"
isolation.tools.log.disable = "true"
isolation.tools.diskWiper.disable = "true"
isolation.tools.diskShrink.disable = "true"
isolation.tools.copy.disable = "true"
isolation.tools.paste.disable = "true"
isolation.tools.setGUIOptions.enable = "false"
log.rotateSize = "100000"
log.keepOld = "10"
vlance.noOprom = "true"
vmxnet.noOprom = "true"

# PXE boot on the e1000 vNIC can be disabled with this directive:
ethernet0.opromsize = "0"


...read more

April 8, 2009

Hardening the VMX File: How Your Servers May Already be 0wned by Your Users

If you've been working with any of the VMware products for even a short amount of time, you've probably explored the .vmx file. This single text file stores almost all of a virtual machine's configurable parameters, and has several optional settings that are not available from the administration tools and must be added by hand to the file. In this tutorial, we'll explore a few parameters that can be used to eliminate some serious vulnerabilities with VMware Tools and lock down the communications channel between a guest VM and the ESX server hosting it.

The .vmx file parameters we'll discuss have already been recommended by different sources on the Internet, so this may seem like old news. But there is very little information describing exactly what they do and what issues they are intended to fix. Many of the lists out there have parameters that don't even apply to the VI3 / ESX 3.5 products! So rather than just post another list, we're going to really explore what each parameter changes or fixes. If you have not gotten around to hardening your virtual machine configuration files, keep reading, as this may open your eyes to some pretty serious security holes in your environment.

I like using a combination of putty and vi to make changes to .vmx files. Just copy the list of parameters at the end of this posting, open the .vmx in vi, and right click with vi in insert mode to paste them all in. If you've never used vi, or just find it scary, see the very bottom for a quick tutorial.

You can also add these parameters from the VI Client by right clicking a VM, choosing Edit Settings..., Options tab, Advanced - General, and clicking the Configuration Parameters... button. But that is a lot of work compared to the vi method. Note that for whatever method you choose to add the parameters, the VM must be shut down. If you add anything to the .vmx of a running VM, your changes are going to be discarded when the VM shuts down.

Seeing is believing
You really have to see the vulnerabilities in VMware Tools with your own eyes to get a sense of their scope. To demonstrate, we're going to bring up a new virtual machine installed with Windows 2003 Server Standard and VMware Tools. We'll add a new local user account to the server named testuser, and add this user to the Remote Desktop Users group. This account should only be a member of the Users and Remote Desktop Users groups, and should not be in the Administrators group. This is exactly how a typical Citrix user account is configured, so if you've got virtualized Citrix servers in your environment, you're going to want to test this out for yourself.

And if you're thinking this doesn't apply to your environment because you have hidden the VMware Tools icon in your VMs, think again, as any user can simply open up a Run window and launch "C:\Program Files\VMware\VMware Tools\VMControlPanel.cpl".

Dis-connectable
We'll start by looking at the ability to connect and disconnect CD-ROM, floppy, and network devices from the VMware tools applet. Open a Remote Desktop Connection session to your test server and log in with the testuser account. Once you have a desktop, right click the VMware Tools icon and open it, then click on the Devices tab. Uncheck the NIC device and click Apply, and then watch as your RDP connection dies. That's right, a non-administrator account just disconnected the VM's NIC, taking it completely off the network! If you have virtual Citrix servers, you should already be getting an idea of just how serious a vulnerability this is.

Lucky for us, this is very easy to disable by simply adding these two directives to the .vmx file:

isolation.device.connectable.disable = "true"
isolation.device.edit.disable = "true"

Timecop
This next VMware Tools vulnerability could also ruin your day if one of your Citrix users gets some malicious ideas. Open another Remote Desktop Connection to your test VM, and log in as testuser. Open the VMware Tools applet again and on the Options tab, uncheck Time synchronization between the virtual machine and the host operating system. Click Apply, and then OK to close the window. Now open the VMware Tools applet again and notice that the change you made as a non-administrator was saved!

If you are using the default W32Time service in your Windows guests, and disabling time synchronization between your guest VMs and ESX hosts, you may be thinking this is not a big deal. But if a user logs in and enables guest-host time synchronization, you now have two processes synchronizing time that are totally unaware of each other. And if you are disabling W32Time in your guests, and relying on the VMware Tools synchronization like we are, this could be a really big problem. If a user changes this without your knowledge, the VM could slip out of synch with the domain and cause you much grief.

Hang on, something else must be going on here, that setting is stored in the .vmx file as the tools.syncTime = "TRUE" parameter, and there is no way a regular Windows user account is changing the .vmx on the ESX host!

Let's see for our ourselves, open up a SSH session to the ESX server hosting our test VM, elevate to root, and run this command with the VMware Tools checked for Time synchronization between the virtual machine and the host operating system:

grep tools.syncTime /PathToTheVM/vm.vmx

Our terminal output tools.syncTime = "TRUE". Now let's make the change to the VMware Tools time sync option as testuser, apply it and close the VMware Tools applet, and run the same grep command on the ESX host. Our terminal returns tools.syncTime = "FALSE". Yikes, this really is happening!

The documentation on the optional .vmx file parameters is pretty sparse, so it took quite a bit of trial and error to finally figure out how to prevent non-privileged users from changing the time synchronization behavior of virtualized servers. As it turns out, the parameter to disable this option, isolation.tools.setOption.disable, is on many lists as a recommended parameter, but I couldn't find a good explanation of what it actually does. When in place, a user can check and uncheck the time synchronization option, click Apply and OK, and it looks like the change might save. But when you look at the .vmx file, the parameter does not change, and when you open the VMware Tools applet again, the check box is back to the state it was before you attempted the change. As far as I can tell, this directive only affects the ability to change the tools.syncTime option, so we are going to apply it:
isolation.tools.setOption.disable = "true"

Wait a second...
I was putting the finishing touches on this article when it occurred to me that I have changed the time sync option with a batch script in the past, with the intent of scheduling the script to flip the time sync bit off and back on again to force an immediate sync with the host ESX server. Could a non-privileged user make this change from the command line? I went back to my notes, and found the command, C:\Program Files\VMware\VMware Tools\VMwareService.exe -cmd "vmx.set_option synctime 0 1", and without the setOption.disable parameter in place, this works for the testuser account from the command line just as it did from the applet!

Ok, that's interesting, set_option matches the setOption.disable parameter we used to close this hole. Now things are becoming a little bit clearer. Let's explore the VMwareService.exe command further, and see if there is a help screen: VMwareService.exe -help

Usage: c:\Program Files\VMware\VMware Tools\VMwareService.exe {-v,-i,-u,-kill,-cmd "<command>"}

Well that's not especially helpful, what are the commands we can send it? There's a little bit of information in the SDK documentation, but not enough. So I finally went and downloaded the source code for the Open Virtual Machine Tools and searched through it. After a lot of grepping, I focused in on a couple of remote procedure calls that can be made through what the developers refer to as the "backdoor".

This command allows you to make log entries to the virtual machine's vmware.log file on the ESX server:

C:\Program Files\VMware\VMware Tools\VMwareService.exe -cmd "log %string %string"

And this one allows you to create custom GuestInfo variables to store information about the virtual machine in the ESX server's memory:

C:\Program Files\VMware\VMware Tools\VMwareService.exe -cmd "info-set guestinfo.%variable %string"

And to read a custom GuestInfo variable:

C:\Program Files\VMware\VMware Tools\VMwareService.exe -cmd "info-get guestinfo.%variable"

Note that these custom GuestInfo variables are not stored permanently. They will survive a VM reboot, but will disappear with a VM shutdown. This command looked familiar, so I went and searched some more and found it in the VMware Scripting API documentation.

These backdoor capabilities are also mentioned in the VI3 Security Hardening white paper from VMware in the Limit Data Flow from the Virtual Machine to the Datastore section. And yes, amazingly both RPC commands can be initiated by a non-privileged user!

Close the backdoor
To see if I could use these commands maliciously, I wrote up a couple of quick scripts and launched them as testuser. For the ESX log spamming attack, I attempted to write 1 KB log entries in a loop that would repeat one million times. I then watched the log file grow, and it got up to about 1 MB in size, and then stopped growing. A quick tail on the log revealed the last message was <<< Log Throttled >>>. So that's a good sign, looks like VMware has placed some kind of sanity check on how these log files can grow. But I'd really prefer that this backdoor didn't exist at all, so let's try this obvious directive and see if it shuts the functionality off:
isolation.tools.log.disable = "true"

Good! That shut off logging through the backdoor, but not logging in general to the vmware.log file, which we want in place for troubleshooting.

Then I checked if it was possible to spam the ESX host's memory with fake custom variables. After writing up a quick VBScript that would cycle through a loop, increment an 'i' variable by one each time, and then create a custom variable 'test-i' and store 32 KB of string data in it, I found that I could only store 32 individual variables. That adds up to be exactly 1 MB, which is exactly how much data the VI3 Security Hardening white paper says we're allowed to write.

The VI3 Security Hardening white paper gives us two options for limiting this backdoor RPC channel. We could turn off setting GuestInfo variables from the VM altogether, with this directive:

isolation.tools.setInfo.disable = "true"

But I can't wholeheartedly recommend that, as it also prevents the VM from sending back it's IP address and DNS name information to the host, which means they are no longer displayed in the VI Client. There is no guarantee this won't break something like VCB or a third-party application.

We also have the option to lower the amount of memory available for storing GuestInfo variables, with this directive:

tools.setInfo.sizeLimit = "some # in bytes"

But really the 1 MB limit is pretty reasonable in the first place. I would suggest that if you can live without a VM's IP and DNS information in the VI Client, completely disabling GuestInfo variables is probably the safest way to go. But since the possibility for breakage is there, we won't list this parameter in a our final recommendations.

Shrinkage factor
If you open up the VMware Tools applet on your test VM again, you'll find the curious Shrink tab. I'm not describing it as curious because of its function, I can see where shrinking a virtual disk could be useful, though I have yet to use this feature in a production environment. The Shrink feature is curious to me because VMware has decided to make the option visible to any user, even non-administrators. When a Shrink operation is running, the system operates at 100% processor usage and becomes almost unusable, making it a nice little DoS attack for someone with malicious intent. However, the Shrink feature will not actually work unless the user has write permission to the root of the drive selected for shrinking, C:\ for instance. In a default Windows 2003 Server installation, the built-in Users group does not have write access to the root of the system partition, or to the roots of new volumes you create with the Disk Management console, so you may not be exposed to the Shrink attack vector.

But if another administrator, or an application, were to add write permissions to the root of a partition for the Users or Everyone groups, the potential to exploit the Shrink feature would become a real threat, especially on a Citrix server! Unless you are using Shrink in your environment, it's best to disable it altogether with these two directives:

isolation.tools.diskWiper.disable = "true"
isolation.tools.diskShrink.disable = "true"

PXE dust off
I came across this next virtual machine vulnerability while troubleshooting a SAN issue. The ESX hosts were rudely disconnected from their iSCSI LUNs for an unknown reason overnight, and when we opened up console connections to the VMs, we were surprised by what we found. All of the VMs were still in a running state, but had rebooted themselves after essentially having their hard drives ripped out while they were powered on. Since the .vmx configuration file for each VM had been loaded into memory on the ESX hosts, ESX didn't care that the iSCSI storage that contained the .vmx file was gone, so the VM went through its list of bootable devices, and landed on the PXE boot option when no others were available.

After seeing this first hand, I realized it could be used as a powerful attack vector. If a malicious person were able to connect a PXE boot server to the network, and then launch a DoS attack against the iSCSI network, they could reboot VMs with their own images loaded with the tools they need to launch further attacks. Fortunately, the PXE boot OPROM on the vlance and vmxnet virtual adapters is easy to disable:

vlance.noOprom = "true"
vmxnet.noOprom = "true"

If you are using the e1000 adapter in any virtual machines, you'll need to disable the PXE boot OPROM using a different method, as there is no .noOprom option for the e1000. The directive below will reduce the memory size that the BIOS makes available to the OPROM to zero, effectively preventing it from loading. Note that this is a per-adapter setting:

ethernet0.opromsize = "0"

No paste for you
To prevent the possibility that confidential information, such as passwords, could leak from the clipboard of the VI Client to the clipboard of a VM, or vice versa, the copy and paste capability between the VI Client and VM console should be disabled. This is easy enough to do with the following directives, and should have little impact on your environment unless you are using the VI Client for day to day administration of your servers:

isolation.tools.copy.disable = "true"
isolation.tools.paste.disable = "true"

A third directive is often mentioned with the copy.disable and paste.disable parameters, isolation.tools.setGUIOptions.enable = "false", but I can not find any documentation on exactly what this parameter does. Another curious thing is that it is sometimes written with GUI in all caps, and sometimes Gui with a capital G. You can also find references to the similar parameter: isolation.tools.setGUIOptions.disable, but again there is no documentation on this command, and after trying all the various forms, I can not find any difference with this directive set to "true" or "false". We'll include it in our list of directives to apply however because it is listed in several security and hardening guides, and it doesn't seem to affect anything whether it's on or off:

isolation.tools.setGUIOptions.enable = "false"

Log bucking
The last set of directives we'll recommend deal with the virtual machine log file, vmware.log. We've already seen how this log could potentially be abused through the virtual machine backdoor, but there is another threat from this file. I actually just witnessed this, we had a virtual machine that showed no signs of trouble, but was failing VCB backups every evening. Thankfully we were using VCB Wrangler to manage our virtual machine backups ( <- shameless plug ), a scheduled script that launches vbcMounter.exe and emails the backup results and a list of the files backed up with each VM. While looking through the email for any obvious signs of what was causing the failures, we noticed that the vmware-####.log files had rotated into the 7000 range! Issuing a ls -la command in the VM's directory from an SSH session, we could see that the log files had all rotated within the last hour. Some process was causing a lot of logging, and luckily we had already put these directives in place to rotate through 10 logs of 100000 bytes each:

log.rotateSize = "100000"
log.keepOld = "10"

Had these parameters not been in place, the VM's log files could have completely consumed the available free space on the VMFS LUN, causing a disaster of unknown potential.

Test test test
If you've made these recommended changes, and are testing things out within a VM, you may have noticed that on the Options tab in the VMware Tools applet, there are still two check boxes that a non-privileged user can modify and save, Show VMware Tools in the taskbar, and Notify if update is available. These two parameters are actually stored in the registry in the HKEY_CURRENT_USER hive and are therefore stored separately for each user account. You can watch them change to see for yourself by opening regedit and drilling down to HKEY_CURRENT_USER\Software\VMware, Inc.\VMware Tools\ShowTray. Since these aren't global parameters, and they don't really do anything (a non-administrator wouldn't be able to install a VMware Tools update), we'll just live with them.

Some are recommending that the file security settings be changed on the C:\Program Files\VMware\VMware Tools folder to restrict non-privileged user access to the VMware Tools settings. VMware has even issued a KB article with instructions on how to restrict the folder and completely disable access. The problem with this method is that even after you have completely restricted the folder, if a user is able to get VMControlPanel.cpl or VMwareService.exe from another source and copy it to their desktop, they can still use the VMware Tools control panel or the command line application to make changes.

You should really try this for yourself, copy VMControlPanel.cpl and VMwareService.exe to the testuser account's desktop folder, and then delete the C:\Program Files\VMware\VMware Tools folder, which will first require stopping all VMware services and killing the processes in Task Manager. Now log in as testuser and successfully make all of the changes discussed in this article! If you reboot the server after deleting the C:\Program Files\VMware\VMware Tools folder the VMware Tools services won't even start, but you are still able to make the changes!

I recommend you forget about this option and add the parameters to each .vmx file in your environment. This is the only way to fully remediate these vulnerabilities, besides uninstalling VMware Tools completely.

That's it for our recommended list of .vmx file hardening parameters. If you do some further research (and you should), you'll find that there are a lot of additional parameters not listed here, and many of them are included in security focused white papers as required parameters. I recommend you extensively test any parameters not listed here before rolling them out in a production environment. All of the parameters included in this article, except for isolation.tools.setOption.disable, vlance.noOprom, and vmxnet.noOprom, are listed as recommended in the VI3 Security Hardening white paper.

Recommended VMX Directives:

isolation.device.connectable.disable = "true"
isolation.device.edit.disable = "true"
isolation.tools.setOption.disable = "true"
isolation.tools.log.disable = "true"
isolation.tools.diskWiper.disable = "true"
isolation.tools.diskShrink.disable = "true"
isolation.tools.copy.disable = "true"
isolation.tools.paste.disable = "true"
isolation.tools.setGUIOptions.enable = "false"
log.rotateSize = "100000"
log.keepOld = "10"
vlance.noOprom = "true"
vmxnet.noOprom = "true"

# PXE boot on the e1000 vNIC can be disabled with this directive:
ethernet0.opromsize = "0"

Live and let vi
Here's a quick vi tutorial for those who haven't discovered its awesome power. Open a SSH session to a test ESX server and type vi at the prompt. If you press i on your keyboard, you'll notice -- INSERT -- appears in the lower left corner. Type in some text, and then press Esc on your keyboard, the -- INSERT -- disappears, and you can't just enter text like before. vi has two modes, command mode and insert mode, and after hitting Esc, you returned to command mode. To edit text like you would in any wysiwyg text editor, simply press i and make sure you see -- INSERT -- in the lower left corner.

The real power of vi lies in its 'modal' nature, but if you are just starting out, make all of your edits in -- INSERT -- mode, hit Esc to enter command mode, and then one of the following:

:wq (that's a colon, a w for write, and a q for quit)

or

:q! (that's a colon, a q for quit, and an exclamation for don't save anything)

Those are the only two commands you need for right now in command mode. If you like your changes, :wq, if you mess up and get lost, :q!

And before making any changes to a .vmx file, make sure the virtual machine is shut down, and back up the .vmx first, with a command like this (as root):

cp /vmfs/volumes/somevolume/somevm/somevm.vmx /root

That will copy the .vmx file to the root user's home directory, where it will be nice and safe in case something happens.

To edit a .vmx file with vi, just type:

vi /vmfs/volumes/somevolume/somevm/somevm.vmx


...read more