Unless you've been living under a rock for a while, you've probably read the recommendations to completely isolate the ESX console network. This sage advice addresses a serious threat in a virtual infrastructure: if a malicious person out in the LAN were able to gain SSH access to an ESX console, they would have an unholy amount of power. With just two commands at a shell prompt, they could snapshot a virtual disk, FTP it off somewhere, and then use an assortment of tools on the completely exposed filesystem from the comfort of their lair. This network isolation best practice could be the single most important security consideration in an environment, and should also be extended to the VMotion and storage networks.In fact, if security were the primary architecture concern, the ESX service console network and VirtualCenter server would be plugged into their own isolated network switch. The only way to log into VirtualCenter or SSH to an ESX server would be to march down the hall and pull out the monitor and keyboard in the server room. But obviously that isn't feasible for most organizations, so some sort of compromise between security and administrative access must be made.
One option is to install two NICs in the VirtualCenter servers; one in the LAN, and one plugged in to an isolated service console network switch. You could further enhance this by placing a physical firewall between the VirtualCenter server and the LAN, only allowing the specific ports needed for Active Directory authentication and VI client access from specific workstations to the VirtualCenter server.
With this design in place, an administrator would be able to get many things done from the comfort of a LAN workstation. But unfortunately there are a few critical elements that require direct network access to the ESX servers, such as virtual machine console access, physical-to-virtual migrations with Converter Enterprise, and VCB backups.
We could require that each workstation that needs VI Client or SSH access to the service console network have a second NIC and be physically patched into the isolated switch. But that's not very scalable, and a burden in all but the smallest of environments. There's a better option, and it provides a higher level of security: a virtual firewall with vNICs in two different port groups that only permits specific traffic between the console network and the LAN.
There are several good virtual firewall options out there, and many of them are ready-to-use virtual appliances. But for this application, a virtual appliance isn't going to cut it. We want to know exactly how the firewall VM was built, what applications were installed, and what services were enabled. Turns out that the Vyatta Community Edition open source networking software is just about perfect for what we're trying to accomplish here.
Vyatta is a full-featured networking operating system that provides routing, firewalling, VPN, IDS, DHCP, dynamic routing protocols, and pretty much all the features you would only expect from one of the major networking vendors. Think of it like a Linux-based version of Cisco IOS that can be installed on x86 hardware. And in the latest version, VC5, just released in March, they've really upped the ante by providing a web-based GUI.
Give it a spin, it's free!
Head over to the Vyatta.org site and you'll find download links that don't require registration. You'll probably notice that Vyatta also offers VC5 in a VMware Appliance format, but we'll be using the live CD image. Check out the video tutorials at the Vyatta.com site as well if you want to get a feel for what we're about to cover.
Once you've got the VC5 ISO file downloaded, set up a new virtual machine, perform a custom install, and give it Vyatta's minimum recommended settings:
- 512 MB of RAM
- 2 GB virtual hard disk
- Other Linux (32-bit) for the guest operating system
- Two virtual NICs, one in a port group on the same vSwitch as the Service Console, and one in a port group that uplinks to the LAN
The version I tested wasn't loading the advanced vmxnet virtual NIC, and instead was defaulting to the pcnet32 device, which is a very basic 10/100 NIC with limited features. So I went ahead and forced the VM to use an Intel PRO/1000 vNIC by editing the .vmx file for the VM and changing the vNIC to the e1000 device:
- Open a SSH session to the ESX server hosting the virtual router
- Browse to the datastore folder that contains the VM, and edit the .vmx with
vi
- Add a line for each vNIC:
ethernet0.virtualDev = "e1000" ethernet1.virtualDev = "e1000"
- And delete these lines to force a regeneration of the MAC addresses:
ethernet0.addressType = "vpx" ethernet0.generatedAddress = "00:50:56:af:18:93" ethernet1.addressType = "vpx" ethernet1.generatedAddress = "00:50:56:bd:27:1e"
- Now from
vipress Esc and then:wqto save the .vmx file
root with the password vyattaAt the prompt enter this command:
install-systemThe Vyatta installer will launch, an interactive script that prompts you for some basic information and confirmation during the install. Just choose all of the defaults during the install script, until you reach the prompt
This will destroy all data on /dev/xxx, type Yes here to continue.You'll be prompted to set the passwords for the system accounts, type
Yes and enter strong passwords for the root and vyatta accounts.When the installer finishes, shut the vm down with this command:
shutdown -h nowWhen the system boots up to the
vyatta login: prompt, log on as user vyatta with the password you established during the install.We first need to give the service console facing interface an IP address. To accomplish this, we'll enter configuration mode and assign an IP address, replacing the IP below with one appropriate for the environment. We're going to set ours to the gateway address we've already set on our ESX servers:
configure
set interfaces ethernet eth0 address 172.20.1.254/24
commit
All changes from the Vyatta console begin with the
configure command to enter configuration mode, and end with a commit command to apply the changes.We should save the changes now, so the configuration we made will survive a reboot of the Vyatta virtual machine. To save the configuration to the hard drive of the VM, just enter:
save
If you've set everything up correctly to this point, you should be able to ping that address from an ESX server.
If you type
exit, you'll exit configuration mode, and be in 'operational' mode. If you have spent any time with IOS, you should be pretty familiar with this type of modal work flow. In fact, you can only view the Vyatta configuration while in operational mode, with the command show configuration, just like in IOS, which doesn't make the show run command available in configuration mode. Also, just like in IOS, if you hit tab, you should get command completion in the Vyatta CLI.I've had enough of configuring the Vyatta VM through the VI Client, so let's enable SSH access and the Vyatta web GUI with these commands:
set service ssh
set service https
commit
save
Now you should be able to use
putty to log in as the user vyatta via SSH. Let's try out the web console, open a web browser from a machine with access to the service console network to the IP address we set up earlier, in our case it will be https://172.20.1.254Log in with the
vyatta user account and passwordIf you take a look around the web GUI, it should be pretty obvious that this is not a system you're going to just intuitively figure out. The GUI represents the CLI commands as objects, so without knowing the CLI logic, the GUI really isn't that helpful. To see what I mean, drill down to interfaces > ethernet > eth0, selecting eth0. Check out the address: attribute, and notice that the path we took through the web GUI reflects the command we entered through the CLI:
set interfaces ethernet eth0 address 172.20.1.254/24So let's configure the LAN facing interface through the web GUI. Drill down to interfaces > ethernet > eth1, selecting eth1. In the address: attribute section in the right pane, enter the IP address you want to use, in our case we'll enter
10.1.1.254/24. The other settings are good, so we'll press the Set button.Just like the CLI, we need to commit the changes before they become active, so press the Commit button from the upper right of the web GUI. If you connect the second vNIC from the VI Client now, you should be able to ping that address from the LAN. If everything is set up correctly, the Vyatta virtual machine should be routing between the service console network and the LAN. If you are not able to ping from both sides, then there's likely a routing or default gateway issue on one or both sides.
Let's configure a few system settings from the web GUI. Select system from the left side, and you'll see we can set the domain-name, gateway-address, host-name, name-server, ntp-server, and time-zone settings.
Start your filtering
In order to follow along with the rest of this tutorial, you'll need a few things in place:
- An isolated ESX service console network.
- A good relationship with the network administrator in your environment, or something to blackmail them with, because they'll need to add a static route to the ESX console network to the layer three device routing between the subnets in your LAN. If you're using RIP or OSPF in your network, the Vyatta VM is capable of running one of those dynamic routing protocols, but that's way beyond the scope of this.
In our test environment, we've got a simplified setup: the ESX hosts are using the service console facing interface IP of the Vyatta VM as their default gateway, and the LAN connected Windows VM we're using to connect to the ESX servers is using the LAN facing interface IP of the Vyatta VM as its default gateway.
ESX Hosts <-> Vyatta VM <-> LAN VM
172.20.1.x : 172.20.1.254 eth0 : 10.1.1.254 eth1 : 10.1.1.33
So at this point, we are routing between the networks and able to ping from both sides of the Vyatta VM. We want to restrict traffic in both directions, protecting the console network from the LAN, and also limiting the access the console network has to the LAN, so we'll need to create two firewall rule sets and apply them to the two interfaces.
This first rule set,
lan_in, grants SSH and VI Client access to a range of ESX hosts (172.20.1.1 to 172.20.1.5) from host 10.1.1.33, and this is how we would add it using the CLI:
vyatta@vyatta:~$ configure
vyatta@vyatta# set firewall name lan_in rule 10 action accept
vyatta@vyatta# set firewall name lan_in rule 10 description "Allow SSH and VI client access from 10.1.1.33"
vyatta@vyatta# set firewall name lan_in rule 10 destination address 172.20.1.1-172.20.1.5
vyatta@vyatta# set firewall name lan_in rule 10 destination port ssh,https,902-903
vyatta@vyatta# set firewall name lan_in rule 10 protocol tcp
vyatta@vyatta# set firewall name lan_in rule 10 source address 10.1.1.33
vyatta@vyatta# set firewall name lan_in rule 10 state established enable
vyatta@vyatta# set firewall name lan_in rule 10 state new enable
vyatta@vyatta# set firewall name lan_in rule 10 state related enable
vyatta@vyatta# set firewall name lan_in rule 10 state invalid disable
vyatta@vyatta# set interfaces ethernet eth1 firewall in name lan_in
vyatta@vyatta# commit
vyatta@vyatta# save
To set up the matching rule set on the interface in the console network, we just flip the source and destination, and configure it like this from the CLI:
vyatta@vyatta:~$ configure
vyatta@vyatta# set firewall name console_in rule 10 action accept
vyatta@vyatta# set firewall name console_in rule 10 description "SSH and VI client replies to 10.1.1.33"
vyatta@vyatta# set firewall name console_in rule 10 destination address 10.1.1.33
vyatta@vyatta# set firewall name console_in rule 10 protocol tcp
vyatta@vyatta# set firewall name console_in rule 10 source address 172.20.1.1-172.20.1.5
vyatta@vyatta# set firewall name console_in rule 10 source port ssh,https,902-903
vyatta@vyatta# set firewall name console_in rule 10 state established enable
vyatta@vyatta# set firewall name console_in rule 10 state related enable
vyatta@vyatta# set firewall name console_in rule 10 state new disable
vyatta@vyatta# set firewall name console_in rule 10 state invalid disable
vyatta@vyatta# set interfaces ethernet eth0 firewall in name console_in
vyatta@vyatta# commit
vyatta@vyatta# save
Now if you attempt to ping an ESX host from the LAN side it should fail as each firewall rule set has an implicit deny as the last rule. In order to restore the ability to ping, you would need to add a rule to both the
lan_in and console_in rule sets.We'll add another accept rule for an additional administrator's workstation. This time we'll do it using the web GUI, so open a web browser, log in to the web GUI, drill down to firewall > name > lan_in > rule, and select rule. In the text box on the right, type 20 and then click Set
>. From the screen that opens up, we will set up the second rule just like the first. Drop down to accept in the action: box, enter a description like Allow SSH and VI client access from 10.1.1.62, enter tcp for the protocol:, and click the Set button.
From the left pane now, select destination, click the Create button, and type 172.20.1.1-172.20.1.5, our range of ESX server IP addresses, in the address: box. In the port: box, enter ssh,https,902-903 and click the Set button.
*By the way, if you're curious as to how the protocol names map to the numbered ports, just type
less /etc/protocolsFrom the left pane again, select source, click the Create button, in the address: box enter 10.1.1.62 and click the Set button.
And finally, from the left pane, select state, click the Create button, check established:, new:, and related:, and then click the Set button.
We need a corresponding accept rule for the reply traffic back to the LAN workstation, so drill down to firewall > name > console_in > rule, and select rule. In the text box on the right, type 20 and then click the Set button.
Drop down the action: box to accept, enter the description SSH and VI client replies to 10.1.1.62, enter tcp for the protocol:, and click the Set button.
From the left pane, select destination, click the Create button, type 10.1.1.62 in the address: box, and click the Set button.
From the left pane again, select source, click the Create button, and in the address: box, enter 172.20.1.1-172.20.1.5. In the port: box, enter ssh,https,902-903 and click the Set button.
And finally, from the left pane, select state, click the Create button, check established: and related:, then click the Set button.
From the top of the web GUI, click the Commit button, and then the Save button, and you should now have VI Client and SSH access to the isolated ESX service console network from the second workstation.
The last configuration we should make to the virtual firewall is to lock down the Vyatta web GUI and SSH access from the LAN to just the workstations we defined above. To do that we'll create another rule set, but his time apply it as local to the interface rather than in. This will filter packets sent directly to the interface IP address.
From the left side of the web GUI, drill down to firewall > name, and select name. In the text box on the right, type lan_admin and then click the Set button.
Enter a description for the new rule set, like "Restricts administrative access from the LAN"
On the left, select rule. In the text box on the right, type 10 and then click the Set button.
Drop down the action: box to accept, enter the description Allow SSH and web GUI access from 10.1.1.33, enter tcp for the protocol:, and click the Set button.
From the left pane, select destination, click the Create button, type ssh,https in the port: box, and click the Set button.
From the left pane again, select source, click the Create button, in the address: box enter 10.1.1.33 and click the Set button.
And finally, from the left pane, select state, click the Create button, check established:, new:, and related:, and then click the Set button.
On the left side, drill down to interfaces > ethernet > eth1 > firewall > local, selecting local, and click the Create button. In the text box on the right, type lan_admin and then click the Set button.
Save the changes by clicking the Commit button, and then the Save button. You could now repeat those steps, creating a rule 20 under rule set
lan_admin for the second administrator's workstation, 10.1.1.62.Gleaming the cube
We've only touched the surface of the Vyatta networking software, and if you head over to the Vyatta website, you can download a lot of good reference material after a quick registration. If you take the time to read some of the documentation, you'll get a real sense for how dense and powerful the Vyatta operating system is, especially for extending virtual networking. Stay tuned, as we'll be doing some interesting projects with Vyatta in the future.
