Introduction
Be it at work or at home, I spend a lot of time in virtual environments consisting mostly of ESXi. As part of the psych0tic network I run storm.psych0tik.net, an ESXi 4.0 box that’s currently running our IRC as well as the psych0tik GSProxy. Managing these environments is a task in itself, and I’ve found that no one tool gives me all of the functionality that I need. Additionally, I’ve found some confusion in what’s what as many of VMWare’s product’s have two names, due to a re-branding from Virtual Infrastructure 4 to vSphere. In this post, I’ll go through the various free management method’s that I’ve come across and use. I’ve also provided a short list of the non-free options. Listing all the features of each method would probably require a blog post or two per item, so I’ll go over some common uses as well as anything I think to be extra useful.
Embedded Web Server
The embedded web server provides a home page populated with links to various resources related to your ESXi 4.0 installation, the ability to browse your server’s datastores, and the ability to browse objects managed by the server. Accessing this service is simple and requires no setup. ESXi 4.0 installs with the VMWare ESX Server httpd and connecting requires nothing more than putting your server’s IP address into your browser (preceded by https://). The connection is protected by SSL (you may have to allow a self-signed certificate) and both the datastore and the objects are protected by basic HTTP authentication, using the same access controls as vSphere or the unsupported console.
vSphere Client (originally branded as the Virtual Infrastructure Client or VIC)
The VMWare vSphere Client provides the ability to monitor and manage multiple physical ESXi servers from the same console. This application runs on a separate machine, under Windows, and connects the the ESXi 4.0 host via the management network. Using the VI API, this tool connects to the hypervisor(s) on ports 443 and 903 and lets you manage the entire infrastructure from the hosts and switches to the server configuration.
For most of my day-to-day management the vSphere client is enough. You are able to start and stop virtual machines, take snapshots, adjust resources, get performance graphs, spawn consoles on the various virtual machines, and adjust settings for the networking, guests, and host. In addition to the direct management of the guests, vSphere also provides an interface for working with your datastore(s) and allows you to upload/download files from the physical server as well as functionality for importing virtual machines.
Another advantage of the vSphere client is that it’s a standard VMWare product and therefore other products may be built with that in mind. Products like Cisco’s Nexus 1000v (not free), a replacement for ESXi’s native networking, integrates nicely with vSphere.
Maintenance Console/SSH
Sometimes I need to go beyond the high-level management of my servers. For example, sometimes I need to clone a virtual machine and, as far as I know, there isn’t a way to do this through the vSphere client . Tasks like this are better suited to the maintenance console, an unsupported command line interface for ESXi 4.0. As I mentioned, this method isn’t supported by VMWare and may void any sort of support you get. (I have a free license, I don’t expect support anyways.) The maintenance console is similar to the Service Console in ESX 3.5, but with reduced functionality. Using busybox, it’s possible to open this management method up to remote users as well as to allow SCP. (In my experience, vSphere’s upload/download is less stable than SCP.)
In the maintenance console, you’ll find access to the datastore as well as the raw virtual machine files. In addition to basic functionality, various VMWare specific tools have been added for manipulating guests or the hypervisor itself. vmkfstools, for instance, allows you to clone guess.
Also accessible through the maintenance console is the vSphere Interactive Shell (vsish) , another undocumented shell. From the bits of information I’ve found and some mucking around with vsish, it seems to be a mixture of an integration of the service console’s /proc nodes and linux’s mcelog, for polling reliability information. One interesting command I did find was to crash the ESXi VMM with a kernel panic: vsish -e set /reliability/crashMe/Panic
The Virtual Infrastructure metashell (vimsh) is yet another unsupported and undocumented shell accessible via the console or SSH. From what I can tell, after ESXi 3.5, this is only accessible through the non-interactive vim-cmd.
libvirt
libvirt isn’t solely meant for managing VMWare’s ESXi 4.0, rather it’s meant to be a toolkit to interact with various hypvervisors and their features. Because libvirt isn’t designed specifically for ESXi 4.0, there are functions of the API that it does not implement, such as managing vSwitches; however, it does provide a common programmable interface for managing in multiple languages. libvirt natively supports C/C++ and python and has bindings for Perl, Ruby, Java, C#, and OCaml. Beyond the limitations libvirt has in it’s ESX drivers, the free version of ESXi 4.0 restricts your access to read-only, disabling more of the functionality provided through libvirt.
virsh
virsh is the virtual shell that comes with libvirt as a method for managing domains under various hypervisors. Since virsh is built on libvirt, it also is unable to manage ESXi’s vSwitches and is restricted by the read-only nature of ESXi 4.0′s free version.
Among the usable features the virsh provides are the ability to list currently active domains, view information about the host’s and guests’ hardware and capabilities and view configurations. Unfortunately, from my testing it seems that a good majority of the more useful features virsh provides, like the ability to connect to a serial console, aren’t easily available (if at all) under the free version of ESXi 4.0.
NON-FREE
The management methods listed below are not free and therefore I have not been able to test them. Any functionality I’ve listed is based on what I’ve read. In most cases, you’ll be able to find a more in-depth description (read – “sales pitch”) in one of the references at the bottom of this post.
The vSphere Remote Command Line Interface (RCLI) provides the functionality that was previously available in the Service Console, specifically the esxcfg-* commands. The RCLI also allows the administrator to script various tasks. The esxcfg-* commands the RCLI relies on are still available via the unsupported maintenance console.
Functionality in libvirt and virsh seems to be increased through the use of a paid version of ESXi 4.0 as both are heavily restricted due to the read-only nature of the free version of the hypervisor. It appears that libvirt would gain a significant amount of functionality, including the ability to connect to a serial console, to manage domain’s state (start, stop, pause), and to edit configurations.
VMWare also offers the vSphere Management Assistant (vMA), a prepackaged linux VM to be used as a centralized management platform on which to deploy scripts and management applications.
Conclusion
ESXi 4.0 provides a variety of methods, free and licensed, to maintain and manage both the hypervisor and the virtual machines. The majority of my day-to-day tasks are covered in the free version of ESXi 4.0 and the tools available to it. The ability to manage domains, clone vmdk’s, retrieve configurations and device information, as well as to write applications based on both libvirt’s drivers and the VI API are all available for free. Paid options provide added function through the RCLI and tools like libvirt or virsh are made more useful. Beyond the obvious methods listed above, the undocumented and unsupported portions of ESXi 4.0 may provide additional functionality and features that will open new possibilities in the future.
Do you have another way to manage some portion of ESXi 4.0? Have I missed some serious piece of functionality in one of these items listed above? Let psych0tik know what you think in the comments!
[References]
Virtual Infrastructure API Reference
Accessing the ESXi maintenance console
Creating Kernel Panics with vsish
libvirt mailing list – questions (and answers) on libvirt’s ESX driver