OK devalias

In the previous post, I covered getting existing VMs out of VMWare ESXi and getting SmartOS ready to launch. Now it's time to boot up SmartOS and start converting our vmdk files.

Once SmartOS boots up, it will check for an existing "zones" zfs volume, which is where the VMs are stored. Since this is the first boot on this hardware, we don't yet have a zones volume. The helpful setup script runs and prompts for networking info, root password, and then asks us on which devices our global zone should exist. It also lists the available devices, using classic Solaris notation: c0t0d0 For the uninitiated, this breaks down to Controller 0, SCSI Target 0, Disk 0. This site covers the notation in more detail, for those interested. Back to the topic at hand!

As mentioned, the setup script is asking us which devices should be used for the global zone. Not so helpfully, we only have the device notation and no other information, like the output of format. Either have only the drives meant for the zones volume connected at first boot or guess the right devices. The setup script will determine which zfs pool layout to use based on the number of drives given. For example, two devices will be put into a mirror, while five will be put into a raidz. It will not allow for raidz2 or raidz3. Luckily, the docs explain how to change this after the fact. The setup script creates the pool and reboots.

If all went well, you'll be greeted with the following.

                _____
             ____   ____            .                   .
            _____   _____           | .-. .  . .-. :--. |-
            __         __           ;|   ||  |(.-' |  | |
            _____   _____       `--'  `-' `;-| `-' '  ' `-'
             ____   ____                  /  ; Joyent Live Image v0.147+
                _____                     `-'   build: 20131213T023304Z

00-1e-2a-b7-57-4e wscons login:

The ASCII art logo is a nice touch. Log in or ssh in as root and bask in the Solaris-bred goodness. Now it's time to take the VMDKs extracted earlier and pull them into SmartOS. This is where networked storage is helpful to keep the large VMDKs accessible. Let's start with a simple CentOS 6.4 system.

We need to know how large of a disk the VM expects. qemu-img provides this sort of info.

[root@00-1e-2a-b7-57-4e ~]# qemu-img info -f vmdk /zones/temp/centos6-disk1.vmdk
image: /zones/temp/centos6-disk1.vmdk
file format: vmdk
virtual size: 40G (42949672960 bytes)
disk size: 6.9G

Next, we need a JSON document describing the attributes of the VM. We'll use the virtual size value to set the disk's size value in megabytes.

{
  "brand": "kvm",
  "vcpus": 1,  
  "ram": 512,
  "resolvers": "192.168.240.10",
  "disks": [
    {
	"boot": true,
      	"model": "virtio",
	"size": 40960
    }
  ],
  "nics": [
    {
	"nic_tag": "admin",
      	"model": "virtio",
      	"ip": "192.168.240.50",
      	"netmask": "255.255.255.0",
      	"gateway": "192.168.240.1",
      	"primary": 1
    }
  ]
}

Pay attention to the value of resolvers. It's slightly more complicated to update that value later, so you want to point it to the correct DNS server. It is worth noting that some operating systems, such as any Windows, do not support virtio without extra drivers. Check out the docs to see all valid attributes.

We can validate our json with vmadm validate.

[root@00-1e-2a-b7-57-4e ~]# vmadm validate create -f centos6.json
VALID 'create' payload for kvm brand VMs.

With that written and validated, rerun the command without validate.

[root@00-1e-2a-b7-57-4e ~]# vmadm create -f centos6.json
Successfully created 2c922230-7878-11e3-981f-0800200c9a66

SmartOS uses UUIDs to access its VMs. Without copy/paste, this is quite cumbersome. Fortunately, by creating persistent dot files and appending this wrapper function, vmadm can use unique alias names in place of the UUID for access.

We can check that the vm was actually created with either vmadm list or zfs list.

[root@00-1e-2a-b7-57-4e ~]# zfs list
NAME                                               USED  AVAIL  REFER  MOUNTPOINT
...
zones/2c922230-7878-11e3-981f-0800200c9a66          82K  10.0G    82K  /zones/2c922230-7878-11e3-981f-0800200c9a66
zones/2c922230-7878-11e3-981f-0800200c9a66-disk0  40.1G  1.36T  40.1G  -
...

It's there! At this point we have an empty VM with hardware characteristics and an IP. The VM is ready to be migrated with help from qemu-img:

[root@00-1e-2a-b7-57-4e ~]# qemu-img convert -f vmdk -O host_device wobble-disk1.vmdk /dev/zvol/rdsk/zones/2c922230-7878-11e3-981f-0800200c9a66-disk0
... lots of time ... 
[root@00-1e-2a-b7-57-4e ~]#

This command takes quite a while and, unfortunately, the version of qemu-img in SmartOS does not have the patched in -p flag to indicate progress. Relax, have some tea, and document things while you wait. Is it done? Good, let's prove it.

Most management of VMs is done through the vmadm command. vmadm list will list all VMs.

[root@00-1e-2a-b7-57-4e ~]# vmadm list
UUID                                  TYPE  RAM      STATE             ALIAS
a608f729-2569-40b0-9b22-2d41734a73c3  KVM   512      running           mongoweb
c9475323-2dec-462f-bc89-c1f1d5274c3c  KVM   512      running           pxe
f873e475-dc43-45b8-8128-b88ef112a2c6  KVM   512      running           dbbackup
91e4473d-13fb-4c96-a64f-3ac5aba9e50b  KVM   1024     running           testweb
ac070f31-8f5a-4211-9ee3-e876937e42ad  KVM   1024     running           puppet
e5476ad0-1f69-4eb9-b974-bd862a52cc11  KVM   1024     running           kms
62d7e737-629e-45d5-9e80-0556b16e2972  KVM   2048     running           db01
b354b725-342f-4300-93c2-b9297cb7553b  KVM   2048     running           ad01

This is my current setup. If the centos6 VM is the only VM in your environment, it will display just that VM and, unless specified in the JSON definition, no alias. There are lots of options to alter output of this command to include more, less, or different information, detailed in the documentation.

There may be some gotchas here to solve before you can ssh in. SmartOS provides VNC access for each VM with a randomized port.

[root@00-1e-2a-b7-57-4e ~]# vmadm info 2c922230-7878-11e3-981f-0800200c9a66
...
  "vnc": {
    "host": "192.168.240.8",
    "port": 57755,
    "display": 51855
  },
...

Using the vnc client of your choice, connect to your system. You may need to send ctl-alt-del to restart, as it might have attempted to boot before the image copied over.

When I moved CentOS 6.4 VMs from ESXi, their ethernet interface changed from eth0 to eth1. Fix by doing the following.

[root@00-1e-2a-b7-57-4e ~]# cd /etc/sysconfig/network-scripts/
[root@00-1e-2a-b7-57-4e ~]# cp ifcfg-eth0 ifcfg-eth1

Then, modify the MAC address line. However, that's not scalable, so the better option is to deploy SmartOS and create the empty VMs first, grab the MAC address out of vmadm get UUID, and then create ifcfg-eth1 before extracting the VM from ESXi. I should add that to list of "things to sketch out."

For Windows Server VMs, you will need to alter their JSON description file to use disk model ide instead of virtio. Grab the correct vmtools iso here and place it in the root zone directory for the windows VM. Now, boot the VM with:

[root@00-1e-2a-b7-57-4e ~]# vmadm boot UUID order=cd,once=d cdrom=/vmtoolsXYZ.iso,ide

Alright, now SmartOS should be serving our ESXi VMs and, hopefully, there should be no problems at all. Now we can dive into SmartOS goodness and really make use of the Solaris-bred technologies that make it so great.