The goal of this document is to create 2 VMs on a host which share the same base image (for disk space purpose), and which can be used to migrate a 3rd VM back and forth
1) setup nested virtualization
Pass kvm_intel.nested=1 on the kernel command line, or 'rmmod kvm_intel' (no VM should be running or this won't be possible), followed by modprobe kvm_intel nested=1
The VMs which are going to use nested virtualization have to use CPU passthrough:
https://fedoraproject.org/wiki/How_to_enable_nested_virtualization_in_KVM
2) setup the shared disk image
Install virt-builder, and add a /etc/virt-builder/repos.d/rhel.conf file with:
[rhel]
uri=http://file.rdu.redhat.com/~rjones/builder/index.asc
See https://mojo.redhat.com/people/mbukatov/blog/2015/09/11/creating-rhel-machines-with-virt-builder for more details
Create base image:
$ virt-builder rhel-7.4 -o rhel7-base.qcow2 --format qcow2 --size 20G --root-password password:secret
I then moved the resulting file to /var/lib/libvirt/images
Create the 2 overlays for our 2 virtual hosts for migration:
$ qemu-img create -f qcow2 -o 'backing_file=./rhel7-base.qcow2' ./rhel7-migration-host1.qcow2
$ qemu-img create -f qcow2 -o 'backing_file=./rhel7-base.qcow2' ./rhel7-migration-host2.qcow2
I also put them in /var/lib/libvirt/images
$ virt-install --connect qemu:///system --name rhel7-migration-host1 --ram 2048 --os-variant rhel7.4 --disk path=/var/lib/libvirt/images/rhel7-migration-host1.qcow2 --graphics spice --import
$ virt-install --connect qemu:///system --name rhel7-migration-host2 --ram 2048 --os-variant rhel7.4 --disk path=/var/lib/libvirt/images/rhel7-migration-host2.qcow2 --graphics spice --import
You should now be able to start these 2 VMs
$ virsh -c qemu:///system start rhel7-migration-host1
$ virsh -c qemu:///system start rhel7-migration-host2
3) guest configuration (NFS) (these guests will be the nested hosts for migration)
I used shared NFS storage, with the NFS server between the baremetal host (won't get into details about the host configuration, iirc I added a "/var/lib/libvirt/images/ 192.168.1.0/24(rw,no_root_squash) 192.168.122.0/24(rw,no_root_squash) 10.32.0.0/16(rw,no_root_squash)" line to /etc/exports, and ran sudo systemctl start nfs)
Be very careful not to use the same disk image from 2 qemu instances running on 2 different machines or you'll get filesystem corruption!!
Configuring NFS on the VMs meant adding the appropriate pool:
natto-nfs/mnt/nfs
And then virsh pool-define pool.xml followed by virsh pool-start natto-nfs.
You also need to instruct selinux that libvirt is going to be using NFS:
$ sudo setsebool -P virt_use_nfs 1
On one of the 2 nested hosts, you can setup a guest VM which will be the one we
migrate around. I used virsh dumpxml on a preexisting VM on the host, changed
its disk path to /mnt/nfs instead of /var/lib/libvirt/images and defined it on
rhel7-migration-host1. It also needs to use cache="none" in its disk definition:
4) more guest configuration! (networking)
In order for migration to work, you need 2 more things:
- stop firewalld ($ sudo systemctl stop firewalld), or even disable it for subsequent boots.
This will allow SPICE connections from the baremetal host, as well as the libvirt daemons to communicate during the migration
- make sure 'ping $(ssh $other_host hostname)' works from the 2 virtual hosts as libvirt migration requires that. Without it, migration was hanging for minutes, and then failing. I edited /etc/hostname and ran hostname -F /etc/hostname to configure the host name, and added the corresponding entry to /etc/hosts on the other migration host as I did not have any DNS setup for that.
5) migrate :)
virsh -d0 -c qemu:///system migrate --graphicsuri spice://192.168.122.xx:5900 rhel7-nfs qemu+ssh://root@192.168.122.yy/system