libvirt apparmor profile is preventing libvirt from running eucalyptus VMs

Daniel, can you please give the output of:
$ dmesg | grep audit

Also, can you give me the paths to the pid file, log file and monitor file for this virtual machine?

Daniel, also can you add the following to /etc/apparmor.d/abstractions/libvirt-qemu and let me know if it fixes it for you?

  /var/lib/eucalyptus/instances/**/console.log w,

Daniel, one last thing (sorry for the multiple requests), can you also give me the xml for this virtual machine:
$ virsh dumpxml i-4CFC08E8

Jamie, thank you for taking a look here. First, after your response, I've been able to modify /etc/apparmor.d/abstractions/libvirt-qemu with the following:

  /var/lib/eucalyptus/instances/**/console.log w,
  /var/lib/eucalyptus/instances/**/kernel r,
  /var/lib/eucalyptus/instances/**/ramdisk r,

in order to allow the NC to start a VM. I believe that kernel can always be 'r' only, but I'm not 100% sure about the initrc (ramdisk). It may be the case that some VMs could potentially modify the initrd on boot.

Regarding pidfile, monitor, log file:

from the commandline -

/usr/bin/kvm -S -M pc-0.11 -m 128 -smp 1 -name i-4CFC08E8 -uuid 9f141023-980c-0577-d143-72fcd2d8b7f1 -nographic -monitor unix:/var/run/libvirt/qemu/i-4CFC08E8.monitor,server,nowait -boot c -kernel /var/lib/eucalyptus/instances/admin/i-4CFC08E8/kernel -initrd /var/lib/eucalyptus/instances/admin/i-4CFC08E8/ramdisk -append root=/dev/sda1 console=ttyS0 -drive file=/var/lib/eucalyptus/instances/admin/i-4CFC08E8/disk,if=scsi,index=0,boot=on -net nic,macaddr=d0:0d:4c:fc:08:e8,vlan=0,model=e1000,name=e1000.0 -net tap,fd=17,vlan=0,name=tap.0 -serial file:/var/lib/eucalyptus/instances/admin/i-4CFC08E8/console.log -parallel none -usb

I at least see the monitor file path (/var/run/libvirt/qemu/i-4CFC08E8.monitor), libvirt doesn't appear to be specifying a pid or logfile path, and so i believe they are going to their default location(s). I can at least confirm that the logfile is being dropped in /var/log/libvirt/qemi//i-4CFC08E8.log (cannot confirm pidfile because the process is dying right away).

example libvirt dumpxml:

Connecting to uri: qemu:///system
<domain type='kvm' id='7'>
  <name>i-516E092C</name>
  <uuid>443555e4-42a5-d231-8bf6-4f862cf33bf9</uuid>
  <memory>131072</memory>
  <currentMemory>131072</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-0.11'>hvm</type>
    <kernel>/var/lib/eucalyptus/instances/admin/i-516E092C/kernel</kernel>
    <initrd>/var/lib/eucalyptus/instances/admin/i-516E092C/ramdisk</initrd>
    <cmdline>root=/dev/sda1 console=ttyS0</cmdline>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <source file='/var/lib/eucalyptus/instances/admin/i-516E092C/disk'/>
      <target dev='sda' bus='scsi'/>
    </disk>
    <interface type='bridge'>
      <mac address='d0:0d:51:6e:09:2c'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='e1000'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/eucalyptus/instances/admin/i-516E092C/console.log'/>
      <target port='0'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/eucalyptus/instances/admin/i-516E092C/console.log'/>
      <target port='0'/>
    </console>
  </devices>
</domain>

Regards

I've found one more runtime issue with the apparmor profile. Eucalyptus can provide the ability to dynamically attach/detach block devices to VMs at runtime using libvirt attach-disk/detach-disk. We currently use AOE for dynamic block devices, and these appear on the node in:

/dev/etherd/e*

when a volume (dynamic block device) is created on a remote storage controller. I believe, then, that the apparmor profile may have to also include:

/dev/etherd/e*

in order for this functionality to work.

I am going to provide a eucalyptus specific workaround in my next libvirt upload, to get this the serial, kernel and initrd working.

How are you using attach-disk/detach-disk? I can't seem to get them working here to test this.

This bug was fixed in the package libvirt - 0.7.0-1ubuntu6

---------------
libvirt (0.7.0-1ubuntu6) karmic; urgency=low

  * debian/apparmor/libvirt-qemu: workaround eucalyptus serial console,
    kernel and initrd location. This should be removed after virt-aa-helper is
    able to get these from XML. (LP: #431090)

 -- Jamie Strandboge <email address hidden> Thu, 17 Sep 2009 11:35:42 -0500

There is another bug, in KVM i suspect, that is preventing attach/detach of block devices from fully working, however, you can see some progress (logs indicating activity on the scsi bus in the VM when you do an attach/detach):

https://bugs.launchpad.net/ubuntu/+source/eucalyptus/+bug/432154

Can you thoroughly test libvirt 0.7.0-1ubuntu8 with eucalyptus? I fixed this bug properly along with attach and detach of devices and disks (using Jaunty kvm as you mentioned in bug #432154). Please make sure that the workaround rules in /etc/apparmor.d/abstractions/libvirt-qemu are no longer present (if the are and you removed them, be sure to shutdown and start the VMs to make sure the updated profile is in effect).

Specifically, I tested:
$ dd if=/dev/zero of=/tmp/foo.img bs=1M count=64
64+0 records in
64+0 records out
67108864 bytes (67 MB) copied, 0.182175 s, 368 MB/s

$ cat > /tmp/431090.xml << EOM
<disk type='block'>
  <driver name='phy'/>
  <source dev='/tmp/foo.img'/>
  <target dev='sdb'/>
</disk>
EOM

$ virsh attach-device <vm name> /tmp/431090.xml
...
Device attached successfully

$ cat /etc/apparmor.d/libvirt/libvirt-<vm uuid>.files | grep 'foo'
  "/tmp/foo.img" rw,

$ virsh detach-device <vm name> /tmp/431090.xml
...
Device detached successfully

$ cat /etc/apparmor.d/libvirt/libvirt-<vm uuid>.files | grep 'foo'
$

$ virsh attach-disk test1 /tmp/foo.img sdc --driver file
...
Disk attached successfully

$ cat /etc/apparmor.d/libvirt/libvirt-<vm uuid>.files | grep 'foo'
  "/tmp/foo.img" rw,

$ virsh detach-disk test1 sdc
...
Disk detached successfully

$ cat /etc/apparmor.d/libvirt/libvirt-<vm uuid>.files | grep 'foo'
$

For good measure, I also added a USB disk with virt-manager while a VM was running and the disk was added to /etc/apparmor.d/libvirt/libvirt-<vm uuid>.files and accessible in the VM via fdisk. I wasn't sure how to add a USB disk using virsh.

Hopefully, eucalyptus will now be fully supported (and protected! :) by the AppArmor security driver.

On the eucalyptus NC, when we try to start a VM, the process is unable to do so with the following error being thrown by libvirt (reported in nc.log):

libvirt: monitor socket did not show up.: Connection refused (code=38)
[Thu Dec 23 13:08:58 2010][002757][EUCAFATAL ] hypervisor failed to start domain
[Thu Dec 23 13:08:59 2010][002757][EUCADEBUG ] doDescribeResource() invoked
[Thu Dec 23 13:09:02 2010][002757][EUCADEBUG ] doDescribeInstances() invoked
[Thu Dec 23 13:09:03 2010][002757][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-3BA00773' (code=42)
[Thu Dec 23 13:09:03 2010][002757][EUCAINFO ] vrun(): [rm -rf /var/lib/eucalyptus/instances//admin/i-3BA00773/]
[Thu Dec 23 13:09:04 2010][002757][EUCAINFO ] stopping the network (vlan=10)
[Thu Dec 23 13:09:08 2010][002757][EUCADEBUG ] doDescribeResource() invoked
[Thu Dec 23 13:09:11 2010][002757][EUCADEBUG ] doDescribeInstances() invoked
[Thu Dec 23 13:09:17 2010][002757][EUCADEBUG ] doDescribeResource() invoked
[Thu Dec 23 13:09:20 2010][002757][EUCAINFO ] doTerminateInstance() invoked (id=i-3BA00773)
[Thu Dec 23 13:09:20 2010][002757][EUCAERROR ] libvirt: Domain not found: no domain with matching name 'i-3BA00773' (code=42)

This the qemu log :

root@cluster-controller-desktop:/var/log/libvirt/qemu# cat i-3BA00773.log
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin /usr/bin/kvm -S -M pc-0.12 -enable-kvm -m 256 -smp 1 -name i-3BA00773 -uuid 46d27510-fc55-666e-e2f5-1e2666a55b02 -nographic -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/i-3BA00773.monitor,server,nowait -monitor chardev:monitor -boot c -kernel /var/lib/eucalyptus/instances//admin/i-3BA00773/kernel -initrd /var/lib/eucalyptus/instances//admin/i-3BA00773/ramdisk -append root=/dev/sda1 console=ttyS0 -drive file=/var/lib/eucalyptus/instances//admin/i-3BA00773/disk,if=scsi,index=0,boot=on,format=raw -net nic,macaddr=d0:0d:3b:a0:07:73,vlan=0,model=e1000,name=e1000.0 -net tap,fd=40,vlan=0,name=tap.0 -chardev file,id=serial0,path=/var/lib/eucalyptus/instances//admin/i-3BA00773/console.log -serial chardev:serial0 -parallel none -usb

qemu: linux kernel too old to load a ram disk