Merge lp:~ltrager/maas/virt_detect into lp:~maas-committers/maas/trunk

[Lee Trager]

If available use systemd-detect-virt to determine if the commissioning node is virtualized

This modifies the virtuality script to use systemd-detect-virt if available. When systemd-detect-virt is not available(i.e commissioning with trusty) it falls back on the old method which only detects qemu. systemd-detect-virt outputs the detected virtualization which is now stored so users can review it and possibly create virt type tags later on.

[Mike Pontillo (mpontillo)]

The python code looks okay to me, but the shell script where you detect the binary looks odd to me. I think I caught a small issue below, though I'm not certain what the behavior would be.

review: Needs Fixing

[Lee Trager (ltrager)]

Thanks for the review, I've updated up the shell script as you suggested.

[Mike Pontillo (mpontillo)]

Looks better now.

You could still probably lose the shell redirection on the fallback grep and just use "grep -q". But I won't block you on that.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/virt_detect into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 189 kB in 0s (443 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubun...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/virt_detect into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 189 kB in 0s (433 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubun...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/virt_detect into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [212 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [93.3 kB]
Fetched 494 kB in 0s (1,166 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu...

=== modified file 'src/metadataserver/models/commissioningscript.py'
--- src/metadataserver/models/commissioningscript.py	2016-06-02 12:06:46 +0000
+++ src/metadataserver/models/commissioningscript.py	2016-06-11 23:59:05 +0000
@@ -231,7 +231,7 @@
     """Process the results of `VIRTUALITY_SCRIPT`.
 
     This adds or removes the *virtual* tag from the node, depending on
-    the presence of the terms "notvirtual" or "virtual" in `output`.
+    whether a virtualization type is listed.
 
     If `exit_status` is non-zero, this function returns without doing
     anything.
@@ -239,16 +239,16 @@
     assert isinstance(output, bytes)
     if exit_status != 0:
         return
+    decoded_output = output.decode('ascii').strip()
     tag, _ = Tag.objects.get_or_create(name='virtual')
-    if b'notvirtual' in output:
+    if 'none' in decoded_output:
         node.tags.remove(tag)
-    elif b'virtual' in output:
+    elif decoded_output == '':
+        logger.warn(
+            "No virtual type reported in VIRTUALITY_SCRIPT output for node "
+            "%s", node.system_id)
+    else:
         node.tags.add(tag)
-    else:
-        logger.warn(
-            "Neither 'virtual' nor 'notvirtual' appeared in the "
-            "captured VIRTUALITY_SCRIPT output for node %s.",
-            node.system_id)
 
 
 _xpath_routers = "/lldp//id[@type='mac']/text()"

=== modified file 'src/metadataserver/models/tests/test_commissioningscript.py'
--- src/metadataserver/models/tests/test_commissioningscript.py	2016-06-02 12:06:46 +0000
+++ src/metadataserver/models/tests/test_commissioningscript.py	2016-06-11 23:59:05 +0000
@@ -266,38 +266,36 @@
     def test_sets_virtual_tag(self):
         node = factory.make_Node()
         self.assertTagsEqual(node, [])
-        set_virtual_tag(node, b"virtual", 0)
+        set_virtual_tag(node, b"qemu", 0)
         self.assertTagsEqual(node, ["virtual"])
 
     def test_removes_virtual_tag(self):
         node = factory.make_Node()
         node.tags.add(self.getVirtualTag())
         self.assertTagsEqual(node, ["virtual"])
-        set_virtual_tag(node, b"notvirtual", 0)
+        set_virtual_tag(node, b"none", 0)
         self.assertTagsEqual(node, [])
 
     def test_output_not_containing_virtual_does_not_set_tag(self):
         logger = self.useFixture(FakeLogger())
         node = factory.make_Node()
         self.assertTagsEqual(node, [])
-        set_virtual_tag(node, b"wibble", 0)
+        set_virtual_tag(node, b"", 0)
         self.assertTagsEqual(node, [])
         self.assertIn(
-            "Neither 'virtual' nor 'notvirtual' appeared in the captured "
-            "VIRTUALITY_SCRIPT output for node %s.\n" % node.system_id,
-            logger.output)
+            "No virtual type reported in VIRTUALITY_SCRIPT output for node "
+            "%s" % node.system_id, logger.output)
 
     def test_output_not_containing_virtual_does_not_remove_tag(self):
         logger = self.useFixture(FakeLogger())
         node = factory.make_Node()
         node.tags.add(self.getVirtualTag())
         self.assertTagsEqual(node, ["virtual"])
-        set_virtual_tag(node, b"wibble", 0)
+        set_virtual_tag(node, b"", 0)
         self.assertTagsEqual(node, ["virtual"])
         self.assertIn(
-            "Neither 'virtual' nor 'notvirtual' appeared in the captured "
-            "VIRTUALITY_SCRIPT output for node %s.\n" % node.system_id,
-            logger.output)
+            "No virtual type reported in VIRTUALITY_SCRIPT output for node "
+            "%s" % node.system_id, logger.output)
 
 
 class TestUpdateHardwareDetails(MAASServerTestCase):

=== modified file 'src/metadataserver/tests/test_api.py'
--- src/metadataserver/tests/test_api.py	2016-05-24 14:53:56 +0000
+++ src/metadataserver/tests/test_api.py	2016-06-11 23:59:05 +0000
@@ -934,21 +934,21 @@
     def test_signal_stores_virtual_tag_on_node_if_virtual(self):
         node = factory.make_Node(status=NODE_STATUS.COMMISSIONING)
         client = make_node_client(node=node)
-        content = 'virtual'.encode('utf-8')
+        content = 'qemu'.encode('utf-8')
         response = call_signal(
             client, script_result=0,
             files={'00-maas-02-virtuality.out': content})
         self.assertEqual(http.client.OK, response.status_code)
         node = reload_object(node)
         self.assertEqual(
-            ["virtual"], [each_tag.name for each_tag in node.tags.all()])
+            ['virtual'], [each_tag.name for each_tag in node.tags.all()])
 
     def test_signal_removes_virtual_tag_on_node_if_not_virtual(self):
         node = factory.make_Node(status=NODE_STATUS.COMMISSIONING)
         tag, _ = Tag.objects.get_or_create(name='virtual')
         node.tags.add(tag)
         client = make_node_client(node=node)
-        content = 'notvirtual'.encode('utf-8')
+        content = 'none'.encode('utf-8')
         response = call_signal(
             client, script_result=0,
             files={'00-maas-02-virtuality.out': content})
@@ -960,7 +960,7 @@
     def test_signal_leaves_untagged_physical_node_unaltered(self):
         node = factory.make_Node(status=NODE_STATUS.COMMISSIONING)
         client = make_node_client(node=node)
-        content = 'notvirtual'.encode('utf-8')
+        content = 'none'.encode('utf-8')
         response = call_signal(
             client, script_result=0,
             files={'00-maas-02-virtuality.out': content})

=== modified file 'src/metadataserver/tests/test_api_status.py'
--- src/metadataserver/tests/test_api_status.py	2016-05-24 14:53:56 +0000
+++ src/metadataserver/tests/test_api_status.py	2016-06-11 23:59:05 +0000
@@ -616,7 +616,7 @@
         tag, _ = Tag.objects.get_or_create(name='virtual')
         node.tags.add(tag)
         client = make_node_client(node=node)
-        content = 'notvirtual'.encode('utf-8')
+        content = 'none'.encode('utf-8')
         payload = {
             'event_type': 'finish',
             'result': 'SUCCESS',

=== modified file 'src/provisioningserver/refresh/node_info_scripts.py'
--- src/provisioningserver/refresh/node_info_scripts.py	2016-06-01 19:30:03 +0000
+++ src/provisioningserver/refresh/node_info_scripts.py	2016-06-11 23:59:05 +0000
@@ -70,16 +70,17 @@
     ip addr
     """)
 
-# Built-in script to detect virtual instances. It will only detect QEMU
-# for now and may need expanding/generalising at some point.
-# XXX ltrager 2016-01-14 - Replace with virt-what
+# Built-in script to detect virtual instances.
 VIRTUALITY_SCRIPT = dedent("""\
     #!/bin/sh
-    grep '^model name.*QEMU.*' /proc/cpuinfo >/dev/null 2>&1
-    if [ $? -eq 0 ]; then
-        echo "virtual"
+    if type systemd-detect-virt 2>&1 > /dev/null; then
+        systemd-detect-virt
+    elif grep -q '^model name.*QEMU.*' /proc/cpuinfo; then
+        # Fall back if systemd-detect-virt isn't available. This method only
+        # detects QEMU virtualization not including KVM.
+        echo "qemu"
     else
-        echo "notvirtual"
+        echo "none"
     fi
     """)