Merge lp:~bladernr/checkbox/1621986-virt-accept-urls into lp:checkbox

[Jeff Lane ]

--image and the config file cna now accept the following items:

http://mywebsite.com/images/
 - relative URL and script will find files in the same format as cloud-images.ubuntu.com, e.d. SERIES-server-ARCH-disk1.img

http://mywebsite.com/images/mycloudimage.img
 - Absolute URL to an image with an arbitrary name.

file:///path/to/local/cloudimage.img
 - URL for local filesystem path to image needed

/path/to/local/cloudimage.img
 - absolute local path to image needed

THe web URLs can point to ANY arbitrary location that has the cloud image we need. THe first one is intended for local mirrors of cloud-image, thelatter to call out any cloud image you want to boot as test.

Also, if you provide NONE of these, the default behaviour remains in that the script will go to cloud-images.ubuntu.com to get the proper image.

The above can be passed via the --image option:

./virtualization kvm --image http://somedomain.net/image.img

or via ENV Var as defined in /etc/xdg/canonical-certification.conf.

Additionally, I did some PEP8 cleanup, and removed a vestigal XenTest class that will never be used.

[Sylvain Pineau (sylvain-pineau)]

Thanks a lot for this url option, just one thing though. If you remove the xen class, please also remove the virtualization/xen* jobs from jobs/virtualization.txt.in

review: Needs Fixing

[lp:~bladernr/checkbox/1621986-virt-accept-urls]

4494. By Jeff Lane  on 2016-09-12

Removed long forgotten Xen jobs from jobs/virtualization.txt.in

[Jeff Lane  (bladernr)]

Whoops... thanks for the reminder. I forgot those were even there. Fixed.

review: Approve

[Sylvain Pineau (sylvain-pineau)]

+1, thanks Jeff

review: Approve

=== modified file 'providers/plainbox-provider-checkbox/bin/virtualization'
--- providers/plainbox-provider-checkbox/bin/virtualization	2016-08-09 19:59:36 +0000
+++ providers/plainbox-provider-checkbox/bin/virtualization	2016-09-12 17:56:36 +0000
@@ -44,13 +44,11 @@
 import tarfile
 import time
 import urllib.request
+from urllib.parse import urlparse
 
 DEFAULT_TIMEOUT = 500
 
 
-class XENTest(object):
-    pass
-
 # The "TAR" type is a tarball that contains both
 # a disk image and a kernel binary. This is useful
 # on architectures that don't (yet) have a bootloader
@@ -110,9 +108,9 @@
         'qemu_bin': 'qemu-system-ppc64',
         'qemu_disk_type': QEMU_DISK_TYPE_VIRTIO,
         'qemu_extra_args': [
-	    '-enable-kvm',
+            '-enable-kvm',
             '-machine', 'pseries,usb=off',
-	    '-cpu', 'POWER8',
+            '-cpu', 'POWER8',
         ],
     },
     's390x': {
@@ -121,7 +119,7 @@
         'qemu_bin': 'qemu-system-s390x',
         'qemu_disk_type': QEMU_DISK_TYPE_VIRTIO,
         'qemu_extra_args': [
-	    '-enable-kvm',
+            '-enable-kvm',
             '-machine', 's390-ccw-virtio-2.5',
         ],
     },
@@ -200,35 +198,68 @@
         self.arch = check_output(['dpkg', '--print-architecture'],
                                  universal_newlines=True).strip()
         self.qemu_config = QEMU_ARCH_CONFIG[self.arch]
-
-    def download_image(self):
-        """
-        Downloads Cloud image for same release as host machine
-        """
-
-        # Check Ubuntu release info. Example {quantal, precise}
-        release = lsb_release.get_lsb_information()["CODENAME"]
-
-        # Construct URL
-        cloud_url = "http://cloud-images.ubuntu.com"
-
+        self.release = lsb_release.get_lsb_information()["CODENAME"]
+
+    def url_to_path(self, image_path):
+        """
+        Test the provided image path to determine if it's a URL or or a simple
+        file path
+        """
+        url = urlparse(image_path)
+        if url.scheme == '' or url.scheme == 'file':
+            # Gives us path wheter we specify a filesystem path or a file URL
+            logging.debug("Cloud image exists locally at %s" % url.path)
+            return url.path
+        elif url.scheme == 'http' or url.scheme == 'ftp':
+            # Gives us the stuff needed to build the URL to download the image
+            return self.download_image(image_path)
+
+    def construct_cloud_filename(self):
+        """
+        Build a URL for official Ubuntu images hosted either at
+        cloud-images.ubuntu.com or on a maas server hosting a mirror of
+        cloud-images.ubuntu.com
+        """
         if self.qemu_config['cloudimg_type'] == CLOUD_IMAGE_TYPE_TAR:
             cloud_iso = "%s-server-cloudimg-%s.tar.gz" % (
-                        release, self.qemu_config['cloudimg_arch'])
+                        self.release, self.qemu_config['cloudimg_arch'])
         elif self.qemu_config['cloudimg_type'] == CLOUD_IMAGE_TYPE_DISK:
             cloud_iso = "%s-server-cloudimg-%s-disk1.img" % (
-                        release, self.qemu_config['cloudimg_arch'])
+                        self.release, self.qemu_config['cloudimg_arch'])
         else:
             logging.error("Unknown cloud image type")
-            return False
-        image_url = "/".join((
-            cloud_url, release, "current", cloud_iso))
+            sys.exit(1)
+        return cloud_iso
 
+    def download_image(self, image_url=None):
+        """
+        Downloads Cloud image for same release as host machine
+        """
+        if image_url is None:
+            # If we have not specified a URL to get our images from, default
+            # to ubuntu.com
+            cloud_url = "http://cloud-images.ubuntu.com"
+            cloud_iso = self.construct_cloud_filename()
+            full_url = "/".join((
+                cloud_url, self.release, "current", cloud_iso))
+        else:
+            url = urlparse(image_url)
+            if url.path.endswith('/') or url.path == '':
+                # If we have a relative URL (MAAS server mirror)
+                cloud_url = image_url
+                cloud_iso = self.construct_cloud_filename()
+                full_url = "/".join((
+                    cloud_url, cloud_iso))
+            else:
+                # Assume anything else is an absolute URL to a remote server
+                cloud_iso = url.path.split('/')[-1]
+                cloud_url = "{}://{}".format(url.scheme, url.netloc)
+                full_url = image_url
         logging.debug("Downloading {}, from {}".format(cloud_iso, cloud_url))
 
         # Attempt download
         try:
-            resp = urllib.request.urlretrieve(image_url, cloud_iso)
+            resp = urllib.request.urlretrieve(full_url, cloud_iso)
         except (IOError,
                 OSError,
                 urllib.error.HTTPError,
@@ -362,6 +393,10 @@
                 logging.debug('No image specified, downloading one now.')
                 # Download cloud image
                 self.image = self.download_image()
+            else:
+                logging.debug('Cloud image location specified: %s.' %
+                              self.image)
+                self.image = self.url_to_path(self.image)
 
             if self.image and os.path.isfile(self.image):
 
@@ -410,35 +445,10 @@
 def test_kvm(args):
     print("Executing KVM Test", file=sys.stderr)
 
-    DEFAULT_CFG = "/etc/xdg/virtualization.cfg"
     image = ""
     timeout = ""
 
-    # Configuration data can come from three sources.
-    # Lowest priority is the config file.
-    config_file = DEFAULT_CFG
-    config = configparser.SafeConfigParser()
-
-    try:
-        config.readfp(open(config_file))
-    except IOError:
-        logging.warn("Config file %s was not found" % DEFAULT_CFG)
-        logging.warn("Now looking for ENV variables and command line arguments instead")
-    else:
-        try:
-            timeout = config.getfloat("KVM", "timeout")
-        except ValueError:
-            logging.warning('Invalid or Empty timeout in config file. '
-                            'Falling back to default')
-        except configparser.NoSectionError as e:
-            logging.exception(e)
-
-        try:
-            image = config.get("KVM", "image")
-        except configparser.NoSectionError:
-            logging.exception('Invalid or Empty image in config file.')
-
-    # Next in priority are environment variables.
+    # First in priority are environment variables.
     if 'KVM_TIMEOUT' in os.environ:
         try:
             timeout = float(os.environ['KVM_TIMEOUT'])

=== modified file 'providers/plainbox-provider-checkbox/jobs/virtualization.txt.in'
--- providers/plainbox-provider-checkbox/jobs/virtualization.txt.in	2016-03-21 16:13:00 +0000
+++ providers/plainbox-provider-checkbox/jobs/virtualization.txt.in	2016-09-12 17:56:36 +0000
@@ -14,27 +14,3 @@
  cloud image.
 _summary:
  Verify KVM guest boots
-
-plugin: shell
-category_id: 2013.com.canonical.plainbox::virtualization
-id: virtualization/xen_ok
-requires: package.name == 'libvirt-bin'
-user: root
-estimated_duration: 1.0
-command: virsh -c xen:/// domstate Domain-0
-_description:
- Test to verify that the Xen Hypervisor is running.
-_summary:
- Verify Xen is running
-
-plugin: shell
-category_id: 2013.com.canonical.plainbox::virtualization
-id: virtualization/xen_check_vm
-depends: virtualization/xen_ok
-user: root
-estimated_duration: 300.0
-command: xen_test /images/xentest.img /images/xentest.xml
-_description:
- Test to check that a Xen domU image can boot and run on Xen on Ubuntu
-_summary:
- Verify Xen domU can boot