Merge ~xnox/snappy-device-agents:compress-file-qcow2 into snappy-device-agents:master

[Dimitri John Ledkov]

compress_file: add support for QCOW2

This will allow deploying ubuntu-cpc QCOW2 cloud-images directly onto HiFive boards.

[Dimitri John Ledkov (xnox)]

qemu-utils package must be install where compress_file is executed.

[Paul Larson (pwlars)]

This seems to work ok after installing qemu-utils, thanks! However, I did try one of those images, and while it does seem to put the image on the sd card and boot it, it doesn't look like any network interface is detected other than loopback, so it never really finishes confirming that it's provisioned.

review: Approve

[Dimitri John Ledkov (xnox)]

@pwlars Horay!

The lack of internets on that image is being fixed. It will need a new kernel upload.

The fact that it sort of, kind of, attempted to boot is perfect at this point!

Please land this if possible.

[Dimitri John Ledkov (xnox)]

The kernel bug is this one https://bugs.launchpad.net/ubuntu/+source/linux-riscv/+bug/1905620

[Devices Certification Bot (ce-certification-qa)]

The merge was fine but running tests failed.

Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/tmpgwmnM1/env/bin/python3
Also creating executable in /tmp/tmpgwmnM1/env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Collecting flake8==2.4.0
  Using cached flake8-2.4.0-py2.py3-none-any.whl (28 kB)
Collecting mccabe<0.4,>=0.2.1
  Using cached mccabe-0.3.1-py2.py3-none-any.whl (7.6 kB)
Collecting pep8<1.6,>=1.5.7
  Using cached pep8-1.5.7-py2.py3-none-any.whl (37 kB)
Collecting pyflakes<0.9,>=0.8.1
  Using cached pyflakes-0.8.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pyflakes, pep8, mccabe, flake8
Successfully installed flake8-2.4.0 mccabe-0.3.1 pep8-1.5.7 pyflakes-0.8.1
running flake8
snappy_device_agents/__init__.py:282:13: F841 local variable 'output' is assigned to but never used

review: Needs Fixing

[Dimitri John Ledkov (xnox)]

fixed up.

[Devices Certification Bot (ce-certification-qa)]

The merge was fine but running tests failed.

Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /tmp/tmp168yxu/env/bin/python3
Also creating executable in /tmp/tmp168yxu/env/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Collecting flake8==2.4.0
  Using cached flake8-2.4.0-py2.py3-none-any.whl (28 kB)
Collecting mccabe<0.4,>=0.2.1
  Using cached mccabe-0.3.1-py2.py3-none-any.whl (7.6 kB)
Collecting pep8<1.6,>=1.5.7
  Using cached pep8-1.5.7-py2.py3-none-any.whl (37 kB)
Collecting pyflakes<0.9,>=0.8.1
  Using cached pyflakes-0.8.1-py2.py3-none-any.whl (35 kB)
Installing collected packages: pyflakes, pep8, mccabe, flake8
Successfully installed flake8-2.4.0 mccabe-0.3.1 pep8-1.5.7 pyflakes-0.8.1
running flake8
snappy_device_agents/__init__.py:282:13: F841 local variable '_' is assigned to but never used

review: Needs Fixing

[Dimitri John Ledkov (xnox)]

fixed up again.....

[Dimitri John Ledkov (xnox)]

I hope with this https://people.canonical.com/~xnox/riscv64/livecd.ubuntu-cpc.disk1.qcow2.img just boots

diff --git a/devices/muxpi/muxpi.py b/devices/muxpi/muxpi.py
index a8ee667..5cf49f1 100644
--- a/devices/muxpi/muxpi.py
+++ b/devices/muxpi/muxpi.py
@@ -39,7 +39,8 @@ class MuxPi:
     IMAGE_PATH_IDS = {'writable/usr/bin/firefox': 'pi-desktop',
                       'writable/etc': 'ubuntu',
                       'writable/system-data': 'core',
-                      'ubuntu-seed/snaps': 'core20'}
+                      'ubuntu-seed/snaps': 'core20',
+                      'cloudimg-rootfs/etc/cloud/cloud.cfg': 'ubuntu-cpc'}
 
     def __init__(self, config, job_data):
         with open(config) as configfile:
@@ -328,6 +329,8 @@ class MuxPi:
                 base = os.path.join(self.mount_point, 'writable')
                 if image_type == 'core':
                     base = os.path.join(base, 'system-data')
+                if image_type == 'ubuntu-cpc':
+                    base = os.path.join(self.mount_point, 'cloudimg-rootfs')
                 ci_path = os.path.join(base, 'var/lib/cloud/seed/nocloud-net')
                 self._run_control('sudo mkdir -p {}'.format(ci_path))
                 write_cmd = "sudo bash -c \"echo '{}' > /{}/{}\""
diff --git a/snappy_device_agents/__init__.py b/snappy_device_agents/__init__.py
index 0455ccb..36844f7 100644
--- a/snappy_device_agents/__init__.py
+++ b/snappy_device_agents/__init__.py
@@ -55,7 +55,9 @@ def filetype(filename):
     magic_headers = {
         b"\x1f\x8b\x08": "gz",
         b"\x42\x5a\x68": "bz2",
-        b"\xfd\x37\x7a\x58\x5a\x00": "xz"}
+        b"\xfd\x37\x7a\x58\x5a\x00": "xz",
+        b"\x51\x46\x49\xfb": "qcow2",
+    }
     with open(filename, 'rb') as f:
         filehead = f.read(1024)
     filetype = "unknown"
@@ -268,6 +270,23 @@ def compress_file(filename):
         with lzma.open(compressed_filename, 'wb') as compressed_image:
             with bz2.BZ2File(filename, 'rb') as old_compressed:
                 shutil.copyfileobj(old_compressed, compressed_image)
+    elif filetype(filename) == 'qcow2':
+        raw_filename = '{}.raw'.format(filename)
+        try:
+            # Remove the original file, unless we already did
+            os.unlink(raw_filename)
+        except FileNotFoundError:
+            pass
+        cmd = ['qemu-img', 'convert', '-O', 'raw', filename, raw_filename]
+        try:
+            subprocess.check_output(cmd, stderr=subprocess.STDOUT)
+        except subprocess.CalledProcessError as e:
+            logger.error('Image Conversion Output:\n %s', e.output)
+            raise
+        with open(raw_filename, 'rb') as uncompressed_image:
+            with lzma.open(compressed_filename, 'wb') as compressed_image:
+                shutil.copyfileobj(uncompressed_image, compressed_image)
+        os.unlink(raw_filename)
     else:
         # filetype is 'unknown' so assumed to be raw image
         with open(filename, 'rb') as uncompressed_image: