Merge lp:~heber013/utah/fix-provisioning-function into lp:utah

[Heber Parrucci]

Fixing broken provisioning function due to initrd file extension change

Fixing broken provisioning function due to initrd file extension change

[Paul Larson (pwlars)]

Looks sane, +1 assuming it works for you

review: Approve

[Jean-Baptiste Lallement (jibel)]

I think the fix is incomplete. casper/initrd and the target directory initrd have the same name and utah fails with ENOTDIR

cat: write error: Broken pipe
Unhandled error in UTAH:
Traceback (most recent call last):
  File "/usr/bin/run_utah_tests.py", line 132, in <module>
    exitstatus = timeout(config.jobtimeout, run_utah_tests)
  File "/usr/lib/python2.7/dist-packages/utah/timeout.py", line 65, in timeout
    return command(*args, **kw)
  File "/usr/bin/run_utah_tests.py", line 121, in run_utah_tests
    exitstatus, locallogs = run_tests(args, _get_machine(args))
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 344, in run_tests
    exitstatus, remote_path = _run(machine, args.runlist, extraopts)
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 256, in _run
    return _install_and_run(machine, runlist_url, extraopts)
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 174, in _install_and_run
    machine.installclient()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 315, in installclient
    self.uploadfiles([deb], tmppath)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 202, in uploadfiles
    self.activecheck()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 364, in activecheck
    self.provisioncheck()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 247, in provisioncheck
    config.installtimeout, self._provision, provision_data)
  File "/usr/lib/python2.7/dist-packages/utah/timeout.py", line 75, in timeout
    retval = command(*args, **kw)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 364, in _provision
    self._create(provision_data)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/vm.py", line 522, in _create
    self._copy_qemu_scripts(tmpdir=tmpdir)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 717, in _copy_qemu_scripts
    self.copy_qemu_scripts_to_path(initrd_work_dir)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 1010, in copy_qemu_scripts_to_path
    shutil.copy2(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 130, in copy2
    copyfile(src, dst)
  File "/usr/lib/python2.7/shutil.py", line 83, in copyfile
    with open(dst, 'wb') as fdst:
IOError: [Errno 20] Not a directory: '/var/lib/utah/vm/utah_vm_V0OnwPlGppPWxuFE/tmpcIvRKW/initrd/qemu-setup.py'

review: Needs Fixing

[Heber Parrucci (heber013)]

I tried fixing _copy_qemu_scripts and some more errors. Now I am getting an error in the function _preseedcasper:

2018-08-06 17:17:13,086 root WARNING: Command (sed -i 1i/scripts/casper-bottom/utah\n[ -e /conf/param.conf ] && . /conf/param.conf /var/lib/utah/vm/utah-21-cosmic-amd64/tmp7jWHqv/initrd.d/scripts/casper-bottom/ORDER) failed with return code: 2

[lp:~heber013/utah/fix-provisioning-function]

1130. By Heber Parrucci on 2018-08-22

Fixing issue when repacking initrd

1131. By Heber Parrucci on 2018-09-06

Fixing utah tests:
* Adding more memory to created VMs
* Adding video type qxl
* Adding retries when starting the VM with vm.create()

1132. By Heber Parrucci on 2018-09-07

Increasing wait period default value when starting the virtual machine

1133. By Heber Parrucci on 2018-09-07

Updating logic on first VM boot

[Jean-Baptiste Lallement (jibel)]

Thanks for this patch.

It fixes the problem with desktop images but breaks server tests.

The error is:

INFO: logpath was already set by /etc/utah/config
2018-09-10 03:00:05,890 cleanup WARNING: OSError when changing directory permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865-cosmic-server-amd64/disk0.qcow2'
2018-09-10 03:00:05,890 cleanup WARNING: OSError when changing directory permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865-cosmic-server-amd64/tmpBBkVPS/initrd.gz'
2018-09-10 03:00:05,891 cleanup WARNING: OSError when changing directory permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865-cosmic-server-amd64/tmpBBkVPS/kernel'
2018-09-10 03:00:05,891 cleanup WARNING: OSError when changing directory permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865-cosmic-server-amd64/tmpBBkVPS/cosmic-server-amd64.iso'
Unhandled error in UTAH:
Traceback (most recent call last):
  File "/usr/bin/run_utah_tests.py", line 132, in <module>
    exitstatus = timeout(config.jobtimeout, run_utah_tests)
  File "/usr/lib/python2.7/dist-packages/utah/timeout.py", line 65, in timeout
    return command(*args, **kw)
  File "/usr/bin/run_utah_tests.py", line 121, in run_utah_tests
    exitstatus, locallogs = run_tests(args, _get_machine(args))
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 344, in run_tests
    exitstatus, remote_path = _run(machine, args.runlist, extraopts)
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 256, in _run
    return _install_and_run(machine, runlist_url, extraopts)
  File "/usr/lib/python2.7/dist-packages/utah/run.py", line 174, in _install_and_run
    machine.installclient()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py", line 316, in installclient
    self.uploadfiles([deb], tmppath)
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 202, in uploadfiles
    self.activecheck()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 366, in activecheck
    self._start()
  File "/usr/lib/python2.7/dist-packages/utah/provisioning/vm.py", line 570, in _start
    self.vm.create()
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: Requested operation is not valid: domain is already running

+ RETCODE=4

review: Needs Fixing

[lp:~heber013/utah/fix-provisioning-function]

1134. By Heber Parrucci on 2018-09-10

Fixing start vm function

[Heber Parrucci (heber013)]

> Thanks for this patch.
>
> It fixes the problem with desktop images but breaks server tests.
>
> The error is:
>
> INFO: logpath was already set by /etc/utah/config
> 2018-09-10 03:00:05,890 cleanup WARNING: OSError when changing directory
> permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865
> -cosmic-server-amd64/disk0.qcow2'
> 2018-09-10 03:00:05,890 cleanup WARNING: OSError when changing directory
> permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865
> -cosmic-server-amd64/tmpBBkVPS/initrd.gz'
> 2018-09-10 03:00:05,891 cleanup WARNING: OSError when changing directory
> permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865
> -cosmic-server-amd64/tmpBBkVPS/kernel'
> 2018-09-10 03:00:05,891 cleanup WARNING: OSError when changing directory
> permissions: [Errno 1] Operation not permitted: '/var/lib/utah/vm/utah-74865
> -cosmic-server-amd64/tmpBBkVPS/cosmic-server-amd64.iso'
> Unhandled error in UTAH:
> Traceback (most recent call last):
> File "/usr/bin/run_utah_tests.py", line 132, in <module>
> exitstatus = timeout(config.jobtimeout, run_utah_tests)
> File "/usr/lib/python2.7/dist-packages/utah/timeout.py", line 65, in timeout
> return command(*args, **kw)
> File "/usr/bin/run_utah_tests.py", line 121, in run_utah_tests
> exitstatus, locallogs = run_tests(args, _get_machine(args))
> File "/usr/lib/python2.7/dist-packages/utah/run.py", line 344, in run_tests
> exitstatus, remote_path = _run(machine, args.runlist, extraopts)
> File "/usr/lib/python2.7/dist-packages/utah/run.py", line 256, in _run
> return _install_and_run(machine, runlist_url, extraopts)
> File "/usr/lib/python2.7/dist-packages/utah/run.py", line 174, in
> _install_and_run
> machine.installclient()
> File "/usr/lib/python2.7/dist-packages/utah/provisioning/provisioning.py",
> line 316, in installclient
> self.uploadfiles([deb], tmppath)
> File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 202,
> in uploadfiles
> self.activecheck()
> File "/usr/lib/python2.7/dist-packages/utah/provisioning/ssh.py", line 366,
> in activecheck
> self._start()
> File "/usr/lib/python2.7/dist-packages/utah/provisioning/vm.py", line 570,
> in _start
> self.vm.create()
> File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1035, in create
> if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
> libvirtError: Requested operation is not valid: domain is already running
>
> + RETCODE=4

It should be fixed now. Thanks!

[Jean-Baptiste Lallement (jibel)]

Thanks. Tested on Xenial with cosmic and bionic images of Desktop and server.

review: Approve

=== modified file 'conf/utah/default-vm.xml'
--- conf/utah/default-vm.xml	2016-02-04 16:28:14 +0000
+++ conf/utah/default-vm.xml	2018-09-10 14:38:14 +0000
@@ -20,8 +20,8 @@
 <domain type='qemu'>
   <name>Utah Example</name>
   <uuid>00000000-0000-0000-0000-000000000000</uuid>
-  <memory>524288</memory>
-  <currentMemory>524288</currentMemory>
+  <memory>1048576</memory>
+  <currentMemory>1048576</currentMemory>
   <vcpu>1</vcpu>
   <os>
     <type arch='i686' machine='pc'>hvm</type>
@@ -49,6 +49,9 @@
       <model type='virtio'/>
     </interface>
     <graphics type='vnc' autoport='yes'/>
+    <video>
+      <model type='qxl' vram='16384' heads='1'/>
+    </video>
   </devices>
 </domain>
 

=== modified file 'utah/provisioning/provisioning.py'
--- utah/provisioning/provisioning.py	2018-07-07 16:12:04 +0000
+++ utah/provisioning/provisioning.py	2018-09-10 14:38:14 +0000
@@ -25,6 +25,7 @@
 import re
 import shutil
 import stat
+import subprocess
 import sys
 import urllib
 import uuid
@@ -321,7 +322,7 @@
                                 'File not found: {filename}; '
                                 'UTAH was probably updated during the run. '
                                 'Please try again.'
-                                .format(filename=deb),
+                                    .format(filename=deb),
                                 retry=True)
                     raise err
                 except AttributeError:
@@ -508,8 +509,8 @@
         raise UTAHProvisioningException(
             '{cls} attempted to call the {method} method '
             'of the base Machine class, which is not implemented'
-            .format(cls=self.__class__.__name__,
-                    method=method))
+                .format(cls=self.__class__.__name__,
+                        method=method))
 
     def cleanfile(self, path):
         """Register a path to be cleaned later.
@@ -558,7 +559,6 @@
 
 
 class CustomInstallMixin(object):
-
     """Provide routines for automating an install from an image."""
 
     def _preparekernel(self, kernel=None, tmpdir=None):
@@ -661,15 +661,24 @@
         if os.path.splitext(initrd)[1] == '.gz':
             self.logger.debug('Using gzip based on file extension')
             pipe.prepend('zcat $IN', 'f-')
+            pipe.append('cpio -ivd 2>/dev/null', '-.')
+            exitstatus = pipe.copy(initrd, '/dev/null')
+            self.initrd_uncompressed_path = os.path.join(tmpdir, 'initrd.d')
+            self.initrd_no_extension = False
         elif os.path.splitext(initrd)[1] == '.lz':
             self.logger.debug('Using lzma based on file extension')
             pipe.prepend('lzcat -S .lz $IN', 'f-')
+            pipe.append('cpio -ivd 2>/dev/null', '-.')
+            exitstatus = pipe.copy(initrd, '/dev/null')
+            self.initrd_uncompressed_path = os.path.join(tmpdir, 'initrd.d')
+            self.initrd_no_extension = False
+        elif not os.path.splitext(initrd)[1]:
+            exitstatus = subprocess.check_call(['unmkinitramfs', initrd, os.getcwd()])
+            self.initrd_uncompressed_path = os.path.join(tmpdir, 'initrd.d', 'main')
+            self.initrd_no_extension = True
         else:
             raise UTAHProvisioningException(
-                'initrd file does have have gz or lz extension: {}'
-                .format(initrd))
-        pipe.append('cpio -ivd 2>/dev/null', '-.')
-        exitstatus = pipe.copy(initrd, '/dev/null')
+                'initrd file does have have gz or lz extension or no extension: {}'.format(initrd))
         if exitstatus != 0:
             # Currently this comes up as 512 when things seem fine
             # TODO: refactor this and check for codes at all stages
@@ -683,10 +692,10 @@
         if tmpdir is None:
             tmpdir = self.tmpdir
         shutil.copyfile(config.sshpublickey,
-                        os.path.join(tmpdir, 'initrd.d', 'utah-ssh-key'))
+                        os.path.join(self.initrd_uncompressed_path, 'utah-ssh-key'))
 
         self.logger.info('Creating latecommand scripts')
-        filename = os.path.join(tmpdir, 'initrd.d', 'utah-latecommand')
+        filename = os.path.join(self.initrd_uncompressed_path, 'utah-latecommand')
         template.write('utah-latecommand.jinja2',
                        filename,
                        user=config.user,
@@ -696,19 +705,22 @@
                        install_type=self.image.installtype,
                        md5=self.image.getmd5())
 
-        filename = os.path.join(tmpdir, 'initrd.d', 'utah-setup')
+        filename = os.path.join(self.initrd_uncompressed_path, 'utah-setup')
         template.write('utah-setup.jinja2',
                        filename,
                        packages=config.installpackages,
                        log_file='/var/log/utah-install')
 
-        filename = os.path.join(tmpdir, 'initrd.d', 'utah-autorun.sh')
+        filename = os.path.join(self.initrd_uncompressed_path, 'utah-autorun.sh')
         template.write('utah-autorun.sh.jinja2',
                        filename,
                        log_file='/var/log/utah-install')
 
     def _copy_qemu_scripts(self, tmpdir=None):
-        initrd_work_dir = os.path.join(tmpdir, 'initrd')
+        if self.initrd_no_extension:
+            initrd_work_dir = self.initrd_uncompressed_path
+        else:
+            initrd_work_dir = os.path.join(tmpdir, 'initrd')
         if not os.path.exists(initrd_work_dir):
             os.makedirs(initrd_work_dir)
         # copy the qemu scripts into initrd
@@ -736,8 +748,8 @@
             if self.image.installtype == 'desktop':
                 self._rewrite_failure_command(preseed)
 
-            output_preseed_filename = os.path.join(tmpdir,
-                                                   'initrd.d', 'preseed.cfg')
+            output_preseed_filename = os.path.join(self.initrd_uncompressed_path,
+                                                   'preseed.cfg')
             with open(output_preseed_filename, 'w') as f:
                 f.write(preseed.dump())
             self.finalpreseed = output_preseed_filename
@@ -766,7 +778,7 @@
             question.prepend('ubiquity ubiquity/summary note')
             question.prepend('ubiquity ubiquity/reboot boolean true')
 
-        filename = os.path.join(tmpdir, 'initrd.d', 'latecommand-wrapper')
+        filename = os.path.join(self.initrd_uncompressed_path, 'latecommand-wrapper')
         target_log_file = '/target{}'.format(log_file)
         template.write('latecommand-wrapper.jinja2',
                        filename,
@@ -838,10 +850,8 @@
         self.logger.info('Inserting preseed into casper')
         if tmpdir is None:
             tmpdir = self.tmpdir
-
-        casper_dir = os.path.join(tmpdir, 'initrd.d', 'scripts',
-                                  'casper-bottom')
-
+        casper_dir = os.path.join(os.path.join(self.initrd_uncompressed_path, 'scripts',
+                                               'casper-bottom'))
         filename = os.path.join(casper_dir, 'utah')
         template.write('casper-preseed-script.jinja2', filename)
         os.chmod(filename, 0755)
@@ -856,7 +866,7 @@
         if exitstatus != 0:
             raise UTAHProvisioningException('Failed to setup casper script')
 
-        casper_file = os.path.join(tmpdir, 'initrd.d', 'etc', 'casper.conf')
+        casper_file = os.path.join(self.initrd_uncompressed_path, 'etc', 'casper.conf')
         tmpfilename = '{}.tmp'.format(casper_file)
         with open(casper_file, 'r') as i:
             with open(tmpfilename, 'w') as o:
@@ -879,7 +889,7 @@
         """
         if tmpdir is None:
             tmpdir = self.tmpdir
-        inittab = os.path.join(tmpdir, 'initrd.d', 'etc', 'inittab')
+        inittab = os.path.join(self.initrd_uncompressed_path, 'etc', 'inittab')
         if os.path.isfile(inittab):
             self.logger.info('Updating inittab')
             with open(inittab, 'a') as myfile:
@@ -889,7 +899,7 @@
                              "-f /var/log/syslog \n")
 
         self.logger.info('Creating rsyslog config file')
-        conffilename = os.path.join(tmpdir, 'initrd.d', '50-utahdefault.conf')
+        conffilename = os.path.join(self.initrd_uncompressed_path, '50-utahdefault.conf')
         with open(conffilename, 'w') as f:
             if self.rsyslog.port:
                 ip = self._ipaddr(config.bridge)
@@ -909,21 +919,30 @@
         self.logger.info('Repacking initrd')
         if tmpdir is None:
             tmpdir = self.tmpdir
-        pipe = pipes.Template()
-        pipe.prepend('find .', '.-')
-        pipe.append('cpio --quiet -o -H newc', '--')
-        # Desktop image loads initrd.gz,
-        # but for physical machines we should stick with lz
-        if self.image.installtype == 'desktop':
-            self.logger.debug('Using lzma because installtype is desktop')
-            pipe.append('lzma -9fc ', '--')
-            initrd = os.path.join(tmpdir, 'initrd.lz')
+        if self.initrd_no_extension:
+            os.chdir(self.initrd_uncompressed_path)
+            pipe = pipes.Template()
+            pipe.prepend('find .', '.-')
+            pipe.append('cpio --quiet -o -H newc', '--')
+            initrd = os.path.join(tmpdir, 'initrd')
+            if pipe.copy('/dev/null', initrd) != 0:
+                raise UTAHProvisioningException('Failed to repack initrd')
         else:
-            self.logger.debug('Using gzip because installtype is not desktop')
-            pipe.append('gzip -9fc ', '--')
-            initrd = os.path.join(tmpdir, 'initrd.gz')
-        if pipe.copy('/dev/null', initrd) != 0:
-            raise UTAHProvisioningException('Failed to repack initrd')
+            pipe = pipes.Template()
+            pipe.prepend('find .', '.-')
+            pipe.append('cpio --quiet -o -H newc', '--')
+            # Desktop image loads initrd.gz,
+            # but for physical machines we should stick with lz
+            if self.image.installtype == 'desktop':
+                self.logger.debug('Using lzma because installtype is desktop')
+                pipe.append('lzma -9fc ', '--')
+                initrd = os.path.join(tmpdir, 'initrd.lz')
+            else:
+                self.logger.debug('Using gzip because installtype is not desktop')
+                pipe.append('gzip -9fc ', '--')
+                initrd = os.path.join(tmpdir, 'initrd.gz')
+            if pipe.copy('/dev/null', initrd) != 0:
+                raise UTAHProvisioningException('Failed to repack initrd')
         return initrd
 
     def _cmdlinesetup(self):

=== modified file 'utah/provisioning/vm.py'
--- utah/provisioning/vm.py	2017-07-06 18:03:17 +0000
+++ utah/provisioning/vm.py	2018-09-10 14:38:14 +0000
@@ -220,6 +220,7 @@
                 os.makedirs(self.directory)
             except OSError as err:
                 raise UTAHVMProvisioningException(err)
+        self.first_boot = True
         self.logger.debug('CustomVM init finished')
 
     def _createdisks(self, disksizes=None):
@@ -552,12 +553,26 @@
         if self.poweroff:
             self.cleanfunction(self.vm.shutdown, force=True)
 
-    def _start(self):
+    def _start(self, retries=6, wait=10):
         """Start the VM."""
         self.logger.info('Starting CustomVM')
+        attempt = 1
         if self.vm is not None:
-            if self.vm.isActive() == 0:
+            # For desktop tests the first boot configures the qemu script to run system tests and
+            # shuts down the machine, so we start the machine and wait until it shuts down,
+            # then we start it again and continue
+            if self.image.installtype == 'desktop' and self.first_boot:
                 self.vm.create()
+                while self.vm.isActive() == 1 and attempt <= retries:
+                    self.logger.info('Waiting for the VM to shut down, attempt number: %s', attempt)
+                    time.sleep(wait)
+                    attempt += 1
+                if self.vm.isActive() == 0:
+                    self.vm.create()
+                    self.first_boot = False
+            else:
+                if self.vm.isActive() == 0:
+                    self.vm.create()
         else:
             raise UTAHVMProvisioningException('Failed to provision VM')