Merge lp:~ltrager/maas/lp1571007 into lp:~maas-committers/maas/trunk

[Lee Trager]

Add more logging to the boot image download process

I've added some more logging to the boot download process. I've added logging when boot resources and image descriptions are being downloaded. If a bootloaded fails to download from a mirror the URL the URL tried is now logged and no exception is raised.

[Gavin Panella (allenap)]

Looks good. I think you ought to slightly change the exception handling, but I'm not going to block you on it.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/lp1571007 into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [93.3 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [92.2 kB]
Fetched 185 kB in 0s (416 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already ...

=== modified file 'src/provisioningserver/boot/utils.py'
--- src/provisioningserver/boot/utils.py	2016-03-28 13:54:47 +0000
+++ src/provisioningserver/boot/utils.py	2016-04-27 18:26:38 +0000
@@ -23,6 +23,7 @@
 import urllib.parse
 import urllib.request
 
+from provisioningserver.import_images.helpers import maaslog
 from provisioningserver.utils.fs import tempdir
 from provisioningserver.utils.shell import call_and_check
 
@@ -46,8 +47,15 @@
     # Build a new opener so that the environment is checked for proxy
     # URLs. Using urllib2.urlopen() means that we'd only be using the
     # proxies as defined when urlopen() was called the first time.
-    response = urllib.request.build_opener().open(url)
-    return response.read()
+    try:
+        response = urllib.request.build_opener().open(url)
+        return response.read()
+    except urllib.error.URLError as e:
+        maaslog.error("Unable to download %s: %s", url, str(e.reason))
+        raise
+    except BaseException as e:
+        maaslog.error("Unable to download %s: %s", url, str(e))
+        raise
 
 
 def get_md5sum(data):

=== modified file 'src/provisioningserver/import_images/boot_resources.py'
--- src/provisioningserver/import_images/boot_resources.py	2016-03-28 13:54:47 +0000
+++ src/provisioningserver/import_images/boot_resources.py	2016-04-27 18:26:38 +0000
@@ -95,7 +95,11 @@
     """
     for _, boot_method in BootMethodRegistry:
         if arches.intersection(boot_method.bootloader_arches) != set():
-            boot_method.install_bootloader(destination)
+            try:
+                boot_method.install_bootloader(destination)
+            except BaseException:
+                maaslog.error(
+                    "Unable to download the %s bootloader.", boot_method.name)
 
 
 def make_arg_parser(doc):

=== modified file 'src/provisioningserver/import_images/download_descriptions.py'
--- src/provisioningserver/import_images/download_descriptions.py	2016-04-12 17:36:35 +0000
+++ src/provisioningserver/import_images/download_descriptions.py	2016-04-27 18:26:38 +0000
@@ -1,4 +1,4 @@
-# Copyright 2014-2015 Canonical Ltd.  This software is licensed under the
+# Copyright 2014-2016 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Download boot resource descriptions from Simplestreams repo.
@@ -192,6 +192,7 @@
     :param keyring: Optional keyring for verifying the repo's signatures.
     :return: A `BootImageMapping` describing available boot resources.
     """
+    maaslog.info("Downloading image descriptions from %s", path)
     mirror, rpath = path_from_mirror_url(path, None)
     policy = get_signing_policy(rpath, keyring)
     reader = UrlMirrorReader(mirror, policy=policy)

=== modified file 'src/provisioningserver/import_images/download_resources.py'
--- src/provisioningserver/import_images/download_resources.py	2015-12-01 18:12:59 +0000
+++ src/provisioningserver/import_images/download_resources.py	2016-04-27 18:26:38 +0000
@@ -225,6 +225,7 @@
     :param keyring_file: Optional path to a keyring file for verifying
         signatures.
     """
+    maaslog.info("Downloading boot resources from %s", path)
     writer = RepoWriter(snapshot_path, store, product_mapping)
     (mirror, rpath) = path_from_mirror_url(path, None)
     policy = get_signing_policy(rpath, keyring_file)