Merge lp:~mpontillo/maas/fresh-images into lp:~maas-committers/maas/trunk

[Mike Pontillo]

Change default image selection to 'daily' from 'releases'. Drive-by fix to revert image sync protocol to http from https, to accommodate customer use cases and because images are already verified cryptographically.

[Mike Pontillo (mpontillo)]

Come to think of it, there should be a constant for this. It shouldn't be 5 places in the code...

[Lee Trager (ltrager)]

Should we document that releases still exists? Will releases ever go away(including making releases a sym link to daily)?

[Blake Rouse (blake-rouse)]

"It shouldn't be 5 places in the code..."? So why not fix that now?

review: Needs Fixing

[Mike Pontillo (mpontillo)]

I was about to. ;-)

[Mike Pontillo (mpontillo)]

I'll document that `releases` still exists, thanks Lee. I expect that a few customers won't want to be on the bleeding edge, so it'll be good for them to know.

[Scott Moser (smoser)]

Do realize that you're not really doing anything other than kicking the can from releases to daily.
The goal of this I believe is to resolve "we need fresh images".
The 'released' stream was not fresh because it required someone to QA that before you promoted it to released. The solution you've provided to this is to just use the daily stream.

That effectively just accepts "there is no QA".

If you're willing to do that, then you might has well just automatically move things from 'daily' to 'released'. Then at least in the future should you ever have a way to apply QA before promoting you don't have to build a "pre-daily" ppa to perform that QA on and then promote from 'pre-daily' to 'daily'.

Note that we clearly have no intent of ever producing bad things in daily, but reality is that people make mistakes (https://bugs.launchpad.net/maas-images/+bug/1582410).

[Mike Pontillo (mpontillo)]

@smoser, I feel like we should converge on a more granular way to select a risk-level for images, to be consistent with what we're doing elsewhere in the company.

For example, you could have:

    stable - equivalent to the latest point release? (14.04.4 for example)
    candidate - moved for every RC when a new release is coming
    beta - automatically moved based on some set of tests
    edge - freshest images (equivalent to daily today)

[Blake Rouse (blake-rouse)]

@smoser - Yes no QA, we know that. But again this is what is in *-updates pocket so if it doesn't work Ubuntu is already in hot water.

I think we should create a updated symlink between:

updated -> daily

So for now we can accept no QA, but later we can add QA and convert updated into its own stream. This will allow us to keep releases as they are today.

[Mike Pontillo (mpontillo)]

(Based on my stake in the ground for what each "risk channel" means, I think if we moved to the model above, `beta` would be the default.)

[Mike Pontillo (mpontillo)]

Alternatively, skip the `beta` channel altogether and we could just do stable/candidate/release, where candidate releases are auto-promoted after testing.

[Mike Pontillo (mpontillo)]

Sorry, I meant stable/candidate/edge.

[Blake Rouse (blake-rouse)]

Looks good. Lets go with this.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/fresh-images into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [93.3 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 188 kB in 0s (432 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest versio...

[Mike Pontillo (mpontillo)]

Filed a bug for this test failure:

https://bugs.launchpad.net/maas/+bug/1582944

Rolling the dice again.

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/fresh-images into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:4 http://security.ubuntu.com/ubuntu xenial-security InRelease [93.3 kB]
Fetched 188 kB in 0s (466 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest versio...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/fresh-images into lp:maas failed. Below is the output from the failed tests.

Get:1 http://security.ubuntu.com/ubuntu xenial-security InRelease [93.3 kB]
Hit:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 188 kB in 0s (451 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest versio...

[MAAS Lander (maas-lander)]

The attempt to merge lp:~mpontillo/maas/fresh-images into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [94.5 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [93.3 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 188 kB in 0s (408 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu3).
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
build-essential is already the newest version (12.1ubuntu2).
bzr is already the newest version (2.7.0-2ubuntu1).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
isc-dhcp-common is already the newest version (4.3.3-5ubuntu12).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
libpq-dev is already the newest version (9.5.2-1).
make is already the newest version (4.1-6).
postgresql is already the newest versio...

[Mike Pontillo (mpontillo)]

I talked to Gavin about this repeatedly-failing test case, and he's still working on getting it passing 100% of the tine, but is out of the office today. So I'll add a skip for now.

=== modified file 'docs/bootsources.rst'
--- docs/bootsources.rst	2016-04-11 16:23:26 +0000
+++ docs/bootsources.rst	2016-05-18 08:18:24 +0000
@@ -20,7 +20,7 @@
     $ maas $PROFILE boot-sources read
     [
         {
-            "url": "https://images.maas.io/ephemeral-v2/releases/",
+            "url": "http://images.maas.io/ephemeral-v2/daily/",
             "keyring_data": "",
             "resource_uri": "<url omitted for readability>",
             "keyring_filename": "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg",

=== modified file 'docs/sstreams-mirror.rst'
--- docs/sstreams-mirror.rst	2016-04-11 16:23:26 +0000
+++ docs/sstreams-mirror.rst	2016-05-18 08:18:24 +0000
@@ -35,6 +35,12 @@
 can use the ``maas`` (see :doc:`maascli`) command, logged in as the admin
 user::
 
+  $ maas <profile> boot-sources create url=http://<server>/images/ephemeral-v2/daily/ keyring_filename=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
+
+Note that if you wish to use older images (which change far less frequently,
+but will be lacking security updates), you can use the ``releases`` stream,
+such as::
+
   $ maas <profile> boot-sources create url=http://<server>/images/ephemeral-v2/releases/ keyring_filename=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
 
 And then initiate the download with::

=== modified file 'src/maasserver/bootsources.py'
--- src/maasserver/bootsources.py	2016-04-23 08:53:54 +0000
+++ src/maasserver/bootsources.py	2016-05-18 08:18:24 +0000
@@ -26,6 +26,7 @@
 from maasserver.utils.orm import transactional
 from maasserver.utils.threads import deferToDatabase
 from provisioningserver.auth import get_maas_user_gpghome
+from provisioningserver.config import DEFAULT_IMAGES_URL
 from provisioningserver.drivers.osystem.ubuntu import UbuntuOS
 from provisioningserver.import_images.download_descriptions import (
     download_all_image_descriptions,
@@ -49,7 +50,7 @@
     """Set default boot source if none is currently defined."""
     if not BootSource.objects.exists():
         source = BootSource.objects.create(
-            url='https://images.maas.io/ephemeral-v2/releases/',
+            url=DEFAULT_IMAGES_URL,
             keyring_filename=(
                 '/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg'))
         # Default is to import newest Ubuntu LTS releases, for only amd64

=== modified file 'src/maasserver/forms.py'
--- src/maasserver/forms.py	2016-05-11 19:01:48 +0000
+++ src/maasserver/forms.py	2016-05-18 08:18:24 +0000
@@ -153,6 +153,7 @@
     IPNetwork,
     valid_ipv6,
 )
+from provisioningserver.config import DEFAULT_IMAGES_URL
 from provisioningserver.logger import get_maas_logger
 from provisioningserver.utils.network import make_network
 from provisioningserver.utils.twisted import (
@@ -1403,8 +1404,8 @@
     boot_source_url = forms.URLField(
         label="Sync URL", required=True,
         help_text=(
-            "URL to sync boot image from. E.g. "
-            "https://images.maas.io/ephemeral-v2/releases/"))
+            "URL to sync boot image from. E.g. %s" % (
+                DEFAULT_IMAGES_URL)))
 
     def __init__(self, *args, **kwargs):
         super(BootSourceSettingsForm, self).__init__(*args, **kwargs)

=== modified file 'src/maasserver/rpc/tests/test_regionservice.py'
--- src/maasserver/rpc/tests/test_regionservice.py	2016-05-17 16:45:31 +0000
+++ src/maasserver/rpc/tests/test_regionservice.py	2016-05-18 08:18:24 +0000
@@ -746,6 +746,7 @@
 
         self.assertThat(service.ports, Equals([sentinel.port]))
 
+    @skip("XXX test fails far too often; bug #1582944")
     @wait_for_reactor
     @inlineCallbacks
     def test_start_up_logs_failure_if_all_endpoint_options_fail(self):

=== modified file 'src/maasserver/tests/test_bootsources.py'
--- src/maasserver/tests/test_bootsources.py	2016-05-12 19:07:37 +0000
+++ src/maasserver/tests/test_bootsources.py	2016-05-18 08:18:24 +0000
@@ -34,6 +34,7 @@
     MAASTransactionServerTestCase,
 )
 from maasserver.tests.test_bootresources import SimplestreamsEnvFixture
+from provisioningserver.config import DEFAULT_IMAGES_URL
 from provisioningserver.import_images import (
     download_descriptions as download_descriptions_module,
 )
@@ -102,7 +103,7 @@
         self.assertAttributes(
             source,
             {
-                'url': 'https://images.maas.io/ephemeral-v2/releases/',
+                'url': DEFAULT_IMAGES_URL,
                 'keyring_filename': (
                     '/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg'),
             })

=== modified file 'src/provisioningserver/config.py'
--- src/provisioningserver/config.py	2016-04-11 16:23:26 +0000
+++ src/provisioningserver/config.py	2016-05-18 08:18:24 +0000
@@ -162,6 +162,8 @@
 # Default result for cluster UUID if not set
 UUID_NOT_SET = None
 
+DEFAULT_IMAGES_URL = "http://images.maas.io/ephemeral-v2/daily/"
+
 
 class BootSourceSelection(Schema):
     """Configuration validator for boot source selection configuration."""
@@ -181,7 +183,7 @@
     if_key_missing = None
 
     url = UnicodeString(
-        if_missing="https://images.maas.io/ephemeral-v2/releases/")
+        if_missing=DEFAULT_IMAGES_URL)
     keyring = UnicodeString(
         if_missing="/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg")
     keyring_data = UnicodeString(if_missing="")

=== modified file 'src/provisioningserver/import_images/tests/test_download_resources.py'
--- src/provisioningserver/import_images/tests/test_download_resources.py	2016-05-12 19:07:37 +0000
+++ src/provisioningserver/import_images/tests/test_download_resources.py	2016-05-18 08:18:24 +0000
@@ -11,6 +11,7 @@
 
 from maastesting.matchers import MockCalledWith
 from maastesting.testcase import MAASTestCase
+from provisioningserver.config import DEFAULT_IMAGES_URL
 from provisioningserver.import_images import download_resources
 from provisioningserver.import_images.product_mapping import ProductMapping
 from simplestreams.objectstores import FileStore
@@ -73,7 +74,7 @@
         snapshot_path = self.make_dir()
         cache_path = os.path.join(storage_path, 'cache')
         file_store = FileStore(cache_path)
-        source_url = "https://images.maas.io/ephemeral-v2/releases/"
+        source_url = DEFAULT_IMAGES_URL
 
         download_resources.download_boot_resources(
             source_url, file_store, snapshot_path, None, None)