Merge lp:~ltrager/maas/new_kernels into lp:~maas-committers/maas/trunk

[Lee Trager]

Add support for new style kernels(e.g hwe-16.04), lowlatency kernels, rolling kernels, and edge kernels. While the old method was able to use an algorithm based on the first letter of the release this looks up the release/version mapping using the UbuntuDistro library. This will require the region to be running on a system with an updated distro-info-data package, otherwise new releases added to the stream won't be deployable.

The new kernel name hwe-16.04-lowlatency-edge is 25 characters, because of this I had to bump the allowable lengths in the BootSourceCache table from 20 to 32.

This adds support for new style kernels(e.g hwe-16.04), lowlatency kernels, rolling kernels, and edge kernels. Old style kernels(e.g hwe-t) are still supported and can be used along side new style kernels.

[Blake Rouse (blake-rouse)]

Lots of comments and identified with some key issues with this branch. Some decisions need to be made to make sure this is the correct path.

review: Needs Fixing

[Lee Trager (ltrager)]

I've updated the branch to include the kflavor in the BootResource and BootSourceCache tables. The kflavor field is now used instead of accessing extra['kflavor'] throughout the region. The rack currently broadly assumes the subarch is the kernel. From how the rack reads the SimpleStream created by the region, to how kernels are stored on the filesystem, and the TGT and GRUB config files are generated. If we make that much change it brings up questions as to how we're presenting hwe_kernels and flavors to the user as well.

I've also allowed any kernel type to be used as the min_hwe_kernel. If a min_hwe_kernel is set and the user does not set a hwe_kernel during deploy MAAS picks a kernel at or above the min_hwe_kernel with the same kflavor.

[Andres Rodriguez (andreserl)]

I have one question, how will this change affect the directories kernels are placed on the system in the Rack Controller?

The real question I want to answer is:

If we are to upgrade from 2.0 to 2.1 and the user doesn't update their images/kernels (or doesn't update the streams to the new version), will MAAS be able to continue to deploy machines with the current images/kernels available already in the FS?

review: Needs Information

[Blake Rouse (blake-rouse)]

Looks good.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~ltrager/maas/new_kernels into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease [95.7 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security InRelease [94.5 kB]
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Fetched 190 kB in 0s (451 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind avahi-utils bash bind9 bind9utils build-essential bzr bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common isc-dhcp-server libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-attr python3-bson python3-convoy python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-netaddr python3-netifaces python3-novaclient python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-simplejson python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
archdetect-deb is already the newest version (1.117ubuntu2).
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~rc+dfsg-1ubuntu2).
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+dfsg-11ubuntu1).
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest ve...

=== modified file 'src/maasserver/bootresources.py'
--- src/maasserver/bootresources.py	2016-08-19 10:22:29 +0000
+++ src/maasserver/bootresources.py	2016-08-31 06:33:18 +0000
@@ -458,8 +458,12 @@
         os = get_os_from_product(product)
         series = product['release']
         arch = product['arch']
-        subarch = product['subarch']
         name = '%s/%s' % (os, series)
+        if 'kflavor' in product and product['kflavor'] != 'generic':
+            subarch = "%s-%s" % (product['subarch'], product['kflavor'])
+        else:
+            subarch = product['subarch']
+
         architecture = '%s/%s' % (arch, subarch)
 
         # Allow a generated resource to be replaced by a sycned resource. This
@@ -488,16 +492,14 @@
                 # replaced with this synced image.
                 resource.rtype = BOOT_RESOURCE_TYPE.SYNCED
 
+        resource.kflavor = product.get('kflavor')
         # Simplestreams content from maas.io includes the following
         # extra fields. Looping through the extra product data and adding it to
         # extra will not work as the product data that is passed into this
         # object store contains additional data that should not be stored into
-        # the database. If kflavor and/or subarches exist in the product then
-        # we store those values to expose in the simplestreams endpoint on the
-        # region.
+        # the database. If subarches exist in the product then we store those
+        # values to expose in the simplestreams endpoint on the region.
         resource.extra = {}
-        if 'kflavor' in product:
-            resource.extra['kflavor'] = product['kflavor']
         if 'subarches' in product:
             resource.extra['subarches'] = product['subarches']
 

=== modified file 'src/maasserver/bootsources.py'
--- src/maasserver/bootsources.py	2016-06-16 20:51:55 +0000
+++ src/maasserver/bootsources.py	2016-08-31 06:33:18 +0000
@@ -165,9 +165,14 @@
                 BootSourceCache.objects.filter(boot_source=bootsource).delete()
                 if not descriptions.is_empty():
                     for spec, item in descriptions.mapping.items():
+                        kflavor = item.get('kflavor')
+                        if kflavor not in (None, 'generic'):
+                            subarch = "%s-%s" % (spec.subarch, kflavor)
+                        else:
+                            subarch = spec.subarch
                         BootSourceCache.objects.create(
                             boot_source=bootsource, os=spec.os,
-                            arch=spec.arch, subarch=spec.subarch,
+                            arch=spec.arch, subarch=subarch, kflavor=kflavor,
                             release=spec.release, label=spec.label,
                             release_codename=item.get('release_codename'),
                             release_title=item.get('release_title'),

=== added file 'src/maasserver/migrations/builtin/maasserver/0081_allow_larger_bootsourcecache_fields.py'
--- src/maasserver/migrations/builtin/maasserver/0081_allow_larger_bootsourcecache_fields.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/migrations/builtin/maasserver/0081_allow_larger_bootsourcecache_fields.py	2016-08-31 06:33:18 +0000
@@ -0,0 +1,42 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import (
+    migrations,
+    models,
+)
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('maasserver', '0080_change_packagerepository_url_type'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='bootsourcecache',
+            name='arch',
+            field=models.CharField(max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='bootsourcecache',
+            name='label',
+            field=models.CharField(max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='bootsourcecache',
+            name='os',
+            field=models.CharField(max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='bootsourcecache',
+            name='release',
+            field=models.CharField(max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='bootsourcecache',
+            name='subarch',
+            field=models.CharField(max_length=32),
+        ),
+    ]

=== added file 'src/maasserver/migrations/builtin/maasserver/0082_add_kflavor.py'
--- src/maasserver/migrations/builtin/maasserver/0082_add_kflavor.py	1970-01-01 00:00:00 +0000
+++ src/maasserver/migrations/builtin/maasserver/0082_add_kflavor.py	2016-08-31 06:33:18 +0000
@@ -0,0 +1,47 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import (
+    migrations,
+    models,
+)
+
+
+def add_kflavor_to_boot_resource(apps, schema_editor):
+    BootResource = apps.get_model('maasserver', 'BootResource')
+    for resource in BootResource.objects.all():
+        if 'kflavor' in resource.extra:
+            resource.kflavor = resource.extra['kflavor']
+            del resource.extra['kflavor']
+            resource.save()
+
+
+def add_kflavor_to_boot_source_cache(apps, schema_editor):
+    BootSourceCache = apps.get_model('maasserver', 'BootSourceCache')
+    for bsc in BootSourceCache.objects.filter(os='ubuntu'):
+        # The kflavor was never stored as part of the BootSourceCache
+        # however previosuly we only had generic kernels in the stream.
+        bsc.kflavor = 'generic'
+        bsc.save()
+
+        
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('maasserver', '0081_allow_larger_bootsourcecache_fields'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='bootresource',
+            name='kflavor',
+            field=models.CharField(max_length=32, blank=True, null=True),
+        ),
+        migrations.RunPython(add_kflavor_to_boot_resource),
+        migrations.AddField(
+            model_name='bootsourcecache',
+            name='kflavor',
+            field=models.CharField(max_length=32, blank=True, null=True),
+        ),
+        migrations.RunPython(add_kflavor_to_boot_source_cache),
+    ]

=== modified file 'src/maasserver/models/bootresource.py'
--- src/maasserver/models/bootresource.py	2016-07-30 01:17:54 +0000
+++ src/maasserver/models/bootresource.py	2016-08-31 06:33:18 +0000
@@ -220,7 +220,8 @@
             return False
         return True
 
-    def get_usable_hwe_kernels(self, name=None, architecture=None):
+    def get_usable_hwe_kernels(
+            self, name=None, architecture=None, kflavor=None):
         """Return the set of usable kernels for architecture and release."""
         if not name:
             name = ''
@@ -229,6 +230,8 @@
         kernels = set()
         for resource in self.filter(
                 architecture__startswith=architecture, name__startswith=name):
+            if kflavor is not None and resource.kflavor != kflavor:
+                continue
             resource_set = resource.get_latest_set()
             if(resource_set is None or
                not resource_set.commissionable or
@@ -240,8 +243,22 @@
             if "subarches" in resource.extra:
                 for subarch in resource.extra["subarches"].split(","):
                     if subarch.startswith("hwe-"):
-                        kernels.add(subarch)
-        return sorted(kernels)
+                        if kflavor is None:
+                            kernels.add(subarch)
+                        else:
+                            # generic kflavors are not included in the subarch.
+                            if kflavor == 'generic':
+                                kparts = subarch.split('-')
+                                if len(kparts) == 2:
+                                    kernels.add(subarch)
+                            else:
+                                if kflavor in subarch:
+                                    kernels.add(subarch)
+        # Make sure kernels named with a version come after the kernels named
+        # with the first letter of release. This switched in Xenial so this
+        # preserves the chronological order of the kernels.
+        return sorted(
+            kernels, key=lambda k: (k.replace('hwe-', '')[0].isdigit(), k))
 
     def get_kpackage_for_node(self, node):
         """Return the kernel package name for the kernel specified."""
@@ -340,11 +357,13 @@
     architecture = CharField(
         max_length=255, blank=False, validators=[validate_architecture])
 
+    kflavor = CharField(max_length=32, blank=True, null=True)
+
     extra = JSONObjectField(blank=True, default="", editable=False)
 
     def __str__(self):
-        return "<BootResource name=%s, arch=%s>" % (
-            self.name, self.architecture)
+        return "<BootResource name=%s, arch=%s, kflavor=%s>" % (
+            self.name, self.architecture, self.kflavor)
 
     @property
     def display_rtype(self):

=== modified file 'src/maasserver/models/bootsourcecache.py'
--- src/maasserver/models/bootsourcecache.py	2016-06-22 17:03:02 +0000
+++ src/maasserver/models/bootsourcecache.py	2016-08-31 06:33:18 +0000
@@ -51,18 +51,26 @@
 
     boot_source = ForeignKey(BootSource, blank=False)
 
-    os = CharField(max_length=20, blank=False, null=False)
-
-    arch = CharField(max_length=20, blank=False, null=False)
-
-    subarch = CharField(max_length=20, blank=False, null=False)
-
-    release = CharField(max_length=20, blank=False, null=False)
-
-    label = CharField(max_length=20, blank=False, null=False)
+    os = CharField(max_length=32, blank=False, null=False)
+
+    arch = CharField(max_length=32, blank=False, null=False)
+
+    subarch = CharField(max_length=32, blank=False, null=False)
+
+    kflavor = CharField(max_length=32, blank=True, null=True)
+
+    release = CharField(max_length=32, blank=False, null=False)
+
+    label = CharField(max_length=32, blank=False, null=False)
 
     release_codename = CharField(max_length=255, blank=True, null=True)
 
     release_title = CharField(max_length=255, blank=True, null=True)
 
     support_eol = DateField(null=True, blank=True)
+
+    def __str__(self):
+        return (
+            "<BootSourceCache os=%s, release=%s, arch=%s, subarch=%s, "
+            "kflavor=%s>" % (
+                self.os, self.release, self.arch, self.subarch, self.kflavor))

=== modified file 'src/maasserver/models/tests/test_bootresource.py'
--- src/maasserver/models/tests/test_bootresource.py	2016-05-25 21:03:50 +0000
+++ src/maasserver/models/tests/test_bootresource.py	2016-08-31 06:33:18 +0000
@@ -594,15 +594,30 @@
             "arch": "armfh",
             "subarch": "hardbank",
             "kernels": [],
-        }))
+            }),
+        ("ubuntu/xenial", {
+            "name": "ubuntu/xenial",
+            "arch": "amd64",
+            "subarch": "generic",
+            "kernels": ["hwe-16.04", "hwe-16.04-lowlatency"],
+            }),
+        )
 
     def test__returns_usable_kernels(self):
         if self.subarch == "generic":
+            generic_kernels = []
             for i in self.kernels:
+                kernel_parts = i.split("-")
+                if len(kernel_parts) > 2:
+                    kflavor = kernel_parts[2]
+                else:
+                    kflavor = "generic"
+                    generic_kernels.append(i)
                 factory.make_usable_boot_resource(
                     name=self.name, rtype=BOOT_RESOURCE_TYPE.SYNCED,
-                    architecture="%s/%s" % (self.arch, i))
+                    architecture="%s/%s" % (self.arch, i), kflavor=kflavor)
         else:
+            generic_kernels = self.kernels
             factory.make_usable_boot_resource(
                 name=self.name, rtype=BOOT_RESOURCE_TYPE.SYNCED,
                 architecture="%s/%s" % (self.arch, self.subarch))
@@ -612,6 +627,12 @@
                 self.name, self.arch),
             "%s should return %s as its usable kernel" % (
                 self.name, self.kernels))
+        self.assertEqual(
+            generic_kernels,
+            BootResource.objects.get_usable_hwe_kernels(
+                self.name, self.arch, 'generic'),
+            "%s should return %s as its usable kernel" % (
+                self.name, generic_kernels))
 
 
 class TestGetKpackageForNode(MAASServerTestCase):

=== modified file 'src/maasserver/rpc/tests/test_regionservice_calls.py'
--- src/maasserver/rpc/tests/test_regionservice_calls.py	2016-08-19 18:11:11 +0000
+++ src/maasserver/rpc/tests/test_regionservice_calls.py	2016-08-31 06:33:18 +0000
@@ -47,7 +47,6 @@
 from maasserver.rpc.regionservice import Region
 from maasserver.rpc.services import update_services
 from maasserver.security import get_shared_secret
-from maasserver.testing.architecture import make_usable_architecture
 from maasserver.testing.eventloop import RegionEventLoopFixture
 from maasserver.testing.factory import factory
 from maasserver.testing.testcase import MAASTransactionServerTestCase
@@ -1104,7 +1103,7 @@
             self.create_node)
 
         params = {
-            'architecture': make_usable_architecture(self),
+            'architecture': factory.make_name('arch'),
             'power_type': factory.make_name('power_type'),
             'power_parameters': dumps({}),
             'mac_addresses': [factory.make_mac_address()],

=== modified file 'src/maasserver/static/js/angular/controllers/node_details.js'
--- src/maasserver/static/js/angular/controllers/node_details.js	2016-07-15 00:42:25 +0000
+++ src/maasserver/static/js/angular/controllers/node_details.js	2016-08-31 06:33:18 +0000
@@ -62,7 +62,7 @@
             },
             min_hwe_kernel: {
                 selected: null,
-                options: GeneralManager.getData("hwe_kernels")
+                options: GeneralManager.getData("min_hwe_kernels")
             },
             zone: {
                 selected: null,

=== modified file 'src/maasserver/static/js/angular/controllers/tests/test_node_details.js'
--- src/maasserver/static/js/angular/controllers/tests/test_node_details.js	2016-07-15 18:00:04 +0000
+++ src/maasserver/static/js/angular/controllers/tests/test_node_details.js	2016-08-31 06:33:18 +0000
@@ -212,7 +212,7 @@
             },
             min_hwe_kernel: {
                 selected: null,
-                options: GeneralManager.getData("hwe_kernels")
+                options: GeneralManager.getData("min_hwe_kernels")
             },
             zone: {
                 selected: null,
@@ -223,7 +223,7 @@
         expect($scope.summary.architecture.options).toBe(
             GeneralManager.getData("architectures"));
         expect($scope.summary.min_hwe_kernel.options).toBe(
-            GeneralManager.getData("hwe_kernels"));
+            GeneralManager.getData("min_hwe_kernels"));
         expect($scope.summary.zone.options).toBe(
             ZonesManager.getItems());
     });

=== modified file 'src/maasserver/static/js/angular/factories/general.js'
--- src/maasserver/static/js/angular/factories/general.js	2016-08-18 15:42:50 +0000
+++ src/maasserver/static/js/angular/factories/general.js	2016-08-31 06:33:18 +0000
@@ -84,6 +84,13 @@
                     polling: false,
                     nextPromise: null
                 },
+                min_hwe_kernels: {
+                    method: "general.min_hwe_kernels",
+                    data: [],
+                    loaded: false,
+                    polling: false,
+                    nextPromise: null
+                },
                 default_min_hwe_kernel: {
                     method: "general.default_min_hwe_kernel",
                     data: { text: '' },

=== modified file 'src/maasserver/static/js/angular/factories/tests/test_general.js'
--- src/maasserver/static/js/angular/factories/tests/test_general.js	2016-08-19 13:29:03 +0000
+++ src/maasserver/static/js/angular/factories/tests/test_general.js	2016-08-31 06:33:18 +0000
@@ -53,8 +53,9 @@
             ["machine_actions", "device_actions", "region_controller_actions",
             "rack_controller_actions", "region_and_rack_controller_actions",
             "architectures", "known_architectures", "pockets_to_disable",
-            "hwe_kernels", "default_min_hwe_kernel", "osinfo", "bond_options",
-            "version", "power_types", "release_options"]);
+            "hwe_kernels", "min_hwe_kernels", "default_min_hwe_kernel",
+            "osinfo", "bond_options", "version", "power_types",
+            "release_options"]);
     });
 
     it("_data.machine_actions has correct data", function() {
@@ -321,6 +322,7 @@
             GeneralManager._data.known_architectures.loaded = true;
             GeneralManager._data.pockets_to_disable.loaded = true;
             GeneralManager._data.hwe_kernels.loaded = true;
+            GeneralManager._data.min_hwe_kernels.loaded = true;
             GeneralManager._data.default_min_hwe_kernel.loaded = true;
             GeneralManager._data.osinfo.loaded = true;
             GeneralManager._data.bond_options.loaded = true;
@@ -584,7 +586,7 @@
             spyOn(GeneralManager, "_loadData").and.returnValue(
                 $q.defer().promise);
             GeneralManager.loadItems();
-            expect(GeneralManager._loadData.calls.count()).toBe(15);
+            expect(GeneralManager._loadData.calls.count()).toBe(16);
         });
 
         it("resolve defer once all resolve", function(done) {
@@ -603,6 +605,7 @@
                 $q.defer(),
                 $q.defer(),
                 $q.defer(),
+                $q.defer(),
                 $q.defer()
             ];
             var i = 0;

=== modified file 'src/maasserver/testing/architecture.py'
--- src/maasserver/testing/architecture.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/testing/architecture.py	2016-08-31 06:33:18 +0000
@@ -22,6 +22,7 @@
     """
     if arch_name is None:
         arch_name = factory.make_name('arch')
+    factory.make_default_ubuntu_release_bootable(arch_name)
     if with_subarch:
         if subarch_name is None:
             subarch_name = factory.make_name('sub')

=== modified file 'src/maasserver/testing/factory.py'
--- src/maasserver/testing/factory.py	2016-08-26 08:04:38 +0000
+++ src/maasserver/testing/factory.py	2016-08-31 06:33:18 +0000
@@ -22,6 +22,7 @@
 from distro_info import UbuntuDistroInfo
 from django.conf import settings
 from django.contrib.auth.models import User
+from django.db import transaction
 from django.test.client import RequestFactory
 from django.utils import timezone
 from maasserver.clusterrpc.power_parameters import get_power_types
@@ -56,6 +57,7 @@
     BootSourceCache,
     BootSourceSelection,
     CacheSet,
+    Config,
     Device,
     DHCPSnippet,
     DNSData,
@@ -112,6 +114,7 @@
 from maasserver.testing import get_data
 from maasserver.utils.converters import round_size_to_nearest_block
 from maasserver.utils.orm import reload_object
+from maasserver.utils.osystems import get_release_from_distro_info
 from maasserver.worker_user import get_worker_user
 import maastesting.factory
 from maastesting.factory import TooManyRandomRetries
@@ -1272,7 +1275,7 @@
     def make_BootSourceCache(self, boot_source=None, os=None, arch=None,
                              subarch=None, release=None, label=None,
                              release_codename=None, release_title=None,
-                             support_eol=None):
+                             support_eol=None, kflavor=None):
         """Create a new `BootSourceCache`."""
         if boot_source is None:
             boot_source = self.make_BootSource()
@@ -1290,7 +1293,7 @@
             boot_source=boot_source, os=os, arch=arch,
             subarch=subarch, release=release, label=label,
             release_codename=release_codename, release_title=release_title,
-            support_eol=support_eol)
+            support_eol=support_eol, kflavor=kflavor)
 
     def make_many_BootSourceCaches(self, number, **kwargs):
         caches = list()
@@ -1404,12 +1407,9 @@
                 self.make_name('key'): self.make_name('value')
                 for _ in range(3)
                 }
-        if kflavor is None:
-            extra['kflavor'] = 'generic'
-        else:
-            extra['kflavor'] = kflavor
         return BootResource.objects.create(
-            rtype=rtype, name=name, architecture=architecture, extra=extra)
+            rtype=rtype, name=name, architecture=architecture, kflavor=kflavor,
+            extra=extra)
 
     def make_BootResourceSet(self, resource, version=None, label=None):
         if version is None:
@@ -1467,6 +1467,26 @@
                 resource_set, filename=filetype, filetype=filetype)
         return resource
 
+    def make_default_ubuntu_release_bootable(self, arch=None):
+        if arch is None:
+            arch = self.make_name('arch')
+        default_osystem = Config.objects.get_config(
+            name='commissioning_osystem')
+        default_series = Config.objects.get_config(
+            name='commissioning_distro_series')
+        default_name = "%s/%s" % (default_osystem, default_series)
+        release = get_release_from_distro_info(default_series)
+        architecture = "%s/hwe-%s" % (arch, release['version'].split()[0])
+        try:
+            return BootResource.objects.get(
+                name=default_name, architecture=architecture,
+                rtype=BOOT_RESOURCE_TYPE.SYNCED)
+        except BootResource.DoesNotExist:
+            with transaction.atomic():
+                return self.make_usable_boot_resource(
+                    name=default_name, architecture=architecture,
+                    kflavor='generic', rtype=BOOT_RESOURCE_TYPE.SYNCED)
+
     def make_BlockDevice(
             self, node=None, name=None, id_path=None, size=None,
             block_size=None, tags=None):

=== modified file 'src/maasserver/testing/osystems.py'
--- src/maasserver/testing/osystems.py	2015-12-01 18:12:59 +0000
+++ src/maasserver/testing/osystems.py	2016-08-31 06:33:18 +0000
@@ -14,6 +14,7 @@
     make_rpc_osystem,
     make_rpc_release,
 )
+from maasserver.models import Node
 from maasserver.testing.factory import factory
 from maasserver.utils import osystems as osystems_module
 
@@ -34,6 +35,10 @@
         make_rpc_release(release)
         for release in releases
         ]
+    # If this is being used to test commissioning make sure the default
+    # Ubuntu release is bootable for every known architecture.
+    for node in Node.objects.all():
+        factory.make_default_ubuntu_release_bootable(node.split_arch()[0])
     return make_rpc_osystem(osystem_name, releases=rpc_releases)
 
 

=== modified file 'src/maasserver/tests/test_bootresources.py'
--- src/maasserver/tests/test_bootresources.py	2016-08-19 10:22:29 +0000
+++ src/maasserver/tests/test_bootresources.py	2016-08-31 06:33:18 +0000
@@ -582,11 +582,13 @@
             AssertConnectionWrapper.connection.connection)
 
 
-def make_product(ftype=None):
+def make_product(ftype=None, kflavor=None):
     """Make product dictionary that is just like the one provided
     from simplsetreams."""
     if ftype is None:
         ftype = factory.pick_choice(BOOT_RESOURCE_FILE_TYPE_CHOICES)
+    if kflavor is None:
+        kflavor = 'generic'
     subarch = factory.make_name('subarch')
     subarches = [factory.make_name('subarch') for _ in range(3)]
     subarches.insert(0, subarch)
@@ -597,7 +599,7 @@
         'arch': factory.make_name('arch'),
         'subarch': subarch,
         'release': factory.make_name('release'),
-        'kflavor': factory.make_name('kflavor'),
+        'kflavor': kflavor,
         'subarches': subarches,
         'version_name': factory.make_name('version'),
         'label': factory.make_name('label'),
@@ -607,7 +609,11 @@
         'path': '/path/to/%s' % name,
         }
     name = '%s/%s' % (product['os'], product['release'])
-    architecture = '%s/%s' % (product['arch'], product['subarch'])
+    if kflavor == 'generic':
+        subarch = product['subarch']
+    else:
+        subarch = "%s-%s" % (product['subarch'], kflavor)
+    architecture = '%s/%s' % (product['arch'], subarch)
     return name, architecture, product
 
 
@@ -697,7 +703,7 @@
         self.assertEqual(BOOT_RESOURCE_TYPE.SYNCED, resource.rtype)
         self.assertEqual(name, resource.name)
         self.assertEqual(architecture, resource.architecture)
-        self.assertEqual(product['kflavor'], resource.extra['kflavor'])
+        self.assertEqual(product['kflavor'], resource.kflavor)
         self.assertEqual(product['subarches'], resource.extra['subarches'])
 
     def test_get_or_create_boot_resource_gets_resource(self):
@@ -708,7 +714,7 @@
         store = BootResourceStore()
         resource = store.get_or_create_boot_resource(product)
         self.assertEqual(expected, resource)
-        self.assertEqual(product['kflavor'], resource.extra['kflavor'])
+        self.assertEqual(product['kflavor'], resource.kflavor)
         self.assertEqual(product['subarches'], resource.extra['subarches'])
 
     def test_get_or_create_boot_resource_calls_prevent_resource_deletion(self):
@@ -736,6 +742,21 @@
         self.assertThat(
             mock_prevent, MockNotCalled())
 
+    def test_get_or_create_boot_resource_adds_kflavor_to_subarch(self):
+        kflavor = factory.make_name('kflavor')
+        _, architecture, product = make_product(kflavor=kflavor)
+        store = BootResourceStore()
+        resource = store.get_or_create_boot_resource(product)
+        self.assertEqual(architecture, reload_object(resource).architecture)
+
+    def test_get_or_create_boot_resources_add_no_kflavor_for_generic(self):
+        _, architecture, product = make_product(kflavor='generic')
+        store = BootResourceStore()
+        resource = store.get_or_create_boot_resource(product)
+        resource = reload_object(resource)
+        self.assertEqual(architecture, resource.architecture)
+        self.assertNotIn('generic', resource.architecture)
+
     def test_get_or_create_boot_resource_set_creates_resource_set(self):
         self.useFixture(SignalsDisabled("largefiles"))
         name, architecture, product = make_product()
@@ -1019,7 +1040,7 @@
         with transaction.atomic():
             resource = factory.make_BootResource(
                 rtype=BOOT_RESOURCE_TYPE.SYNCED, name=name,
-                architecture=architecture)
+                architecture=architecture, kflavor='generic')
             release_name = resource.name.split('/')[1]
             resource_set = factory.make_BootResourceSet(
                 resource, version=product['version_name'])

=== modified file 'src/maasserver/utils/osystems.py'
--- src/maasserver/utils/osystems.py	2016-06-29 09:52:24 +0000
+++ src/maasserver/utils/osystems.py	2016-08-31 06:33:18 +0000
@@ -5,15 +5,17 @@
 __all__ = [
     'get_distro_series_initial',
     'get_release_requires_key',
+    'get_release_version_from_string',
     'list_all_releases_requiring_keys',
+    'list_all_usable_hwe_kernels',
     'list_all_usable_osystems',
     'list_all_usable_releases',
-    'list_all_usable_hwe_kernels',
+    'list_commissioning_choices',
     'list_hwe_kernel_choices',
     'list_osystem_choices',
     'list_release_choices',
-    'list_commissioning_choices',
     'make_hwe_kernel_ui_text',
+    'release_a_newer_than_b',
     'validate_hwe_kernel',
     ]
 
@@ -21,6 +23,7 @@
 
 from distro_info import UbuntuDistroInfo
 from django.core.exceptions import ValidationError
+from django.db.models import Q
 from maasserver.clusterrpc.osystems import gen_all_known_operating_systems
 from maasserver.models import (
     BootResource,
@@ -85,18 +88,20 @@
 def make_hwe_kernel_ui_text(hwe_kernel):
     if not hwe_kernel:
         return hwe_kernel
-    release_letter = hwe_kernel.replace('hwe-', '')
-    boot_sources = BootSourceCache.objects.filter(
-        release__startswith=release_letter,
-        subarch=hwe_kernel)
-    if len(boot_sources) > 0:
-        return "%s (%s)" % (boot_sources[0].release, hwe_kernel)
-    else:
-        ubuntu = UbuntuDistroInfo()
-        for release in ubuntu.all:
-            if release.startswith(release_letter):
-                return "%s (%s)" % (release, hwe_kernel)
-    return hwe_kernel
+    # Fall back on getting it from DistroInfo.
+    kernel_list = hwe_kernel.split('-')
+    if len(kernel_list) >= 2:
+        kernel = kernel_list[1]
+    else:
+        kernel = hwe_kernel
+    # Try to get the release name from the SimpleStream
+    ubuntu_release = get_release_from_db(kernel)
+    if ubuntu_release is None:
+        ubuntu_release = get_release_from_distro_info(kernel)
+    if ubuntu_release is None:
+        return hwe_kernel
+    else:
+        return "%s (%s)" % (ubuntu_release['series'], hwe_kernel)
 
 
 def list_hwe_kernel_choices(hwe_kernels):
@@ -248,32 +253,138 @@
     return osystem, release
 
 
+def get_release_from_distro_info(string):
+    """Convert an Ubuntu release or version into a release dict.
+
+    This data is pulled from the UbuntuDistroInfo library which contains
+    additional information such as the release, EOL, and code name."""
+    ubuntu = UbuntuDistroInfo()
+    release_found = False
+    # We can only look at release names for 12.04+ as previous versions
+    # have overlapping first letters(e.g Warty and Wily) which break looking
+    # up old style kernels(e.g hwe-w).
+    for row in ubuntu._rows:
+        if (
+                int(row['version'].split('.')[0]) >= 12 and
+                row['series'].startswith(string) or
+                row['version'].startswith(string)):
+            release_found = True
+            break
+    if release_found:
+        return row
+    else:
+        return None
+
+
+def get_release_from_db(string):
+    """Convert an Ubuntu release, version, or subarch into a release dict.
+
+    This does not contain the release, eol, or created dates like
+    get_release_from_distro_info does."""
+    bsc = BootSourceCache.objects.filter(
+        (
+            Q(subarch="hwe-%s" % string) &
+            Q(release_title__startswith=string) | Q(release__startswith=string)
+        ) |
+        Q(release__startswith=string) |
+        Q(release_title__startswith=string)).first()
+    if bsc is None:
+        return None
+    elif None in (bsc.release_title, bsc.release, bsc.release_codename):
+        return None
+    else:
+        return {
+            'version': bsc.release_title,
+            'eol-server': bsc.support_eol,
+            'series': bsc.release,
+            'codename': bsc.release_codename,
+        }
+
+
+def get_release_version_from_string(string):
+    """Convert an Ubuntu release, version, or kernel into a version tuple.
+
+    Takes a string input represneting an Ubuntu release, version, or hwe_kernel
+    and returns a version tuple. The return value is a three integer tuple
+    representing an Ubuntu major, minor values(e.g 16, 4 for Xenial) and a
+    weight. The weight is used to give edge kernels a higher value when
+    compared to a non-edge kernel. Rolling kernels and releases are given a
+    very high value (999, 999) to always be the higher value during comparison.
+
+    Input: xenial, 16.04, hwe-16.04, hwe-16.04-lowlatency
+    Output: (16, 4, 0)
+
+    Input: hwe-16.04-edge
+    Output: (16, 4, 1)
+
+    Input: rolling, hwe-rolling, hwe-rolling-lowlatency
+    Output: (999, 999, 0)
+
+    Input: hwe-rolling-edge, hwe-rolling-lowlatency-edge
+    Output: (999, 999, 1)
+    """
+    parts = string.split('-')
+    parts_len = len(parts)
+    if parts_len == 1:
+        # Just the release name, e.g xenial or 16.04
+        release = string
+        edge = False
+    elif parts_len == 2:
+        # hwe kernel, e.g hwe-x or hwe-16.04
+        release = parts[1]
+        edge = False
+    elif parts_len == 3:
+        # hwe edge or lowlatency kernel,
+        # e.g hwe-16.04-edge or hwe-16.04-lowlatency
+        release = parts[1]
+        if parts[2] == 'edge':
+            edge = True
+        else:
+            edge = False
+    elif parts_len == 4:
+        # hwe edge lowlatency kernel, e.g hwe-16.04-lowlatency-edge
+        release = parts[1]
+        if parts[3] == 'edge':
+            edge = True
+        else:
+            edge = False
+    else:
+        raise ValueError("Unknown release or kernel %s!" % string)
+
+    if release == 'rolling':
+        # Rolling kernels are always the latest
+        version = [999, 999]
+    else:
+        # First try to get release info from the SimpleStream
+        ubuntu_release = get_release_from_db(release)
+        if ubuntu_release is None:
+            # Fall back on using the UbuntuDistroInfo library
+            ubuntu_release = get_release_from_distro_info(release)
+        if ubuntu_release is None:
+            raise ValueError(
+                "%s not found amoungst the known Ubuntu releases!" % string)
+        # Remove 'LTS' from version if it exists
+        version = ubuntu_release['version'].split(' ')[0]
+        # Convert the version into a list of ints
+        version = [int(seg) for seg in version.split('.')]
+    if edge:
+        # Ensure edge kernels are viewed as newer than non-edge
+        version += [1]
+    else:
+        version += [0]
+    return tuple(version)
+
+
 def release_a_newer_than_b(a, b):
     """Compare two Ubuntu releases and return true if a >= b.
 
-    The release names can be the full release name(e.g Precise, Trusty), or
-    a hardware enablement(e.g hwe-p, hwe-t). The function wraps around the
-    letter 'p' as Precise was the first version of Ubuntu MAAS supported
+    The release names can be the full release name(e.g Precise, Trusty),
+    release versions(e.g 12.04, 16.04), or an hwe kernel(e.g hwe-p, hwe-16.04,
+    hwe-rolling-lowlatency-edge).
     """
-    def get_release_num(release):
-        release = release.lower()
-        if 'hwe-' in release:
-            release = release.replace('hwe-', '')
-        return ord(release[0])
-
-    # Compare release versions based off of the first letter of their
-    # release name or the letter in hwe-<letter>. Wrap around the letter
-    # 'p' as that is the first version of Ubuntu MAAS supported.
-    num_a = get_release_num(a)
-    num_b = get_release_num(b)
-    num_wrap = ord('p')
-
-    if((num_a >= num_wrap and num_b >= num_wrap and num_a >= num_b) or
-       (num_a < num_wrap and num_b >= num_wrap and num_a < num_b) or
-       (num_a < num_wrap and num_b < num_wrap and num_a >= num_b)):
-        return True
-    else:
-        return False
+    ver_a = get_release_version_from_string(a)
+    ver_b = get_release_version_from_string(b)
+    return ver_a >= ver_b
 
 
 def validate_hwe_kernel(
@@ -306,10 +417,10 @@
             subarch)
 
     os_release = osystem + '/' + distro_series
-    usable_kernels = BootResource.objects.get_usable_hwe_kernels(
-        os_release, arch)
 
     if hwe_kernel and hwe_kernel.startswith('hwe-'):
+        usable_kernels = BootResource.objects.get_usable_hwe_kernels(
+            os_release, arch)
         if hwe_kernel not in usable_kernels:
             raise ValidationError(
                 '%s is not available for %s on %s.' %
@@ -324,6 +435,13 @@
                 (hwe_kernel, min_hwe_kernel))
         return hwe_kernel
     elif(min_hwe_kernel and min_hwe_kernel.startswith('hwe-')):
+        kernel_parts = min_hwe_kernel.split('-')
+        if len(kernel_parts) == 2:
+            kflavor = 'generic'
+        else:
+            kflavor = kernel_parts[2]
+        usable_kernels = BootResource.objects.get_usable_hwe_kernels(
+            os_release, arch, kflavor)
         for i in usable_kernels:
             if(release_a_newer_than_b(i, min_hwe_kernel) and
                release_a_newer_than_b(i, distro_series)):
@@ -331,7 +449,11 @@
         raise ValidationError(
             '%s has no kernels availible which meet min_hwe_kernel(%s).' %
             (distro_series, min_hwe_kernel))
-    return 'hwe-' + distro_series[0]
+    for kernel in BootResource.objects.get_usable_hwe_kernels(
+            os_release, arch, 'generic'):
+        if release_a_newer_than_b(kernel, distro_series):
+            return kernel
+    raise ValidationError('%s has no kernels available.' % distro_series)
 
 
 def validate_min_hwe_kernel(min_hwe_kernel):
@@ -341,4 +463,5 @@
     usable_kernels = BootResource.objects.get_usable_hwe_kernels()
     if min_hwe_kernel not in usable_kernels:
         raise ValidationError('%s is not a usable kernel.' % min_hwe_kernel)
-    return min_hwe_kernel
+    else:
+        return min_hwe_kernel

=== modified file 'src/maasserver/utils/tests/test_osystems.py'
--- src/maasserver/utils/tests/test_osystems.py	2016-06-29 09:52:24 +0000
+++ src/maasserver/utils/tests/test_osystems.py	2016-08-31 06:33:18 +0000
@@ -25,7 +25,10 @@
 from maasserver.utils import osystems as osystems_module
 from maasserver.utils.osystems import (
     get_distro_series_initial,
+    get_release_from_db,
+    get_release_from_distro_info,
     get_release_requires_key,
+    get_release_version_from_string,
     list_all_releases_requiring_keys,
     list_all_usable_osystems,
     list_all_usable_releases,
@@ -35,6 +38,7 @@
     make_hwe_kernel_ui_text,
     release_a_newer_than_b,
     validate_hwe_kernel,
+    validate_min_hwe_kernel,
     validate_osystem_and_distro_series,
 )
 from maastesting.matchers import MockAnyCall
@@ -290,13 +294,10 @@
         self.assertEqual('trusty (hwe-t)', make_hwe_kernel_ui_text('hwe-t'))
 
     def test_make_hwe_kernel_ui_returns_kernel_when_none_found(self):
-        # Since this is testing that our fall final fall back returns just the
-        # kernel name when the release isn't found in BootSourceCache or
-        # UbuntuDistroInfo we patch out UbuntuDistroInfo so nothing is found.
-        self.patch(UbuntuDistroInfo, 'all').value = []
+        unknown_kernel = factory.make_name('kernel')
         self.assertEqual(
-            'hwe-m',
-            make_hwe_kernel_ui_text('hwe-m'))
+            unknown_kernel,
+            make_hwe_kernel_ui_text(unknown_kernel))
 
 
 class TestValidateOsystemAndDistroSeries(MAASServerTestCase):
@@ -339,20 +340,102 @@
             validate_osystem_and_distro_series(osystem['name'], release + '*'))
 
 
+class TestGetReleaseVersionFromString(MAASServerTestCase):
+
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        ubuntu = UbuntuDistroInfo()
+        # We can't test with releases older than Precise as they have duplicate
+        # names(e.g Wily and Warty) which will break the old style kernel
+        # tests.
+        valid_releases = [
+            row for row in ubuntu._rows
+            if int(row['version'].split('.')[0]) >= 12
+        ]
+        release = random.choice(valid_releases)
+        # Remove 'LTS' from version if it exists
+        version_str = release['version'].split(' ')[0]
+        # Convert the version into a list of ints
+        version_tuple = tuple([int(seg) for seg in version_str.split('.')])
+
+        self.scenarios = (
+            ("Release name", {
+                "string": release['series'],
+                "expected": version_tuple + tuple([0]),
+            }),
+            ("Release version", {
+                "string": version_str,
+                "expected": version_tuple + tuple([0]),
+            }),
+            ("Old style kernel", {
+                "string": "hwe-%s" % release['series'][0],
+                "expected": version_tuple + tuple([0]),
+            }),
+            ("New style kernel", {
+                "string": "hwe-%s" % version_str,
+                "expected": version_tuple + tuple([0]),
+            }),
+            ("New style edge kernel", {
+                "string": "hwe-%s-edge" % version_str,
+                "expected": version_tuple + tuple([1]),
+            }),
+            ("New style low latency kernel", {
+                "string": "hwe-%s-lowlatency" % version_str,
+                "expected": version_tuple + tuple([0]),
+            }),
+            ("New style edge low latency kernel", {
+                "string": "hwe-%s-lowlatency-edge" % version_str,
+                "expected": version_tuple + tuple([1]),
+            }),
+            ("Rolling kernel", {
+                "string": "hwe-rolling",
+                "expected": tuple([999, 999, 0]),
+            }),
+            ("Rolling edge kernel", {
+                "string": "hwe-rolling-edge",
+                "expected": tuple([999, 999, 1]),
+            }),
+            ("Rolling lowlatency kernel", {
+                "string": "hwe-rolling-lowlatency",
+                "expected": tuple([999, 999, 0]),
+            }),
+            ("Rolling lowlatency edge kernel", {
+                "string": "hwe-rolling-lowlatency-edge",
+                "expected": tuple([999, 999, 1]),
+            }),
+        )
+
+    def test_get_release_version_from_string(self):
+        self.assertEquals(
+            self.expected,
+            get_release_version_from_string(self.string))
+
+
 class TestReleaseANewerThanB(MAASServerTestCase):
 
-    def test_release_a_newer_than_b(self):
-        # Since we wrap around 'p' we want to use 'p' as our starting point
-        alphabet = ([chr(i) for i in range(ord('p'), ord('z') + 1)] +
-                    [chr(i) for i in range(ord('a'), ord('p'))])
-        previous_true = 0
-        for i in alphabet:
-            true_count = 0
-            for j in alphabet:
-                if release_a_newer_than_b('hwe-' + i, j):
-                    true_count += 1
-            previous_true += 1
-            self.assertEqual(previous_true, true_count)
+    def test_a_newer_than_b_true(self):
+        self.assertTrue(
+            release_a_newer_than_b(
+                'hwe-rolling',
+                factory.make_kernel_string(can_be_release_or_version=True)))
+
+    def test_a_equal_to_b_true(self):
+        string = factory.make_kernel_string(can_be_release_or_version=True)
+        self.assertTrue(release_a_newer_than_b(string, string))
+
+    def test_a_less_than_b_false(self):
+        self.assertFalse(
+            release_a_newer_than_b(
+                factory.make_kernel_string(can_be_release_or_version=True),
+                'hwe-rolling'))
+
+    def test_accounts_for_edge(self):
+        self.assertFalse(
+            release_a_newer_than_b('hwe-rolling', 'hwe-rolling-edge'))
+
+    def test_kernel_flavor_doesnt_make_difference(self):
+        self.assertTrue(release_a_newer_than_b(
+            'hwe-rolling', 'hwe-rolling-lowlatency'))
 
 
 class TestValidateHweKernel(MAASServerTestCase):
@@ -474,3 +557,108 @@
         self.assertThat(
             mock_get_config, MockAnyCall('commissioning_distro_series'))
         self.assertEqual('hwe-v', kernel)
+
+
+class TestValidateMinHweKernel(MAASServerTestCase):
+
+    def test_validates_kernel(self):
+        kernel = factory.make_kernel_string(generic_only=True)
+        self.patch(
+            BootResource.objects,
+            'get_usable_hwe_kernels').return_value = (kernel,)
+        self.assertEquals(kernel, validate_min_hwe_kernel(kernel))
+
+    def test_returns_empty_string_when_none(self):
+        self.assertEquals("", validate_min_hwe_kernel(None))
+
+    def test_raises_exception_when_not_found(self):
+        self.assertRaises(
+            ValidationError,
+            validate_min_hwe_kernel, factory.make_kernel_string())
+
+    def test_raises_exception_when_lowlatency(self):
+        self.assertRaises(
+            ValidationError, validate_min_hwe_kernel, 'hwe-16.04-lowlatency')
+
+
+class TestGetReleaseFromDistroInfo(MAASServerTestCase):
+
+    def pick_release(self):
+        ubuntu = UbuntuDistroInfo()
+        supported_releases = [
+            release for release in ubuntu._rows
+            if int(release['version'].split('.')[0]) >= 12
+        ]
+        return random.choice(supported_releases)
+
+    def test_finds_by_series(self):
+        release = self.pick_release()
+        self.assertItemsEqual(
+            release, get_release_from_distro_info(release['series']))
+
+    def test_finds_by_series_first_letter(self):
+        release = self.pick_release()
+        self.assertItemsEqual(
+            release, get_release_from_distro_info(release['series'][0]))
+
+    def test_finds_by_version(self):
+        release = self.pick_release()
+        self.assertItemsEqual(
+            release, get_release_from_distro_info(release['version']))
+
+    def test_returns_none_when_not_found(self):
+        self.assertIsNone(
+            get_release_from_distro_info(factory.make_name('string')))
+
+
+class TestGetReleaseFromDB(MAASServerTestCase):
+
+    def make_boot_source_cache(self):
+        # Disable boot sources signals otherwise the test fails due to unrun
+        # post-commit tasks at the end of the test.
+        self.useFixture(SignalsDisabled("bootsources"))
+        ubuntu = UbuntuDistroInfo()
+        supported_releases = [
+            release for release in ubuntu._rows
+            if int(release['version'].split('.')[0]) >= 12
+        ]
+        release = random.choice(supported_releases)
+        subarch = "hwe-%s" % release['version'].split(' ')[0]
+        factory.make_BootSourceCache(
+            os='ubuntu',
+            arch=factory.make_name('arch'),
+            subarch=subarch,
+            release=release['series'],
+            release_codename=release['codename'],
+            release_title=release['version'],
+            support_eol=release['eol-server'],
+        )
+        return release
+
+    def test_finds_by_subarch(self):
+        release = self.make_boot_source_cache()
+        self.assertEquals(
+            release['series'],
+            get_release_from_db(release['version'].split(' ')[0])['series'])
+
+    def test_finds_by_release(self):
+        release = self.make_boot_source_cache()
+        self.assertEquals(
+            release['version'],
+            get_release_from_db(release['series'])['version'])
+
+    def test_finds_by_release_first_letter(self):
+        release = self.make_boot_source_cache()
+        self.assertEquals(
+            release['version'],
+            get_release_from_db(release['series'][0])['version'])
+
+    def test_finds_by_version(self):
+        release = self.make_boot_source_cache()
+        self.assertItemsEqual(
+            release['series'],
+            get_release_from_db(release['version'])['series'])
+
+    def test_returns_none_when_not_found(self):
+        self.assertIsNone(
+            get_release_from_db(factory.make_name('string')))

=== modified file 'src/maasserver/websockets/handlers/general.py'
--- src/maasserver/websockets/handlers/general.py	2016-08-18 15:42:50 +0000
+++ src/maasserver/websockets/handlers/general.py	2016-08-31 06:33:18 +0000
@@ -46,6 +46,7 @@
             'known_architectures',
             'pockets_to_disable',
             'hwe_kernels',
+            'min_hwe_kernels',
             'default_min_hwe_kernel',
             'osinfo',
             'machine_actions',
@@ -77,6 +78,16 @@
         return list_hwe_kernel_choices(
             BootResource.objects.get_usable_hwe_kernels())
 
+    def min_hwe_kernels(self, params):
+        """Return all supported min_hwe_kernels.
+
+        This filters out all non-generic kernel flavors. The user can select
+        the flavor during deployment.
+        """
+        return list_hwe_kernel_choices(
+            BootResource.objects.get_usable_hwe_kernels()
+        )
+
     def default_min_hwe_kernel(self, params):
         """Return the default_min_hwe_kernel."""
         return Config.objects.get_config('default_min_hwe_kernel')

=== modified file 'src/maasserver/websockets/handlers/tests/test_general.py'
--- src/maasserver/websockets/handlers/tests/test_general.py	2016-08-18 15:42:50 +0000
+++ src/maasserver/websockets/handlers/tests/test_general.py	2016-08-31 06:33:18 +0000
@@ -29,6 +29,12 @@
 
 class TestGeneralHandler(MAASServerTestCase):
 
+    def setUp(self):
+        super().setUp()
+        # Disable boot sources signals otherwise the test fails due to unrun
+        # post-commit tasks at the end of the test.
+        self.useFixture(SignalsDisabled("bootsources"))
+
     def dehydrate_actions(self, actions, node_type=None):
         return [
             {
@@ -40,6 +46,41 @@
             if node_type is None or node_type in action.for_type
             ]
 
+    def make_boot_sources(self):
+        kernels = []
+        ubuntu = UbuntuDistroInfo()
+        for row in ubuntu._rows:
+            release_year = int(row['version'].split('.')[0])
+            if release_year < 12:
+                continue
+            elif release_year < 16:
+                style = row['series'][0]
+            else:
+                style = row['version']
+            for kflavor in [
+                    'generic', 'lowlatency', 'edge', 'lowlatency-edge']:
+                if kflavor == 'generic':
+                    kernel = "hwe-%s" % style
+                else:
+                    kernel = "hwe-%s-%s" % (style, kflavor)
+                arch = factory.make_name('arch')
+                architecture = "%s/%s" % (arch, kernel)
+                release = row['series'].split(' ')[0]
+                factory.make_usable_boot_resource(
+                    name="ubuntu/" + release,
+                    kflavor=kflavor,
+                    extra={'subarches': kernel},
+                    architecture=architecture,
+                    rtype=BOOT_RESOURCE_TYPE.SYNCED)
+                factory.make_BootSourceCache(
+                    os="ubuntu",
+                    arch=arch,
+                    subarch=kernel,
+                    release=release)
+                kernels.append(
+                    (kernel, '%s (%s)' % (release, kernel)))
+        return kernels
+
     def test_architectures(self):
         arches = [
             "%s/%s" % (factory.make_name("arch"), factory.make_name("subarch"))
@@ -63,41 +104,19 @@
             handler.pockets_to_disable({}))
 
     def test_hwe_kernels(self):
-        ubuntu_releases = UbuntuDistroInfo()
-        expected_output = []
-        # Disable boot sources signals otherwise the test fails due to unrun
-        # post-commit tasks at the end of the test.
-        self.useFixture(SignalsDisabled("bootsources"))
-        # Start with the first release MAAS supported. We do this
-        # because the lookup between hwe- kernel and release can fail
-        # when multiple releases start with the same letter. For
-        # example both warty(4.10) and wily(15.10) will have an hwe-w
-        # kernel. Because of this the mapping between kernel and
-        # release will pick the release which was downloaded
-        # first. Since precise no release has used the same first
-        # letter so we do not have this problem with supported
-        # releases.
-        for release in ubuntu_releases.all[
-                ubuntu_releases.all.index('precise'):]:
-            kernel = 'hwe-' + release[0]
-            arch = factory.make_name('arch')
-            architecture = "%s/%s" % (arch, kernel)
-            factory.make_usable_boot_resource(
-                name="ubuntu/" + release,
-                extra={'subarches': kernel},
-                architecture=architecture,
-                rtype=BOOT_RESOURCE_TYPE.SYNCED)
-            factory.make_BootSourceCache(
-                os="ubuntu",
-                arch=arch,
-                subarch=kernel,
-                release=release)
-            expected_output.append((kernel, '%s (%s)' % (release, kernel)))
+        expected_output = self.make_boot_sources()
         handler = GeneralHandler(factory.make_User(), {})
         self.assertItemsEqual(
             sorted(expected_output, key=lambda choice: choice[0]),
             sorted(handler.hwe_kernels({}), key=lambda choice: choice[0]))
 
+    def test_hwe_min_kernels(self):
+        expected_output = self.make_boot_sources()
+        handler = GeneralHandler(factory.make_User(), {})
+        self.assertItemsEqual(
+            sorted(expected_output, key=lambda choice: choice[0]),
+            sorted(handler.min_hwe_kernels({}), key=lambda choice: choice[0]))
+
     def test_osinfo(self):
         handler = GeneralHandler(factory.make_User(), {})
         osystem = make_osystem_with_releases(self)

=== modified file 'src/maastesting/factory.py'
--- src/maastesting/factory.py	2016-08-18 11:02:26 +0000
+++ src/maastesting/factory.py	2016-08-31 06:33:18 +0000
@@ -32,6 +32,7 @@
 import urllib.request
 from uuid import uuid1
 
+from distro_info import UbuntuDistroInfo
 from maastesting.fixtures import TempDirectory
 from netaddr import (
     IPAddress,
@@ -619,5 +620,30 @@
             cmd=[self.make_name("command")],
             output=factory.make_bytes())
 
+    def make_kernel_string(
+            self, can_be_release_or_version=False, generic_only=False):
+        ubuntu = UbuntuDistroInfo()
+        # Only select from MAAS supported releases so we don't have to deal
+        # with versions name overlap(e.g Warty and Wily).
+        supported_releases = [
+            release for release in ubuntu._rows
+            if int(release['version'].split('.')[0]) >= 12
+        ]
+        release = random.choice(supported_releases)
+        # Remove 'LTS' from version if it exists
+        version_str = release['version'].split(' ')[0]
+        strings = [
+            "hwe-%s" % release['series'][0], "hwe-%s" % version_str,
+            "hwe-%s-edge" % version_str,
+        ]
+        if not generic_only:
+            strings += [
+                "hwe-%s-lowlatency" % version_str,
+                "hwe-%s-lowlatency-edge" % version_str,
+            ]
+        if can_be_release_or_version:
+            strings += [release['series'], version_str]
+        return random.choice(strings)
+
 # Create factory singleton.
 factory = Factory()

=== modified file 'src/provisioningserver/import_images/download_descriptions.py'
--- src/provisioningserver/import_images/download_descriptions.py	2016-06-16 20:51:55 +0000
+++ src/provisioningserver/import_images/download_descriptions.py	2016-08-31 06:33:18 +0000
@@ -36,7 +36,7 @@
     """Return a subset of dict `item` for storing in a boot images dict."""
     keys_to_keep = [
         'content_id', 'product_name', 'version_name', 'path', 'subarches',
-        'release_codename', 'release_title', 'support_eol']
+        'release_codename', 'release_title', 'support_eol', 'kflavor']
     compact_item = {
         key: item[key]
         for key in keys_to_keep
@@ -94,12 +94,14 @@
         self.boot_images_dict.set(
             base_image._replace(subarch=subarch), compact_item)
 
-        # HWE resources with generic, should map to the HWE that ships with
-        # that release.
-        hwe_arch = 'hwe-%s' % release[0]
-        if subarch == hwe_arch and 'generic' in subarches:
-            self.boot_images_dict.set(
-                base_image._replace(subarch='generic'), compact_item)
+        if os == 'ubuntu' and item.get('version') is not None:
+            # HWE resources with generic, should map to the HWE that ships with
+            # that release. Starting with Xenial kernels changed from using the
+            # naming format hwe-<letter> to hwe-<release>. Look for both.
+            hwe_archs = ["hwe-%s" % item['version'], "hwe-%s" % release[0]]
+            if subarch in hwe_archs and 'generic' in subarches:
+                self.boot_images_dict.set(
+                    base_image._replace(subarch='generic'), compact_item)
 
     def sync(self, reader, path):
         try:

=== modified file 'src/provisioningserver/import_images/tests/test_boot_resources.py'
--- src/provisioningserver/import_images/tests/test_boot_resources.py	2016-07-06 18:30:19 +0000
+++ src/provisioningserver/import_images/tests/test_boot_resources.py	2016-08-31 06:33:18 +0000
@@ -476,6 +476,7 @@
         self.assertItemsEqual(
             [
                 'content_id',
+                'kflavor',
                 'path',
                 'product_name',
                 'version_name',

=== modified file 'src/provisioningserver/import_images/tests/test_download_descriptions.py'
--- src/provisioningserver/import_images/tests/test_download_descriptions.py	2016-07-06 18:43:07 +0000
+++ src/provisioningserver/import_images/tests/test_download_descriptions.py	2016-08-31 06:33:18 +0000
@@ -234,12 +234,14 @@
 class TestRepoDumper(MAASTestCase):
     """Tests for `RepoDumper`."""
 
-    def make_item(self, os=None, release=None, arch=None,
+    def make_item(self, os=None, release=None, version=None, arch=None,
                   subarch=None, subarches=None, label=None):
         if os is None:
             os = factory.make_name('os')
         if release is None:
             release = factory.make_name('release')
+        if version is None:
+            version = factory.make_name('version')
         if arch is None:
             arch = factory.make_name('arch')
         if subarch is None:
@@ -257,6 +259,7 @@
             'path': factory.make_name('path'),
             'os': os,
             'release': release,
+            'version': version,
             'arch': arch,
             'subarch': subarch,
             'subarches': ','.join(subarches),
@@ -307,7 +310,7 @@
             label=item['label'])
         self.assertEqual(compat_item, boot_images_dict.mapping[image_spec])
 
-    def test_insert_item_sets_generic_to_release_item_for_hwe(self):
+    def test_insert_item_sets_generic_to_release_item_for_hwe_letter(self):
         boot_images_dict = BootImageMapping()
         dumper = RepoDumper(boot_images_dict)
         os = 'ubuntu'
@@ -338,6 +341,37 @@
             label=label)
         self.assertEqual(compat_item, boot_images_dict.mapping[image_spec])
 
+    def test_insert_item_sets_generic_to_release_item_for_hwe_version(self):
+        boot_images_dict = BootImageMapping()
+        dumper = RepoDumper(boot_images_dict)
+        os = 'ubuntu'
+        release = 'xenial'
+        arch = 'amd64'
+        label = 'release'
+        hwep_subarch = 'hwe-16.04'
+        hwep_subarches = ['generic', 'hwe-16.04', 'hwe-16.10']
+        hwes_subarch = 'hwe-16.10'
+        hwes_subarches = ['generic', 'hwe-16.04', 'hwe-16.10']
+        hwep_item, compat_item = self.make_item(
+            os=os, release=release,
+            arch=arch, subarch=hwep_subarch,
+            subarches=hwep_subarches, label=label)
+        hwes_item, _ = self.make_item(
+            os=os, release=release,
+            arch=arch, subarch=hwes_subarch,
+            subarches=hwes_subarches, label=label)
+        self.patch(
+            download_descriptions,
+            'products_exdata').side_effect = [hwep_item, hwes_item]
+        for _ in range(2):
+            dumper.insert_item(
+                sentinel.data, sentinel.src, sentinel.target,
+                sentinel.pedigree, sentinel.contentsource)
+        image_spec = make_image_spec(
+            os=os, release=release, arch=arch, subarch='generic',
+            label=label)
+        self.assertEqual(compat_item, boot_images_dict.mapping[image_spec])
+
     def test_sync_does_not_propagate_ioerror(self):
         mock_sync = self.patch(download_descriptions.BasicMirrorWriter, "sync")
         mock_sync.side_effect = IOError()

=== modified file 'utilities/check-imports'
--- utilities/check-imports	2016-08-15 16:41:18 +0000
+++ utilities/check-imports	2016-08-31 06:33:18 +0000
@@ -147,6 +147,7 @@
 StandardLibraries = Pattern(
     map("{0}|{0}.**".format, python_standard_libs))
 TestingLibraries = Pattern(
+    "distro_info.UbuntuDistroInfo",
     "django_nose|django_nose.**",
     "dns|dns.**",
     "fixtures|fixtures.**",