Merge lp:~jtv/maas/multi-arch-imports into lp:~maas-maintainers/maas/new-import-script-integration

[Jeroen T. Vermeulen]

Support multiple architectures in one simplestreams source.

This can be used for good or for evil. Good: specify a single source for completely identical, run-of-the-mill downloads — think “amd64 and oh, all the same stuff for i386 as well please.” Evil: “I want i386/generic plus amd64/hwe-t, but let's just specify them as one source and watch my company's bandwidth crumble under the load of 4 instead of 2 sets of images.”

There's a comment to warn people of the potential for evil. If you specify something like "i386 and armhf, generic and highbank" you probably get just what you'd expect: i386/generic, armhf/generic, armhf/highbank (with the third item actually being just an alias for the second, so bad example). But I'm not too interested in the minutiae right now; when in doubt, specify multiple sources. I think this even comes out clearer than before.

Jeroen

[Raphaël Badin (rvb)]

Looks good!

[0]

26 + #
27 + # Be careful: these constraints tend to multiply. For example if you

I'd use another verb here: 'tend' implies that this happens frequently but opens the door for "not always".

[1]

[...] If you only want some of
31 + # those combinations, keep them as separate sources.

Maybe mention that you can also use separate 'selections'.

review: Approve

=== modified file 'etc/maas/pserv.yaml'
--- etc/maas/pserv.yaml	2014-03-21 13:31:55 +0000
+++ etc/maas/pserv.yaml	2014-03-25 02:05:44 +0000
@@ -40,17 +40,37 @@
 
 ## Boot configuration.
 boot:
+  ## Storage location for boot images.
+  #
+  # These images can get quite large: some files are hundreds of megabytes,
+  # and you may need multiple copies for different architectures, OS releases,
+  # etc.
   storage: "/var/lib/maas/boot-resources/"
+
+  ## Sources of downloadable boot images.
+  #
+  # Each source downloads from one simplestreams URL, though it is possible to
+  # have multiple sources using the same URL.  For example, if you want to
+  # download "release" images for one OS release and just the "beta-2" images
+  # for another, specify two separate sources.  Otherwise, the importer would
+  # have to download both kinds of images for both releases.
   sources:
     - path: "http://maas.ubuntu.com/images/ephemeral/releases/"
       keyring: "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg"
+      ## Filters for this source's imports.
+      #
+      # Be careful: these items tend to multiply.  For example if you specify
+      # architectures "i386" and "amd64", and subarchitectures "generic",
+      # "hwe-s", and "hwe-t", the import script will need to import 6 images,
+      # for each possible combination.  If you only want some of those
+      # combinations, keep them as separate sources.
       selections:
         - release: "*"
-          arch: "*"
+          arches: ["*"]
           subarches: ["*"]
 ## other source example:
 #    - path: "http://hyperscale.ubuntu.com/images/"
 #      selections:
-#        - release: *
-#          arch: *
-#          subarches: ['*']
+#        - release: "*"
+#          arches: ["*"]
+#          subarches: ["*"]

=== modified file 'src/provisioningserver/config.py'
--- src/provisioningserver/config.py	2014-03-22 17:26:36 +0000
+++ src/provisioningserver/config.py	2014-03-25 02:05:44 +0000
@@ -137,7 +137,7 @@
     if_key_missing = None
 
     release = String(if_missing="*")
-    arch = String(if_missing="*")
+    arches = Set(if_missing=["*"])
     subarches = Set(if_missing=['*'])
 
 

=== modified file 'src/provisioningserver/import_images/boot_resources.py'
--- src/provisioningserver/import_images/boot_resources.py	2014-03-24 14:16:59 +0000
+++ src/provisioningserver/import_images/boot_resources.py	2014-03-25 02:05:44 +0000
@@ -131,7 +131,7 @@
     for filter_dict in filters:
         item_matches = (
             value_passes_filter(filter_dict['release'], release) and
-            value_passes_filter(filter_dict['arch'], arch) and
+            value_passes_filter_list(filter_dict['arches'], arch) and
             value_passes_filter_list(filter_dict['subarches'], subarch)
         )
         if item_matches:
@@ -151,7 +151,7 @@
     :param filters: list of dicts each of which contains 'arch', 'subarch',
         'release' keys; function takes d[arch][subarch][release] chain to the
         first dict only if filters contain at least one dict with
-        d['arch'] == arch, subarch in d['subarch'], d['release'] == release;
+        arch in d['arches'], subarch in d['subarch'], d['release'] == release;
         dict may have '*' as a value for 'arch' and 'release' keys and as a
         member of 'subarch' list -- in that case key-specific check always
         passes.

=== modified file 'src/provisioningserver/tests/test_config.py'
--- src/provisioningserver/tests/test_config.py	2014-03-22 17:26:36 +0000
+++ src/provisioningserver/tests/test_config.py	2014-03-25 02:05:44 +0000
@@ -144,7 +144,7 @@
                         '/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg'),
                     'selections': [
                         {
-                            'arch': '*',
+                            'arches': ['*'],
                             'release': '*',
                             'subarches': ['*'],
                         },