Merge lp:~dannf/maas-images/arm64-netinst-restructure into lp:maas-images

[dann frazier]

[Scott Moser (smoser)]

for reference... here is what I did to verify this is sane.

- added bin/netboot-mirror and README.di-scraping
  these probalby woudl have been helpful for you.. in seeing what is going
  on. and explaining things.

- ran from your proposed merge:
    ./bin/netboot-mirror --no-item-download out-new.d
  and from original:
    ./bin/netboot-mirror --no-item-download out-old.d

  That created out-old.d and out-new.d which contains streams data for the NetbootMirrorReader

- got a list of fields in there, and flattened and sorted output.
  run:
    sstream-query --pretty out-old.d/streams/v1/com.ubuntu.installer:released:netboot.json
  That dumps all fields for each item... looks like:
    {'arch': 'i386',
     'basename': 'linux',
     'content_id': 'com.ubuntu.installer:released:netboot',
     'format': 'products:1.0',
     'ftype': 'kernel',
     'image-format': 'default',
     'item_name': 'ututgenke0.default',
     'item_url': 'out-old.d/files/utopic/i386/20101020ubuntu349/utopic/generic/kernel.default',
     'kernel-flavor': 'generic',
     'kernel-release': 'utopic',
     'md5': '4237d65fc81b909684447bf54fd02ce9',
     'path': 'files/utopic/i386/20101020ubuntu349/utopic/generic/kernel.default',
     'pocket': 'release',
     'product_name': 'com.ubuntu.installer:netboot:14.10:i386',
     'pubdate': 'Fri, 17 Oct 2014 15:53:00 +0000',
     'release': 'utopic',
     'sha256': '479619cc6522efb1ceb591bb425fd85d3735e2306de24a1e4345de01a209017c',
     'size': '6077104',
     'version': '14.10',
     'version_name': '20101020ubuntu349'}

- cleaned up the list of fields i wanted to compare to make sure old and new are similar.
  then, sorted that output and diffed them.

  base="%(product_name)s/%(version_name)s/%(item_name)s"
  extra="%(release)s %(version)s %(pocket)s %(kernel-flavor)s %(ftype)s %(image-format)s %(kernel-release)s %(md5)s %(path)s"
  path=streams/v1/com.ubuntu.installer:released:netboot.json

  sstream-query --output-format="$base $extra" out-old.d/$path | sort > old.list
  sstream-query --output-format="$base $extra" out-new.d/$path | sort > new.list

  Then, just diff them
   http://paste.ubuntu.com/11503207/

  Your new had only new entries, and only in wily... so that seems good.

=== modified file 'meph2/netinst.py'
--- meph2/netinst.py	2015-02-12 22:14:06 +0000
+++ meph2/netinst.py	2015-02-19 15:59:01 +0000
@@ -61,6 +61,10 @@
     "generic-lpae": "glp",
 }
 
+DTB_TO_FORMAT = {
+    "apm-mustang.dtb": "xgene"
+}
+
 # #
 # # Under a path like: MIRROR/precise-updates/main/installer-i386/
 # #  we find a listing of directories like:
@@ -236,7 +240,7 @@
     # or a tuple of release-kernel, kernel-flavor, initrd-flavor, filetype
 
     # at the moment the only 'initrd-flavor' that we're supporting is netboot
-    if path.find("netboot") < 0:
+    if path.find("netboot") < 0 and path.find("device-tree") < 0:
         return None
     iflavor = "netboot"
 
@@ -255,7 +259,18 @@
     # generic/xgene/uInitrd is generic flavor,
     # xgene-uboot or xgene-uboot-mustang image
     imgfmt = "default"
-    if (len(toks) == 3 and toks[0] == "generic" and
+    if (len(toks) == 4 and toks[1] == "generic" and (toks[3].startswith('uI'))):
+        path = "%s-netboot/%s/%s" % (release, toks[1], toks[3])
+        imgfmt = toks[2]
+        # xgene-uboot -> xgene
+        imgfmt = re.sub("-uboot$", "", imgfmt)
+    elif (len(toks) == 2 and toks[0] == "device-tree" and
+        toks[1].endswith('dtb')):
+        path = "%s-netboot/generic/%s" % (release, toks[1])
+        if toks[1] in DTB_TO_FORMAT:
+            imgfmt = DTB_TO_FORMAT[toks[1]]
+    # trusty & utopic used this layout for arm64
+    elif (len(toks) == 3 and toks[0] == "generic" and
             (toks[2].startswith('uI') or toks[2].endswith('dtb'))):
         path = "%s-netboot/%s/%s" % (release, toks[0], toks[2])
         imgfmt = toks[1]
@@ -345,7 +360,7 @@
 
     versions = {}
 
-    regex = re.compile(".*netboot")
+    regex = re.compile(".*(netboot|device-tree)")
     for (di_ver, pubdate) in usable:
         versions[di_ver] = {'items': {}}
         curp = '/'.join((url, di_ver, 'images',))