Merge ~ltrager/maas-images:daily_to_candidate into maas-images:master

[Lee Trager]

Rename daily to candidate

The daily stream is being renamed candidate and we'll be creating a new stable stream. Images produced by lp:maas-images need to be created using the candidate label so inserting works properly. This branch won't be landed until CPC is ready to create both streams.

[Alberto Donato (ack)]

LGTM.

As mentioned on MM maybe we could keep a feature branch where to land these changed until the whole feature is complete.

review: Approve

bae0560... by Lee Trager on 2020-09-11

Merge branch 'master' into daily_to_candidate

35f0383... by Lee Trager on 2020-10-06

Merge branch 'master' into daily_to_candidate

diff --git a/README b/README
index df2a9ab..050bdcf 100644
--- a/README
+++ b/README
@@ -1,6 +1,6 @@
-This builds maas ephemeral images from daily builds of cloud images.
+This builds maas ephemeral images from candidate builds of cloud images.
 Ie, it creates stuff that goes on
-  http://images.maas.io/ephemeral-v2/daily/
+  http://images.maas.io/ephemeral-v3/candidate
 from stuff on
   http://cloud-images.ubuntu.com/daily
 
@@ -50,7 +50,7 @@ To force a build, use '--target=force'
     copied from maas.ubuntu.com for all other products, but will only
     create new content for products that match the provided filters.
 
-  If the maas ephemeral data is up to date with the latest cloud image daily
+  If the maas ephemeral data is up to date with the latest cloud image candidate
   then meph2-cloudimg-sync will not do anything.
 
 In order to 'rebuild' something.  To create a new '20160101.1' even when
@@ -114,10 +114,10 @@ an existing stream.  This option (meph2-util insert) is additive only.
 
 === insert data ===
 Given output of a meph2-cloudimg-sync command above in build-output.d
-and local daily stream directory daily-stream.d you can add the new data
+and local candidate stream directory candidate-stream.d you can add the new data
 with:
 
-  meph2-util insert build-output.d daily-stream.d
+  meph2-util insert build-output.d candidate-stream.d
 
 If you do not want to sign (create .gpg and .sjson files), then you will
 have to use '--no-sign'.
@@ -126,23 +126,23 @@ You can use '--dry-run' to only report what would be done.
 
 === Promoting Builds ===
 Promoting is done on a specific version of a product.
-You promote a daily build to a released build with a given label.
+You promote a candidate build to a released build with a given label.
 Likely that label is 'release'.
 
    # list the product-ids and version are available for promotion
    $ sstream-query --output-format="%(product_name)s %(version_name)s" --max=1 \
-      daily.d/streams/v1/index.json arch=i386 | sort -u
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-p 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-r 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-s 20140414
-   com.ubuntu.maas.daily:v2:boot:13.10:i386:hwe-s 20140416
-   com.ubuntu.maas.daily:v2:boot:14.04:i386:hwe-t 20140416.1
+      candidate.d/streams/v1/index.json arch=i386 | sort -u
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-p 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-r 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-s 20140414
+   com.ubuntu.maas.candidate:v3:boot:13.10:i386:hwe-s 20140416
+   com.ubuntu.maas.candidate:v3:boot:14.04:i386:hwe-t 20140416.1
 
    # promote the listed version
    $ meph2-util promote --label=release \
-        daily.d/streams/v1/index.json releases.d/ 20140414 \
-        product_name=com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q
+        candidate.d/streams/v1/index.json stable.d/ 20140414 \
+        product_name=com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q
 
 
 == cleaning old images ==
@@ -158,7 +158,7 @@ clean-md is the meph-util subcommand that allows you to keep only a given
 number of builds.  For example, to keep the latest 3 versions of every
 product in the meta-data, you would run:
 
-  meph2-util clean-md --max=3 daily-sream.d/
+  meph2-util clean-md --max=3 candidate-stream.d/
 
 This *only* modifies the meta-data, it does not delete files, use find-orphans
 and reap-orphans for that.
@@ -176,7 +176,7 @@ Example usage:
  my-orphans.json with any new items, but will not update first-orphaned
  timestamp.
 
-  $ meph2-util find-orphans my-orphans.json ./daily/ ./daily/streams/v1/*.json
+  $ meph2-util find-orphans my-orphans.json ./candidate/ ./candidate/streams/v1/*.json
 
 More complex usage:
  Its possible that 2 streams dirs could share a data directory, similar
@@ -184,11 +184,11 @@ More complex usage:
  directory tree layout (note, 'path' elements would have 'data' in them in
  this example):
 
-   daily/
+   candidate/
      data -> ../data/
      streams/v1/index.json
-     streams/v1/daily.json
-   released/
+     streams/v1/candidate.json
+   stable/
      data -> ../data
      streams/v1/index.json
      streams/v1/released.json
@@ -199,10 +199,10 @@ More complex usage:
  Then, we could run meph-util find-orphans with:
 
    $ meph2-util find-orphans my-orphans.json ./data/ \
-       ./daily/streams/v1/index.json released/streams/v1/index.json
+       ./candidate/streams/v1/index.json stable/streams/v1/index.json
 
  That will look through files in data/ that are not referenced by path
- elements in either the daily/streams/v1/ or released/streams/v1/ files.
+ elements in either the candidate/streams/v1/ or stable/streams/v1/ files.
 
  There is no --dry-run for finding orphans as it is non-destructive.
 
diff --git a/README.testing b/README.testing
index 1622e4a..f5d558a 100644
--- a/README.testing
+++ b/README.testing
@@ -1,28 +1,28 @@
 To get local copies of maas ephemerals, you can do:
 
   burl=http://maas.ubuntu.com/images/ephemeral-v2
-  for stream in releases daily; do
+  for stream in releases candidate; do
      sstream-mirror --no-item-download \
         $burl/$stream/streams/v1/index.json "$stream.d"; done
 
 
 Then, to test promote you could do:
    $ sstream-query --output-format="%(product_name)s %(version_name)s" --max=1 \
-      daily.d/streams/v1/index.json  arch=i386 | sort -u
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-p 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-r 20140414
-   com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-s 20140414
-   com.ubuntu.maas.daily:v2:boot:13.10:i386:hwe-s 20140416
-   com.ubuntu.maas.daily:v2:boot:14.04:i386:hwe-t 20140416.1
+      candidate.d/streams/v1/index.json  arch=i386 | sort -u
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-p 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-r 20140414
+   com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-s 20140414
+   com.ubuntu.maas.candidate:v3:boot:13.10:i386:hwe-s 20140416
+   com.ubuntu.maas.candidate:v3:boot:14.04:i386:hwe-t 20140416.1
 
    $ sstream-query \
         --output-format="%(product_name)s %(version_name)s/%(item_name)s" \
         releases.d/streams/v1/index.json | sort -u > before
 
    $ meph2-util promote --no-sign --skip-file-copy --label=release \
-        daily.d/streams/v1/index.json releases.d/ 20140414 \
-        product_name=com.ubuntu.maas.daily:v2:boot:12.04:i386:hwe-q
+        candidate.d/streams/v1/index.json releases.d/ 20140414 \
+        product_name=com.ubuntu.maas.candidate:v3:boot:12.04:i386:hwe-q
 
    $ sstream-query \
         --output-format="%(product_name)s %(version_name)s/%(item_name)s" \
diff --git a/conf/bootloaders.yaml b/conf/bootloaders.yaml
index cf83162..7fd275e 100644
--- a/conf/bootloaders.yaml
+++ b/conf/bootloaders.yaml
@@ -1,5 +1,5 @@
-product_id: "com.ubuntu.maas.daily:1:{os}:{firmware_platform}:{arch}"
-content_id: "com.ubuntu.maas:daily:1:bootloader-download"
+product_id: "com.ubuntu.maas.candidate:1:{os}:{firmware_platform}:{arch}"
+content_id: "com.ubuntu.maas:candidate:1:bootloader-download"
 
 bootloaders:
  - firmware-platform: pxe
diff --git a/conf/centos.yaml b/conf/centos.yaml
index 55bc311..3c06369 100644
--- a/conf/centos.yaml
+++ b/conf/centos.yaml
@@ -1,7 +1,7 @@
 image_index: http://cloud.centos.org/centos/{version}/images/image-index
 
-product_id: "com.ubuntu.maas.daily:centos-bases:{version}:{arch}"
-content_id: "com.ubuntu.maas:daily:centos-bases-download"
+product_id: "com.ubuntu.maas.candidate:centos-bases:{version}:{arch}"
+content_id: "com.ubuntu.maas:candidate:centos-bases-download"
 os: centos
 arch: amd64
 base_mirror: http://mirrors.edge.kernel.org
diff --git a/conf/meph-v3.yaml b/conf/meph-v3.yaml
index 594918b..a7e45a8 100644
--- a/conf/meph-v3.yaml
+++ b/conf/meph-v3.yaml
@@ -207,9 +207,9 @@ enable_di: False
 ##    so for now, we just don't have a stream for that.
 ##  * kernel meta package in all cases here is linux-image-<flavor>
 
-default_target: http://images.maas.io/ephemeral-v3/daily/streams/v1/com.ubuntu.maas:daily:v3:download.json
-content_id: com.ubuntu.maas:daily:v3:download
-product_id_pre: com.ubuntu.maas.daily:v3:boot
+default_target: http://images.maas.io/ephemeral-v3/candidate/streams/v1/com.ubuntu.maas:candidate:v3:download.json
+content_id: com.ubuntu.maas:candidate:v3:download
+product_id_pre: com.ubuntu.maas.candidate:v3:boot
 
 # By default maas-images uses the krel in the boot path. This worked for the v2
 # stream because there was only one kernel per release/arch/krel/kflavor. The
diff --git a/meph2/commands/cloudimg_sync.py b/meph2/commands/cloudimg_sync.py
index a118a52..2ea1903 100755
--- a/meph2/commands/cloudimg_sync.py
+++ b/meph2/commands/cloudimg_sync.py
@@ -16,8 +16,8 @@ import os
 import sys
 import yaml
 
-CLOUD_IMAGES_DAILY = ("http://cloud-images.ubuntu.com/daily/"
-                      "streams/v1/com.ubuntu.cloud:daily:download.json")
+CLOUD_IMAGES_CANDIDATE = ("http://cloud-images.ubuntu.com/candidate/"
+                      "streams/v1/com.ubuntu.cloud:candidate:download.json")
 
 FORCE_URL = "force"  # a fake target url that will have nothing in it
 DEFAULT_ARCHES = {
@@ -36,7 +36,7 @@ def v2_to_cloudimg_products(prodtree, rebuilds={}):
     # this turns a v2 products tree into a cloud-image products tree.
     # it pays attention only to products with krel == release
     # (in an attempt to only get "primary")
-    ret = util.empty_iid_products("com.ubuntu.cloud:daily:download")
+    ret = util.empty_iid_products("com.ubuntu.cloud:candidate:download")
     # rebuilds is {cloudimgYYYYMMDD:newYYYYMMDD}
     for product in prodtree.get('products'):
         if not (prodtree['products'][product].get('krel') ==
@@ -44,8 +44,8 @@ def v2_to_cloudimg_products(prodtree, rebuilds={}):
             continue
 
         # com.ubuntu.maas:boot:12.04:amd64:hwe-s =>
-        # com.ubuntu.cloud.daily:server:12.04:amd64
-        tprod = ("com.ubuntu.cloud.daily:server:%(version)s:%(arch)s" %
+        # com.ubuntu.cloud.candidate:server:12.04:amd64
+        tprod = ("com.ubuntu.cloud.candidate:server:%(version)s:%(arch)s" %
                  prodtree['products'][product])
 
         if tprod not in ret['products']:
@@ -99,7 +99,7 @@ class CloudImg2Meph2Sync(mirrors.BasicMirrorWriter):
         self.rebuilds = rebuilds
 
     def load_products(self, path=None, content_id=None):
-        if content_id != "com.ubuntu.cloud:daily:download":
+        if content_id != "com.ubuntu.cloud:candidate:download":
             raise ValueError("Not expecting to sync with content_id: %s" %
                              content_id)
 
@@ -172,7 +172,7 @@ class CloudImg2Meph2Sync(mirrors.BasicMirrorWriter):
             fp.write(util.dump_data(index))
 
     def filter_index_entry(self, data, src, pedigree):
-        if pedigree[0] != "com.ubuntu.cloud:daily:download":
+        if pedigree[0] != "com.ubuntu.cloud:candidate:download":
             LOG.info("skipping index entry %s" % '/'.join(pedigree))
             return False
         return True
@@ -209,7 +209,7 @@ def main():
     parser.add_argument('--proposed', action='store_true', default=False)
     parser.add_argument('--rebuild', action='append', default=[],
                         help='rebuild version name YYYYMMDD:YYYMMDD.1')
-    parser.add_argument('--source', default=CLOUD_IMAGES_DAILY,
+    parser.add_argument('--source', default=CLOUD_IMAGES_CANDIDATE,
                         help='cloud images mirror')
     parser.add_argument('--target', default=None,
                         help="maas ephemeral v2 or v3 mirror, overrides "
diff --git a/meph2/commands/flags.py b/meph2/commands/flags.py
index f3ca85d..f85060d 100644
--- a/meph2/commands/flags.py
+++ b/meph2/commands/flags.py
@@ -59,7 +59,7 @@ SUBCOMMANDS = {
             ]
     },
     'promote': {
-        'help': 'promote a product/version from daily to release',
+        'help': 'promote a product/version from candidate to stable',
         'opts': [
             COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'],
             COMMON_FLAGS['keyring'],
diff --git a/meph2/commands/meph2_util.py b/meph2/commands/meph2_util.py
index 295442d..f33b4be 100755
--- a/meph2/commands/meph2_util.py
+++ b/meph2/commands/meph2_util.py
@@ -135,7 +135,7 @@ class InsertBareMirrorWriter(BareMirrorWriter):
 
 class ReleasePromoteMirror(InsertBareMirrorWriter):
     # this does not do reference counting or .data/ storage
-    # it converts a daily item to a release item and inserts it.
+    # it converts a candidate item to a release item and inserts it.
 
     # we take care of writing file in insert_products
     insert_index_entry = BareMirrorWriter._noop
@@ -145,7 +145,7 @@ class ReleasePromoteMirror(InsertBareMirrorWriter):
                                                    objectstore=objectstore)
         self.label = label
 
-    def rel2daily(self, ptree):
+    def rel2candidate(self, ptree):
         ret = copy.deepcopy(ptree)
         ret['content_id'] = self.fixed_content_id(ret['content_id'])
 
@@ -155,25 +155,25 @@ class ReleasePromoteMirror(InsertBareMirrorWriter):
             del ptree['products'][oname]
 
     def fixed_content_id(self, content_id):
-        # when promoting from daily, our content ids get ':daily' removed
-        #  com.ubuntu.maas:daily:v2:download => com.ubuntu.maas:v2:download
-        return(content_id.replace(":daily", ""))
+        # when promoting from candidate, our content ids get ':candidate' removed
+        #  com.ubuntu.maas:candidate:v2:download => com.ubuntu.maas:v2:download
+        return(content_id.replace(":candidate", ""))
 
     def fixed_pedigree(self, pedigree):
         return tuple([self.fixed_product_id(pedigree[0])] + list(pedigree[1:]))
 
     def fixed_product_id(self, product_id):
-        # when promoting from daily, product ids get '.daily' removed
-        #  com.ubuntu.maas.daily:v2:boot:13.10:armhf:generic-lpae ->
+        # when promoting from candidate, product ids get '.candidate' removed
+        #  com.ubuntu.maas.candidate:v2:boot:13.10:armhf:generic-lpae ->
         #     com.ubuntu.maas:v2:boot:13.10:armhf:generic-lpae
-        return product_id.replace(".daily:", ":")
+        return product_id.replace(".candidate:", ":")
 
     def load_products(self, path, content_id):
         # this loads the released products, but returns it in form
-        # of daily products
+        # of candidate products
         ret = super(ReleasePromoteMirror, self).load_products(
             path=path, content_id=self.fixed_content_id(content_id))
-        return self.rel2daily(ret)
+        return self.rel2candidate(ret)
 
     def insert_item(self, data, src, target, pedigree, contentsource):
         ret = super(ReleasePromoteMirror, self).insert_item(
diff --git a/meph2/commands/mimport.py b/meph2/commands/mimport.py
index 6344d3f..224a248 100644
--- a/meph2/commands/mimport.py
+++ b/meph2/commands/mimport.py
@@ -67,7 +67,7 @@ def import_remote_config(args, product_tree, cfgdata):
             print("Creating new product %s" % product_id)
             product_tree['products'][product_id] = {
                 'subarches': 'generic',
-                'label': 'daily',
+                'label': 'candidate',
                 'subarch': 'generic',
                 'arch': arch,
                 'os': os_name,
@@ -175,7 +175,7 @@ def import_bootloaders(args, product_tree, cfgdata):
         if product_tree['products'].get(product_id) is None:
             print("Creating new product %s" % product_id)
             product_tree['products'][product_id] = {
-                'label': 'daily',
+                'label': 'candidate',
                 'arch': firmware_platform['arch'],
                 'arches': firmware_platform['arches'],
                 'os': firmware_platform['os'],
diff --git a/meph2/stream.py b/meph2/stream.py
index 3142401..47a9abd 100644
--- a/meph2/stream.py
+++ b/meph2/stream.py
@@ -11,7 +11,7 @@ import yaml
 
 from simplestreams.log import LOG
 
-ALL_ITEM_TAGS = {'label': 'daily', 'os': 'ubuntu'}
+ALL_ITEM_TAGS = {'label': 'candidate', 'os': 'ubuntu'}
 
 PATH_COMMON = "%(release)s/%(arch)s/"
 BOOT_COMMON = PATH_COMMON + "%(version_name)s/%(kname)s/%(flavor)s"
diff --git a/tools/genstream-muc b/tools/genstream-muc
index 0134afa..758836a 100755
--- a/tools/genstream-muc
+++ b/tools/genstream-muc
@@ -162,7 +162,7 @@ class FakeProduct(object):
         self._versions[version] = d
 
     def get_label(self, version):
-        return 'daily'
+        return 'candidate'
 
     def makepaths(self, target, makedata=makedata_filepath):
         for version_id, ver in self._versions.items():