Merge ~ltrager/maas-images:version_remove_copy into maas-images:master

[Lee Trager]

Add the ability to remove or copy versions from a stream.

Example - remove a version from all Bionic images
./maas-images/bin/meph2-util remove-version ~/remove-version/ 20191004 release=bionic

Example - copy a version for all amd64 Bionic images
/maas-images/bin/meph2-util copy-version ~/copy-version/ 20191004 20191022 release=bionic arch=amd64

[Newell Jensen (newell-jensen)]

LGTM

review: Approve

0c3ba47... by Lee Trager on 2019-10-23

Add documentation

[Robert C Jennings (rcj)]

Lee, the docs look good, thank you. Should we copy an image forward from $serial to $serial.1 to test this out? Or at least, how did you test this?

[Lee Trager (ltrager)]

For development and testing I created a local mirror of the images and ran remove-version and copy-version on it. I verified with diff that the modifications happened as expected.

diff --git a/doc/copying-product-version.txt b/doc/copying-product-version.txt
new file mode 100644
index 0000000..6008d0b
--- /dev/null
+++ b/doc/copying-product-version.txt
@@ -0,0 +1,36 @@
+In the case that a bad image is published it may be quicker to copy a working
+version to a new version than wait for a fixed version to appear at
+cloud-images.ubuntu.com. This process is safer than just removing a broken
+version as it mimics a new version being added to the stream. This guarantees
+all clients, not just MAAS, will get the fixed image.
+
+It is also suggested to create a backup of the stream metadata in
+maas-v3-images/streams/v1 so you can use diff to verify only versions you wish
+to remove were removed.
+
+Basic usage:
+meph2-util copy-version data_d from_version to_version
+
+data_d - The path to the directory containing the stream you wish to modify.
+from_version - The version you wish to copy from
+to_version - The version you wish to copy to
+
+Example - Copy 20191004 to 20191022 on all products
+meph2-util copy-version /path/to/stream 20191004 20191022
+
+Filters:
+You may also add filters to the copy-version command. Only products matching
+those filters will have the specified version copied. Filters may be any field
+described in the product.
+
+Example - Copy the version 20191004 to 20191022 on all AMD64 Bionic and Xenial
+products.
+meph2-util copy-version /path/to/stream 20191004 20191022 \
+    'release~(bionic|xenial)' arch=amd64
+
+Optional Arguments:
+ -n, --dry-run - Only show what will be copied, do not modify the stream.
+ -u, --no-sign - Do not sign the stream when done. A stream can be signed later
+                 with the meph2-util sign command.
+ --keyring     - Specify the keyring to use when verifying the stream. Defaults
+ to /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
diff --git a/doc/removing-product-version.txt b/doc/removing-product-version.txt
new file mode 100644
index 0000000..df8dfe6
--- /dev/null
+++ b/doc/removing-product-version.txt
@@ -0,0 +1,51 @@
+In the case that a bad image is published it may be quicker to remove it from
+the stream than wait for a fixed version to appear at cloud-images.ubuntu.com.
+MAAS will import the latest version available from the stream, if a version is
+removed the previous version will be automatically retrieved.
+
+Before beginning make sure any import process which imports from
+cloud-images.ubuntu.com is disabled so the image isn't recreated.
+
+It is also suggested to create a backup of the stream metadata in
+maas-v3-images/streams/v1 so you can use diff to verify only versions you wish
+to remove were removed.
+
+Basic usage:
+meph2-util remove-version data_d version
+
+data_d - The path to the directory containing the stream you wish to modify.
+version - The version of the product you wish to remove.
+
+Example - Remove the version 20191021 from all products
+meph2-util remove-version /path/to/stream 20191021
+
+Filters:
+You may also add filters to the remove-version command. Only products matching
+those filters will have the specified version removed. Filters may be any field
+described in the product.
+
+Example - Remove the version 20191021 from all AMD64 Bionic and Xenial
+products.
+meph2-util remove-version /path/to/stream 20191021 \
+    'release~(bionic|xenial)' arch=amd64
+
+Optional Arguments:
+ -n, --dry-run - Only show what will be removed, do not modify the stream.
+ -u, --no-sign - Do not sign the stream when done. A stream can be signed later
+                 with the meph2-util sign command.
+ --keyring     - Specify the keyring to use when verifying the stream. Defaults
+ to /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
+
+Clean up:
+meph2-util remove-version only modifies the stream metadata. The image files
+themselves are left on the filesystem. To automatically remove them run the
+following commands
+
+Find all orphaned files and write the list to /tmp/orphans:
+meph2-util find-orphans /tmp/orphans /path/to/stream
+
+Delete all orphaned files:
+meph2-util reap-orphans /tmp/orphans /path/to/stream --older 0
+
+Remove orphan file:
+rm /tmp/orphans
diff --git a/meph2/commands/flags.py b/meph2/commands/flags.py
index e2cffa5..d0c3d61 100644
--- a/meph2/commands/flags.py
+++ b/meph2/commands/flags.py
@@ -24,6 +24,8 @@ COMMON_FLAGS = {
     'keyring': (('--keyring',),
                 {'help': 'gpg keyring to check sjson',
                  'default': DEF_KEYRING}),
+    'filters': ('filters', {'nargs': '*', 'default': []}),
+    'version': ('version', {'help': 'the version_id to promote.'}),
 }
 
 SUBCOMMANDS = {
@@ -33,7 +35,7 @@ SUBCOMMANDS = {
             COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'],
             COMMON_FLAGS['keyring'],
             COMMON_FLAGS['src'], COMMON_FLAGS['target'],
-            ('filters', {'nargs': '*', 'default': []}),
+            COMMON_FLAGS['filters'],
         ]
     },
     'import': {
@@ -68,8 +70,7 @@ SUBCOMMANDS = {
              {'help': 'do not copy files, only metadata [TEST_ONLY]',
               'action': 'store_true', 'default': False}),
             COMMON_FLAGS['src'], COMMON_FLAGS['target'],
-            ('version', {'help': 'the version_id to promote.'}),
-            ('filters', {'nargs': '+', 'default': []}),
+            COMMON_FLAGS['version'], COMMON_FLAGS['filters'],
         ]
     },
     'clean-md': {
@@ -78,7 +79,7 @@ SUBCOMMANDS = {
             COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'],
             COMMON_FLAGS['keyring'],
             ('max', {'type': int}), ('target', {}),
-            ('filters', {'nargs': '*', 'default': []}),
+            COMMON_FLAGS['filters'],
         ]
     },
     'find-orphans': {
@@ -106,6 +107,24 @@ SUBCOMMANDS = {
             COMMON_FLAGS['data_d'], COMMON_FLAGS['no-sign'],
         ],
     },
+    'remove-version': {
+        'help': 'Remove a version from a product',
+        'opts': [
+            COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'],
+            COMMON_FLAGS['keyring'], COMMON_FLAGS['data_d'],
+            COMMON_FLAGS['version'], COMMON_FLAGS['filters'],
+        ],
+    },
+    'copy-version': {
+        'help': 'Copy a version of a product to a new version',
+        'opts': [
+            COMMON_FLAGS['dry-run'], COMMON_FLAGS['no-sign'],
+            COMMON_FLAGS['keyring'], COMMON_FLAGS['data_d'],
+            ('from_version', {'help': 'the version_id to copy from.'}),
+            ('to_version', {'help': 'the version_id to copy to.'}),
+            COMMON_FLAGS['filters']
+        ],
+    },
 }
 
 # vi: ts=4 expandtab syntax=python
diff --git a/meph2/commands/meph2_util.py b/meph2/commands/meph2_util.py
index 4109634..0b114dc 100755
--- a/meph2/commands/meph2_util.py
+++ b/meph2/commands/meph2_util.py
@@ -403,6 +403,71 @@ def main_sign(args):
     return 0
 
 
+def main_remove_version(args):
+    filter_list = filters.get_filters(args.filters)
+    product_streams = util.load_product_streams(args.data_d)
+    resign = False
+
+    for product_stream in product_streams:
+        product_stream_path = os.path.join(args.data_d, product_stream)
+        content = util.load_content(product_stream_path)
+        products = content['products']
+        write_stream = False
+        for product, data in products.items():
+            if (
+                    filters.filter_dict(filter_list, data) and
+                    args.version in data['versions']):
+                print('Removing %s from %s' % (args.version, product))
+                if not args.dry_run:
+                    del data['versions'][args.version]
+                    resign = write_stream = True
+        if write_stream:
+            with open(product_stream_path, 'wb') as f:
+                f.write(util.dump_data(content).strip())
+    if resign:
+        util.gen_index_and_sign(args.data_d, not args.no_sign)
+    return 0
+
+
+def main_copy_version(args):
+    filter_list = filters.get_filters(args.filters)
+    product_streams = util.load_product_streams(args.data_d)
+    resign = False
+
+    for product_stream in product_streams:
+        product_stream_path = os.path.join(args.data_d, product_stream)
+        content = util.load_content(product_stream_path)
+        products = content['products']
+        write_stream = False
+        for product, data in products.items():
+            if (
+                    filters.filter_dict(filter_list, data) and
+                    args.from_version in data['versions']):
+                print('Copying %s to %s in %s' % (
+                    args.from_version, args.to_version, product))
+                if not args.dry_run:
+                    new_version = copy.deepcopy(
+                        data['versions'][args.from_version])
+                    for item in new_version['items'].values():
+                        old_path = os.path.join(args.data_d, item['path'])
+                        item['path'] = item['path'].replace(
+                            args.from_version, args.to_version)
+                        new_path = os.path.join(args.data_d, item['path'])
+                        if not os.path.exists(new_path):
+                            os.makedirs(
+                                os.path.dirname(new_path), exist_ok=True)
+                            shutil.copy(
+                                old_path, new_path, follow_symlinks=False)
+                    data['versions'][args.to_version] = new_version
+                    resign = write_stream = True
+        if write_stream:
+            with open(product_stream_path, 'wb') as f:
+                f.write(util.dump_data(content).strip())
+    if resign:
+        util.gen_index_and_sign(args.data_d, not args.no_sign)
+    return 0
+
+
 def main_import(args):
     """meph2-util import wraps the preferred command 'meph2-import'.
 
diff --git a/meph2/util.py b/meph2/util.py
index 017e5ef..f990eb9 100644
--- a/meph2/util.py
+++ b/meph2/util.py
@@ -264,15 +264,19 @@ def dump_data(data, end_cr=True):
     return bytestr
 
 
+def load_content(path):
+    if not os.path.exists(path):
+        return {}
+    with scontentsource.UrlContentSource(path) as tcs:
+        return sutil.load_content(tcs.read())
+
+
 def load_products(path, product_streams):
     products = {}
     for product_stream in product_streams:
         product_stream_path = os.path.join(path, product_stream)
-        if os.path.exists(product_stream_path):
-            with scontentsource.UrlContentSource(
-                    product_stream_path) as tcs:
-                product_listing = sutil.load_content(tcs.read())
-                products.update(product_listing['products'])
+        product_listing = load_content(product_stream_path)
+        products.update(product_listing['products'])
     return products