Merge ~johnsca/layer-snap:feature/lp/1845559/snap-coherence into ~stub/layer-snap:master

[Kevin W Monroe]

Add support for snap coherence

Add additional functions and flags to create and manage cohort snapshots.

This is in support of k8s snap coherence:

https://bugs.launchpad.net/charm-kubernetes-master/+bug/1845559

It has been working well for me as a way to manage snap cohorts amongst k8s applications. When coupled with https://github.com/johnsca/charm-snap-store-proxy, I've been able to hold k8s snaps at specific revs, e.g.:

$ juju run --unit snap-store-proxy/0 'sudo snap-proxy list-overrides kubectl'
kubectl 1.16/stable amd64 1342 (upstream 1309)

$ juju run --application kubernetes-master 'sudo snap list kubectl'
- Stdout: |
    Name Version Rev Tracking Publisher Notes
    kubectl 1.16.3-beta.0 1342 1.16 canonical* classic,in-cohort
  UnitId: kubernetes-master/1
- Stdout: |
    Name Version Rev Tracking Publisher Notes
    kubectl 1.16.3-beta.0 1342 1.16 canonical* classic,in-cohort
  UnitId: kubernetes-master/0

There's still work to do in the k8s charms and docs, but this is ready for upstream layer-snap consideration.

[Stuart Bishop (stub)]

Code is good.

If create_cohort_snapshot and join_cohort_snapshot are to be part of the public API, they should be documented in README.md; I haven't wired up any documentation generation and snap.py contains a mix of public API and support functions in any case (the line isn't clear, so I guess that makes all the methods public). Minimally would be fine. Some care may be needed to explain expected behavior when the refresh() method is called on snap that is part of a cohort.

[Stuart Bishop (stub)]

Yup, looks good.

review: Approve

diff --git a/README.md b/README.md
index 1ba2ebe..c9f6f16 100644
--- a/README.md
+++ b/README.md
@@ -146,6 +146,20 @@ package::
   snap updated if the snap or arguments have changed. If the snap was
   installed from the Snap Store, `snap refresh` is run to update the snap.
 
+* `create_cohort_snapshot(snapname)`. Creates a new cohort snapshot and
+  returns the associated key. A cohort snapshot allows snaps on different
+  machines to coordinate their refreshes by sharing the cohort snapshot key.
+
+* `join_cohort_snapshot(snapname, cohort_key)`. Joins a cohort snapshot.
+  When in a cohort snapshot, new snap revisions will not be automatically
+  applied; instead, the leader should periodically check for an available
+  refresh, then create a new cohort snapshot and distribute the key in
+  a controlled fashion to roll out updates.
+
+* `is_refresh_available(snapname)`. Check whether the given snap can be
+  updated. Also available as an automatically managed flag, of the form
+  `snap.refresh-available.{snapname}`.
+
 * `remove(snapname)`. The snap is removed.
 
 Keyword arguments correspond to the layer.yaml options and snap command line
diff --git a/lib/charms/layer/snap.py b/lib/charms/layer/snap.py
index 1254351..f760d69 100644
--- a/lib/charms/layer/snap.py
+++ b/lib/charms/layer/snap.py
@@ -17,6 +17,8 @@
 import os
 import subprocess
 
+import yaml
+
 from charmhelpers.core import hookenv
 from charms import layer
 from charms import reactive
@@ -28,6 +30,14 @@ def get_installed_flag(snapname):
     return 'snap.installed.{}'.format(snapname)
 
 
+def get_refresh_available_flag(snapname):
+    return 'snap.refresh-available.{}'.format(snapname)
+
+
+def get_local_flag(snapname):
+    return 'snap.local.{}'.format(snapname)
+
+
 def get_disabled_flag(snapname):
     return 'snap.disabled.{}'.format(snapname)
 
@@ -44,6 +54,7 @@ def install(snapname, **kw):
     function is called.
     '''
     installed_flag = get_installed_flag(snapname)
+    local_flag = get_local_flag(snapname)
     if reactive.is_flag_set(installed_flag):
         refresh(snapname, **kw)
     else:
@@ -53,6 +64,7 @@ def install(snapname, **kw):
                 _install_store(snapname, **kw)
             else:
                 _install_local(res_path, **kw)
+                reactive.set_flag(local_flag)
         else:
             _install_store(snapname, **kw)
         reactive.set_flag(installed_flag)
@@ -68,6 +80,19 @@ def is_installed(snapname):
     return reactive.is_flag_set(get_installed_flag(snapname))
 
 
+def is_local(snapname):
+    return reactive.is_flag_set(get_local_flag(snapname))
+
+
+def get_installed_snaps():
+    '''Return a list of snaps which are installed by this layer.
+    '''
+    flag_prefix = 'snap.installed.'
+    return [flag[len(flag_prefix):]
+            for flag in reactive.get_flags()
+            if flag.startswith(flag_prefix)]
+
+
 def refresh(snapname, **kw):
     '''Update a snap.
 
@@ -83,14 +108,18 @@ def refresh(snapname, **kw):
     # upload a zero byte resource, but then we would need to uninstall
     # the snap before reinstalling from the store and that has the
     # potential for data loss.
+    local_flag = get_local_flag(snapname)
     if hookenv.has_juju_version('2.0'):
         res_path = _resource_get(snapname)
         if res_path is False:
             _refresh_store(snapname, **kw)
+            reactive.clear_flag(local_flag)
         else:
             _install_local(res_path, **kw)
+            reactive.set_flag(local_flag)
     else:
         _refresh_store(snapname, **kw)
+        reactive.clear_flag(local_flag)
 
 
 def remove(snapname):
@@ -317,6 +346,7 @@ def _install_store(snapname, **kw):
         hookenv.log('Installation successful cmd="{}" output="{}"'
                     .format(cmd, out),
                     level=hookenv.DEBUG)
+        reactive.clear_flag(get_local_flag(snapname))
     except subprocess.CalledProcessError as cp:
         hookenv.log('Installation failed cmd="{}" returncode={} output="{}"'
                     .format(cmd, cp.returncode, cp.output),
@@ -347,3 +377,60 @@ def _resource_get(snapname):
     if res_path and os.stat(res_path).st_size != 0:
         return res_path
     return False
+
+
+def get_available_refreshes():
+    '''Return a list of snaps which have refreshes available.
+    '''
+    out = subprocess.check_output(['snap', 'refresh', '--list']).decode('utf8')
+    if out == 'All snaps up to date.':
+        return []
+    else:
+        return [l.split()[0] for l in out.splitlines()[1:]]
+
+
+def is_refresh_available(snapname):
+    '''Check whether a new revision is available for the given snap.
+    '''
+    return reactive.is_flag_set(get_refresh_available_flag(snapname))
+
+
+def _check_refresh_available(snapname):
+    return snapname in get_available_refreshes()
+
+
+def create_cohort_snapshot(snapname):
+    '''Create a new cohort key for the given snap.
+
+    Cohort keys represent a snapshot of the revision of a snap at the time
+    the key was created. These keys can then be used on any machine to lock
+    the revision of the snap until a new cohort is joined (or the key expires,
+    after 90 days). This is used to maintain consistency of the revision of
+    the snap across units or applications, and to manage the refresh of the
+    snap in a controlled manner.
+
+    Returns a cohort key.
+    '''
+    out = subprocess.check_output(['snap', 'create-cohort', snapname])
+    data = yaml.safe_load(out.decode('utf8'))
+    return data['cohorts'][snapname]['cohort-key']
+
+
+def join_cohort_snapshot(snapname, cohort_key):
+    '''Refresh the snap into the given cohort.
+
+    If the snap was previously in a cohort, this will update the revision
+    to that of the new cohort snapshot. Note that this does not change the
+    channel that the snap is in, only the revision within that channel.
+    '''
+    if is_local(snapname):
+        # joining a cohort can override a locally installed snap
+        hookenv.log('Skipping joining cohort for local snap: '
+                    '{}'.format(snapname))
+        return
+    subprocess.check_output(['snap', 'refresh', snapname,
+                             '--cohort', cohort_key])
+    # even though we just refreshed to the latest in the cohort, it's
+    # slightly possible that there's a newer rev available beyond the cohort
+    reactive.toggle_flag(get_refresh_available_flag(snapname),
+                         _check_refresh_available(snapname))
diff --git a/reactive/snap.py b/reactive/snap.py
index 882280a..669a56d 100644
--- a/reactive/snap.py
+++ b/reactive/snap.py
@@ -32,7 +32,7 @@ from charmhelpers.core.host import write_file
 from charms import layer
 from charms import reactive
 from charms.layer import snap
-from charms.reactive import register_trigger, when, when_not
+from charms.reactive import register_trigger, when, when_not, toggle_flag
 from charms.reactive.helpers import data_changed
 
 
@@ -85,6 +85,17 @@ def install():
         snap.connect_all()
 
 
+def check_refresh_available():
+    # Do nothing if we don't have kernel support yet
+    if not kernel_supported():
+        return
+
+    available_refreshes = snap.get_available_refreshes()
+    for snapname in snap.get_installed_snaps():
+        toggle_flag(snap.get_refresh_available_flag(snapname),
+                    snapname in available_refreshes)
+
+
 def refresh():
     # Do nothing if we don't have kernel support yet
     if not kernel_supported():
@@ -332,3 +343,4 @@ hookenv.atstart(ensure_path)
 hookenv.atstart(update_snap_proxy)
 hookenv.atstart(configure_snap_store_proxy)
 hookenv.atstart(install)
+hookenv.atstart(check_refresh_available)