Merge lp:~afreiberger/landscape-client-charm/add-juju-annotations into lp:landscape-client-charm

[Drew Freiberger]

Add cronjob and charm config for registering juju agents as Landscape annotations.

Based on previous efforts (which are then obsolete) by:
* @xavpaice https://code.launchpad.net/~xavpaice/landscape-client-charm/add_tags/+merge/407578
* @exsdev https://code.launchpad.net/~exsdev/landscape-client-charm/juju-app-annotator
* @stephanpampel https://code.launchpad.net/~stephanpampel/landscape-client-charm/landscape-client-charm/+merge/411409

Add hook and cronjob for registering juju apps as Landscape annotations
Can be enabled via config flag juju-annotations-enabled (default=false)

Details 'juju-annotations-enabled':
        Periodically updates landscape annotations (key:value) for each juju agent
        found on the machine and the juju availability-zone (AZ) of the machine.
        Key is 'juju-${application}' OR 'juju-az-${AZ}' if an AZ is set.
        Value is 'deployed=true' OR 'deployed=false' if the agent got removed.
        Example: juju-landscape-client:deployed=true
        Cronjob runs every 5 min, update frequency depends on exchange-interval.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 74
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1498/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

75. By Drew Freiberger on 2021-12-15

Address import issues

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 75
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1499/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

76. By Drew Freiberger on 2021-12-15

py3 updates for write-juju-annotations.py

77. By Drew Freiberger on 2021-12-15

Rename annotations script

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 77
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1500/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

78. By Drew Freiberger on 2021-12-15

fix data-path for juju annotations cronjob

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 78
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1501/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

79. By Drew Freiberger on 2021-12-15

Fixed path to annotations script

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 79
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1502/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

80. By Drew Freiberger on 2021-12-15

Annotations - Detect multiple dashes in unit names and add unit number along with app name

[Drew Freiberger (afreiberger)]

I've completed manual functional testing and am satisfied with this feature implementation now.

Ready for review by Landscape team.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 80
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1503/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

81. By Drew Freiberger on 2021-12-16

Add clarification to config.yaml for annotations

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 81
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1505/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

82. By Drew Freiberger on 2021-12-16

Fix ordering of config-changed for juju-annotations cronjob

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 82
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1506/

review: Needs Fixing (test results)

[lp:~afreiberger/landscape-client-charm/add-juju-annotations]

83. By Drew Freiberger on 2021-12-16

Fix regex for annotations

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1507/

review: Needs Fixing (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1510/

review: Needs Fixing (test results)

[Drew Freiberger (afreiberger)]

Update from landscape team is that this is going to be superceded by work being done to allow client-side tag manipulation directly. This branch is being held available as the underlying branch published at cs:~afreiberger/landscape-client.

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1517/

review: Needs Fixing (test results)

[Simon Poirier (simpoir)]

First, this changes needs some test coverage (for both hooks and the externally-called `write_juju_annotations.py` script), and fixing broken tests (which might be mostly covered by my comments below).

review: Needs Fixing

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1521/

review: Needs Fixing (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1536/

review: Needs Fixing (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1539/

review: Needs Fixing (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1544/

review: Needs Fixing (test results)

[🤖 Landscape Builder (landscape-builder)]

review: Abstain (executing tests)

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Fail
Revno: 83
Branch: lp:~afreiberger/landscape-client-charm/add-juju-annotations
Jenkins: https://jenkins.canonical.com/landscape/job/latch-test-xenial/1549/

review: Needs Fixing (test results)

Unmerged revisions

83. By Drew Freiberger on 2021-12-16

Fix regex for annotations

82. By Drew Freiberger on 2021-12-16

Fix ordering of config-changed for juju-annotations cronjob

81. By Drew Freiberger on 2021-12-16

Add clarification to config.yaml for annotations

80. By Drew Freiberger on 2021-12-15

Annotations - Detect multiple dashes in unit names and add unit number along with app name

79. By Drew Freiberger on 2021-12-15

Fixed path to annotations script

78. By Drew Freiberger on 2021-12-15

fix data-path for juju annotations cronjob

77. By Drew Freiberger on 2021-12-15

Rename annotations script

76. By Drew Freiberger on 2021-12-15

py3 updates for write-juju-annotations.py

75. By Drew Freiberger on 2021-12-15

Address import issues

74. By Drew Freiberger on 2021-12-15

Update juju-annotations to be an external root cron

=== modified file 'config.yaml'
--- config.yaml	2019-02-03 14:50:36 +0000
+++ config.yaml	2021-12-16 20:35:21 +0000
@@ -21,7 +21,7 @@
   origin:
     description: |
       *DEPRECATED*. See install_sources and install_keys
-      
+
       Origin of ppa or private deb repository from which to install
       landscape-client.  May be one of the following:
       distro (default), ppa:somecustom/ppa or a full APT url source
@@ -131,4 +131,15 @@
       regardless of the setting created by the unattended-upgrades package.
     type: boolean
     default: False
-
+  juju-annotations-enabled:
+    description: |
+      Periodically updates landscape annotations (key:value) for each juju agent
+      found on the machine and the juju availability-zone (AZ) of the machine.
+      Key is 'juju-${application}' OR 'juju-az-${AZ}' if an AZ is set.
+      Additionally, annotations for 'juju-${application}-${unit_number}' will
+      be set. Values are 'deployed=true' OR 'deployed=false' denoting presence
+      of these agents.
+      Example: juju-landscape-client:deployed=true
+      Cronjob runs every 5 min, update frequency depends on exchange-interval.
+    type: boolean
+    default: False

=== modified file 'hooks/hooks.py'
--- hooks/hooks.py	2019-04-04 16:07:48 +0000
+++ hooks/hooks.py	2021-12-16 20:35:21 +0000
@@ -18,7 +18,8 @@
 from ceph import (
     get_ceph_client_path, write_ceph_client_keyring, write_ceph_client_config)
 
-from install import install
+from install import (
+    install, create_annotations_cronjob, remove_annotations_cronjob)
 
 CERT_FILE = "/etc/ssl/certs/landscape_server_ca.crt"
 
@@ -62,6 +63,12 @@
             else:
                 set_config[key] = value
 
+    # Manage toggling of juju-annotations cronjob
+    if service_config["juju-annotations-enabled"]:
+        create_annotations_cronjob()
+    else:
+        remove_annotations_cronjob()
+
     relation_data.update(set_config)
     if service_config.get("disable-unattended-upgrades"):
         Cnf = apt_pkg.Configuration()

=== modified file 'hooks/install.py'
--- hooks/install.py	2019-02-03 14:50:36 +0000
+++ hooks/install.py	2021-12-16 20:35:21 +0000
@@ -7,14 +7,19 @@
 
 import subprocess
 import sys
+from os import environ, remove
 
 from charmhelpers import fetch
 import charmhelpers.core.hookenv as hookenv
+import charmhelpers.core.host as host
 
 
 hooks = hookenv.Hooks()
 
 
+ANNOTATIONS_CRON_PATH = '/etc/cron.d/landscape-client-juju-annotations'
+
+
 @hooks.hook("install")
 def install(fetch=fetch, hookenv=hookenv):
     charm_config = hookenv.config()
@@ -34,6 +39,11 @@
             "install", "-o", "landscape", "-g", "root", "-m", "755", "-d",
             data_path])
 
+    if charm_config["juju-annotations-enabled"]:
+        create_annotations_cronjob()
+    else:
+        remove_annotations_cronjob()
+
 
 def add_apt_source(url, fetch=fetch):
     """Add an apt source entry, with passed in key.
@@ -54,6 +64,30 @@
     return fetch.add_source(url, key)
 
 
+def create_annotations_cronjob():
+    hookenv.log('Create juju-annotations cronjob', hookenv.DEBUG)
+    unit_name = hookenv.local_unit()
+    agent_name = 'unit-' + unit_name.replace('/', '-')
+    zone_env = ""
+    if environ['JUJU_AVAILABILITY_ZONE']:
+        az = environ['JUJU_AVAILABILITY_ZONE']
+        zone_env = "JUJU_AVAILABILITY_ZONE=%s" % az
+    data_path = hookenv.config().get("data-path")
+    if not data_path:
+        data_path = "/var/lib/landscape/client"
+    cronjob = (
+            "*/5 * * * * root %s "
+            "/var/lib/juju/agents/%s/charm/scripts/write_juju_annotations.py "
+            "%s\n"
+        ) % (zone_env, agent_name, data_path)
+    host.write_file(ANNOTATIONS_CRON_PATH, cronjob)
+
+
+def remove_annotations_cronjob():
+    hookenv.log('Remove juju-annotations cronjob', hookenv.DEBUG)
+    remove(ANNOTATIONS_CRON_PATH)
+
+
 if __name__ == '__main__':
     try:
         sys.exit(hooks.execute(sys.argv))

=== added directory 'scripts'
=== added file 'scripts/write_juju_annotations.py'
--- scripts/write_juju_annotations.py	1970-01-01 00:00:00 +0000
+++ scripts/write_juju_annotations.py	2021-12-16 20:35:21 +0000
@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+"""Update landscape-client juju info annotations.
+
+usage: [JUJU_AVAILABILITY_ZONE="zone1"] write-juju-annotations [lds-data-path]
+"""
+
+import sys
+import os
+import re
+
+
+JUJU_AGENT_DIR = "/var/lib/juju/agents"
+LDS_CLIENT_DATA_PATH = "/var/lib/landscape/client"
+
+
+def write_juju_annotations(argv):
+    """Write annotation files in the '$DATA_PATH/annotations.d' folder.
+
+    For each currently deployed juju unit and one annotation for the juju
+    availability zone (if JUJU_AVAILABILITY_ZONE is set). The
+    annotation-value/file-content is 'deployed=true'.
+
+    For previously-deployed/removed units change the annotation value to
+    'deployed=false'.
+
+    This is used in conjunction with lp:charm-landscape-tagger.
+    """
+    if len(argv) == 2:
+        data_path = argv[1]
+    else:
+        data_path = LDS_CLIENT_DATA_PATH
+    annotations_folder = os.path.join(data_path, 'annotations.d')
+    unit_regex = re.compile(r"^unit-(((\w|-)+)-\d+)")
+    prefix = 'juju-'
+
+    target_annotations = []
+    for agent in os.listdir(JUJU_AGENT_DIR):
+        result = unit_regex.match(agent)
+        if result:
+            unit = result[1]
+            app = result[2]
+            target_annotations.append(prefix + unit)
+            target_annotations.append(prefix + app)
+
+    current_annotations = \
+        [a for a in os.listdir(annotations_folder) if a.startswith(prefix)]
+
+    if "JUJU_AVAILABILITY_ZONE" in os.environ:
+        target_annotations.append(
+            f"juju-az-{os.environ['JUJU_AVAILABILITY_ZONE']}"
+        )
+
+    all_annotations = list(set(target_annotations + current_annotations))
+    for annotation in all_annotations:
+        annotation_file = os.path.join(annotations_folder, annotation)
+        with open(annotation_file, "w") as f:
+            if annotation in target_annotations:
+                f.write("deployed=true")
+            else:
+                f.write("deployed=false")
+
+
+if __name__ == '__main__':
+    write_juju_annotations(sys.argv)