Merge lp:~cprov/adt-continuous-deployer/better-monitor into lp:adt-continuous-deployer

[Celso Providelo]

Refactoring monitor.py and also adding support for basic auto-deployed service listing with list.py script.

Refactoring monitor.py and also adding support for basic service listing via list.py:

{{{
stg-ue-ci-engineering@wendigo:~/adt-continuous-deployer$ ./list.py
* mojo-ue-adt-cloud-service:
        11a8f85b: Thu Apr 16 22:05:46 2015 (4 units)
* mojo-ue-adt-cloud-worker:
        e240a00e: Thu Apr 23 18:50:44 2015 (7 units)
* mojo-ue-adt-image-mapper:
        c1ec6cc8: Wed Apr 15 19:20:48 2015 (4 units)
* mojo-ue-adt-result-checker:
        13d54727: Wed Apr 15 21:35:30 2015 (2 units)
* mojo-ue-core-image-publisher:
        324e91c7: Thu Apr 23 04:15:42 2015 (2 units)
* mojo-ue-core-image-tester:
        057b67aa: Thu Apr 23 19:20:44 2015 (2 units)
* mojo-ue-core-image-watcher:
        1aeb45ff: Thu Apr 23 02:10:52 2015 (2 units)
* mojo-ue-core-result-checker:
        428911cd: Tue Apr 14 15:40:40 2015 (2 units)
}}}

Removal of obsolete deployments still supported by `./monitor.py mojo-ue-adt-cloud-worker -m 0`.

Also fixing some broken and ignored tests from the shortening-identifier change (must double-check tarmac config).

[lp:~cprov/adt-continuous-deployer/better-monitor]

45. By Celso Providelo on 2015-04-24

Sorted service list

[Paul Larson (pwlars)]

Nice, this will be really useful

review: Approve

[Evan (ev)]

Does pretty-printing the deployments and purging old deployments really belong in the same program, especially as the algorithm to decide what old environments grows in complexity?

See inline comment.

review: Needs Information

[Para Siva (psivaa)]

An inline comment about a typo for logstash env variable.
Also tested the 'list' option to be working.

[lp:~cprov/adt-continuous-deployer/better-monitor]

46. By Celso Providelo on 2015-04-27

Fixing typo.

[Celso Providelo (cprov)]

Psivaa, nice catch, typo fixed.

I still have to address Ev's comment/suggestion for preserving the old monitor.py behavior and creating a list.py script for the pretty-print task.

[lp:~cprov/adt-continuous-deployer/better-monitor]

47. By Celso Providelo on 2015-04-27

Dedicated script for listing auto-deployed services (list.py), monitor.py preserves previous behaviour.

[Celso Providelo (cprov)]

Ev and others,

I've restored monitor.py behaviour (refactored on top of the common nova functions) and introduced list.py for listing auto-deployed services nicely.

Let me know what do you think.

[Para Siva (psivaa)]

LGTM now, approving.
A docstring for list.py would be a bonus.

review: Approve

[lp:~cprov/adt-continuous-deployer/better-monitor]

48. By Celso Providelo on 2015-04-27

Docstring fixing.

[Francis Ginther (fginther)]

I like it

review: Approve

=== added file 'ci_automation/nova.py'
--- ci_automation/nova.py	1970-01-01 00:00:00 +0000
+++ ci_automation/nova.py	2015-04-27 18:16:41 +0000
@@ -0,0 +1,92 @@
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""CI Automation: nova helpers."""
+
+import io
+import os
+import stat
+import subprocess
+
+
+def nova_list():
+    """Default 'nova list' accessor."""
+    args = [
+        'nova',
+        'list',
+        '--minimal',
+    ]
+
+    output = subprocess.check_output(
+        args, stderr=subprocess.DEVNULL).decode('utf-8').strip()
+
+    return output
+
+
+def get_services_map(base_dir, nova_output=None):
+    """Return a services map dictionary.
+
+    'base_dir' is the path where deployment identifier can be found.
+
+    If 'nova_output' is not provided `nova_list` accessor is used to get
+    this information.
+
+    Returns a dictionary key-ed by service name (middle term of the mojo
+    stage used in the deployment, e.g. 'mojo-ue-core-image-watcher'):
+
+    {
+        'mojo-ue-core-image-watcher': {
+            'DEADBEEF': {
+                'path': <identifier_path>,
+                'epoch': <identifier_ST_MTIME>,
+                'units': [<unit_name>, <unit_name>, ...],
+            },
+            'ABCD1234': {
+                'path': <identifier_path>,
+                'epoch': <identifier_ST_MTIME>,
+                'units': [<unit_name>, <unit_name>, ...],
+            },
+        },
+    }
+
+    Manually deployed environments, i.e. no identifiers on disk, are ignored.
+    """
+    if nova_output is None:
+        nova_output = nova_list()
+
+    services = {}
+
+    for line in io.StringIO(nova_output).readlines()[3:-1]:
+        unit_name = line.split('|')[2].strip()
+        service_name = '-'.join(unit_name.split('-')[1:-3])
+        identifier = unit_name.split('-')[-3]
+
+        path = os.path.join(base_dir, identifier)
+        if not os.path.exists(path):
+            continue
+
+        service = services.setdefault(service_name, {})
+        deployment = service.setdefault(identifier, {})
+        units = deployment.setdefault('units', [])
+        units.append(unit_name)
+
+        if deployment.get('path') is None:
+            deployment.update({
+                'path': path,
+                'epoch': os.stat(path)[stat.ST_MTIME],
+            })
+
+    return services
 
=== modified file 'ci_automation/tests/test_branch.py'
--- ci_automation/tests/test_branch.py	2015-04-02 19:29:27 +0000
+++ ci_automation/tests/test_branch.py	2015-04-27 18:16:41 +0000
@@ -99,13 +99,13 @@
         self.assertEqual(
             ('\n'.join(['bob/devel\t1',
                         'lp:foo;revno=1\t1']),
-             '90945bed4b6e27d500ae1ae05c7bba1c'),
+             '7ebe5386'),
             get_identifier(self.branch, stage))
         self.branch_add_stage('the-builder/devel')
         self.assertEqual(
             ('\n'.join(['bob/devel\t1',
                         'lp:foo;revno=1\t1']),
-             '90945bed4b6e27d500ae1ae05c7bba1c'),
+             '7ebe5386'),
             get_identifier(self.branch, stage))
 
     def test_get_identifier_spec_changes(self):
@@ -115,7 +115,7 @@
         self.assertEqual(
             ('\n'.join(['bob/devel\t1',
                         'lp:foo;revno=1\t1']),
-             '90945bed4b6e27d500ae1ae05c7bba1c'),
+             '7ebe5386'),
             get_identifier(self.branch, stage))
         repo_path = os.path.join(self.branch, stage, 'repo')
         with open(repo_path, 'w'):
@@ -125,7 +125,7 @@
         self.assertEqual(
             ('\n'.join(['bob/devel\t2',
                         'lp:foo;revno=1\t1']),
-             'dce1f2c4a1a1634f49a4572fa8630765'),
+             'fd2e0853'),
             get_identifier(self.branch, stage))
 
     def test_get_identifier_collect_changes(self):
@@ -136,7 +136,7 @@
         self.assertEqual(
             ('\n'.join(['bob/devel\t2',
                         'lp:foo;revno=7\t7']),
-             'f4bdf7cb82d6f1a9fc725d1bf70eef75'),
+             '77ff4c59'),
             get_identifier(self.branch, stage))
         self.branch_update_collect(
             stage, '\n'.join(['foo lp:~cprov/foo;revno=7',
@@ -145,7 +145,7 @@
             ('\n'.join(['bob/devel\t3',
                         'lp:~cprov/foo;revno=7\t7',
                         'lp:bar;revno=5\t5']),
-             '261a6c824fa28680e7e0869194a12152'),
+             '7a1b03fa'),
             get_identifier(self.branch, stage))
 
 
 
=== renamed file 'test_monitor.py' => 'ci_automation/tests/test_nova.py'
--- test_monitor.py	2015-04-08 17:59:46 +0000
+++ ci_automation/tests/test_nova.py	2015-04-27 18:16:41 +0000
@@ -1,93 +1,102 @@
-import logging
+#!/usr/bin/env python3
+#
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+"""CI Automation tests for nova helpers."""
+
 import os
 import shutil
+import sys
 import tempfile
 import time
 import unittest
 
-from monitor import (
-    get_service_deployments,
-    get_service_info,
-    parse_nova_output,
-)
-
-
-class TestServiceWatcher(unittest.TestCase):
+
+HERE = os.path.abspath(os.path.dirname(__file__))
+sys.path.append(os.path.join(HERE, '../..'))
+
+from ci_automation.nova import get_services_map
+
+
+class TestNova(unittest.TestCase):
+
+    identifiers = (
+        '12345',
+        'abcdef',
+        '56789',
+    )
+    service_names = (
+        'mojo-ue-core-result-watcher',
+        'mojo-ue-core-image-watcher',
+        'mojo-ue-core-image-publisher',
+        'mojo-ue-core-image-tester',
+    )
+
     def setUp(self):
-        self.prefix = 'juju-mojo-ue'
-        self.identifiers = [
-            '12345',
-            'abcdef',
-            '56789',
-        ]
-        self.service_names = [
-            'core-result-watcher',
-            'core-image-watcher',
-            'core-image-publisher',
-            'core-image-tester',
-        ]
         self.base_dir = tempfile.mkdtemp()
         self.addCleanup(shutil.rmtree, self.base_dir)
-        self.unit_name_pattern = '{}-(.*)-(.*)-machine-\d+'.format(self.prefix)
-
-        # generate some fake 'nova list' output
+
+    def generate_identifiers(self):
+        """Generate deployment identifiers on disk."""
+        for ident in self.identifiers:
+            # get unique timestamps
+            time.sleep(.1)
+            os.makedirs(os.path.join(self.base_dir, ident))
+
+    def create_output(self):
+        """Return some fake 'nova list' output."""
         count = 0
-        self.output = ""
+        output = ""
         for ident in self.identifiers:
             count += 1
-
-            for service_name in self.service_names:
-                self.output = '{}| id-{} | {}-{}-{}-machine-0 |\n'.format(
-                    self.output, count, self.prefix, service_name, ident)
-
-        #self._create_tmp_files()
-
-    def _create_tmp_files(self):
-        for ident in self.identifiers:
-            # get unique timestamps
-            time.sleep(2)
-            os.makedirs(os.path.join(self.base_dir, ident))
-
-    def test_parse_nova_output_valid(self):
-
-        units = parse_nova_output(self.output, self.service_names[0])
-        logging.warn("JOE: units: %s", units)
-
-        self.assertEqual(len(self.identifiers), len(units))
-
-    def test_get_service_info_valid(self):
-        unit_name = '{}-{}-{}-machine-0'.format(
-            self.prefix,
-            self.service_names[0],
-            self.identifiers[0],
-        )
-
-        service_name, identifier = get_service_info(unit_name,
-                                                    self.unit_name_pattern)
-
-        self.assertEqual(self.service_names[0], service_name)
-        self.assertEqual(self.identifiers[0], identifier)
-
-    def test_get_service_info_invalid(self):
-        unit_name = '{}-{}-{}-machine-0'.format(
-            self.prefix,
-            self.service_names[0],
-            self.identifiers[0],
-        )
-
-        service_name, identifier = get_service_info("junk-%s" % unit_name,
-                                                    '')
-
-        self.assertEqual(None, service_name)
-        self.assertEqual(None, identifier)
-
-    def test_get_service_deployments(self):
-        service_deployments = get_service_deployments(self.service_names[0],
-                                                      self.output,
-                                                      self.unit_name_pattern)
-
-        self.assertEqual(1, len(service_deployments))
-        for key in service_deployments:
-            self.assertEqual(len(self.identifiers),
-                             len(service_deployments[key]),
-                             "mismatch for '{}'".format(key))
+            for name in self.service_names:
+                output += '| id-{} | juju-{}-{}-machine-0 |\n'.format(
+                    count, name, ident)
+        return output
+
+    def test_service_map_ignores_missing_identifiers(self):
+        # Deployments with no identifiers on disk are ignored (they were
+        # probably deployed manually with mojo.py, not with cd.py via cron).
+        service_map = get_services_map(
+            self.base_dir, nova_output=self.create_output())
+        self.assertEqual({}, service_map)
+
+    def test_service_map(self):
+        self.generate_identifiers()
+        service_map = get_services_map(
+            self.base_dir, nova_output=self.create_output())
+
+        # service_map is key-ed by services names.
+        self.assertEqual(
+            sorted(self.service_names), sorted(service_map.keys()))
+
+        # Each service points to a dictionary key-ed by the deployment
+        # identifier.
+        service_contents = service_map['mojo-ue-core-image-watcher']
+        self.assertEqual(
+            ['56789', 'abcdef'], sorted(service_contents.keys()))
+
+        # Each deployment contains 'path' (identifier file on disk), 'epoch'
+        # (identifier file mtime) and 'units' (list of full unit names,
+        # including the bootstrap node).
+        deployment_contents = service_contents['abcdef']
+        self.assertEqual(
+            ['epoch', 'path', 'units'], sorted(deployment_contents.keys()))
+
+
+if __name__ == '__main__':
+    unittest.main()
 
=== modified file 'ci_automation/utils.py'
--- ci_automation/utils.py	2015-04-02 19:29:27 +0000
+++ ci_automation/utils.py	2015-04-27 18:16:41 +0000
@@ -16,6 +16,8 @@
 
 """CI Automation: miscelaneous utilities."""
 
+import logging
+import os
 import subprocess
 import textwrap
 
@@ -61,3 +63,31 @@
                 {}"""
                 ''').format(stdout.decode(), stderr.decode())
         )
+
+
+def setup_logger():
+    """Return a configured logger instance.
+
+    Defaults to INFO level and if 'CI_LOGSTASH_HOST' environment variable
+    is set and python-logstash is reachable, adds a corresponding 'logstash'
+    handler.
+    """
+    logging.basicConfig(
+        format='%(asctime)s  %(name)s %(levelname)s: %(message)s')
+    logger = logging.getLogger()
+    logger.setLevel(logging.INFO)
+
+    # Optional remote logging via logstash.
+    logstash_host = os.environ.get('CI_LOGSTASH_HOST')
+    if logstash_host is not None:
+        try:
+            import logstash
+        except ImportError:
+            print('Follow the README instructions for installing '
+                  'python-logstash')
+            return 1
+        else:
+            logger.addHandler(
+                logstash.LogstashHandler(logstash_host, 5959, 1))
+
+    return logger
 
=== added file 'list.py'
--- list.py	1970-01-01 00:00:00 +0000
+++ list.py	2015-04-27 18:16:41 +0000
@@ -0,0 +1,56 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2015 Canonical
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+"""CI Automation: script for listing auto-deployed services."""
+
+import argparse
+import datetime
+import os
+import sys
+
+
+from ci_automation.nova import get_services_map
+from ci_automation.utils import setup_logger
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Check the number of deployments for a service")
+    parser.add_argument('--base-dir', '-b',
+                        default=os.path.expanduser("~/ci-cd-identifiers"),
+                        help="The directory where identifiers are stored "
+                             " (eg. ~/ci-cd-identifiers)")
+    args = parser.parse_args()
+    logger = setup_logger()
+
+    services = get_services_map(args.base_dir)
+    sorted_services = sorted(services.items(), key=lambda s: s[0])
+    for srv, deployments in sorted_services:
+        print('* {}:'.format(srv))
+        sorted_deployments = sorted(
+            deployments.items(), key=lambda d: d[1]['epoch'], reverse=True)
+        for ident, contents in sorted_deployments:
+            date_created = datetime.datetime.utcfromtimestamp(
+                contents['epoch']).ctime()
+            print('\t{}: {} ({} units)'.format(
+                ident, date_created, len(contents['units'])))
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())
 
=== modified file 'monitor.py'
--- monitor.py	2015-04-10 20:20:17 +0000
+++ monitor.py	2015-04-27 18:16:41 +0000
@@ -15,155 +15,51 @@
 # You should have received a copy of the GNU General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
+"""CI Automation: monitoring (purging obsolete deployments) services."""
 
 import argparse
-import subprocess
-import io
-import logging
-import operator
 import os
-import re
 import sys
 
-from stat import ST_MTIME
-
-
-def parse_args():
-    desc = "Check the number of deployments for a service"
-    parser = argparse.ArgumentParser(description=desc)
-    parser.add_argument('--base-dir', '-b',
-                        default=os.path.expanduser("~/ci-cd-identifiers"),
-                        help="The directory where identifiers are stored "
-                             " (eg. ~/ci-cd-identifiers)")
-    parser.add_argument('--max-deployments', '-m', type=int, default=2,
-                        help="The maximum number of deployments to keep.")
-    parser.add_argument('stage_dirname',
-                        help="the mojo stage directory name (eg. "
-                             "mojo-ue-core-image-tester")
+
+from ci_automation.nova import get_services_map
+from ci_automation.utils import setup_logger
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Check the number of deployments for a service")
+    parser.add_argument(
+        '--base-dir', '-b',
+        default=os.path.expanduser("~/ci-cd-identifiers"),
+        help="The directory where identifiers are stored "
+        " (eg. ~/ci-cd-identifiers)")
+    parser.add_argument(
+        '--max-deployments', '-m', type=int, default=2,
+        help="The maximum number of deployments to keep.")
+    parser.add_argument(
+        'stage_dirname',
+        help="the mojo stage directory name (eg. mojo-ue-core-image-tester")
 
     args = parser.parse_args()
-
-    return args
-
-
-def get_service_info(unit_name, unit_name_pattern):
-
-    unit_name_regex = re.compile(unit_name_pattern)
-    match = unit_name_regex.match(unit_name)
-    stage_dirname = None
-    identifier = None
-
-    if match and len(match.groups()) == 2:
-        stage_dirname = match.group(1)
-        identifier = match.group(2)
-
-    return (stage_dirname, identifier)
-
-
-def nova_list():
-    args = [
-        'nova',
-        'list',
-        '--minimal',
-    ]
-
-    output = subprocess.check_output(
-        args, stderr=subprocess.DEVNULL).decode('utf-8').strip()
-
-    return output
-
-
-def parse_nova_output(output, stage_dirname):
-    units = []
-    buf = io.StringIO(output)
-    for line in buf.readlines():
-        # skip everything but the instance entries
-        if stage_dirname not in line:
-            continue
-        data = line.split('|')
-        instance_id = data[1].strip()
-        unit_name = data[2].strip()
-        units.append({'id': instance_id, 'name': unit_name})
-
-    return units
-
-
-def check_deployments(service_deployments, max_deployments, base_dir, logger):
-    """
-    :param service_deployments: dictionary of the form:
-        {'stage_dirname': ['identifier_1', 'identifier_2', ...]}
-    """
-    for key in service_deployments:
-        if len(service_deployments[key]) > max_deployments:
-            data = []
-            for identifier in service_deployments[key]:
-                path = os.path.join(base_dir, identifier)
-                # Manually deployed environments (i.e. no identifiers
-                # recorded on disk) are ignored and left alone.
-                if not os.path.exists(path):
-                    continue
-                data.append((identifier, os.stat(path)[ST_MTIME], path))
-            data = sorted(data, key=operator.itemgetter(1), reverse=True)
-            for identifier, _, _ in data[max_deployments:]:
-                extra = {
-                    "service": "ci/cd",
-                    "deployment_status": "DESTROYED",
-                    "deployment_name": key,
-                    "deployment_identifier": identifier,
-                }
-                logger.info(
-                    "Destroying deployment %s", identifier, extra=extra)
-
-
-def get_service_deployments(stage_dirname, output, unit_name_pattern):
-    service_deployments = {stage_dirname: []}
-
-    units = parse_nova_output(output, stage_dirname)
-
-    for unit in units:
-        service_name, identifier = get_service_info(unit['name'],
-                                                    unit_name_pattern)
-        # skip unparsable unit names
-        if service_name is None or identifier is None:
-            continue
-
-        service_deployments[stage_dirname].append(identifier)
-
-    return service_deployments
-
-
-def main():
-    args = parse_args()
-
-    base_dir = args.base_dir
-    max_deployments = args.max_deployments
-    stage_dirname = args.stage_dirname
-
-    unit_name_pattern = 'juju-({})-(.*)-machine-\d+'.format(stage_dirname)
-
-    logging.basicConfig(
-        format='%(asctime)s  %(name)s %(levelname)s: %(message)s')
-    logger = logging.getLogger()
-    logger.setLevel(logging.INFO)
-
-    # Optional remote logging via logstash.
-    logstash_host = os.environ.get('CI_LOGSTASH_HOST')
-    if logstash_host is not None:
-        try:
-            import logstash
-        except ImportError:
-            print('Follow the README instructions for installing '
-                  'python-logstash')
-            return 1
-        else:
-            logger.addHandler(
-                logstash.LogstashHandler(logstash_host, 5959, 1))
-
-    output = nova_list()
-    service_deployments = get_service_deployments(
-        stage_dirname, output, unit_name_pattern)
-    check_deployments(
-        service_deployments, max_deployments, base_dir, logger)
+    logger = setup_logger()
+
+    deployments = get_services_map(args.base_dir).get(args.stage_dirname)
+    if deployments is None:
+        print('No deployments found!')
+        return
+
+    sorted_deployments = sorted(
+        deployments.items(), key=lambda d: d[1]['epoch'], reverse=True)
+    for identifier, _ in sorted_deployments[args.max_deployments:]:
+        extra = {
+            "service": "ci/cd",
+            "deployment_status": "DESTROYED",
+            "deployment_name": args.stage_dirname,
+            "deployment_identifier": identifier,
+        }
+        logger.info(
+            "Destroying deployment %s", identifier, extra=extra)
 
     return 0