Merge lp:~free.ekanayaka/landscape-charm/enable-juju-sync into lp:~landscape/landscape-charm/trunk

[Free Ekanayaka]

This branch enables the juju-sync process on the leader unit, if
the deployment is a standalone deployment.

In order to fix the main bug, I also had to fix a side bug, i.e. handling
the relation with the landscape-hosted charm, so we know which deployment
mode we're deployed with (and hence whether to fire juju-sync or not). Since
the final branch is not too big, I'm pushing everything in one go.

This branch enables the juju-sync process on the leader unit, if
the deployment is a standalone deployment.

In order to fix the main bug, I also had to fix a side bug, i.e. handling
the relation with the landscape-hosted charm, so we know which deployment
mode we're deployed with (and hence whether to fire juju-sync or not). Since
the final branch is not too big, I'm pushing everything in one go.

To test this branch grab the juju-deployer config from

https://pastebin.canonical.com/128932/

and deploy with

juju-deployer -vdW -w180 -c landscape.yaml landscape

You'll see that the landscape-server unit has RUN_JUJU_SYNC=yes in
its /etc/default/landscape-server file and that the juju-sync
process is actually running on it.

If you now run "juju add-unit landscape-server" and wait, the new
unit won't be running juju-sync, as it's not the leader.

Furthermore if you grab the landscape-hosted charm (lp:landscape-hosted-charm),
deploy it and relate it to landscape-server you'll see that the
leader unit no longer runs juju-sync.

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Success
Revno: 243
Branch: lp:~free.ekanayaka/landscape-charm/enable-juju-sync
Jenkins: https://ci.lscape.net/job/latch-test/398/

review: Approve (test results)

[Adam Collard (adam-collard)]

I got a conflict when merging into trunk (r241)

Text conflict in hooks/lib/tests/test_templates.py
1 conflicts encountered.

review: Needs Fixing

[lp:~free.ekanayaka/landscape-charm/enable-juju-sync]

244. By Free Ekanayaka on 2015-04-07

Merge from trunk, solve conflicts

[Free Ekanayaka (free.ekanayaka)]

> I got a conflict when merging into trunk (r241)
>
> Text conflict in hooks/lib/tests/test_templates.py
> 1 conflicts encountered.

Fixed.

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Success
Revno: 244
Branch: lp:~free.ekanayaka/landscape-charm/enable-juju-sync
Jenkins: https://ci.lscape.net/job/latch-test/412/

review: Approve (test results)

[Adam Collard (adam-collard)]

3 comments inline

review: Needs Information

[Alberto Donato (ack)]

Code looks good, one question inline (and some minor comments).

review: Needs Information

[Free Ekanayaka (free.ekanayaka)]

Thanks Adam, I've fixed/replied to all comments.

[lp:~free.ekanayaka/landscape-charm/enable-juju-sync]

245. By Free Ekanayaka on 2015-04-07

Use python to create symlinks

246. By Free Ekanayaka on 2015-04-07

Merge from trunk, fix conflicts

247. By Free Ekanayaka on 2015-04-07

Fix typo

[Free Ekanayaka (free.ekanayaka)]

Thanks Alberto, all fixed.

[🤖 Landscape Builder (landscape-builder)]

Command: make ci-test
Result: Success
Revno: 247
Branch: lp:~free.ekanayaka/landscape-charm/enable-juju-sync
Jenkins: https://ci.lscape.net/job/latch-test/415/

review: Approve (test results)

[Adam Collard (adam-collard)]

LGTM, +1 with a couple of docstring cleanups inline below.

review: Approve

[Alberto Donato (ack)]

Looks good, +1

minor comments inline

review: Approve

[Free Ekanayaka (free.ekanayaka)]

All fixed.

[lp:~free.ekanayaka/landscape-charm/enable-juju-sync]

248. By Free Ekanayaka on 2015-04-08

Address review comments

=== added file 'hooks/hosted-relation-changed'
--- hooks/hosted-relation-changed	1970-01-01 00:00:00 +0000
+++ hooks/hosted-relation-changed	2015-04-08 12:40:27 +0000
@@ -0,0 +1,9 @@
+#!/usr/bin/python
+import sys
+
+from lib.services import ServicesHook
+
+
+if __name__ == "__main__":
+    hook = ServicesHook()
+    sys.exit(hook())

=== added file 'hooks/lib/callbacks/filesystem.py'
--- hooks/lib/callbacks/filesystem.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/callbacks/filesystem.py	2015-04-08 12:40:27 +0000
@@ -0,0 +1,32 @@
+# Filesystem-related callbacks
+
+import os
+
+from charmhelpers.core.services.base import ManagerCallback
+
+CONFIGS_DIR = "/opt/canonical/landscape/configs"
+
+
+class EnsureConfigDir(ManagerCallback):
+    """Ensure that the config dir for the configured deployment mode exists.
+
+    XXX This is a temporary workaround till we make the Landscape server
+        code always look at the same location for configuration files.
+    """
+    def __init__(self, configs_dir=CONFIGS_DIR):
+        self._configs_dir = configs_dir
+
+    def __call__(self, manager, service_name, event_name):
+        service = manager.get_service(service_name)
+
+        # Lookup the deployment mode
+        for data in service.get("required_data"):
+            if "hosted" in data:
+                deployment_mode = data["hosted"][0]["deployment-mode"]
+                break
+
+        # Create a symlink for the config directory
+        config_link = os.path.join(self._configs_dir, deployment_mode)
+        if not os.path.exists(config_link):
+            standalone_dir = os.path.join(self._configs_dir, "standalone")
+            os.symlink(standalone_dir, config_link)

=== added file 'hooks/lib/callbacks/tests/test_filesystem.py'
--- hooks/lib/callbacks/tests/test_filesystem.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/callbacks/tests/test_filesystem.py	2015-04-08 12:40:27 +0000
@@ -0,0 +1,29 @@
+import os
+
+from fixtures import TempDir
+
+from charmhelpers.core.services.base import ServiceManager
+
+from lib.tests.helpers import HookenvTest
+from lib.callbacks.filesystem import EnsureConfigDir
+
+
+class EnsureConfigDirTest(HookenvTest):
+
+    with_hookenv_monkey_patch = True
+
+    def setUp(self):
+        super(EnsureConfigDirTest, self).setUp()
+        self.configs_dir = self.useFixture(TempDir())
+        self.callback = EnsureConfigDir(self.configs_dir.path)
+
+    def test_options(self):
+        """
+        The callback creates a config dir symlink if needed.
+        """
+        manager = ServiceManager([{
+            "service": "landscape",
+            "required_data": [{"hosted": [{"deployment-mode": "edge"}]}],
+        }])
+        self.callback(manager, "landscape", None)
+        self.assertIsNotNone(os.lstat(self.configs_dir.join("edge")))

=== added file 'hooks/lib/relations/hosted.py'
--- hooks/lib/relations/hosted.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/relations/hosted.py	2015-04-08 12:40:27 +0000
@@ -0,0 +1,37 @@
+from charmhelpers.core.services.helpers import RelationContext
+
+from lib.hook import HookError
+
+DEPLOYMENT_MODES = ("standalone", "edge", "staging", "production")
+
+
+class HostedRequirer(RelationContext):
+    """Relation data requirer for the 'hosted' interface.
+
+    This relation acquires information from the landscape-hosted subordinate,
+    which will affect local configuration (for instance 'deployment-mode').
+
+    If we're not related to any landscape-hosted subordinate, then this data
+    manager will simply fall back to stock data setting a 'standalone' mode.
+    """
+    name = "hosted"
+    interface = "landscape-hosted"
+    required_keys = [
+        "deployment-mode",  # Can be standalone/edge/staging/production.
+    ]
+
+    def get_data(self):
+        super(HostedRequirer, self).get_data()
+        data = self.get(self.name)
+        if data is None:
+            # This means that we're not currently related to landscape-hosted,
+            # so we set the deployment mode to standalone.
+            self.update({self.name: [{"deployment-mode": "standalone"}]})
+        else:
+            # We're related to landscape-hosted, and it's safe to assume that
+            # there's exactly one unit we're related to, since landscape-hosted
+            # is a subordinate charm.
+            deployment_mode = data[0]["deployment-mode"]
+            if deployment_mode not in DEPLOYMENT_MODES:
+                raise HookError(
+                    "Invalid deployment-mode '%s'" % deployment_mode)

=== added file 'hooks/lib/relations/tests/test_hosted.py'
--- hooks/lib/relations/tests/test_hosted.py	1970-01-01 00:00:00 +0000
+++ hooks/lib/relations/tests/test_hosted.py	2015-04-08 12:40:27 +0000
@@ -0,0 +1,61 @@
+from lib.tests.helpers import HookenvTest
+from lib.tests.sample import SAMPLE_HOSTED_DATA
+from lib.relations.hosted import HostedRequirer
+from lib.hook import HookError
+
+
+class HostedRequirerTest(HookenvTest):
+
+    with_hookenv_monkey_patch = True
+
+    def test_required_keys(self):
+        """
+        The HostedRequirer class defines all keys that are required to
+        be set on the cluster relation in order for the relation to be
+        considered ready.
+        """
+        self.assertItemsEqual(
+            ["deployment-mode"],
+            HostedRequirer.required_keys)
+
+    def test_not_related(self):
+        """
+        When the landscape-server service is not related to landscape-hosted
+        the deployment-mode is standalone.
+        """
+        relation = HostedRequirer()
+        self.assertTrue(relation.is_ready())
+        self.assertEqual("standalone", SAMPLE_HOSTED_DATA["deployment-mode"])
+        self.assertEqual([SAMPLE_HOSTED_DATA], relation["hosted"])
+
+    def test_related(self):
+        """
+        When the landscape-server service is related to landscape-hosted
+        the deployment-mode is the one set on the relation.
+        """
+        hosted_data = {"deployment-mode": "production"}
+        self.hookenv.relations = {
+            "hosted": {
+                "hosted:1": {
+                    "landscape-hosted/1": hosted_data,
+                }
+            }
+        }
+        relation = HostedRequirer()
+        self.assertTrue(relation.is_ready())
+        self.assertEqual([hosted_data], relation["hosted"])
+
+    def test_invalid_deployment_mode(self):
+        """
+        The deployment mode set on the relation must be a valid one, otherwise
+        an error is raised.
+        """
+        hosted_data = {"deployment-mode": "foo"}
+        self.hookenv.relations = {
+            "hosted": {
+                "hosted:1": {
+                    "landscape-hosted/1": hosted_data,
+                }
+            }
+        }
+        self.assertRaises(HookError, HostedRequirer)

=== modified file 'hooks/lib/services.py'
--- hooks/lib/services.py	2015-04-07 13:54:01 +0000
+++ hooks/lib/services.py	2015-04-08 12:40:27 +0000
@@ -12,7 +12,10 @@
 from lib.relations.haproxy import HAProxyProvider, OFFLINE_FOLDER
 from lib.relations.landscape import (
     LandscapeLeaderContext, LandscapeRequirer, LandscapeProvider)
+from lib.relations.hosted import HostedRequirer
 from lib.callbacks.scripts import SchemaBootstrap, LSCtl
+from lib.callbacks.filesystem import CONFIGS_DIR, EnsureConfigDir
+
 
 SERVICE_CONF = "/etc/landscape/service.conf"
 DEFAULT_FILE = "/etc/default/landscape-server"
@@ -26,11 +29,13 @@
     proceed with the configuration if ready.
     """
     def __init__(self, hookenv=hookenv, cluster=cluster, host=host,
-                 subprocess=subprocess, offline_dir=OFFLINE_FOLDER):
+                 subprocess=subprocess, configs_dir=CONFIGS_DIR,
+                 offline_dir=OFFLINE_FOLDER):
         super(ServicesHook, self).__init__(hookenv=hookenv)
         self._cluster = cluster
         self._host = host
         self._subprocess = subprocess
+        self._configs_dir = configs_dir
         self._offline_dir = offline_dir
 
     def _run(self):
@@ -51,6 +56,7 @@
                 LandscapeRequirer(leader_context),
                 PostgreSQLRequirer(),
                 RabbitMQRequirer(),
+                HostedRequirer(),
                 {"config": hookenv.config(),
                  "is_leader": is_leader},
             ],
@@ -61,6 +67,7 @@
                 render_template(
                     owner="landscape", group="root", perms=0o640,
                     source="landscape-server", target=DEFAULT_FILE),
+                EnsureConfigDir(configs_dir=self._configs_dir),
                 SchemaBootstrap(subprocess=self._subprocess),
             ],
             "start": LSCtl(subprocess=self._subprocess),

=== modified file 'hooks/lib/tests/sample.py'
--- hooks/lib/tests/sample.py	2015-03-26 09:02:12 +0000
+++ hooks/lib/tests/sample.py	2015-04-08 12:40:27 +0000
@@ -22,3 +22,7 @@
     "openid-provider-url": "http://openid-host/",
     "openid-logout-url": "http://openid-host/logout",
 }
+
+SAMPLE_HOSTED_DATA = {
+    "deployment-mode": "standalone",
+}

=== modified file 'hooks/lib/tests/test_services.py'
--- hooks/lib/tests/test_services.py	2015-04-07 13:54:01 +0000
+++ hooks/lib/tests/test_services.py	2015-04-08 12:40:27 +0000
@@ -1,10 +1,14 @@
+import os
+
+from fixtures import TempDir
+
 from charmhelpers.core import templating
 
 from lib.tests.helpers import HookenvTest
 from lib.tests.stubs import ClusterStub, HostStub, SubprocessStub
 from lib.tests.sample import (
     SAMPLE_DB_UNIT_DATA, SAMPLE_LEADER_CONTEXT_DATA, SAMPLE_AMQP_UNIT_DATA,
-    SAMPLE_CONFIG_OPENID_DATA)
+    SAMPLE_CONFIG_OPENID_DATA, SAMPLE_HOSTED_DATA)
 from lib.services import ServicesHook, SERVICE_CONF, DEFAULT_FILE
 from lib.tests.offline_fixture import OfflineDir
 
@@ -18,10 +22,12 @@
         self.cluster = ClusterStub()
         self.host = HostStub()
         self.subprocess = SubprocessStub()
-        self.offline_dir = self.useFixture(OfflineDir()).path
+        self.offline_dir = self.useFixture(OfflineDir())
+        self.configs_dir = self.useFixture(TempDir())
         self.hook = ServicesHook(
             hookenv=self.hookenv, cluster=self.cluster, host=self.host,
-            subprocess=self.subprocess, offline_dir=self.offline_dir)
+            subprocess=self.subprocess, offline_dir=self.offline_dir.path,
+            configs_dir=self.configs_dir.path)
 
         # XXX Monkey patch the templating API, charmhelpers doesn't sport
         #     any dependency injection here as well.
@@ -94,6 +100,7 @@
             "db": [SAMPLE_DB_UNIT_DATA],
             "leader": SAMPLE_LEADER_CONTEXT_DATA,
             "amqp": [SAMPLE_AMQP_UNIT_DATA],
+            "hosted": [SAMPLE_HOSTED_DATA],
             "config": {},
             "is_leader": True,
         }
@@ -111,6 +118,33 @@
         self.assertEqual(
             ["/usr/bin/lsctl", "restart"], call2[0])
 
+    def test_ready_with_non_standalone_deployment_mode(self):
+        """
+        If deployment-mode is set to 'edge' an appropriate config symlink will
+        be created
+        """
+        hosted_data = SAMPLE_HOSTED_DATA.copy()
+        hosted_data["deployment-mode"] = "edge"
+        self.hookenv.relations = {
+            "db": {
+                "db:1": {
+                    "postgresql/0": SAMPLE_DB_UNIT_DATA,
+                },
+            },
+            "amqp": {
+                "amqp:1": {
+                    "rabbitmq-server/0": SAMPLE_AMQP_UNIT_DATA,
+                },
+            },
+            "hosted": {
+                "hosted:1": {
+                    "landscape-hosted/0": hosted_data,
+                },
+            },
+        }
+        self.hook()
+        self.assertIsNotNone(os.lstat(self.configs_dir.join("edge")))
+
     def test_ready_with_openid_configuration(self):
         """
         OpenID configuration is passed in to service.conf generation if
@@ -135,6 +169,7 @@
             "leader": SAMPLE_LEADER_CONTEXT_DATA,
             "amqp": [SAMPLE_AMQP_UNIT_DATA],
             "config": SAMPLE_CONFIG_OPENID_DATA,
+            "hosted": [SAMPLE_HOSTED_DATA],
             "is_leader": True,
         }
 

=== modified file 'hooks/lib/tests/test_templates.py'
--- hooks/lib/tests/test_templates.py	2015-04-06 08:34:00 +0000
+++ hooks/lib/tests/test_templates.py	2015-04-08 12:40:27 +0000
@@ -4,7 +4,8 @@
 
 from lib.tests.helpers import TemplateTest
 from lib.tests.sample import (
-    SAMPLE_DB_UNIT_DATA, SAMPLE_LEADER_CONTEXT_DATA, SAMPLE_AMQP_UNIT_DATA)
+    SAMPLE_DB_UNIT_DATA, SAMPLE_LEADER_CONTEXT_DATA, SAMPLE_AMQP_UNIT_DATA,
+    SAMPLE_HOSTED_DATA)
 
 
 class ServiceConfTest(TemplateTest):
@@ -21,6 +22,7 @@
             "db": [SAMPLE_DB_UNIT_DATA],
             "amqp": [SAMPLE_AMQP_UNIT_DATA],
             "leader": SAMPLE_LEADER_CONTEXT_DATA,
+            "hosted": [SAMPLE_HOSTED_DATA],
             "config": {},
         }
         buffer = StringIO(self.template.render(context))
@@ -36,6 +38,7 @@
             "landscape-token", config.get("landscape", "secret-token"))
         self.assertFalse(config.has_option("landscape", "openid-provider-url"))
         self.assertFalse(config.has_option("landscape", "openid-logout-url"))
+        self.assertEqual("standalone", config.get("global", "deployment-mode"))
 
     def test_render_with_openid(self):
         """
@@ -46,6 +49,7 @@
             "db": [SAMPLE_DB_UNIT_DATA],
             "amqp": [SAMPLE_AMQP_UNIT_DATA],
             "leader": SAMPLE_LEADER_CONTEXT_DATA,
+            "hosted": [SAMPLE_HOSTED_DATA],
             "config": {
                 "openid-provider-url": "http://openid-host/",
                 "openid-logout-url": "http://openid-host/logout",
@@ -73,6 +77,8 @@
         on a particular unit.
         """
         context = {
+            "hosted": [SAMPLE_HOSTED_DATA],
+            "config": {},
             "is_leader": True,
         }
         buffer = StringIO(self.template.render(context)).readlines()
@@ -83,7 +89,58 @@
         On a non-leader unit, cron scripts are not enabled by default.
         """
         context = {
+            "hosted": [SAMPLE_HOSTED_DATA],
+            "config": {},
             "is_leader": False,
         }
         buffer = StringIO(self.template.render(context)).readlines()
         self.assertIn('RUN_CRON="no"\n', buffer)
+
+    def test_render_juju_sync(self):
+        """
+        If the landscape-server unit is the leader and we're in standalone
+        mode, juju-sync will be run.
+        """
+        context = {
+            "db": [SAMPLE_DB_UNIT_DATA],
+            "amqp": [SAMPLE_AMQP_UNIT_DATA],
+            "leader": SAMPLE_LEADER_CONTEXT_DATA,
+            "hosted": [SAMPLE_HOSTED_DATA],
+            "config": {},
+            "is_leader": True,
+        }
+        buffer = StringIO(self.template.render(context)).readlines()
+        self.assertIn('RUN_JUJU_SYNC="yes"\n', buffer)
+
+    def test_render_juju_sync_not_leader(self):
+        """
+        If the landscape-server unit is not the leader, juju-sync
+        won't be run.
+        """
+        context = {
+            "db": [SAMPLE_DB_UNIT_DATA],
+            "amqp": [SAMPLE_AMQP_UNIT_DATA],
+            "leader": SAMPLE_LEADER_CONTEXT_DATA,
+            "hosted": [SAMPLE_HOSTED_DATA],
+            "config": {},
+            "is_leader": False,
+        }
+        buffer = StringIO(self.template.render(context)).readlines()
+        self.assertIn('RUN_JUJU_SYNC="no"\n', buffer)
+
+    def test_render_juju_sync_not_standalone(self):
+        """
+        If the deployment mode is not standalone, juju-sync won't be run.
+        """
+        hosted_data = SAMPLE_HOSTED_DATA.copy()
+        hosted_data["deployment-mode"] = "production"
+        context = {
+            "db": [SAMPLE_DB_UNIT_DATA],
+            "amqp": [SAMPLE_AMQP_UNIT_DATA],
+            "leader": SAMPLE_LEADER_CONTEXT_DATA,
+            "hosted": [hosted_data],
+            "config": {},
+            "is_leader": True,
+        }
+        buffer = StringIO(self.template.render(context)).readlines()
+        self.assertIn('RUN_JUJU_SYNC="no"\n', buffer)

=== modified file 'templates/landscape-server'
--- templates/landscape-server	2015-04-06 08:34:00 +0000
+++ templates/landscape-server	2015-04-08 12:40:27 +0000
@@ -15,7 +15,11 @@
 RUN_PINGSERVER="yes"
 RUN_APISERVER="yes"
 RUN_PACKAGEUPLOADSERVER="no"
+{% if is_leader and hosted[0]["deployment-mode"] == "standalone" %}
+RUN_JUJU_SYNC="yes"
+{% else %}
 RUN_JUJU_SYNC="no"
+{% endif %}
 RUN_PACKAGESEARCH="no"
 
 # To run cron jobs on this server when RUN_ALL="no" set RUN_CRON to "yes".

=== modified file 'templates/service.conf'
--- templates/service.conf	2015-04-02 16:37:14 +0000
+++ templates/service.conf	2015-04-08 12:40:27 +0000
@@ -3,6 +3,7 @@
 [global]
 # Syslog address is either a socket path or server:port string.
 syslog-address = /dev/log
+deployment-mode = {{ hosted[0]["deployment-mode"] }}
 
 [stores]
 host = {{ postgres["host"] }}:{{ postgres["port"] }}