Merge ~smoser/cloud-init:bug/1745663-use-lxd-console into cloud-init:master

[Scott Moser]

tests: add support for logs in with snap and future lxd 3.

This puts in place detection for if 'show-log' will work with lxc
client, and uses that if present. The 'lxc console --show-log' is
not expected to work until lxd/liblxc3.0. That should come in a
few months. The hope is that when that function arrives, this
code will move over to using it.

For other scenarios (all current lxd installs) this will now
support getting logs from a snap installed lxd or a package installed
lxd via the old 'lxc.console.logfile'.

If installed from snap, a platform error will be raised until
the user does:
   sudo mkdir --mode=1777 -p /var/snap/lxd/common/consoles

LP: #1745663

see commit message

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:d941491687592392bdfc71d9f2c28d6222732f47
https://jenkins.ubuntu.com/server/job/cloud-init-ci/740/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/740/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:1d6cda52bc1ad7f08512a09325ada547f09e9a73
https://jenkins.ubuntu.com/server/job/cloud-init-ci/743/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/743/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

LGTM

Tried this out locally and works much better :) thanks!

review: Approve

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:3aaf53c796e7a6e35b3f291dfda42a981fd67eac
https://jenkins.ubuntu.com/server/job/cloud-init-ci/747/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/747/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

Josh,
I added some hackish support for getting log output today.
It should work in both snap lxd (tested) and package lxd (untested) if the lxd is local. Right now this only works on a local lxd. I do hope at some point we could point it at a remote, but that is work to do.

The first time you run installed from snap, it will fail and raise exception like:
ERROR - stage: set up and collect data for os: xenial encountered error: Unable to log with snap lxc. Please run:
  sudo mkdir --mode=1777 -p /var/snap/lxd/common/consoles: unexpected error in platform.

I'll be happy to remove the log to local file at some later date.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:a9b594501674590ded8a282444215cfc905ea1a9
https://jenkins.ubuntu.com/server/job/cloud-init-ci/748/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/748/rebuild

review: Approve (continuous-integration)

[Joshua Powers (powersj)]

+1 - I'm good with this as well. It does seem a little hacky, but to get us to 3.0 that is fine and we still get console logs :)

First run (abbreviated output):
https://paste.ubuntu.com/26490439/

Second run:
https://paste.ubuntu.com/26490453/

Console on 2nd run:
http://paste.ubuntu.com/26490455/

review: Approve

diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index 5ea88e5..d4f9135 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -44,8 +44,9 @@ def collect_console(instance, base_dir):
     LOG.debug('getting console log for %s to %s', instance, logfile)
     try:
         data = instance.console_log()
-    except NotImplementedError:
-        data = b'instance.console_log: not implemented'
+    except NotImplementedError as e:
+        # args[0] is hacky, but thats all I see to get at the message.
+        data = b'NotImplementedError:' + e.args[0].encode()
     with open(logfile, "wb") as fp:
         fp.write(data)
 
diff --git a/tests/cloud_tests/platforms/lxd/instance.py b/tests/cloud_tests/platforms/lxd/instance.py
index d2d2a1f..0488da5 100644
--- a/tests/cloud_tests/platforms/lxd/instance.py
+++ b/tests/cloud_tests/platforms/lxd/instance.py
@@ -6,7 +6,9 @@ import os
 import shutil
 from tempfile import mkdtemp
 
-from cloudinit.util import subp, ProcessExecutionError
+from cloudinit.util import load_yaml, subp, ProcessExecutionError, which
+from tests.cloud_tests import LOG
+from tests.cloud_tests.util import PlatformError
 
 from ..instances import Instance
 
@@ -15,6 +17,8 @@ class LXDInstance(Instance):
     """LXD container backed instance."""
 
     platform_name = "lxd"
+    _console_log_method = None
+    _console_log_file = None
 
     def __init__(self, platform, name, properties, config, features,
                  pylxd_container):
@@ -30,8 +34,8 @@ class LXDInstance(Instance):
         super(LXDInstance, self).__init__(
             platform, name, properties, config, features)
         self.tmpd = mkdtemp(prefix="%s-%s" % (type(self).__name__, name))
-        self._setup_console_log()
         self.name = name
+        self._setup_console_log()
 
     @property
     def pylxd_container(self):
@@ -39,21 +43,6 @@ class LXDInstance(Instance):
         self._pylxd_container.sync()
         return self._pylxd_container
 
-    def _setup_console_log(self):
-        logf = os.path.join(self.tmpd, "console.log")
-
-        # doing this ensures we can read it. Otherwise it ends up root:root.
-        with open(logf, "w") as fp:
-            fp.write("# %s\n" % self.name)
-
-        cfg = "lxc.console.logfile=%s" % logf
-        orig = self._pylxd_container.config.get('raw.lxc', "")
-        if orig:
-            orig += "\n"
-        self._pylxd_container.config['raw.lxc'] = orig + cfg
-        self._pylxd_container.save()
-        self._console_log_file = logf
-
     def _execute(self, command, stdin=None, env=None):
         if env is None:
             env = {}
@@ -97,19 +86,80 @@ class LXDInstance(Instance):
         """
         self.pylxd_container.files.put(remote_path, data)
 
+    @property
+    def console_log_method(self):
+        if self._console_log_method is not None:
+            return self._console_log_method
+
+        client = which('lxc')
+        if not client:
+            raise PlatformError("No 'lxc' client.")
+
+        elif _has_proper_console_support():
+            self._console_log_method = 'show-log'
+        elif client.startswith("/snap"):
+            self._console_log_method = 'logfile-snap'
+        else:
+            self._console_log_method = 'logfile-tmp'
+
+        LOG.debug("Set console log method to %s", self._console_log_method)
+        return self._console_log_method
+
+    def _setup_console_log(self):
+        method = self.console_log_method
+        if not method.startswith("logfile-"):
+            return
+
+        if method == "logfile-snap":
+            log_dir = "/var/snap/lxd/common/consoles"
+            if not os.path.exists(log_dir):
+                raise PlatformError(
+                    "Unable to log with snap lxc.  Please run:\n"
+                    "  sudo mkdir --mode=1777 -p %s" % log_dir)
+        elif method == "logfile-tmp":
+            log_dir = "/tmp"
+        else:
+            raise PlatformError(
+                "Unexpected value for console method: %s" % method)
+
+        # doing this ensures we can read it. Otherwise it ends up root:root.
+        log_file = os.path.join(log_dir, self.name)
+        with open(log_file, "w") as fp:
+            fp.write("# %s\n" % self.name)
+
+        cfg = "lxc.console.logfile=%s" % log_file
+        orig = self._pylxd_container.config.get('raw.lxc', "")
+        if orig:
+            orig += "\n"
+        self._pylxd_container.config['raw.lxc'] = orig + cfg
+        self._pylxd_container.save()
+        self._console_log_file = log_file
+
     def console_log(self):
         """Console log.
 
-        @return_value: bytes of this instance’s console
+        @return_value: bytes of this instance's console
         """
-        if not os.path.exists(self._console_log_file):
-            raise NotImplementedError(
-                "Console log '%s' does not exist. If this is a remote "
-                "lxc, then this is really NotImplementedError.  If it is "
-                "A local lxc, then this is a RuntimeError."
-                "https://github.com/lxc/lxd/issues/1129")
-        with open(self._console_log_file, "rb") as fp:
-            return fp.read()
+
+        if self._console_log_file:
+            if not os.path.exists(self._console_log_file):
+                raise NotImplementedError(
+                    "Console log '%s' does not exist. If this is a remote "
+                    "lxc, then this is really NotImplementedError.  If it is "
+                    "A local lxc, then this is a RuntimeError."
+                    "https://github.com/lxc/lxd/issues/1129")
+            with open(self._console_log_file, "rb") as fp:
+                return fp.read()
+
+        try:
+            stdout, stderr = subp(
+                ['lxc', 'console', '--show-log', self.name], decode=False)
+            return stdout
+        except ProcessExecutionError as e:
+            raise PlatformError(
+                "console log",
+                "Console log failed [%d]: stdout=%s stderr=%s" % (
+                    e.exit_code, e.stdout, e.stderr))
 
     def reboot(self, wait=True):
         """Reboot instance."""
@@ -146,7 +196,37 @@ class LXDInstance(Instance):
         if self.platform.container_exists(self.name):
             raise OSError('container {} was not properly removed'
                           .format(self.name))
+        if self._console_log_file and os.path.exists(self._console_log_file):
+            os.unlink(self._console_log_file)
         shutil.rmtree(self.tmpd)
         super(LXDInstance, self).destroy()
 
+
+def _has_proper_console_support():
+    stdout, _ = subp(['lxc', 'info'])
+    info = load_yaml(stdout)
+    reason = None
+    if 'console' not in info.get('api_extensions', []):
+        reason = "LXD server does not support console api extension"
+    else:
+        dver = info.get('environment', {}).get('driver_version', "")
+        if dver.startswith("2.") or dver.startwith("1."):
+            reason = "LXD Driver version not 3.x+ (%s)" % dver
+        else:
+            try:
+                stdout, stderr = subp(['lxc', 'console', '--help'],
+                                      decode=False)
+                if not (b'console' in stdout and b'log' in stdout):
+                    reason = "no '--log' in lxc console --help"
+            except ProcessExecutionError as e:
+                reason = "no 'console' command in lxc client"
+
+    if reason:
+        LOG.debug("no console-support: %s", reason)
+        return False
+    else:
+        LOG.debug("console-support looks good")
+        return True
+
+
 # vi: ts=4 expandtab