Merge lp:~andreserl/maas/rsyslog_enlistment into lp:~maas-committers/maas/trunk

[Andres Rodriguez]

Add rsyslog support cloud-init logging during enlistment and commissioning (and the ephemeral environment in general).

This changes re-enables rsyslog for the enlistment environment, commissioning and the rest of the ephemeral environments for cloud-init ONLY. This only works on Xenial+ due to the version of cloud-init, but will NOT create any regressions to previous versions as cloud-init will just ignore the setting.

[Mike Pontillo (mpontillo)]

Please get rid of the hard-coded port number before this lands and make it more DRY; I've suggested what to do below.

review: Needs Fixing

[Mike Pontillo (mpontillo)]

A couple minor corrections below.

[Andres Rodriguez (andreserl)]

Hi Mike,

responses inline!

[Mike Pontillo (mpontillo)]

Not getting the port number from /etc/services could be considered a bug, because you should be able to change that, restart your services, and have everything move to the correct port. But based on the config file you pointed me to at debian/extras/20-maas.conf, I agree that's problematic.

If you insist on hard-coding it, then please put it in a common location, such as at the top of compose_preseed.py; don't duplicate it in two places. For example:

# Default port for rsyslog usage in the ephemeral environment
DEFAULT_RSYSLOG_PORT = 514

(I was thinking of suggesting placing it in enum.py, but at least if you do it this way it's consistent with DEFAULT_PORT in maasserver/eventloop.py)

If you don't want to refactor this to have a common function to get the host:port, then I'd be okay with the compromise of grabbing the port number from a common location. But I would just go ahead and put an accessor function in compose_preseed.py to return just the port number in that case anyway, since then none of the other code has to change if/when we refactor it to grab the port number dynamically.

[Mike Pontillo (mpontillo)]

Looks better. Thanks for the fix.

review: Approve

[MAAS Lander (maas-lander)]

The attempt to merge lp:~andreserl/maas/rsyslog_enlistment into lp:maas failed. Below is the output from the failed tests.

Get:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [116 kB]
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [1,108 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,514 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/multiverse Sources [182 kB]
Get:8 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,434 kB]
Get:9 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,250 kB]
Get:10 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [146 kB]
Fetched 17.7 MB in 3s (5,098 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-1ubuntu1).
archdetect-deb is already the newest version (1.114ubuntu4).
authbind is already the newest version (2.1.1+nmu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-5).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-5).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu2).
dh-apport is already the newest version (...

[Andres Rodriguez (andreserl)]

review: Approve

=== modified file 'contrib/preseeds_v2/enlist_userdata'
--- contrib/preseeds_v2/enlist_userdata	2016-03-01 19:02:08 +0000
+++ contrib/preseeds_v2/enlist_userdata	2016-03-24 23:53:00 +0000
@@ -25,6 +25,10 @@
          primary:  "http://ports.ubuntu.com/ubuntu-ports"
          security: "http://ports.ubuntu.com/ubuntu-ports"
 
+rsyslog:
+   remotes:
+     maas: "{{syslog_host_port}}"
+
 power_state:
    delay: now
    mode: poweroff

=== modified file 'src/maasserver/compose_preseed.py'
--- src/maasserver/compose_preseed.py	2016-03-24 17:44:54 +0000
+++ src/maasserver/compose_preseed.py	2016-03-24 23:53:00 +0000
@@ -24,6 +24,9 @@
 )
 import yaml
 
+# Default port for RSYSLOG
+RSYSLOG_PORT = 514
+
 
 def get_apt_proxy_for_node(node):
     """Return the APT proxy for the `node`."""
@@ -38,6 +41,13 @@
         return None
 
 
+def get_rsyslog_host_port(node):
+    """Return the rsyslog host and port to use."""
+    # TODO: In the future, we can make this configurable
+    return "%s:%d" % (get_maas_facing_server_host(
+        node.get_boot_rack_controller()), RSYSLOG_PORT)
+
+
 def get_system_info():
     """Return the system info which includes the APT mirror information."""
     return {
@@ -184,6 +194,12 @@
                 'token_secret': token.secret,
             },
         },
+        # This configure rsyslog for the ephemeral environment
+        'rsyslog': {
+            'remotes': {
+                'maas': get_rsyslog_host_port(node),
+            }
+        },
     }
     # Add the system configuration information.
     cloud_config.update(get_system_info())

=== modified file 'src/maasserver/preseed.py'
--- src/maasserver/preseed.py	2016-03-24 17:44:54 +0000
+++ src/maasserver/preseed.py	2016-03-24 23:53:00 +0000
@@ -32,6 +32,7 @@
 from maasserver.compose_preseed import (
     compose_cloud_init_preseed,
     compose_preseed,
+    RSYSLOG_PORT,
 )
 from maasserver.enum import (
     FILESYSTEM_TYPE,
@@ -602,6 +603,7 @@
         'release': release,
         'server_host': server_host,
         'server_url': absolute_reverse('machines_handler', base_url=base_url),
+        'syslog_host_port': '%s:%d' % (server_host, RSYSLOG_PORT),
         'metadata_enlist_url': absolute_reverse('enlist', base_url=base_url),
         'enable_http_proxy': Config.objects.get_config('enable_http_proxy'),
         'http_proxy': Config.objects.get_config('http_proxy'),

=== modified file 'src/maasserver/tests/test_compose_preseed.py'
--- src/maasserver/tests/test_compose_preseed.py	2016-03-24 17:44:54 +0000
+++ src/maasserver/tests/test_compose_preseed.py	2016-03-24 23:53:00 +0000
@@ -114,6 +114,9 @@
             KeysEqual(
                 'consumer_key', 'endpoint', 'token_key', 'token_secret',
                 'type'))
+        self.assertThat(
+            preseed['rsyslog']['remotes'],
+            KeysEqual('maas'))
         self.assertSystemInfo(preseed)
 
     def test_compose_preseed_for_commissioning_node_has_header(self):

=== modified file 'src/maasserver/tests/test_preseed.py'
--- src/maasserver/tests/test_preseed.py	2016-03-24 17:44:54 +0000
+++ src/maasserver/tests/test_preseed.py	2016-03-24 23:53:00 +0000
@@ -427,7 +427,7 @@
             ['osystem', 'release', 'metadata_enlist_url', 'server_host',
              'server_url', 'main_archive_hostname', 'main_archive_directory',
              'ports_archive_hostname', 'ports_archive_directory',
-             'enable_http_proxy', 'http_proxy'],
+             'enable_http_proxy', 'http_proxy', 'syslog_host_port'],
             context)
 
     def test_get_preseed_context_archive_refs(self):