Merge ~raharper/curtin:feature/write-pollinate-user-data into curtin:master

[Ryan Harper]

Add pollinate user-agent configuration support.

The pollinate client in Ubuntu images supports custom data to be
appended to the user-agent value sent when contacting the pollen
server. This patch allows users to add any key/value pairs which
will be added to the pollinate client configuration.

By default, curtin will configure pollinate with the curtin version
and if being deployed by MAAS, the MAAS version. Users can also
disable configuration of the pollinate client.

Additionally fix the vmtest webhook logger to encode writes when
handling GETs.

Add pollinate user-agent configuration support

The pollinate client in Ubuntu images supports custom data to be
appended to the user-agent value sent when contacting the pollen
server. This patch allows users to add any key/value pairs which
will be added to the pollinate client configuration.

By default, curtin will configure pollinate with the curtin version
and if being deployed by MAAS, the MAAS version. Users can also
disable configuration of the pollinate client.

Additional fixes
vmtest:
  report_webhook_logger didn't encode writes when handling GET request

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b732e66eb53fbb2a494119f3d3756ef63f3be7f8
https://jenkins.ubuntu.com/server/job/curtin-ci/829/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/829
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/829
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/829
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/829

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/829/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

i don't love the pollinate daemon install.
is it guaranteed that pollinate daemon is running and available before pollinate runs ?

we really should change pollinate to log somewhere what its user-agent is, or provide 'pollinate --user-agent' and just print it.

[Ryan Harper (raharper)]

1) we install pollen during the curtin install
2) we restart the service in our boot scripts
3) we manually invoke pollinate in boot scripts

So, yes, we ensure local pollen server is running and then force a pollinate client connection to the localhost service.

If that happens, we can modify the vmtest.

[Scott Moser (smoser)]

fyi
https://code.launchpad.net/~smoser/pollinate/trunk.print-user-agent/+merge/337395

[Scott Moser (smoser)]

if we can get that into pollinate, then we should put this test into the base class, and always collect the output of 'pollinate --print-user-agent' and check it is either error or has what we expect.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:eb38070b9ea3b8df9ded7c605de909061bed611f
https://jenkins.ubuntu.com/server/job/curtin-ci/852/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/852
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/852
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/852
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/852

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/852/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

i think this is good, but lets just use --print-user-agent.
one other inlinecomment.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:941e606e02bac80918c1a24ea75100acc5d9abe9
https://jenkins.ubuntu.com/server/job/curtin-ci/858/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/858
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/858
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/858
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/858

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/858/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:3a38395ef703e7cc62f9619e8d58e5d065ac0b52
https://jenkins.ubuntu.com/server/job/curtin-ci/859/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/859
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/859
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/859
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/859

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/859/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Couple minor inlines remaining in case you haven't already fixed them.

review: Approve

[Ryan Harper (raharper)]

Thanks Chad,

I've taken your suggestions and added additional exception handling in get_maas_version.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c8e5b4faabb2051a56946f25eb47591bea2c9c8b
https://jenkins.ubuntu.com/server/job/curtin-ci/862/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/862
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/862
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/862
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/862

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/862/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:82350f997ccfe803ee75a7a7f983631c35459a87
https://jenkins.ubuntu.com/server/job/curtin-ci/866/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/866
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/866
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/866
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/866

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/866/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

we suggested to change the maas version to something with 'vmtest' in it to not be confusin.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:7c2f63b7d0555f481115f4ed5200ed523bee865b
https://jenkins.ubuntu.com/server/job/curtin-ci/869/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-amd64/869
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/869
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/869
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/869

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/869/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

LGTM! thanks for the changes to integration test data.

review: Approve

[Chad Smith (chad.smith)]

Thank you for your merge proposal.

Your branch has been set to 'Work in progress'.
Please set the branch back to 'Needs Review' after resolving the issues below.

Thanks again,
Your friendly neighborhood cloud-init robot.

Please fix the following issues:
------------------------------
No commit message in Launchpad
------------------------------

For more information, see commit message guidelines at
https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#do-these-things-for-each-feature-or-bug

review: Needs Fixing

[Chad Smith (chad.smith)]

Thank you for your merge proposal.

Your branch has been set to 'Work in progress'.
Please set the branch back to 'Needs Review' after resolving the issues below.

Thanks again,
Your friendly neighborhood curtin robot.

Please fix the following issues:
------------------------------
No commit message in Launchpad
------------------------------

For more information, see commit message guidelines at
https://cloudinit.readthedocs.io/en/latest/topics/hacking.html#do-these-things-for-each-feature-or-bug

review: Needs Fixing

diff --git a/curtin/commands/curthooks.py b/curtin/commands/curthooks.py
index 64e5180..9e51a65 100644
--- a/curtin/commands/curthooks.py
+++ b/curtin/commands/curthooks.py
@@ -16,8 +16,10 @@ from curtin import futil
 from curtin.log import LOG
 from curtin import swap
 from curtin import util
+from curtin import version as curtin_version
 from curtin.reporter import events
 from curtin.commands import apply_net, apt_config
+from curtin.url_helper import get_maas_version
 
 from . import populate_one_subcmd
 
@@ -717,6 +719,61 @@ def system_upgrade(cfg, target):
     util.system_upgrade(target=target)
 
 
+def inject_pollinate_user_agent_config(ua_cfg, target):
+    """Write out user-agent config dictionary to pollinate's
+    user-agent file (/etc/pollinate/add-user-agent) in target.
+    """
+    if not isinstance(ua_cfg, dict):
+        raise ValueError('ua_cfg is not a dictionary: %s', ua_cfg)
+
+    pollinate_cfg = util.target_path(target, '/etc/pollinate/add-user-agent')
+    comment = "# written by curtin"
+    content = "\n".join(["%s/%s %s" % (ua_key, ua_val, comment)
+                         for ua_key, ua_val in ua_cfg.items()]) + "\n"
+    util.write_file(pollinate_cfg, content=content)
+
+
+def handle_pollinate_user_agent(cfg, target):
+    """Configure the pollinate user-agent if provided configuration
+
+    pollinate:
+        user_agent: false  # disable writing out a user-agent string
+
+    # custom agent key/value pairs
+    pollinate:
+       user_agent:
+          key1: value1
+          key2: value2
+
+    No config will result in curtin fetching:
+      curtin version
+      maas version (via endpoint URL, if present)
+    """
+
+    pcfg = cfg.get('pollinate')
+    if not isinstance(pcfg, dict):
+        pcfg = {'user_agent': {}}
+
+    uacfg = pcfg.get('user_agent', {})
+    if uacfg is False:
+        return
+
+    # set curtin version
+    uacfg['curtin'] = curtin_version.version_string()
+
+    # maas configures a curtin reporting webhook handler with
+    # an endpoint URL.  This url is used to query the MAAS REST
+    # api to extract the exact maas version.
+    maas_reporting = cfg.get('reporting', {}).get('maas', None)
+    if maas_reporting:
+        endpoint = maas_reporting.get('endpoint')
+        maas_version = get_maas_version(endpoint)
+        if maas_version:
+            uacfg['maas'] = maas_version['version']
+
+    inject_pollinate_user_agent_config(uacfg, target)
+
+
 def handle_cloudconfig(cfg, base_dir=None):
     """write cloud-init configuration files into base_dir.
 
@@ -1012,6 +1069,12 @@ def curthooks(args):
             description="updating packages on target system"):
         system_upgrade(cfg, target)
 
+    with events.ReportEventStack(
+            name=stack_prefix + '/pollinate-user-agent',
+            reporting_enabled=True, level="INFO",
+            description="configuring pollinate user-agent on target system"):
+        handle_pollinate_user_agent(cfg, target)
+
     # If a crypttab file was created by block_meta than it needs to be copied
     # onto the target system, and update_initramfs() needs to be run, so that
     # the cryptsetup hooks are properly configured on the installed system and
diff --git a/curtin/url_helper.py b/curtin/url_helper.py
index 8343ebe..d4d43a9 100644
--- a/curtin/url_helper.py
+++ b/curtin/url_helper.py
@@ -117,6 +117,66 @@ def download(url, path, reporthook=None, data=None):
         wfp.close()
 
 
+def get_maas_version(endpoint):
+    """ Attempt to return the MAAS version via api calls to the specified
+        endpoint.
+
+        MAAS endpoint url looks like this:
+
+        http://10.245.168.2/MAAS/metadata/status/node-f0462064-20f6-11e5-990a-d4bed9a84493
+
+        We need the MAAS_URL, which is http://10.245.168.2
+
+        Returns a maas version dictionary:
+        {'subversion': '16.04.1',
+         'capabilities': ['networks-management', 'static-ipaddresses',
+                          'ipv6-deployment-ubuntu', 'devices-management',
+                          'storage-deployment-ubuntu',
+                          'network-deployment-ubuntu',
+                          'bridging-interface-ubuntu',
+                          'bridging-automatic-ubuntu'],
+         'version': '2.1.5+bzr5596-0ubuntu1'
+        }
+    """
+    # https://docs.ubuntu.com/maas/devel/en/api indicates that
+    # we leave 1.0 in here for maas 1.9 endpoints
+    MAAS_API_SUPPORTED_VERSIONS = ["1.0", "2.0"]
+
+    try:
+        parsed = urlparse(endpoint)
+    except AttributeError as e:
+        LOG.warn('Failed to parse endpoint URL: %s', e)
+        return None
+
+    maas_host = "%s://%s" % (parsed.scheme, parsed.netloc)
+    maas_api_version_url = "%s/MAAS/api/version/" % (maas_host)
+
+    try:
+        result = geturl(maas_api_version_url)
+    except UrlError as e:
+        LOG.warn('Failed to query MAAS API version URL: %s', e)
+        return None
+
+    api_version = result.decode('utf-8')
+    if api_version not in MAAS_API_SUPPORTED_VERSIONS:
+        LOG.warn('Endpoint "%s" API version "%s" not in MAAS supported'
+                 'versions: "%s"', endpoint, api_version,
+                 MAAS_API_SUPPORTED_VERSIONS)
+        return None
+
+    maas_version_url = "%s/MAAS/api/%s/version/" % (maas_host, api_version)
+    maas_version = None
+    try:
+        result = geturl(maas_version_url)
+        maas_version = json.loads(result.decode('utf-8'))
+    except UrlError as e:
+        LOG.warn('Failed to query MAAS version via URL: %s', e)
+    except (ValueError, TypeError):
+        LOG.warn('Failed to load MAAS version result: %s', result)
+
+    return maas_version
+
+
 def _get_headers(headers=None):
     allheaders = DEFAULT_HEADERS.copy()
     if headers is not None:
diff --git a/doc/topics/config.rst b/doc/topics/config.rst
index fdc524f..76e520d 100644
--- a/doc/topics/config.rst
+++ b/doc/topics/config.rst
@@ -23,6 +23,7 @@ Curtin's top level config keys are as follows:
 - kexec (``kexec``)
 - multipath (``multipath``)
 - network (``network``)
+- pollinate (``pollinate``)
 - power_state (``power_state``)
 - proxy (``proxy``)
 - reporting (``reporting``)
@@ -348,6 +349,36 @@ Configure networking (see Networking section for details).
            - type: dhcp4
 
 
+pollinate
+~~~~~~~~~
+Configure pollinate user-agent
+
+Curtin will automatically include Curtin's version in the pollinate user-agent.
+If a MAAS server is being used, Curtin will query the MAAS version and include
+this value as well.
+
+**user_agent**: [*<mapping>* | *<boolean>*]
+
+Mapping is a dictionary of key value pairs which will result in the string
+'key/value' being present in the pollinate user-agent string sent to the
+pollen server.
+
+Setting the ``user_agent`` value to false will disable writting of the
+user-agent string.
+
+**Example**::
+
+  pollinate:
+     user_agent:
+         curtin: 17.1-33-g92fbc491
+         maas: 2.1.5+bzr5596-0ubuntu1
+         machine: bob27
+         app: 63.12
+
+  pollinate:
+     user_agent: false
+
+
 power_state
 ~~~~~~~~~~~
 Curtin can configure the target machine into a specific power state after
diff --git a/examples/tests/pollinate-useragent.yaml b/examples/tests/pollinate-useragent.yaml
new file mode 100644
index 0000000..dcf373c
--- /dev/null
+++ b/examples/tests/pollinate-useragent.yaml
@@ -0,0 +1,4 @@
+pollinate:
+   user_agent:
+       curtin-vmtest-app1: 32.1.5+bzr5596-0ubuntu1
+       curtin-vmtest-app2: uc16.snap1/edge
diff --git a/tests/unittests/test_curthooks.py b/tests/unittests/test_curthooks.py
index e4360e8..a8275c7 100644
--- a/tests/unittests/test_curthooks.py
+++ b/tests/unittests/test_curthooks.py
@@ -808,4 +808,76 @@ class TestCurthooksWriteFiles(CiTestCase):
             path=cc_target + expected_cfg['file1']['path'],
             perms='0644')
 
+
+class TestCurthooksPollinate(CiTestCase):
+    def setUp(self):
+        super(TestCurthooksPollinate, self).setUp()
+        self.add_patch('curtin.version.version_string', 'mock_curtin_version')
+        self.add_patch('curtin.util.write_file', 'mock_write')
+        self.add_patch('curtin.commands.curthooks.get_maas_version',
+                       'mock_maas_version')
+        self.target = self.tmp_dir()
+
+    def test_handle_pollinate_user_agent_disable(self):
+        """ handle_pollinate_user_agent does nothing if disabled """
+        cfg = {'pollinate': {'user_agent': False}}
+        curthooks.handle_pollinate_user_agent(cfg, self.target)
+        self.assertEqual(0, self.mock_curtin_version.call_count)
+        self.assertEqual(0, self.mock_maas_version.call_count)
+        self.assertEqual(0, self.mock_write.call_count)
+
+    def test_handle_pollinate_user_agent_default(self):
+        """ handle_pollinate_user_agent checks curtin/maas version by default
+        """
+        cfg = {'reporting': {'maas': {'endpoint': 'http://127.0.0.1/foo'}}}
+        curthooks.handle_pollinate_user_agent(cfg, self.target)
+        self.assertEqual(1, self.mock_curtin_version.call_count)
+        self.assertEqual(1, self.mock_maas_version.call_count)
+        self.assertEqual(1, self.mock_write.call_count)
+
+    def test_handle_pollinate_user_agent_default_no_maas(self):
+        """ handle_pollinate_user_agent checks curtin version, skips maas """
+        cfg = {}
+        curthooks.handle_pollinate_user_agent(cfg, self.target)
+        self.assertEqual(1, self.mock_curtin_version.call_count)
+        self.assertEqual(0, self.mock_maas_version.call_count)
+        self.assertEqual(1, self.mock_write.call_count)
+
+    @patch('curtin.commands.curthooks.inject_pollinate_user_agent_config')
+    def test_handle_pollinate_user_agent_custom(self, mock_inject):
+        """ handle_pollinate_user_agent merges custom with default config """
+        self.mock_curtin_version.return_value = 'curtin-version'
+        cfg = {'pollinate': {'user_agent': {'myapp': 'myversion'}}}
+        curthooks.handle_pollinate_user_agent(cfg, self.target)
+        self.assertEqual(1, self.mock_curtin_version.call_count)
+        self.assertEqual(0, self.mock_maas_version.call_count)
+        expected_cfg = {
+            'curtin': 'curtin-version',
+            'myapp': 'myversion',
+        }
+        mock_inject.assert_called_with(expected_cfg, self.target)
+
+
+class TestCurthooksInjectPollinate(CiTestCase):
+    def setUp(self):
+        super(TestCurthooksInjectPollinate, self).setUp()
+        self.target = self.tmp_dir()
+        self.user_agent = os.path.join(self.target,
+                                       'etc/pollinate/add-user-agent')
+
+    def test_inject_ua_output(self):
+        cfg = {'mykey': 'myvalue', 'foobar': 127}
+        expected_content = [
+            "mykey/myvalue # written by curtin\n",
+            "foobar/127 # written by curtin\n"
+        ]
+        curthooks.inject_pollinate_user_agent_config(cfg, self.target)
+        content = open(self.user_agent).readlines()
+        for line in expected_content:
+            self.assertIn(line, content)
+
+    def test_inject_ua_raises_exception(self):
+        with self.assertRaises(ValueError):
+            curthooks.inject_pollinate_user_agent_config(None, self.target)
+
 # vi: ts=4 expandtab syntax=python
diff --git a/tests/unittests/test_url_helper.py b/tests/unittests/test_url_helper.py
index 8fb3565..9cc9a5d 100644
--- a/tests/unittests/test_url_helper.py
+++ b/tests/unittests/test_url_helper.py
@@ -1,6 +1,8 @@
 # This file is part of curtin. See LICENSE file for copyright and license info.
 
 import filecmp
+import json
+import mock
 
 from curtin import url_helper
 
@@ -22,3 +24,99 @@ class TestDownload(CiTestCase):
         url_helper.download("file://" + src_file, target_file)
         self.assertTrue(filecmp.cmp(src_file, target_file),
                         "Downloaded file differed from source file.")
+
+
+class TestGetMaasVersion(CiTestCase):
+    @mock.patch('curtin.url_helper.geturl')
+    def test_get_maas_version(self, mock_get_url):
+        """verify we fetch maas version from api """
+        host = '127.0.0.1'
+        endpoint = 'http://%s/MAAS/metadata/node-1230348484' % host
+        maas_version = {'version': '2.1.5+bzr5596-0ubuntu1'}
+        maas_api_version = '2.0'
+        mock_get_url.side_effect = iter([
+            maas_api_version.encode('utf-8'),
+            ("%s" % json.dumps(maas_version)).encode('utf-8'),
+        ])
+        result = url_helper.get_maas_version(endpoint)
+        self.assertEqual(maas_version, result)
+        mock_get_url.assert_has_calls([
+            mock.call('http://%s/MAAS/api/version/' % host),
+            mock.call('http://%s/MAAS/api/%s/version/' % (host,
+                                                          maas_api_version))
+        ])
+
+    @mock.patch('curtin.url_helper.LOG')
+    @mock.patch('curtin.url_helper.geturl')
+    def test_get_maas_version_unsupported(self, mock_get_url, mock_log):
+        """return value is None if endpoint returns unsupported MAAS version
+        """
+        host = '127.0.0.1'
+        endpoint = 'http://%s/MAAS/metadata/node-1230348484' % host
+        supported = ['1.0', '2.0']
+        maas_api_version = 'garbage'
+        mock_get_url.side_effect = iter([
+            maas_api_version.encode('utf-8'),
+        ])
+        result = url_helper.get_maas_version(endpoint)
+        self.assertIsNone(result)
+        self.assertTrue(mock_log.warn.called)
+        mock_log.warn.assert_called_with(
+            'Endpoint "%s" API version "%s" not in MAAS supported'
+            'versions: "%s"', endpoint, maas_api_version, supported)
+        mock_get_url.assert_has_calls([
+            mock.call('http://%s/MAAS/api/version/' % host),
+        ])
+
+    def test_get_maas_version_no_endpoint(self):
+        """verify we return None with invalid endpoint """
+        host = None
+        result = url_helper.get_maas_version(host)
+        self.assertEqual(None, result)
+
+    @mock.patch('curtin.url_helper.LOG')
+    @mock.patch('curtin.url_helper.geturl')
+    def test_get_maas_version_bad_json(self, mock_get_url, mock_log):
+        """return value is None if endpoint returns invalid json """
+        host = '127.0.0.1'
+        endpoint = 'http://%s/MAAS/metadata/node-1230348484' % host
+        maas_api_version = '2.0'
+        bad_json = '{1237'.encode('utf-8')
+        mock_get_url.side_effect = iter([
+            maas_api_version.encode('utf-8'),
+            bad_json,
+        ])
+        result = url_helper.get_maas_version(endpoint)
+        self.assertIsNone(result)
+        self.assertTrue(mock_log.warn.called)
+        mock_log.warn.assert_called_with(
+            'Failed to load MAAS version result: %s', bad_json)
+        mock_get_url.assert_has_calls([
+            mock.call('http://%s/MAAS/api/version/' % host),
+            mock.call('http://%s/MAAS/api/%s/version/' % (host,
+                                                          maas_api_version))
+        ])
+
+    @mock.patch('curtin.url_helper.LOG')
+    @mock.patch('curtin.url_helper.geturl')
+    def test_get_maas_version_network_error_returns_none(self, mock_get_url,
+                                                         mock_log):
+        """return value is None if endpoint fails to respond"""
+        host = '127.0.0.1'
+        endpoint = 'http://%s/MAAS/metadata/node-1230348484' % host
+        maas_api_version = '2.0'
+        exception = url_helper.UrlError('404')
+        mock_get_url.side_effect = iter([
+            maas_api_version.encode('utf-8'),
+            exception,
+        ])
+        result = url_helper.get_maas_version(endpoint)
+        self.assertIsNone(result)
+        self.assertTrue(mock_log.warn.called)
+        mock_log.warn.assert_called_with(
+            'Failed to query MAAS version via URL: %s', exception)
+        mock_get_url.assert_has_calls([
+            mock.call('http://%s/MAAS/api/version/' % host),
+            mock.call('http://%s/MAAS/api/%s/version/' % (host,
+                                                          maas_api_version))
+        ])
diff --git a/tests/vmtests/report_webhook_logger.py b/tests/vmtests/report_webhook_logger.py
index 954fa12..e95397c 100755
--- a/tests/vmtests/report_webhook_logger.py
+++ b/tests/vmtests/report_webhook_logger.py
@@ -76,7 +76,7 @@ class ServerHandler(http_server.SimpleHTTPRequestHandler):
         self._message = None
         self.send_response(200)
         self.end_headers()
-        self.wfile.write("content of %s\n" % self.path)
+        self.wfile.write(("content of %s\n" % self.path).encode('utf-8'))
 
     def do_POST(self):
         length = int(self.headers['Content-Length'])
diff --git a/tests/vmtests/test_pollinate_useragent.py b/tests/vmtests/test_pollinate_useragent.py
new file mode 100644
index 0000000..e5ad64d
--- /dev/null
+++ b/tests/vmtests/test_pollinate_useragent.py
@@ -0,0 +1,66 @@
+# This file is part of curtin. See LICENSE file for copyright and license info.
+
+from . import VMBaseClass
+from .releases import base_vm_classes as relbase
+
+import re
+import textwrap
+from unittest import SkipTest
+
+
+class TestPollinateUserAgent(VMBaseClass):
+    # Test configuring pollinate useragent
+    conf_file = "examples/tests/pollinate-useragent.yaml"
+    extra_disks = []
+    extra_nics = []
+    collect_scripts = VMBaseClass.collect_scripts + [textwrap.dedent("""
+        cd OUTPUT_COLLECT_D
+        find /etc/network/interfaces.d > find_interfacesd
+        cp -a /etc/pollinate etc_pollinate
+        pollinate --print-user-agent > pollinate_print_user_agent
+        """)]
+
+    def test_pollinate_user_agent(self):
+        self.output_files_exist(["pollinate_print_user_agent"])
+        agent_values = self.load_collect_file("pollinate_print_user_agent")
+        if len(agent_values) == 0:
+            pollver = re.search('pollinate\s(?P<version>\S+)',
+                                self.load_collect_file("debian-packages.txt"))
+            msg = ("pollinate client '%s' does not support "
+                   "--print-user-agent'" % pollver['version'])
+            raise SkipTest(msg)
+
+        # curtin version is always present
+        curtin_ua = "curtin/%s #from curtin" % self.get_curtin_version()
+        ua_values = self.load_collect_file("etc_pollinate/add-user-agent")
+        for line in ua_values.splitlines() + [curtin_ua]:
+            """Each line is:
+               key/value # comment goes here
+
+               or
+
+               key/value
+
+               So we break at the first space
+            """
+            ua_val = line.split()[0]
+            # escape + and . that are likely in maas/curtin version strings
+            regex = r'%s' % ua_val.replace('+', '\+').replace('.', '\.')
+            hit = re.search(regex, agent_values)
+            self.assertIsNotNone(hit)
+            self.assertEqual(ua_val, hit.group())
+
+
+class TrustyTestPollinateUserAgent(relbase.trusty, TestPollinateUserAgent):
+    __test__ = True
+
+
+class XenialTestPollinateUserAgent(relbase.xenial, TestPollinateUserAgent):
+    __test__ = True
+
+
+class BionicTestPollinateUserAgent(relbase.bionic, TestPollinateUserAgent):
+    __test__ = True
+
+
+# vi: ts=4 expandtab syntax=python