Merge ~chad.smith/cloud-init:ubuntu/xenial into cloud-init:ubuntu/xenial

[Chad Smith]

Sync tip of cloud-init master to get SRU-related zfs fixes from tip into Xenial-proposed.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:cfa0a509976cf8a0db7f54939f272204579ca3be
https://jenkins.ubuntu.com/server/job/cloud-init-ci/968/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/968/rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

Looks good, only delta is the pocket name:

% git diff raharper/ubuntu/xenial/sru-20180403-lp1759406 chad.smith-cloudinit/ubuntu/xenial
diff --git a/debian/changelog b/debian/changelog
index bb49b7a..a917a56 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,4 +1,4 @@
-cloud-init (18.2-3-gb27f713a-0ubuntu1~16.04.1) xenial; urgency=medium
+cloud-init (18.2-3-gb27f713a-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

   * debian/new-upstream-snapshot: Remove script, now maintained elsewhere.
   * New upstream snapshot. (LP: #1759406)
@@ -7,7 +7,7 @@ cloud-init (18.2-3-gb27f713a-0ubuntu1~16.04.1) xenial; urgency=medium
     - cc_resizefs, util: handle no /dev/zfs
     - doc: Fix links in OpenStack datasource documentation. [Dominic Schlegel]

- -- Ryan Harper <email address hidden> Tue, 03 Apr 2018 16:10:29 -0500
+ -- Chad Smith <email address hidden> Tue, 03 Apr 2018 14:29:11 -0600

 cloud-init (18.2-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

[Ryan Harper (raharper)]

OK, -proposed should be there after discussion in IRC with Chad.

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:d6441b7f42e1d8756d09f47ca4dc3854ad2ad6a9
https://jenkins.ubuntu.com/server/job/cloud-init-ci/973/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/973/rebuild

review: Approve (continuous-integration)

diff --git a/cloudinit/config/cc_resizefs.py b/cloudinit/config/cc_resizefs.py
index c8e1752..013e69b 100644
--- a/cloudinit/config/cc_resizefs.py
+++ b/cloudinit/config/cc_resizefs.py
@@ -251,6 +251,8 @@ def handle(name, cfg, _cloud, log, args):
     if fs_type == 'zfs':
         zpool = devpth.split('/')[0]
         devpth = util.get_device_info_from_zpool(zpool)
+        if not devpth:
+            return  # could not find device from zpool
         resize_what = zpool
 
     info = "dev=%s mnt_point=%s path=%s" % (devpth, mount_point, resize_what)
diff --git a/cloudinit/config/cc_users_groups.py b/cloudinit/config/cc_users_groups.py
index f363000..b215e95 100644
--- a/cloudinit/config/cc_users_groups.py
+++ b/cloudinit/config/cc_users_groups.py
@@ -34,16 +34,16 @@ config keys for an entry in ``users`` are as follows:
     - ``homedir``: Optional. Home dir for user. Default is ``/home/<username>``
     - ``inactive``: Optional. Mark user inactive. Default: false
     - ``lock_passwd``: Optional. Disable password login. Default: true
-    - ``no-create-home``: Optional. Do not create home directory. Default:
+    - ``no_create_home``: Optional. Do not create home directory. Default:
       false
-    - ``no-log-init``: Optional. Do not initialize lastlog and faillog for
+    - ``no_log_init``: Optional. Do not initialize lastlog and faillog for
       user. Default: false
-    - ``no-user-group``: Optional. Do not create group named after user.
+    - ``no_user_group``: Optional. Do not create group named after user.
       Default: false
     - ``passwd``: Hash of user password
-    - ``primary-group``: Optional. Primary group for user. Default to new group
+    - ``primary_group``: Optional. Primary group for user. Default to new group
       named after user.
-    - ``selinux-user``: Optional. SELinux user for user's login. Default to
+    - ``selinux_user``: Optional. SELinux user for user's login. Default to
       default SELinux user.
     - ``shell``: Optional. The user's login shell. The default is to set no
       shell, which results in a system-specific default being used.
@@ -51,9 +51,9 @@ config keys for an entry in ``users`` are as follows:
       a Snappy user through ``snap create-user``. If an Ubuntu SSO account is
       associated with the address, username and SSH keys will be requested from
       there. Default: none
-    - ``ssh-authorized-keys``: Optional. List of ssh keys to add to user's
+    - ``ssh_authorized_keys``: Optional. List of ssh keys to add to user's
       authkeys file. Default: none
-    - ``ssh-import-id``: Optional. SSH id to import for user. Default: none
+    - ``ssh_import_id``: Optional. SSH id to import for user. Default: none
     - ``sudo``: Optional. Sudo rule to use, or list of sudo rules to use.
       Default: none.
     - ``system``: Optional. Create user as system user with no home directory.
@@ -89,18 +89,18 @@ config keys for an entry in ``users`` are as follows:
           homedir: <home directory>
           inactive: <true/false>
           lock_passwd: <true/false>
-          no-create-home: <true/false>
-          no-log-init: <true/false>
-          no-user-group: <true/false>
+          no_create_home: <true/false>
+          no_log_init: <true/false>
+          no_user_group: <true/false>
           passwd: <password>
-          primary-group: <primary group>
-          selinux-user: <selinux username>
+          primary_group: <primary group>
+          selinux_user: <selinux username>
           shell: <shell path>
           snapuser: <email>
-          ssh-authorized-keys:
+          ssh_authorized_keys:
               - <key>
               - <key>
-          ssh-import-id: <id>
+          ssh_import_id: <id>
           sudo: <sudo config>
           system: <true/false>
           uid: <user id>
diff --git a/cloudinit/util.py b/cloudinit/util.py
index 0ab2c48..acdc0d8 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -2249,7 +2249,15 @@ def get_mount_info_freebsd(path):
 
 
 def get_device_info_from_zpool(zpool):
-    (zpoolstatus, err) = subp(['zpool', 'status', zpool])
+    # zpool has 10 second timeout waiting for /dev/zfs LP: #1760173
+    if not os.path.exists('/dev/zfs'):
+        LOG.debug('Cannot get zpool info, no /dev/zfs')
+        return None
+    try:
+        (zpoolstatus, err) = subp(['zpool', 'status', zpool])
+    except ProcessExecutionError as err:
+        LOG.warning("Unable to get zpool status of %s: %s", zpool, err)
+        return None
     if len(err):
         return None
     r = r'.*(ONLINE).*'
diff --git a/debian/changelog b/debian/changelog
index 2872eb9..7a26815 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,8 +1,14 @@
-cloud-init (18.2-0ubuntu1~16.04.2) UNRELEASED; urgency=medium
+cloud-init (18.2-4-g05926e48-0ubuntu1~16.04.1) xenial-proposed; urgency=medium
 
   * debian/new-upstream-snapshot: Remove script, now maintained elsewhere.
-
- -- Scott Moser <[e-mail]>  Wed, 28 Mar 2018 16:09:50 -0400
+  * New upstream snapshot. (LP: #1759406)
+    - tests: fix integration tests to support lxd 3.0 release
+    - correct documentation to match correct attribute name usage.
+      [Dominic Schlegel]
+    - cc_resizefs, util: handle no /dev/zfs
+    - doc: Fix links in OpenStack datasource documentation. [Dominic Schlegel]
+
+ -- Chad Smith <[e-mail]>  Tue, 03 Apr 2018 17:01:55 -0600
 
 cloud-init (18.2-0ubuntu1~16.04.1) xenial-proposed; urgency=medium
 
diff --git a/doc/examples/cloud-config-user-groups.txt b/doc/examples/cloud-config-user-groups.txt
index 0554d1f..7bca24a 100644
--- a/doc/examples/cloud-config-user-groups.txt
+++ b/doc/examples/cloud-config-user-groups.txt
@@ -10,20 +10,20 @@ users:
   - default
   - name: foobar
     gecos: Foo B. Bar
-    primary-group: foobar
+    primary_group: foobar
     groups: users
-    selinux-user: staff_u
+    selinux_user: staff_u
     expiredate: 2012-09-01
-    ssh-import-id: foobar
+    ssh_import_id: foobar
     lock_passwd: false
     passwd: $6$j212wezy$7H/1LT4f9/N3wpgNunhsIqtMj62OKiS3nyNwuizouQc3u7MbYCarYeAHWYPYb2FT.lbioDm2RrkJPb9BZMN1O/
   - name: barfoo
     gecos: Bar B. Foo
     sudo: ALL=(ALL) NOPASSWD:ALL
     groups: users, admin
-    ssh-import-id: None
+    ssh_import_id: None
     lock_passwd: true
-    ssh-authorized-keys:
+    ssh_authorized_keys:
       - <ssh pub key 1>
       - <ssh pub key 2>
   - name: cloudy
@@ -37,10 +37,10 @@ users:
 #   gecos: The user name's real name, i.e. "Bob B. Smith"
 #   homedir: Optional. Set to the local path you want to use. Defaults to
 #           /home/<username>
-#   primary-group: define the primary group. Defaults to a new group created
+#   primary_group: define the primary group. Defaults to a new group created
 #           named after the user.
 #   groups:  Optional. Additional groups to add the user to. Defaults to none
-#   selinux-user:  Optional. The SELinux user for the user's login, such as
+#   selinux_user:  Optional. The SELinux user for the user's login, such as
 #           "staff_u". When this is omitted the system will select the default
 #           SELinux user.
 #   lock_passwd: Defaults to true. Lock the password to disable password login
@@ -66,11 +66,11 @@ users:
 #               should use SSH authentication only.
 #
 #               You have thus been warned.
-#   no-create-home: When set to true, do not create home directory.
-#   no-user-group: When set to true, do not create a group named after the user.
-#   no-log-init: When set to true, do not initialize lastlog and faillog database.
-#   ssh-import-id: Optional. Import SSH ids
-#   ssh-authorized-keys: Optional. [list] Add keys to user's authorized keys file
+#   no_create_home: When set to true, do not create home directory.
+#   no_user_group: When set to true, do not create a group named after the user.
+#   no_log_init: When set to true, do not initialize lastlog and faillog database.
+#   ssh_import_id: Optional. Import SSH ids
+#   ssh_authorized_keys: Optional. [list] Add keys to user's authorized keys file
 #   sudo: Defaults to none. Set to the sudo string you want to use, i.e.
 #           ALL=(ALL) NOPASSWD:ALL. To add multiple rules, use the following
 #           format.
diff --git a/doc/rtd/topics/datasources/openstack.rst b/doc/rtd/topics/datasources/openstack.rst
index 607b70f..43592de 100644
--- a/doc/rtd/topics/datasources/openstack.rst
+++ b/doc/rtd/topics/datasources/openstack.rst
@@ -5,7 +5,7 @@ OpenStack
 
 This datasource supports reading data from the
 `OpenStack Metadata Service
-<http://docs.openstack.org/admin-guide/compute-networking-nova.html#metadata-service>`_.
+<https://docs.openstack.org/nova/latest/admin/networking-nova.html#metadata-service>`_.
 
 Configuration
 -------------
@@ -59,6 +59,6 @@ upgrade packages and install ``htop`` on all instances:
   {"cloud-init": "#cloud-config\npackage_upgrade: True\npackages:\n - htop"}
 
 For more general information about how cloud-init handles vendor data,
-including how it can be disabled by users on instances, see `Vendor Data`_.
+including how it can be disabled by users on instances, see :doc:`/topics/vendordata`.
 
 .. vi: textwidth=78
diff --git a/integration-requirements.txt b/integration-requirements.txt
index 45baac6..df3a73e 100644
--- a/integration-requirements.txt
+++ b/integration-requirements.txt
@@ -12,8 +12,8 @@ boto3==1.5.9
 paramiko==2.4.0
 
 # lxd backend
-# 01/10/2018: enables use of lxd as snap support
-git+https://github.com/lxc/pylxd.git@0722955260a6557e6d2ffde1896bfe0707bbca27
+# 04/03/2018: enables use of lxd 3.0
+git+https://github.com/lxc/pylxd.git@1a85a12a23401de6e96b1aeaf59ecbff2e88f49d
 
 
 # finds latest image information
diff --git a/tests/cloud_tests/platforms/lxd/instance.py b/tests/cloud_tests/platforms/lxd/instance.py
index 0488da5..0d957bc 100644
--- a/tests/cloud_tests/platforms/lxd/instance.py
+++ b/tests/cloud_tests/platforms/lxd/instance.py
@@ -210,7 +210,7 @@ def _has_proper_console_support():
         reason = "LXD server does not support console api extension"
     else:
         dver = info.get('environment', {}).get('driver_version', "")
-        if dver.startswith("2.") or dver.startwith("1."):
+        if dver.startswith("2.") or dver.startswith("1."):
             reason = "LXD Driver version not 3.x+ (%s)" % dver
         else:
             try:
diff --git a/tests/cloud_tests/testcases/examples/including_user_groups.yaml b/tests/cloud_tests/testcases/examples/including_user_groups.yaml
index 469d03c..77528d9 100644
--- a/tests/cloud_tests/testcases/examples/including_user_groups.yaml
+++ b/tests/cloud_tests/testcases/examples/including_user_groups.yaml
@@ -16,7 +16,7 @@ cloud_config: |
     - default
     - name: foobar
       gecos: Foo B. Bar
-      primary-group: foobar
+      primary_group: foobar
       groups: users
       expiredate: 2038-01-19
       lock_passwd: false
diff --git a/tests/cloud_tests/testcases/modules/user_groups.yaml b/tests/cloud_tests/testcases/modules/user_groups.yaml
index 22b5d70..675dfb8 100644
--- a/tests/cloud_tests/testcases/modules/user_groups.yaml
+++ b/tests/cloud_tests/testcases/modules/user_groups.yaml
@@ -15,7 +15,7 @@ cloud_config: |
     - default
     - name: foobar
       gecos: Foo B. Bar
-      primary-group: foobar
+      primary_group: foobar
       groups: users
       expiredate: 2038-01-19
       lock_passwd: false
diff --git a/tests/data/merge_sources/expected7.yaml b/tests/data/merge_sources/expected7.yaml
index 25284f0..d32988e 100644
--- a/tests/data/merge_sources/expected7.yaml
+++ b/tests/data/merge_sources/expected7.yaml
@@ -4,20 +4,20 @@ users:
   - default
   - name: foobar
     gecos: Foo B. Bar
-    primary-group: foobar
+    primary_group: foobar
     groups: users
-    selinux-user: staff_u
+    selinux_user: staff_u
     expiredate: 2012-09-01
-    ssh-import-id: foobar
+    ssh_import_id: foobar
     lock-passwd: false
     passwd: $6$j212wezy$7H/1LT4f9/N3wpgNunhsIqtMj62OKiS3nyNwuizouQc3u7MbYCarYeAHWYPYb2FT.lbioDm2RrkJPb9BZMN1O/
   - name: barfoo
     gecos: Bar B. Foo
     sudo: ALL=(ALL) NOPASSWD:ALL
     groups: users, admin
-    ssh-import-id: None
+    ssh_import_id: None
     lock-passwd: true
-    ssh-authorized-keys:
+    ssh_authorized_keys:
       - <ssh pub key 1>
       - <ssh pub key 2>
   - name: cloudy
@@ -29,10 +29,10 @@ users:
   - sue
   - name: foobar_jr
     gecos: Foo B. Bar Jr
-    primary-group: foobar
+    primary_group: foobar
     groups: users
-    selinux-user: staff_u
+    selinux_user: staff_u
     expiredate: 2012-09-01
-    ssh-import-id: foobar
+    ssh_import_id: foobar
     lock-passwd: false
     passwd: $6$j212wezy$7H/1LT4f9/N3wpgNunhsIqtMj62OKiS3nyNwuizouQc3u7MbYCarYeAHWYPYb2FT.lbioDm2RrkJPb9BZMN1O/
diff --git a/tests/data/merge_sources/source7-1.yaml b/tests/data/merge_sources/source7-1.yaml
index 8fb9b32..6405fc9 100644
--- a/tests/data/merge_sources/source7-1.yaml
+++ b/tests/data/merge_sources/source7-1.yaml
@@ -4,20 +4,20 @@ users:
   - default
   - name: foobar
     gecos: Foo B. Bar
-    primary-group: foobar
+    primary_group: foobar
     groups: users
-    selinux-user: staff_u
+    selinux_user: staff_u
     expiredate: 2012-09-01
-    ssh-import-id: foobar
+    ssh_import_id: foobar
     lock-passwd: false
     passwd: $6$j212wezy$7H/1LT4f9/N3wpgNunhsIqtMj62OKiS3nyNwuizouQc3u7MbYCarYeAHWYPYb2FT.lbioDm2RrkJPb9BZMN1O/
   - name: barfoo
     gecos: Bar B. Foo
     sudo: ALL=(ALL) NOPASSWD:ALL
     groups: users, admin
-    ssh-import-id: None
+    ssh_import_id: None
     lock-passwd: true
-    ssh-authorized-keys:
+    ssh_authorized_keys:
       - <ssh pub key 1>
       - <ssh pub key 2>
   - name: cloudy
diff --git a/tests/data/merge_sources/source7-2.yaml b/tests/data/merge_sources/source7-2.yaml
index 1e26201..0cd2897 100644
--- a/tests/data/merge_sources/source7-2.yaml
+++ b/tests/data/merge_sources/source7-2.yaml
@@ -6,11 +6,11 @@ users:
   - sue
   - name: foobar_jr
     gecos: Foo B. Bar Jr
-    primary-group: foobar
+    primary_group: foobar
     groups: users
-    selinux-user: staff_u
+    selinux_user: staff_u
     expiredate: 2012-09-01
-    ssh-import-id: foobar
+    ssh_import_id: foobar
     lock-passwd: false
     passwd: $6$j212wezy$7H/1LT4f9/N3wpgNunhsIqtMj62OKiS3nyNwuizouQc3u7MbYCarYeAHWYPYb2FT.lbioDm2RrkJPb9BZMN1O/
 
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index 8685b8e..5010190 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -366,8 +366,11 @@ class TestMountinfoParsing(helpers.ResourceUsingTestCase):
         expected = ('none', 'tmpfs', '/run/lock')
         self.assertEqual(expected, util.parse_mount_info('/run/lock', lines))
 
+    @mock.patch('cloudinit.util.os')
     @mock.patch('cloudinit.util.subp')
-    def test_get_device_info_from_zpool(self, zpool_output):
+    def test_get_device_info_from_zpool(self, zpool_output, m_os):
+        # mock /dev/zfs exists
+        m_os.path.exists.return_value = True
         # mock subp command from util.get_mount_info_fs_on_zpool
         zpool_output.return_value = (
             self.readResource('zpool_status_simple.txt'), ''
@@ -376,9 +379,31 @@ class TestMountinfoParsing(helpers.ResourceUsingTestCase):
         ret = util.get_device_info_from_zpool('vmzroot')
         self.assertEqual('gpt/system', ret)
         self.assertIsNotNone(ret)
+        m_os.path.exists.assert_called_with('/dev/zfs')
 
+    @mock.patch('cloudinit.util.os')
+    def test_get_device_info_from_zpool_no_dev_zfs(self, m_os):
+        # mock /dev/zfs missing
+        m_os.path.exists.return_value = False
+        # save function return values and do asserts
+        ret = util.get_device_info_from_zpool('vmzroot')
+        self.assertIsNone(ret)
+
+    @mock.patch('cloudinit.util.os')
+    @mock.patch('cloudinit.util.subp')
+    def test_get_device_info_from_zpool_handles_no_zpool(self, m_sub, m_os):
+        """Handle case where there is no zpool command"""
+        # mock /dev/zfs exists
+        m_os.path.exists.return_value = True
+        m_sub.side_effect = util.ProcessExecutionError("No zpool cmd")
+        ret = util.get_device_info_from_zpool('vmzroot')
+        self.assertIsNone(ret)
+
+    @mock.patch('cloudinit.util.os')
     @mock.patch('cloudinit.util.subp')
-    def test_get_device_info_from_zpool_on_error(self, zpool_output):
+    def test_get_device_info_from_zpool_on_error(self, zpool_output, m_os):
+        # mock /dev/zfs exists
+        m_os.path.exists.return_value = True
         # mock subp command from util.get_mount_info_fs_on_zpool
         zpool_output.return_value = (
             self.readResource('zpool_status_simple.txt'), 'error'