Merge ~smoser/cloud-init:bug/1724354-debug-only-on-missing-json-schema into cloud-init:master

[Scott Moser]

schema: Log debug instead of warning when jsonschema is not available.

When operating in expected path, cloud-init should avoid logging with
warning. That causes 'WARNING' messages in /var/log/cloud-init.log.
By default, warnings also go to the console.

Since jsonschema is a optional dependency, and not present on xenial
and zesty, cloud-init should not warn there.

Also here:
* Add a test to integration tests to assert that there are no
  warnings in /var/log/cloud-init.log.
* Update one integration test that did show warning and the related
  documentation and examples.

LP: #1724354

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:526ae49fa43ba0528ce4ffc30cfa56a1fe433221
https://jenkins.ubuntu.com/server/job/cloud-init-ci/402/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/402/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:1dc8e693e88f31913e27bee780dc1bbee11f652d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/403/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/403/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:5a46b85c35ebeb25aa69eb1115b36dba861b2487
https://jenkins.ubuntu.com/server/job/cloud-init-ci/404/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/404/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:bc70e1578b9adcfd739342a2e6f3bf8d38ac26a5
https://jenkins.ubuntu.com/server/job/cloud-init-ci/405/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/405/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9a07f80e8e61c1d4d1ce4991e0a791c7ecf9fa35
https://jenkins.ubuntu.com/server/job/cloud-init-ci/406/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/406/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:24afb7d32ba3deb4097f8812122b5a7bf6279806
https://jenkins.ubuntu.com/server/job/cloud-init-ci/407/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/407/rebuild

review: Needs Fixing (continuous-integration)

[Chad Smith (chad.smith)]

Once CI passes on this I'm good.

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/408/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/408/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Ran tox locally and all good here. Looks like remote service timeouts for centos/yum etc as the failing points on CI tests. unrelated to this changeset.

  py27: commands succeeded
  py3: commands succeeded
  flake8: commands succeeded
  xenial: commands succeeded
  pylint: commands succeeded
  congratulations :)

review: Approve

diff --git a/cloudinit/config/cc_users_groups.py b/cloudinit/config/cc_users_groups.py
index b80d1d3..f363000 100644
--- a/cloudinit/config/cc_users_groups.py
+++ b/cloudinit/config/cc_users_groups.py
@@ -15,7 +15,8 @@ options, see the ``Including users and groups`` config example.
 Groups to add to the system can be specified as a list under the ``groups``
 key. Each entry in the list should either contain a the group name as a string,
 or a dictionary with the group name as the key and a list of users who should
-be members of the group as the value.
+be members of the group as the value. **Note**: Groups are added before users,
+so any users in a group list must already exist on the system.
 
 The ``users`` config key takes a list of users to configure. The first entry in
 this list is used as the default user for the system. To preserve the standard
diff --git a/cloudinit/config/schema.py b/cloudinit/config/schema.py
index bb291ff..ca7d0d5 100644
--- a/cloudinit/config/schema.py
+++ b/cloudinit/config/schema.py
@@ -74,7 +74,7 @@ def validate_cloudconfig_schema(config, schema, strict=False):
     try:
         from jsonschema import Draft4Validator, FormatChecker
     except ImportError:
-        logging.warning(
+        logging.debug(
             'Ignoring schema validation. python-jsonschema is not present')
         return
     validator = Draft4Validator(schema, format_checker=FormatChecker())
diff --git a/doc/examples/cloud-config-user-groups.txt b/doc/examples/cloud-config-user-groups.txt
index 9c5202f..0554d1f 100644
--- a/doc/examples/cloud-config-user-groups.txt
+++ b/doc/examples/cloud-config-user-groups.txt
@@ -1,8 +1,8 @@
 # Add groups to the system
-# The following example adds the ubuntu group with members foo and bar and
-# the group cloud-users.
+# The following example adds the ubuntu group with members 'root' and 'sys'
+# and the empty group cloud-users.
 groups:
-  - ubuntu: [foo,bar]
+  - ubuntu: [root,sys]
   - cloud-users
 
 # Add users to the system. Users are added after groups are added.
diff --git a/tests/cloud_tests/testcases/base.py b/tests/cloud_tests/testcases/base.py
index bb545ab..b2b5b4b 100644
--- a/tests/cloud_tests/testcases/base.py
+++ b/tests/cloud_tests/testcases/base.py
@@ -72,6 +72,10 @@ class CloudTestCase(unittest.TestCase):
         result = self.get_status_data(self.get_data_file('result.json'))
         self.assertEqual(len(result['errors']), 0)
 
+    def test_no_warnings_in_log(self):
+        """Warnings should not be found in the log."""
+        self.assertNotIn("WARN", self.get_data_file('cloud-init.log'))
+
 
 class PasswordListTest(CloudTestCase):
     """Base password test case class."""
diff --git a/tests/cloud_tests/testcases/examples/including_user_groups.py b/tests/cloud_tests/testcases/examples/including_user_groups.py
index 67af527..93b7a82 100644
--- a/tests/cloud_tests/testcases/examples/including_user_groups.py
+++ b/tests/cloud_tests/testcases/examples/including_user_groups.py
@@ -40,4 +40,10 @@ class TestUserGroups(base.CloudTestCase):
         out = self.get_data_file('user_cloudy')
         self.assertRegex(out, r'cloudy:x:[0-9]{3,4}:')
 
+    def test_user_root_in_secret(self):
+        """Test root user is in 'secret' group."""
+        user, _, groups = self.get_data_file('root_groups').partition(":")
+        self.assertIn("secret", groups.split(),
+                      msg="User root is not in group 'secret'")
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/examples/including_user_groups.yaml b/tests/cloud_tests/testcases/examples/including_user_groups.yaml
index 0aa7ad2..469d03c 100644
--- a/tests/cloud_tests/testcases/examples/including_user_groups.yaml
+++ b/tests/cloud_tests/testcases/examples/including_user_groups.yaml
@@ -8,7 +8,7 @@ cloud_config: |
   #cloud-config
   # Add groups to the system
   groups:
-    - secret: [foobar,barfoo]
+    - secret: [root]
     - cloud-users
 
   # Add users to the system. Users are added after groups are added.
@@ -24,7 +24,7 @@ cloud_config: |
     - name: barfoo
       gecos: Bar B. Foo
       sudo: ALL=(ALL) NOPASSWD:ALL
-      groups: cloud-users
+      groups: [cloud-users, secret]
       lock_passwd: true
     - name: cloudy
       gecos: Magic Cloud App Daemon User
@@ -49,5 +49,8 @@ collect_scripts:
   user_cloudy: |
     #!/bin/bash
     getent passwd cloudy
+  root_groups: |
+    #!/bin/bash
+    groups root
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/user_groups.py b/tests/cloud_tests/testcases/modules/user_groups.py
index 67af527..93b7a82 100644
--- a/tests/cloud_tests/testcases/modules/user_groups.py
+++ b/tests/cloud_tests/testcases/modules/user_groups.py
@@ -40,4 +40,10 @@ class TestUserGroups(base.CloudTestCase):
         out = self.get_data_file('user_cloudy')
         self.assertRegex(out, r'cloudy:x:[0-9]{3,4}:')
 
+    def test_user_root_in_secret(self):
+        """Test root user is in 'secret' group."""
+        user, _, groups = self.get_data_file('root_groups').partition(":")
+        self.assertIn("secret", groups.split(),
+                      msg="User root is not in group 'secret'")
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/user_groups.yaml b/tests/cloud_tests/testcases/modules/user_groups.yaml
index 71cc9da..22b5d70 100644
--- a/tests/cloud_tests/testcases/modules/user_groups.yaml
+++ b/tests/cloud_tests/testcases/modules/user_groups.yaml
@@ -7,7 +7,7 @@ cloud_config: |
   #cloud-config
   # Add groups to the system
   groups:
-    - secret: [foobar,barfoo]
+    - secret: [root]
     - cloud-users
 
   # Add users to the system. Users are added after groups are added.
@@ -23,7 +23,7 @@ cloud_config: |
     - name: barfoo
       gecos: Bar B. Foo
       sudo: ALL=(ALL) NOPASSWD:ALL
-      groups: cloud-users
+      groups: [cloud-users, secret]
       lock_passwd: true
     - name: cloudy
       gecos: Magic Cloud App Daemon User
@@ -48,5 +48,8 @@ collect_scripts:
   user_cloudy: |
     #!/bin/bash
     getent passwd cloudy
+  root_groups: |
+    #!/bin/bash
+    groups root
 
 # vi: ts=4 expandtab