Merge ~powersj/cloud-init:test-set-password-list into cloud-init:master

[Joshua Powers]

test: Add test for setting password as list

This adds an integration test for setting passwords when given
as a list, rather than a string. This also updates the docs and
tests so that Random is now RANDOM as is correct.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:2ded8b9e37bb65786655cee48fca3ff48ae1c585
https://jenkins.ubuntu.com/server/job/cloud-init-ci/115/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/115
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/115
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/115
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/115
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/115

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/115/rebuild

review: Approve (continuous-integration)

5be8342... by Joshua Powers on 2017-03-16

Adding in smoser's commits to both files

a881acb... by Joshua Powers on 2017-03-16

Adding in smoser's commits to both files

9cd40a2... by Joshua Powers on 2017-03-16

Revrting expire on accident

41bf34e... by Joshua Powers on 2017-03-16

Revrting expire on accident

[Joshua Powers (powersj)]

Pulled in smoser's changes from https://paste.ubuntu.com/24191229/ and applied them to both set_password_list and set_password_list_string

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:41bf34e63ecf25c9960fd820542d8919517e3c16
https://jenkins.ubuntu.com/server/job/cloud-init-ci/123/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/123/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/123/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/123/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/123/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/123/rebuild

review: Needs Fixing (continuous-integration)

d3c2e6c... by Joshua Powers on 2017-03-17

Fix flake8 errors

cc3aaf5... by Joshua Powers on 2017-03-17

Fix flake8 errors

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:cc3aaf5db84ee135409a2345570893dfbcd71d37
https://jenkins.ubuntu.com/server/job/cloud-init-ci/124/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/124
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/124
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/124
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/124

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/124/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

Josh, this looks great. Thank you.

The only thing I don't like now is the duplication.

There are 2 ways i can think of to reduce it:
a.) both tests extend a base class that has the test_* functions.
   this is just less flexible than 'b', but because these things are identical now, its the easiest thing.

I started to write in text how you could do this, but found that it was just as easy to do it myself and describe it that way. I think acceptable: http://paste.ubuntu.com/24195190/

The thing i'm not sure of is if we need to somehow make sure that the 'PasswordListTest' base class somehow doesnt run. In curtin vmtest we always did this with __test__ = False. I'm happy that it doesnt appear we need this, but I didn't yet run a full simple 'run' without specific -t

b.) some helper functions for testing this, and then the test_* functions will have call helper functions and have less in them.

aba194b... by Joshua Powers on 2017-03-17

Using class + subclass for password list

This reduces the duplication of test cases since we are doing the
exact same thing for each.

c95f455... by Joshua Powers on 2017-03-17

Using class + subclass for password list

This reduces the duplication of test cases since we are doing the
exact same thing for each.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c95f4552408d63519cd72ab0344e12099f57a015
https://jenkins.ubuntu.com/server/job/cloud-init-ci/126/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/126
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/126
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/126
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/126

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/126/rebuild

review: Approve (continuous-integration)

8451d0e... by Joshua Powers on 2017-03-17

Adding testcase as a parameter and enabling tests

c194276... by Joshua Powers on 2017-03-17

Adding testcase as a parameter and enabling tests

[Joshua Powers (powersj)]

I added class/subclass method you proposed, which I like a lot. There is no reason for duplicate code. My only concern is base.py filling up with a bunch of code, but I think that is ok.

However, I had to modify the parameters slightly so nose actually realizes there are tests to run and added the __test__ = True variable. Both were required for tests to be discovered and ran; I did try without each. I also added the description change that we recently added to curtin since we are now doing this.

Here is output of a run with those two tests:
https://paste.ubuntu.com/24196020/

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c1942769e99f2de99970800e6d86b326620f60c2
https://jenkins.ubuntu.com/server/job/cloud-init-ci/127/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/127
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/127
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/127
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/127

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/127/rebuild

review: Approve (continuous-integration)

d707e81... by Joshua Powers on 2017-03-17

Removing uncessary test case parameters

68b67d1... by Joshua Powers on 2017-03-17

Merge branch 'test-set-password-list' of git+ssh://git.launchpad.net/~powersj/cloud-init into test-set-password-list

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:68b67d18b10ea5fdf8e51d7946b0bf17e6edbf48
https://jenkins.ubuntu.com/server/job/cloud-init-ci/130/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/130
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/130
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/130
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/130

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/130/rebuild

review: Approve (continuous-integration)

[Scott Moser (smoser)]

thanks josh

review: Approve

diff --git a/cloudinit/config/cc_set_passwords.py b/cloudinit/config/cc_set_passwords.py
index 1611704..8440e59 100755
--- a/cloudinit/config/cc_set_passwords.py
+++ b/cloudinit/config/cc_set_passwords.py
@@ -47,7 +47,7 @@ enabled, disabled, or left to system defaults using ``ssh_pwauth``.
     chpasswd:
         list: |
             user1:password1
-            user2:Random
+            user2:RANDOM
             user3:password3
             user4:R
 
@@ -57,7 +57,7 @@ enabled, disabled, or left to system defaults using ``ssh_pwauth``.
     chpasswd:
         list:
             - user1:password1
-            - user2:Random
+            - user2:RANDOM
             - user3:password3
             - user4:R
 """
diff --git a/tests/cloud_tests/configs/modules/set_password_list.yaml b/tests/cloud_tests/configs/modules/set_password_list.yaml
index 3612904..a1eadd7 100644
--- a/tests/cloud_tests/configs/modules/set_password_list.yaml
+++ b/tests/cloud_tests/configs/modules/set_password_list.yaml
@@ -6,22 +6,26 @@ cloud_config: |
   ssh_pwauth: yes
   users:
     - name: tom
-      password: $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
+      # md5 gotomgo
+      passwd: "$1$S7$tT1BEDIYrczeryDQJfdPe0"
       lock_passwd: false
     - name: dick
-      password: $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
+      # md5 gocubsgo
+      passwd: "$1$ssisyfpf$YqvuJLfrrW6Cg/l53Pi1n1"
       lock_passwd: false
     - name: harry
-      password: $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
+      # sha512 goharrygo
+      passwd: "$6$LF$9Z2p6rWK6TNC1DC6393ec0As.18KRAvKDbfsGJEdWN3sRQRwpdfoh37EQ3yUh69tP4GSrGW5XKHxMLiKowJgm/"
       lock_passwd: false
     - name: jane
-      password: $1$xyz$sPMsLNmf66Ohl.ol6JvzE.
+      # sha256 gojanego
+      passwd: "$5$iW$XsxmWCdpwIW8Yhv.Jn/R3uk6A4UaicfW5Xp7C9p9pg."
       lock_passwd: false
   chpasswd:
-    list: |
-      tom:mypassword123!
-      dick:R
-      harry:Random
+    list:
+      - tom:mypassword123!
+      - dick:RANDOM
+      - harry:RANDOM
 collect_scripts:
   shadow: |
     #!/bin/bash
diff --git a/tests/cloud_tests/configs/modules/set_password_list_string.yaml b/tests/cloud_tests/configs/modules/set_password_list_string.yaml
new file mode 100644
index 0000000..cbb71be
--- /dev/null
+++ b/tests/cloud_tests/configs/modules/set_password_list_string.yaml
@@ -0,0 +1,37 @@
+#
+# Set password of list of users as a string
+#
+cloud_config: |
+  #cloud-config
+  ssh_pwauth: yes
+  users:
+    - name: tom
+      # md5 gotomgo
+      passwd: "$1$S7$tT1BEDIYrczeryDQJfdPe0"
+      lock_passwd: false
+    - name: dick
+      # md5 gocubsgo
+      passwd: "$1$ssisyfpf$YqvuJLfrrW6Cg/l53Pi1n1"
+      lock_passwd: false
+    - name: harry
+      # sha512 goharrygo
+      passwd: "$6$LF$9Z2p6rWK6TNC1DC6393ec0As.18KRAvKDbfsGJEdWN3sRQRwpdfoh37EQ3yUh69tP4GSrGW5XKHxMLiKowJgm/"
+      lock_passwd: false
+    - name: jane
+      # sha256 gojanego
+      passwd: "$5$iW$XsxmWCdpwIW8Yhv.Jn/R3uk6A4UaicfW5Xp7C9p9pg."
+      lock_passwd: false
+  chpasswd:
+    list: |
+      tom:mypassword123!
+      dick:RANDOM
+      harry:RANDOM
+collect_scripts:
+  shadow: |
+    #!/bin/bash
+    cat /etc/shadow
+  sshd_config: |
+    #!/bin/bash
+    grep '^PasswordAuth' /etc/ssh/sshd_config
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/base.py b/tests/cloud_tests/testcases/base.py
index 5395b9a..51ce2b4 100644
--- a/tests/cloud_tests/testcases/base.py
+++ b/tests/cloud_tests/testcases/base.py
@@ -2,6 +2,7 @@
 
 from cloudinit import util as c_util
 
+import crypt
 import json
 import unittest
 
@@ -14,6 +15,9 @@ class CloudTestCase(unittest.TestCase):
     conf = None
     _cloud_config = None
 
+    def shortDescription(self):
+        return None
+
     @property
     def cloud_config(self):
         """
@@ -78,4 +82,52 @@ class CloudTestCase(unittest.TestCase):
         result = self.get_status_data(self.get_data_file('result.json'))
         self.assertEqual(len(result['errors']), 0)
 
+
+class PasswordListTest(CloudTestCase):
+    def test_shadow_passwords(self):
+        shadow = self.get_data_file('shadow')
+        users = {}
+        dupes = []
+        for line in shadow.splitlines():
+            user, encpw = line.split(":")[0:2]
+            if user in users:
+                dupes.append(user)
+            users[user] = encpw
+
+        jane_enc = "$5$iW$XsxmWCdpwIW8Yhv.Jn/R3uk6A4UaicfW5Xp7C9p9pg."
+        self.assertEqual([], dupes)
+        self.assertEqual(jane_enc, users['jane'])
+
+        # shadow entry is $N$salt$, so we encrypt with the same format
+        # and salt and expect the result.
+        tom = "mypassword123!"
+        fmtsalt = users['tom'][0:users['tom'].rfind("$") + 1]
+        tom_enc = crypt.crypt(tom, fmtsalt)
+        self.assertEqual(tom_enc, users['tom'])
+
+        harry_enc = ("$6$LF$9Z2p6rWK6TNC1DC6393ec0As.18KRAvKDbfsG"
+                     "JEdWN3sRQRwpdfoh37EQ3yUh69tP4GSrGW5XKHxMLiKowJgm/")
+        dick_enc = "$1$ssisyfpf$YqvuJLfrrW6Cg/l53Pi1n1"
+
+        # these should have been changed to random values.
+        self.assertNotEqual(harry_enc, users['harry'])
+        self.assertTrue(users['harry'].startswith("$"))
+        self.assertNotEqual(dick_enc, users['dick'])
+        self.assertTrue(users['dick'].startswith("$"))
+
+        self.assertNotEqual(users['harry'], users['dick'])
+
+    def test_shadow_expected_users(self):
+        """Test every tom, dick, and harry user in shadow"""
+        out = self.get_data_file('shadow')
+        self.assertIn('tom:', out)
+        self.assertIn('dick:', out)
+        self.assertIn('harry:', out)
+        self.assertIn('jane:', out)
+
+    def test_sshd_config(self):
+        """Test sshd config allows passwords"""
+        out = self.get_data_file('sshd_config')
+        self.assertIn('PasswordAuthentication yes', out)
+
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_password_list.py b/tests/cloud_tests/testcases/modules/set_password_list.py
index b764362..5671de1 100644
--- a/tests/cloud_tests/testcases/modules/set_password_list.py
+++ b/tests/cloud_tests/testcases/modules/set_password_list.py
@@ -4,22 +4,7 @@
 from tests.cloud_tests.testcases import base
 
 
-class TestPasswordList(base.CloudTestCase):
-    """Test password module"""
-
-    # TODO: Verify dick and harry passwords are random
-    # TODO: Verify tom's password was changed
-
-    def test_shadow(self):
-        """Test every tom, dick, and harry user in shadow"""
-        out = self.get_data_file('shadow')
-        self.assertIn('tom:', out)
-        self.assertIn('dick:', out)
-        self.assertIn('harry:', out)
-
-    def test_sshd_config(self):
-        """Test sshd config allows passwords"""
-        out = self.get_data_file('sshd_config')
-        self.assertIn('PasswordAuthentication yes', out)
+class TestPasswordList(base.PasswordListTest):
+    """Test password setting via list in chpasswd/list"""
 
 # vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/set_password_list_string.py b/tests/cloud_tests/testcases/modules/set_password_list_string.py
new file mode 100644
index 0000000..9e77c87
--- /dev/null
+++ b/tests/cloud_tests/testcases/modules/set_password_list_string.py
@@ -0,0 +1,10 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""cloud-init Integration Test Verify Script"""
+from tests.cloud_tests.testcases import base
+
+
+class TestPasswordListString(base.PasswordListTest):
+    """Test password setting via string in chpasswd/list"""
+
+# vi: ts=4 expandtab