Merge lp:~daniel-thewatkins/cloud-init/shim_fixes into lp:~cloud-init-dev/cloud-init/trunk

[Dan Watkins]

[lp:~daniel-thewatkins/cloud-init/shim_fixes]

1139. By Scott Moser on 2015-09-01

Handle symlink mount points in mount_cb.

The Azure data source now uses a /dev/disk symlink to identify devices,
but the dereferenced version of this appears in the mount table.
mount_cb therefore doesn't identify when a disk is already mounted, and
attempts to mount it a second time (which fails with NTFS).

1140. By Scott Moser on 2015-09-02

sync curtin reporting changes back to cloud-init

this pulls in the reporting framework that we'd been working on
in curtin back to cloud-init.

1141. By Scott Moser on 2015-09-03

reporting/events: catch a final remaining incorrect use of 'reporting'

1142. By Scott Moser on 2015-09-08

NoCloud: fix consumption of vendor-data

the content of vendordata was was being assigned to vendordata,
rather than vendordata_raw. The result was that it is not processed
for includes or part handlers or other things as it is in other
datasources.

1143. By Scott Moser on 2015-09-08

  power_state: support 'condition' argument

  if 'condition' is provided to config in power_state, then
  consult it before powering off.

  This allows the user to shut down only if a condition is met, and
  leave the system in a debuggable state otherwise.

  An example is as simple as:
   power_state:
     mode: poweroff
     condition: ['sh', '-c', '[ -f /disable-poweroff ]']

1144. By Scott Moser on 2015-09-11

snappy: enable ssh on snappy if ssh keys are provided or password auth

If the user provides ssh keys or requests password auth in their
user-data then we will enable ssh by default.

1145. By Ben Howard on 2015-09-15

Change Snappy SSH enabled default from false to 'auto' (LP: #1494816)

1146. By Scott Moser on 2015-09-17

webhook: report with json data

the handler was passing a dictionary to readurl
which was then passing that on to requests.request as 'data'.
the requests library would urlencode that, but we want the
json data posted instead.

[Scott Moser (smoser)]

Odd_Bloke, it seems maybe the part find_endpoint after 'No endpoint found in DHCP config.' could/should be its own method . and then some tests on it. maybe 'extract_value_from_leases_key' or something like that.

other thing, is if you're just reading your two test cases that you added, its very difficult to see how they're any different . since the only difference is in fact the ip address. at very lease a comment would be good there.

[lp:~daniel-thewatkins/cloud-init/shim_fixes]

1147. By Scott Moser on 2015-09-29

MAAS: fix issues with url_helper and oauth module

This would cause problems in the event that we actually had a bad
clock. We add a retry in the main (for test) also, to ensure that
the oauth timestamp fix gets in place.

1148. By Dan Watkins on 2015-10-09

Handle escaped quotes in WALinuxAgentShim.find_endpoint.

This fixes bug 1488891.

1149. By Dan Watkins on 2015-10-09

Handle colons in packed strings in WALinuxAgentShim.find_endpoint.

This fixes bug 1488896.

1150. By Dan Watkins on 2015-10-09

Convert test helper to staticmethod.

1151. By Dan Watkins on 2015-10-09

Refactor WALinuxAgentShim.find_endpoint to use a helper method for IP address unpacking.

1152. By Dan Watkins on 2015-10-09

Refactor tests to test helper method directly, and remove need for test helper.

Unmerged revisions

=== modified file 'cloudinit/sources/helpers/azure.py'
--- cloudinit/sources/helpers/azure.py	2015-05-08 15:52:58 +0000
+++ cloudinit/sources/helpers/azure.py	2015-10-09 13:01:25 +0000
@@ -216,6 +216,21 @@
             self.openssl_manager.clean_up()
 
     @staticmethod
+    def get_ip_from_lease_value(lease_value):
+        unescaped_value = lease_value.replace('\\', '')
+        if len(unescaped_value) > 4:
+            hex_string = ''
+            for hex_pair in unescaped_value.split(':'):
+                if len(hex_pair) == 1:
+                    hex_pair = '0' + hex_pair
+                hex_string += hex_pair
+            packed_bytes = struct.pack(
+                '>L', int(hex_string.replace(':', ''), 16))
+        else:
+            packed_bytes = unescaped_value.encode('utf-8')
+        return socket.inet_ntoa(packed_bytes)
+
+    @staticmethod
     def find_endpoint():
         LOG.debug('Finding Azure endpoint...')
         content = util.load_file('/var/lib/dhcp/dhclient.eth0.leases')
@@ -225,16 +240,7 @@
                 value = line.strip(' ').split(' ', 2)[-1].strip(';\n"')
         if value is None:
             raise Exception('No endpoint found in DHCP config.')
-        if ':' in value:
-            hex_string = ''
-            for hex_pair in value.split(':'):
-                if len(hex_pair) == 1:
-                    hex_pair = '0' + hex_pair
-                hex_string += hex_pair
-            value = struct.pack('>L', int(hex_string.replace(':', ''), 16))
-        else:
-            value = value.encode('utf-8')
-        endpoint_ip_address = socket.inet_ntoa(value)
+        endpoint_ip_address = WALinuxAgentShim.get_ip_from_lease_value(value)
         LOG.debug('Azure endpoint found at %s', endpoint_ip_address)
         return endpoint_ip_address
 

=== modified file 'tests/unittests/test_datasource/test_azure_helper.py'
--- tests/unittests/test_datasource/test_azure_helper.py	2015-05-15 20:28:24 +0000
+++ tests/unittests/test_datasource/test_azure_helper.py	2015-10-09 13:01:25 +0000
@@ -90,48 +90,64 @@
         self.assertRaises(Exception,
                           azure_helper.WALinuxAgentShim.find_endpoint)
 
-    def _build_lease_content(self, ip_address, use_hex=True):
-        ip_address_repr = ':'.join(
-            [hex(int(part)).replace('0x', '')
-             for part in ip_address.split('.')])
-        if not use_hex:
-            ip_address_repr = struct.pack(
-                '>L', int(ip_address_repr.replace(':', ''), 16))
-            ip_address_repr = '"{0}"'.format(ip_address_repr.decode('utf-8'))
+    @staticmethod
+    def _build_lease_content(encoded_address):
         return '\n'.join([
             'lease {',
             ' interface "eth0";',
-            ' option unknown-245 {0};'.format(ip_address_repr),
+            ' option unknown-245 {0};'.format(encoded_address),
             '}'])
 
+    def test_latest_lease_used(self):
+        encoded_addresses = ['5:4:3:2', '4:3:2:1']
+        file_content = '\n'.join([self._build_lease_content(encoded_address)
+                                  for encoded_address in encoded_addresses])
+        self.load_file.return_value = file_content
+        self.assertEqual(encoded_addresses[-1].replace(':', '.'),
+                         azure_helper.WALinuxAgentShim.find_endpoint())
+
+
+class TestExtractIpAddressFromLeaseValue(TestCase):
+
     def test_hex_string(self):
-        ip_address = '98.76.54.32'
-        file_content = self._build_lease_content(ip_address)
-        self.load_file.return_value = file_content
-        self.assertEqual(ip_address,
-                         azure_helper.WALinuxAgentShim.find_endpoint())
+        ip_address, encoded_address = '98.76.54.32', '62:4c:36:20'
+        self.assertEqual(
+            ip_address,
+            azure_helper.WALinuxAgentShim.get_ip_from_lease_value(
+                encoded_address
+            ))
 
     def test_hex_string_with_single_character_part(self):
-        ip_address = '4.3.2.1'
-        file_content = self._build_lease_content(ip_address)
-        self.load_file.return_value = file_content
-        self.assertEqual(ip_address,
-                         azure_helper.WALinuxAgentShim.find_endpoint())
+        ip_address, encoded_address = '4.3.2.1', '4:3:2:1'
+        self.assertEqual(
+            ip_address,
+            azure_helper.WALinuxAgentShim.get_ip_from_lease_value(
+                encoded_address
+            ))
 
     def test_packed_string(self):
-        ip_address = '98.76.54.32'
-        file_content = self._build_lease_content(ip_address, use_hex=False)
-        self.load_file.return_value = file_content
-        self.assertEqual(ip_address,
-                         azure_helper.WALinuxAgentShim.find_endpoint())
-
-    def test_latest_lease_used(self):
-        ip_addresses = ['4.3.2.1', '98.76.54.32']
-        file_content = '\n'.join([self._build_lease_content(ip_address)
-                                  for ip_address in ip_addresses])
-        self.load_file.return_value = file_content
-        self.assertEqual(ip_addresses[-1],
-                         azure_helper.WALinuxAgentShim.find_endpoint())
+        ip_address, encoded_address = '98.76.54.32', 'bL6 '
+        self.assertEqual(
+            ip_address,
+            azure_helper.WALinuxAgentShim.get_ip_from_lease_value(
+                encoded_address
+            ))
+
+    def test_packed_string_with_escaped_quote(self):
+        ip_address, encoded_address = '100.72.34.108', 'dH\\"l'
+        self.assertEqual(
+            ip_address,
+            azure_helper.WALinuxAgentShim.get_ip_from_lease_value(
+                encoded_address
+            ))
+
+    def test_packed_string_containing_a_colon(self):
+        ip_address, encoded_address = '100.72.58.108', 'dH:l'
+        self.assertEqual(
+            ip_address,
+            azure_helper.WALinuxAgentShim.get_ip_from_lease_value(
+                encoded_address
+            ))
 
 
 class TestGoalStateParsing(TestCase):