Merge ~chad.smith/cloud-init:bug/1780481-fix-get-linux-distro into cloud-init:master

[Chad Smith]

ubuntu,centos,debian: get_linux_distro to align with platform.dist

A recent commit added get_linux_distro to replace the deprecated python
platform.dist module behavior before it is dropped from python. It added
behavior that was compliant on OpenSuSE and SLES, by returning
(<distro_name>, <distro_version>, <cpu-arch>).

Fix get_linux_distro to behave more like the specific distribution's
platform.dist on ubuntu, centos and debian, which will return the
distribution release codename as the third element instead of <cpu-arch>.

SLES and OpenSUSE will retain their current behavior.

Examples follow:
('sles', '15', 'x86_64')
('opensuse', '42.3', 'x86_64')
('debian', '9', 'stretch')
('ubuntu', '16.04', 'xenial')
('centos', '7', 'Core')

LP: #1780481

[Robert Schweikert (rjschwei)]

LGTM +1

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:2099b4ce71ce2deba68e38b1443746ea0b125b9c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/141/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/141/rebuild

review: Needs Fixing (continuous-integration)

a2f229b... by Chad Smith on 2018-07-07

need to mock /etc/os-release for azure util.is_FreeBSD checks

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:a2f229bcd228816ff3f2047ae3f3b4fb93fc86ba
https://jenkins.ubuntu.com/server/job/cloud-init-ci/142/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/142/rebuild

review: Needs Fixing (continuous-integration)

b1ea4ac... by Chad Smith on 2018-07-07

not bytes mr smith

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:b1ea4acedb2a107f62fdcde639640ce2b8b007bc
https://jenkins.ubuntu.com/server/job/cloud-init-ci/143/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/143/rebuild

review: Approve (continuous-integration)

9ae57dd... by Chad Smith on 2018-07-09

shlex instead

[Scott Moser (smoser)]

2 comments
a.) it seems wrong to have sles and opensuse reatin their old behavior
b.) can get_linux_distro use load_shell_content ?

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:9ae57dd91778fd655f62e64e9d2e6adfa44c77dc
https://jenkins.ubuntu.com/server/job/cloud-init-ci/144/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/144/rebuild

review: Needs Fixing (continuous-integration)

a871140... by Chad Smith on 2018-07-09

use load_shell_content per smoser's review

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:a8711404dc56287eefdffd5e6a88ebc4de4cb250
https://jenkins.ubuntu.com/server/job/cloud-init-ci/145/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/145/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

looks good to me with a fix of c-i

review: Approve

4634539... by Chad Smith on 2018-07-09

proper indentation of debian /etc/os-release content. mock is_FreeBSD to avoid having to whitebox mock load_file calls

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:46345394b6d8ec96f13ade59c9fffb2885349ade
https://jenkins.ubuntu.com/server/job/cloud-init-ci/146/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/146/rebuild

review: Approve (continuous-integration)

diff --git a/cloudinit/tests/test_util.py b/cloudinit/tests/test_util.py
index 17853fc..6a31e50 100644
--- a/cloudinit/tests/test_util.py
+++ b/cloudinit/tests/test_util.py
@@ -26,8 +26,51 @@ OS_RELEASE_SLES = dedent("""\
     CPE_NAME="cpe:/o:suse:sles:12:sp3"\n
 """)
 
+OS_RELEASE_OPENSUSE = dedent("""\
+NAME="openSUSE Leap"
+VERSION="42.3"
+ID=opensuse
+ID_LIKE="suse"
+VERSION_ID="42.3"
+PRETTY_NAME="openSUSE Leap 42.3"
+ANSI_COLOR="0;32"
+CPE_NAME="cpe:/o:opensuse:leap:42.3"
+BUG_REPORT_URL="https://bugs.opensuse.org"
+HOME_URL="https://www.opensuse.org/"
+""")
+
+OS_RELEASE_CENTOS = dedent("""\
+    NAME="CentOS Linux"
+    VERSION="7 (Core)"
+    ID="centos"
+    ID_LIKE="rhel fedora"
+    VERSION_ID="7"
+    PRETTY_NAME="CentOS Linux 7 (Core)"
+    ANSI_COLOR="0;31"
+    CPE_NAME="cpe:/o:centos:centos:7"
+    HOME_URL="https://www.centos.org/"
+    BUG_REPORT_URL="https://bugs.centos.org/"
+
+    CENTOS_MANTISBT_PROJECT="CentOS-7"
+    CENTOS_MANTISBT_PROJECT_VERSION="7"
+    REDHAT_SUPPORT_PRODUCT="centos"
+    REDHAT_SUPPORT_PRODUCT_VERSION="7"
+""")
+
+OS_RELEASE_DEBIAN = dedent("""\
+    PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
+    NAME="Debian GNU/Linux"
+    VERSION_ID="9"
+    VERSION="9 (stretch)"
+    ID=debian
+    HOME_URL="https://www.debian.org/"
+    SUPPORT_URL="https://www.debian.org/support"
+    BUG_REPORT_URL="https://bugs.debian.org/"
+""")
+
 OS_RELEASE_UBUNTU = dedent("""\
     NAME="Ubuntu"\n
+    # comment test
     VERSION="16.04.3 LTS (Xenial Xerus)"\n
     ID=ubuntu\n
     ID_LIKE=debian\n
@@ -310,7 +353,31 @@ class TestGetLinuxDistro(CiTestCase):
         m_os_release.return_value = OS_RELEASE_UBUNTU
         m_path_exists.side_effect = TestGetLinuxDistro.os_release_exists
         dist = util.get_linux_distro()
-        self.assertEqual(('ubuntu', '16.04', platform.machine()), dist)
+        self.assertEqual(('ubuntu', '16.04', 'xenial'), dist)
+
+    @mock.patch('cloudinit.util.load_file')
+    def test_get_linux_centos(self, m_os_release, m_path_exists):
+        """Verify we get the correct name and release name on CentOS."""
+        m_os_release.return_value = OS_RELEASE_CENTOS
+        m_path_exists.side_effect = TestGetLinuxDistro.os_release_exists
+        dist = util.get_linux_distro()
+        self.assertEqual(('centos', '7', 'Core'), dist)
+
+    @mock.patch('cloudinit.util.load_file')
+    def test_get_linux_debian(self, m_os_release, m_path_exists):
+        """Verify we get the correct name and release name on Debian."""
+        m_os_release.return_value = OS_RELEASE_DEBIAN
+        m_path_exists.side_effect = TestGetLinuxDistro.os_release_exists
+        dist = util.get_linux_distro()
+        self.assertEqual(('debian', '9', 'stretch'), dist)
+
+    @mock.patch('cloudinit.util.load_file')
+    def test_get_linux_opensuse(self, m_os_release, m_path_exists):
+        """Verify we get the correct name and machine arch on OpenSUSE."""
+        m_os_release.return_value = OS_RELEASE_OPENSUSE
+        m_path_exists.side_effect = TestGetLinuxDistro.os_release_exists
+        dist = util.get_linux_distro()
+        self.assertEqual(('opensuse', '42.3', platform.machine()), dist)
 
     @mock.patch('platform.dist')
     def test_get_linux_distro_no_data(self, m_platform_dist, m_path_exists):
diff --git a/cloudinit/util.py b/cloudinit/util.py
index 6da9511..d0b0e90 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -579,16 +579,24 @@ def get_cfg_option_int(yobj, key, default=0):
 def get_linux_distro():
     distro_name = ''
     distro_version = ''
+    flavor = ''
     if os.path.exists('/etc/os-release'):
-        os_release = load_file('/etc/os-release')
-        for line in os_release.splitlines():
-            if line.strip().startswith('ID='):
-                distro_name = line.split('=')[-1]
-                distro_name = distro_name.replace('"', '')
-            if line.strip().startswith('VERSION_ID='):
-                # Lets hope for the best that distros stay consistent ;)
-                distro_version = line.split('=')[-1]
-                distro_version = distro_version.replace('"', '')
+        os_release = load_shell_content(load_file('/etc/os-release'))
+        distro_name = os_release.get('ID', '')
+        distro_version = os_release.get('VERSION_ID', '')
+        if 'sles' in distro_name or 'suse' in distro_name:
+            # RELEASE_BLOCKER: We will drop this sles ivergent behavior in
+            # before 18.4 so that get_linux_distro returns a named tuple
+            # which will include both version codename and architecture
+            # on all distributions.
+            flavor = platform.machine()
+        else:
+            flavor = os_release.get('VERSION_CODENAME', '')
+            if not flavor:
+                match = re.match(r'[^ ]+ \((?P<codename>[^)]+)\)',
+                                 os_release.get('VERSION'))
+                if match:
+                    flavor = match.groupdict()['codename']
     else:
         dist = ('', '', '')
         try:
@@ -606,7 +614,7 @@ def get_linux_distro():
                             'expansion may have unexpected results')
         return dist
 
-    return (distro_name, distro_version, platform.machine())
+    return (distro_name, distro_version, flavor)
 
 
 def system_info():
diff --git a/tests/unittests/test_datasource/test_azure_helper.py b/tests/unittests/test_datasource/test_azure_helper.py
index af9d3e1..26b2b93 100644
--- a/tests/unittests/test_datasource/test_azure_helper.py
+++ b/tests/unittests/test_datasource/test_azure_helper.py
@@ -85,7 +85,9 @@ class TestFindEndpoint(CiTestCase):
         self.dhcp_options.return_value = {"eth0": {"unknown_245": "5:4:3:2"}}
         self.assertEqual('5.4.3.2', wa_shim.find_endpoint(None))
 
-    def test_latest_lease_used(self):
+    @mock.patch('cloudinit.sources.helpers.azure.util.is_FreeBSD')
+    def test_latest_lease_used(self, m_is_freebsd):
+        m_is_freebsd.return_value = False  # To avoid hitting load_file
         encoded_addresses = ['5:4:3:2', '4:3:2:1']
         file_content = '\n'.join([self._build_lease_content(encoded_address)
                                   for encoded_address in encoded_addresses])