Merge ~smoser/cloud-init:feature/freebsd-variant into cloud-init:master

[Scott Moser]

freebsd: Make freebsd a variant, fix unittests and tools/build-on-freebsd.

 - Simplify the logic of 'variant' in util.system_info
   much of the data from
   https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
 - fix get_resource_disk_on_freebsd when running on a system without
   an Azure resource disk.
 - fix tools/build-on-freebsd to replace oauth with oauthlib and add
   bash which is a dependency for tests.
 - update a fiew places that were checking for freebsd but not using
   the util.is_FreeBSD()

[Scott Moser (smoser)]

this replaces some of the changes from
 https://code.launchpad.net/~redriver/cloud-init/+git/cloud-init/+merge/325718

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:2151902fa7a5cba7e420a14f51c851ee06d83d30
https://jenkins.ubuntu.com/server/job/cloud-init-ci/538/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/538/console
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/538
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/538
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/538
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/538

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/538/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:c6af5b9a1648c208c82b3a2704668391abadd8ab
https://jenkins.ubuntu.com/server/job/cloud-init-ci/539/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-amd64/539
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/539
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/539
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/539
    SUCCESS: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/539

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/539/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Looks pretty good, I'd like to see other unit tests specifically looking at return vals from util.system_info() but that can come in a later branch. Minor nits but code looks good.
I have no bsd system available otherwise I'd check it out with a couple of runs.

[Scott Moser (smoser)]

actually, on further thought, lets leve suse out right now.
there is a 'sles' distro (cloudinit/distros/sles.py) but per
 https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
we would see 'suse' as the dist.

not sure what the right change is.

i didn't *change* any behavior here i dont think.
so i'd rather leave it is and just enable the freebsd.

and we can/should fix suse/sles later.

[Chad Smith (chad.smith)]

+1 on that suggestion

diff --git a/cloudinit/config/cc_growpart.py b/cloudinit/config/cc_growpart.py
index d2bc6e6..bafca9d 100644
--- a/cloudinit/config/cc_growpart.py
+++ b/cloudinit/config/cc_growpart.py
@@ -214,7 +214,7 @@ def device_part_info(devpath):
 
     # FreeBSD doesn't know of sysfs so just get everything we need from
     # the device, like /dev/vtbd0p2.
-    if util.system_info()["platform"].startswith('FreeBSD'):
+    if util.is_FreeBSD():
         m = re.search('^(/dev/.+)p([0-9])$', devpath)
         return (m.group(1), m.group(2))
 
diff --git a/cloudinit/config/cc_power_state_change.py b/cloudinit/config/cc_power_state_change.py
index c1c6fe7..eba58b0 100644
--- a/cloudinit/config/cc_power_state_change.py
+++ b/cloudinit/config/cc_power_state_change.py
@@ -71,7 +71,7 @@ def givecmdline(pid):
         # Example output from procstat -c 1
         #   PID COMM             ARGS
         #     1 init             /bin/init --
-        if util.system_info()["platform"].startswith('FreeBSD'):
+        if util.is_FreeBSD():
             (output, _err) = util.subp(['procstat', '-c', str(pid)])
             line = output.splitlines()[1]
             m = re.search('\d+ (\w|\.|-)+\s+(/\w.+)', line)
diff --git a/cloudinit/sources/DataSourceAzure.py b/cloudinit/sources/DataSourceAzure.py
index 71e7c55..4fe0d63 100644
--- a/cloudinit/sources/DataSourceAzure.py
+++ b/cloudinit/sources/DataSourceAzure.py
@@ -101,7 +101,7 @@ def get_dev_storvsc_sysctl():
         sysctl_out, err = util.subp(['sysctl', 'dev.storvsc'])
     except util.ProcessExecutionError:
         LOG.debug("Fail to execute sysctl dev.storvsc")
-        return None
+        sysctl_out = ""
     return sysctl_out
 
 
diff --git a/cloudinit/util.py b/cloudinit/util.py
index ec68925..c93b6d7 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -573,7 +573,7 @@ def is_ipv4(instr):
 
 
 def is_FreeBSD():
-    return system_info()['platform'].startswith('FreeBSD')
+    return system_info()['variant'] == "freebsd"
 
 
 def get_cfg_option_bool(yobj, key, default=False):
@@ -598,37 +598,29 @@ def get_cfg_option_int(yobj, key, default=0):
 def system_info():
     info = {
         'platform': platform.platform(),
+        'system': platform.system(),
         'release': platform.release(),
         'python': platform.python_version(),
         'uname': platform.uname(),
-        'dist': platform.linux_distribution(),  # pylint: disable=W1505
+        'dist': platform.dist(),  # pylint: disable=W1505
     }
-    plat = info['platform'].lower()
-    # Try to get more info about what it actually is, in a format
-    # that we can easily use across linux and variants...
-    if plat.startswith('darwin'):
-        info['variant'] = 'darwin'
-    elif plat.endswith("bsd"):
-        info['variant'] = 'bsd'
-    elif plat.startswith('win'):
-        info['variant'] = 'windows'
-    elif 'linux' in plat:
-        # Try to get a single string out of these...
-        linux_dist, _version, _id = info['dist']
-        linux_dist = linux_dist.lower()
-        if linux_dist in ('ubuntu', 'linuxmint', 'mint'):
-            info['variant'] = 'ubuntu'
+    system = info['system'].lower()
+    var = 'unknown'
+    if system == "linux":
+        linux_dist = info['dist'][0].lower()
+        if linux_dist in ('centos', 'fedora', 'debian'):
+            var = linux_dist
+        elif linux_dist in ('ubuntu', 'linuxmint', 'mint'):
+            var = 'ubuntu'
+        elif linux_dist == 'redhat':
+            var = 'rhel'
         else:
-            for prefix, variant in [('redhat', 'rhel'),
-                                    ('centos', 'centos'),
-                                    ('fedora', 'fedora'),
-                                    ('debian', 'debian')]:
-                if linux_dist.startswith(prefix):
-                    info['variant'] = variant
-        if 'variant' not in info:
-            info['variant'] = 'linux'
-    if 'variant' not in info:
-        info['variant'] = 'unknown'
+            var = 'linux'
+    elif system in ('windows', 'darwin', "freebsd"):
+        var = system
+
+    info['variant'] = var
+
     return info
 
 
diff --git a/config/cloud.cfg.tmpl b/config/cloud.cfg.tmpl
index 5af2a88..f4b9069 100644
--- a/config/cloud.cfg.tmpl
+++ b/config/cloud.cfg.tmpl
@@ -2,7 +2,7 @@
 # The top level settings are used as module
 # and system configuration.
 
-{% if variant in ["bsd"] %}
+{% if variant in ["freebsd"] %}
 syslog_fix_perms: root:wheel
 {% endif %}
 # A set of users which may be applied and/or used by various modules
@@ -13,7 +13,7 @@ users:
 
 # If this is set, 'root' will not be able to ssh in and they
 # will get a message to login instead as the default $user
-{% if variant in ["bsd"] %}
+{% if variant in ["freebsd"] %}
 disable_root: false
 {% else %}
 disable_root: true
@@ -30,7 +30,7 @@ ssh_pwauth:   0
 # This will cause the set+update hostname module to not operate (if true)
 preserve_hostname: false
 
-{% if variant in ["bsd"] %}
+{% if variant in ["freebsd"] %}
 # This should not be required, but leave it in place until the real cause of
 # not beeing able to find -any- datasources is resolved.
 datasource_list: ['ConfigDrive', 'Azure', 'OpenStack', 'Ec2']
@@ -53,13 +53,13 @@ cloud_init_modules:
  - write-files
  - growpart
  - resizefs
-{% if variant not in ["bsd"] %}
+{% if variant not in ["freebsd"] %}
  - disk_setup
  - mounts
 {% endif %}
  - set_hostname
  - update_hostname
-{% if variant not in ["bsd"] %}
+{% if variant not in ["freebsd"] %}
  - update_etc_hosts
  - ca-certs
  - rsyslog
@@ -87,7 +87,7 @@ cloud_config_modules:
  - apt-pipelining
  - apt-configure
 {% endif %}
-{% if variant not in ["bsd"] %}
+{% if variant not in ["freebsd"] %}
  - ntp
 {% endif %}
  - timezone
@@ -108,7 +108,7 @@ cloud_final_modules:
  - landscape
  - lxd
 {% endif %}
-{% if variant not in ["bsd"] %}
+{% if variant not in ["freebsd"] %}
  - puppet
  - chef
  - salt-minion
@@ -130,10 +130,8 @@ cloud_final_modules:
 # (not accessible to handlers/transforms)
 system_info:
    # This will affect which distro class gets used
-{% if variant in ["centos", "debian", "fedora", "rhel", "ubuntu"] %}
+{% if variant in ["centos", "debian", "fedora", "rhel", "ubuntu", "freebsd"] %}
    distro: {{ variant }}
-{% elif variant in ["bsd"] %}
-   distro: freebsd
 {% else %}
    # Unknown/fallback distro.
    distro: ubuntu
@@ -182,7 +180,7 @@ system_info:
       cloud_dir: /var/lib/cloud/
       templates_dir: /etc/cloud/templates/
    ssh_svcname: sshd
-{% elif variant in ["bsd"] %}
+{% elif variant in ["freebsd"] %}
    # Default user name + that default users groups (if added/used)
    default_user:
      name: freebsd
diff --git a/tests/unittests/test_handler/test_handler_ntp.py b/tests/unittests/test_handler/test_handler_ntp.py
index c4299d9..7f27864 100644
--- a/tests/unittests/test_handler/test_handler_ntp.py
+++ b/tests/unittests/test_handler/test_handler_ntp.py
@@ -62,7 +62,7 @@ class TestNtp(FilesystemMockingTestCase):
     def test_ntp_rename_ntp_conf(self):
         """When NTP_CONF exists, rename_ntp moves it."""
         ntpconf = self.tmp_path("ntp.conf", self.new_root)
-        os.mknod(ntpconf)
+        util.write_file(ntpconf, "")
         with mock.patch("cloudinit.config.cc_ntp.NTP_CONF", ntpconf):
             cc_ntp.rename_ntp_conf()
         self.assertFalse(os.path.exists(ntpconf))
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index 014aa6a..a73fd26 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -20,6 +20,9 @@ except ImportError:
     import mock
 
 
+BASH = util.which('bash')
+
+
 class FakeSelinux(object):
 
     def __init__(self, match_what):
@@ -544,17 +547,17 @@ class TestReadSeeded(helpers.TestCase):
 
 class TestSubp(helpers.TestCase):
 
-    stdin2err = ['bash', '-c', 'cat >&2']
+    stdin2err = [BASH, '-c', 'cat >&2']
     stdin2out = ['cat']
     utf8_invalid = b'ab\xaadef'
     utf8_valid = b'start \xc3\xa9 end'
     utf8_valid_2 = b'd\xc3\xa9j\xc8\xa7'
-    printenv = ['bash', '-c', 'for n in "$@"; do echo "$n=${!n}"; done', '--']
+    printenv = [BASH, '-c', 'for n in "$@"; do echo "$n=${!n}"; done', '--']
 
     def printf_cmd(self, *args):
         # bash's printf supports \xaa.  So does /usr/bin/printf
         # but by using bash, we remove dependency on another program.
-        return(['bash', '-c', 'printf "$@"', 'printf'] + list(args))
+        return([BASH, '-c', 'printf "$@"', 'printf'] + list(args))
 
     def test_subp_handles_utf8(self):
         # The given bytes contain utf-8 accented characters as seen in e.g.
diff --git a/tools/build-on-freebsd b/tools/build-on-freebsd
index ccc10b4..ff9153a 100755
--- a/tools/build-on-freebsd
+++ b/tools/build-on-freebsd
@@ -8,6 +8,7 @@ fail() { echo "FAILED:" "$@" 1>&2; exit 1; }
 # Check dependencies:
 depschecked=/tmp/c-i.dependencieschecked
 pkgs="
+   bash
    dmidecode
    e2fsprogs
    py27-Jinja2
@@ -16,7 +17,7 @@ pkgs="
    py27-configobj
    py27-jsonpatch
    py27-jsonpointer
-   py27-oauth
+   py27-oauthlib
    py27-prettytable
    py27-requests
    py27-serial
@@ -35,9 +36,6 @@ touch $depschecked
 python setup.py build
 python setup.py install -O1 --skip-build --prefix /usr/local/ --init-system sysvinit_freebsd
 
-# Install the correct config file:
-cp config/cloud.cfg-freebsd /etc/cloud/cloud.cfg
-
 # Enable cloud-init in /etc/rc.conf:
 sed -i.bak -e "/cloudinit_enable=.*/d" /etc/rc.conf
 echo 'cloudinit_enable="YES"' >> /etc/rc.conf