Merge ~harlowja/cloud-init:tpl-cloud-cfg into cloud-init:master

[Joshua Harlow]

Make config/cloud.cfg a template

Certain settings in cloud.cfg don't make sense on different and/or other
distros, and certain modules are routinely being not applied by various
downstream distributors so to aid in this split of cloud.cfg that is
happening downstream anyways we might as well turn cloud.cfg into a
template that all can share and customize (if needed) for certain distros
(this makes it easier to identify common themes and differences and
resolve those).

This change does just that and adjust needed other tooling to correctly
know how to include the renamed and new cloud.cfg files.

[Scott Moser (smoser)]

Largely seems ok.

A few things though:
 a.) seems unrelated to modify the rpm spec
 b.) what about setup.py?
 c.) shouldnt' we 'git ignore config/cloud.cfg' ?

[Scott Moser (smoser)]

review: Needs Fixing

[Scott Moser (smoser)]

please drop the unrelated change.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:8a445f0811e8deca935f5781a84e8d08067b1b1e
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~harlowja/cloud-init/+git/cloud-init/+merge/307485/+edit-commit-message

https://jenkins.ubuntu.com/server/job/cloud-init-ci/77/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-arm64/77/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-ppc64el/77/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=metal-s390x/77/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-amd64/77/console
    FAILURE: https://jenkins.ubuntu.com/server/job/cloud-init-ci/nodes=vm-i386/77/console

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/77/rebuild

review: Needs Fixing (continuous-integration)

[Scott Moser (smoser)]

This needs
a.) rebase
b.) droping of unrelated rpm spec change
c.) check to see that all rendered is as expected.

[Scott Moser (smoser)]

Hi,
I've kind of taken this over at
 https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+ref/feature/tpl-cloud-cfg

I'll move this to work in progress.

[Scott Moser (smoser)]

I've added all this and put a MP at
 https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/325192

[Scott Moser (smoser)]

Marked this merged. the general templatizatio of cloud.cfg is done now.
thanks for your initial work Josh.

[Scott Moser (smoser)]

Marked this merged. the general templatizatio of cloud.cfg is done now.
thanks for your initial work Josh.

diff --git a/Makefile b/Makefile
index 5d35dcc..496f1e9 100644
--- a/Makefile
+++ b/Makefile
@@ -66,6 +66,9 @@ check_version:
 	    "not equal to code version $(CODE_VERSION)"; exit 2; \
 	    else true; fi
 
+config/cloud.cfg:
+	./tools/render-ud  config/cloud.cfg.tpl > config/cloud.cfg
+
 clean_pyc:
 	@find . -type f -name "*.pyc" -delete
 
@@ -75,10 +78,10 @@ clean: clean_pyc
 yaml:
 	@$(CWD)/tools/validate-yaml.py $(YAML_FILES)
 
-rpm:
+rpm: config/cloud.cfg
 	./packages/brpm --distro $(distro)
 
-deb:
+deb: config/cloud.cfg
 	./packages/bddeb
 
 .PHONY: test pyflakes pyflakes3 clean pep8 rpm deb yaml check_version
diff --git a/cloudinit/util.py b/cloudinit/util.py
index eb3e589..cba1ff0 100644
--- a/cloudinit/util.py
+++ b/cloudinit/util.py
@@ -579,13 +579,40 @@ def get_cfg_option_int(yobj, key, default=0):
 
 
 def system_info():
-    return {
-        'platform': platform.platform(),
-        'release': platform.release(),
-        'python': platform.python_version(),
-        'uname': platform.uname(),
-        'dist': platform.linux_distribution(),
+    info = {
+         'platform': platform.platform(),
+         'release': platform.release(),
+         'python': platform.python_version(),
+         'uname': platform.uname(),
+         'dist': platform.linux_distribution(),
     }
+    plat = info['platform'].lower()
+    # Try to get more info about what it actually is, in a format
+    # that we can easily use across linux and variants...
+    if plat.startswith('darwin'):
+        info['variant'] = 'darwin'
+    elif plat.endswith("bsd"):
+        info['variant'] = 'bsd'
+    elif plat.startswith('win'):
+        info['variant'] = 'windows'
+    elif 'linux' in plat:
+        # Try to get a single string out of these...
+        linux_dist, _version, _id = info['dist']
+        linux_dist = linux_dist.lower()
+        if linux_dist in ('ubuntu', 'linuxmint', 'mint'):
+            info['variant'] = 'ubuntu'
+        else:
+            for prefix, variant in [('redhat', 'rhel'),
+                                    ('centos', 'rhel'),
+                                    ('fedora', 'fedora'),
+                                    ('debian', 'debian')]:
+                if linux_dist.startswith(prefix):
+                    info['variant'] = variant
+        if 'variant' not in info:
+            info['variant'] = 'linux'
+    if 'variant' not in info:
+        info['variant'] = 'unknown'
+    return info
 
 
 def get_cfg_option_list(yobj, key, default=None):
diff --git a/config/cloud.cfg b/config/cloud.cfg
deleted file mode 100644
index d608dc8..0000000
--- a/config/cloud.cfg
+++ /dev/null
@@ -1,116 +0,0 @@
-# The top level settings are used as module
-# and system configuration.
-
-# A set of users which may be applied and/or used by various modules
-# when a 'default' entry is found it will reference the 'default_user'
-# from the distro configuration specified below
-users:
-   - default
-
-# If this is set, 'root' will not be able to ssh in and they 
-# will get a message to login instead as the above $user (ubuntu)
-disable_root: true
-
-# This will cause the set+update hostname module to not operate (if true)
-preserve_hostname: false
-
-# Example datasource config
-# datasource: 
-#    Ec2: 
-#      metadata_urls: [ 'blah.com' ]
-#      timeout: 5 # (defaults to 50 seconds)
-#      max_wait: 10 # (defaults to 120 seconds)
-
-# The modules that run in the 'init' stage
-cloud_init_modules:
- - migrator
- - ubuntu-init-switch
- - seed_random
- - bootcmd
- - write-files
- - growpart
- - resizefs
- - disk_setup
- - mounts
- - set_hostname
- - update_hostname
- - update_etc_hosts
- - ca-certs
- - rsyslog
- - users-groups
- - ssh
-
-# The modules that run in the 'config' stage
-cloud_config_modules:
-# Emit the cloud config ready event
-# this can be used by upstart jobs for 'start on cloud-config'.
- - emit_upstart
- - ssh-import-id
- - locale
- - set-passwords
- - grub-dpkg
- - apt-pipelining
- - apt-configure
- - ntp
- - timezone
- - disable-ec2-metadata
- - runcmd
- - byobu
-
-# The modules that run in the 'final' stage
-cloud_final_modules:
- - snappy
- - package-update-upgrade-install
- - fan
- - landscape
- - lxd
- - puppet
- - chef
- - salt-minion
- - mcollective
- - rightscale_userdata
- - scripts-vendor
- - scripts-per-once
- - scripts-per-boot
- - scripts-per-instance
- - scripts-user
- - ssh-authkey-fingerprints
- - keys-to-console
- - phone-home
- - final-message
- - power-state-change
-
-# System and/or distro specific settings
-# (not accessible to handlers/transforms)
-system_info:
-   # This will affect which distro class gets used
-   distro: ubuntu
-   # Default user name + that default users groups (if added/used)
-   default_user:
-     name: ubuntu
-     lock_passwd: True
-     gecos: Ubuntu
-     groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
-     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
-     shell: /bin/bash
-   # Other config here will be given to the distro class and/or path classes
-   paths:
-      cloud_dir: /var/lib/cloud/
-      templates_dir: /etc/cloud/templates/
-      upstart_dir: /etc/init/
-   package_mirrors:
-     - arches: [i386, amd64]
-       failsafe:
-         primary: http://archive.ubuntu.com/ubuntu
-         security: http://security.ubuntu.com/ubuntu
-       search:
-         primary:
-           - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
-           - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
-           - http://%(region)s.clouds.archive.ubuntu.com/ubuntu/
-         security: []
-     - arches: [armhf, armel, default]
-       failsafe:
-         primary: http://ports.ubuntu.com/ubuntu-ports
-         security: http://ports.ubuntu.com/ubuntu-ports
-   ssh_svcname: ssh
diff --git a/config/cloud.cfg-freebsd b/config/cloud.cfg-freebsd
deleted file mode 100644
index be664f5..0000000
--- a/config/cloud.cfg-freebsd
+++ /dev/null
@@ -1,88 +0,0 @@
-# The top level settings are used as module
-# and system configuration.
-
-syslog_fix_perms: root:wheel
-
-# This should not be required, but leave it in place until the real cause of
-# not beeing able to find -any- datasources is resolved.
-datasource_list: ['ConfigDrive', 'OpenStack', 'Ec2']
-
-# A set of users which may be applied and/or used by various modules
-# when a 'default' entry is found it will reference the 'default_user'
-# from the distro configuration specified below
-users:
-   - default
-
-# If this is set, 'root' will not be able to ssh in and they 
-# will get a message to login instead as the above $user (ubuntu)
-disable_root: false
-
-# This will cause the set+update hostname module to not operate (if true)
-preserve_hostname: false
-
-# Example datasource config
-# datasource: 
-#    Ec2: 
-#      metadata_urls: [ 'blah.com' ]
-#      timeout: 5 # (defaults to 50 seconds)
-#      max_wait: 10 # (defaults to 120 seconds)
-
-# The modules that run in the 'init' stage
-cloud_init_modules:
-# - migrator
- - seed_random
- - bootcmd
-# - write-files
- - growpart
- - resizefs
- - set_hostname
- - update_hostname
-# - update_etc_hosts
-# - ca-certs
-# - rsyslog
- - users-groups
- - ssh
-
-# The modules that run in the 'config' stage
-cloud_config_modules:
-# - disk_setup
-# - mounts
- - ssh-import-id
- - locale
- - set-passwords
- - package-update-upgrade-install
-# - landscape
- - timezone
-# - puppet
-# - chef
-# - salt-minion
-# - mcollective
- - disable-ec2-metadata
- - runcmd
-# - byobu
-
-# The modules that run in the 'final' stage
-cloud_final_modules:
- - rightscale_userdata
- - scripts-vendor
- - scripts-per-once
- - scripts-per-boot
- - scripts-per-instance
- - scripts-user
- - ssh-authkey-fingerprints
- - keys-to-console
- - phone-home
- - final-message
- - power-state-change
-
-# System and/or distro specific settings
-# (not accessible to handlers/transforms)
-system_info:
-   distro: freebsd
-   default_user:
-     name: freebsd
-     lock_passwd: True
-     gecos: FreeBSD
-     groups: [wheel]
-     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
-     shell: /bin/tcsh
diff --git a/config/cloud.cfg.tpl b/config/cloud.cfg.tpl
new file mode 100644
index 0000000..831139b
--- /dev/null
+++ b/config/cloud.cfg.tpl
@@ -0,0 +1,198 @@
+## template:jinja
+
+# The top level settings are used as module
+# and system configuration.
+
+{% if platform.variant in ["bsd"] %}
+syslog_fix_perms: root:wheel
+{% endif %}
+
+# A set of users which may be applied and/or used by various modules
+# when a 'default' entry is found it will reference the 'default_user'
+# from the distro configuration specified below
+users:
+   - default
+
+# If this is set, 'root' will not be able to ssh in and they
+# will get a message to login instead as the default $user
+{% if platform.variant in ["bsd"] %}
+disable_root: false
+{% else %}
+disable_root: true
+{% endif %}
+
+# This will cause the set+update hostname module to not operate (if true)
+preserve_hostname: false
+
+# Example datasource config
+# datasource:
+#    Ec2:
+#      metadata_urls: [ 'blah.com' ]
+#      timeout: 5 # (defaults to 50 seconds)
+#      max_wait: 10 # (defaults to 120 seconds)
+
+{% if platform.variant in ["bsd"] %}
+# This should not be required, but leave it in place until the real cause of
+# not beeing able to find -any- datasources is resolved.
+datasource_list: ['ConfigDrive', 'OpenStack', 'Ec2']
+{% endif %}
+
+# The modules that run in the 'init' stage
+cloud_init_modules:
+ - migrator
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+ - ubuntu-init-switch
+{% endif %}
+ - seed_random
+ - bootcmd
+# Setup disks and filesystems ... before we do much else.
+ - growpart
+ - resizefs
+ - disk_setup
+ - mounts
+# (end block of disk/fs modules)
+{% if platform.variant not in ["bsd"] %}
+ - write-files
+{% endif %}
+ - set_hostname
+ - update_hostname
+{% if platform.variant not in ["bsd"] %}
+ - update_etc_hosts
+ - ca-certs
+ - rsyslog
+{% endif %}
+ - users-groups
+ - ssh
+
+# The modules that run in the 'config' stage
+cloud_config_modules:
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+# Emit the cloud config ready event
+# this can be used by upstart jobs for 'start on cloud-config'.
+ - emit_upstart
+{% endif %}
+ - ssh-import-id
+ - locale
+ - set-passwords
+{% if platform.variant in ["rhel", "fedora"] %}
+ - spacewalk
+ - yum-add-repo
+{% endif %}
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+ - grub-dpkg
+ - apt-pipelining
+ - apt-configure
+{% endif %}
+ - ntp
+ - timezone
+ - disable-ec2-metadata
+ - runcmd
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+ - byobu
+{% endif %}
+
+# The modules that run in the 'final' stage
+cloud_final_modules:
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+ - snappy
+{% endif %}
+ - package-update-upgrade-install
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+ - fan
+ - landscape
+ - lxd
+{% endif %}
+{% if platform.variant not in ["bsd"] %}
+ # These are all grouped together (but typically only one of them
+ # actually is doing anything, since most people don't run many of these
+ # package/configuration management systems at the same time).
+ - puppet
+ - chef
+ - salt-minion
+ - mcollective
+ # (end block of configuration management 'like' modules)
+{% endif %}
+ - rightscale_userdata
+ - scripts-vendor
+ - scripts-per-once
+ - scripts-per-boot
+ - scripts-per-instance
+ - scripts-user
+ - ssh-authkey-fingerprints
+ - keys-to-console
+ - phone-home
+ - final-message
+ - power-state-change
+
+# System and/or distro specific settings
+# (not accessible to handlers/transforms)
+system_info:
+   # This will affect which distro class gets used...
+{% if platform.variant in ["ubuntu"] %}
+   distro: ubuntu
+{% elif platform.variant in ["fedora"] %}
+   distro: fedora
+{% elif platform.variant in ["debian"] %}
+   distro: debian
+{% elif platform.variant in ["rhel"] %}
+   distro: rhel
+{% elif platform.variant in ["bsd"] %}
+   distro: freebsd
+{% else %}
+   # Unknown/fallback distro.
+   distro: ubuntu
+{% endif %}
+{% if platform.variant in ["ubuntu", "unknown", "debian"] %}
+   # Default user name + that default users groups (if added/used)
+   default_user:
+     name: ubuntu
+     lock_passwd: True
+     gecos: Ubuntu
+     groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
+     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+     shell: /bin/bash
+   # Other config here will be given to the distro class and/or path classes
+   paths:
+      cloud_dir: /var/lib/cloud/
+      templates_dir: /etc/cloud/templates/
+      upstart_dir: /etc/init/
+   package_mirrors:
+     - arches: [i386, amd64]
+       failsafe:
+         primary: http://archive.ubuntu.com/ubuntu
+         security: http://security.ubuntu.com/ubuntu
+       search:
+         primary:
+           - http://%(ec2_region)s.ec2.archive.ubuntu.com/ubuntu/
+           - http://%(availability_zone)s.clouds.archive.ubuntu.com/ubuntu/
+           - http://%(region)s.clouds.archive.ubuntu.com/ubuntu/
+         security: []
+     - arches: [armhf, armel, default]
+       failsafe:
+         primary: http://ports.ubuntu.com/ubuntu-ports
+         security: http://ports.ubuntu.com/ubuntu-ports
+   ssh_svcname: ssh
+{% elif platform.variant in ["rhel", "fedora"] %}
+   # Default user name + that default users groups (if added/used)
+   default_user:
+     name: fedora
+     lock_passwd: True
+     gecos: Fedora Cloud User
+     groups: [wheel, adm, systemd-journal]
+     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+     shell: /bin/bash
+   # Other config here will be given to the distro class and/or path classes
+   paths:
+      cloud_dir: /var/lib/cloud/
+      templates_dir: /etc/cloud/templates/
+   ssh_svcname: sshd
+{% elif platform.variant in ["bsd"] %}
+    # Default user name + that default users groups (if added/used)
+    default_user:
+      name: freebsd
+      lock_passwd: True
+      gecos: FreeBSD
+      groups: [wheel]
+      sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+      shell: /bin/tcsh
+{% endif %}
diff --git a/packages/redhat/cloud-init.spec.in b/packages/redhat/cloud-init.spec.in
index d0ae048..9fb8721 100644
--- a/packages/redhat/cloud-init.spec.in
+++ b/packages/redhat/cloud-init.spec.in
@@ -95,6 +95,10 @@ rm -rf \$RPM_BUILD_ROOT%{python_sitelib}/tests
 mkdir -p \$RPM_BUILD_ROOT/%{_sharedstatedir}/cloud
 mkdir -p \$RPM_BUILD_ROOT/%{_libexecdir}/%{name}
 
+# Remove these for now (not sure if they work)...
+rm \$RPM_BUILD_ROOT/%{_sysconfdir}/NetworkManager/dispatcher.d/hook-network-manager
+rm \$RPM_BUILD_ROOT/%{_sysconfdir}/dhcp/dhclient-exit-hooks.d/hook-dhclient
+
 #if $systemd
 mkdir -p         \$RPM_BUILD_ROOT/%{_unitdir}
 cp -p systemd/*  \$RPM_BUILD_ROOT/%{_unitdir}
diff --git a/tools/render-ud b/tools/render-ud
new file mode 100755
index 0000000..0af7d3c
--- /dev/null
+++ b/tools/render-ud
@@ -0,0 +1,26 @@
+#!/usr/bin/env python
+
+import os
+import sys
+
+if "avoid-pep8-E402-import-not-top-of-file":
+    _tdir = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+    sys.path.insert(0, _tdir)
+    from cloudinit import templater
+    from cloudinit import util
+
+
+def main():
+    fn = sys.argv[1]
+    tpl_params = {
+        'platform': util.system_info(),
+    }
+    with open(fn, 'rb') as fh:
+        contents = fh.read()
+    contents = (templater.render_string(contents, tpl_params))
+    print(contents)
+    util.load_yaml(contents)
+
+
+if __name__ == '__main__':
+    main()