Merge ~smoser/cloud-init:cleanup/drop-ubuntu-init-switch into cloud-init:master

[Scott Moser]

upstart: do not package upstart jobs, drop ubuntu-init-switch module.

The ubuntu-init-switch module allowed the use to launch an instance that
was booted with upstart and have it switch its init system to systemd and
then reboot itself. It was only useful for the time period when Ubuntu was
transitioning to systemd but only produced images using upstart.

Also, do not run setup with --init-system=upstart. This means that by
default, debian packages built with packages/bddeb will not have upstart
unit files included. No other removal is done here.

8555ddd... by Scott Moser on 2017-08-30

packages/debian/rules.in: do not install upstart jobs.

cloud-init trunk is only actively maintained on Ubuntu for 16.04 and beyond.
That means that there is no supported Ubuntu that uses upstart by default.

So we will just make the package not include the upstart jobs.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:801c0cf7da205dc2bb0313eb99a53df2b2b7c01b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/230/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/230/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

We need to drop the module from doc/rtd/topics/modules.rst too.

.. automodule:: cloudinit.config.cc_ubuntu_init_switch

[Chad Smith (chad.smith)]

so that I understand intent here. We are leaving upstart logic in place so that tip could be used to still generate upstart-based packages if we wanted. We just don't want to package that init system by default anymore right?

[Scott Moser (smoser)]

> so that I understand intent here. We are leaving upstart logic in place so
> that tip could be used to still generate upstart-based packages if we wanted.
> We just don't want to package that init system by default anymore right?

correct.
we leave it all in place, but the upstream deb doesn't package it.

9167502... by Scott Moser on 2017-08-30

remove docs for ubuntu-init-switch

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:8555dddadd54187e0a5902d0b356f0803a50e896
https://jenkins.ubuntu.com/server/job/cloud-init-ci/232/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/232/rebuild

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:91675024ed03e6dec2116aa48975746bc1ff66af
https://jenkins.ubuntu.com/server/job/cloud-init-ci/234/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/234/rebuild

review: Approve (continuous-integration)

[Chad Smith (chad.smith)]

Hrm, I'm still seeing upstart jobs included in /etc/init/*conf

dpkg -L cloud-init | grep etc/init
/etc/init
/etc/init/cloud-init-blocknet.conf
/etc/init/cloud-init-container.conf
/etc/init/cloud-init-nonet.conf
/etc/init/cloud-final.conf
/etc/init/cloud-init-local.conf
/etc/init/cloud-log-shutdown.conf
/etc/init/cloud-init.conf
/etc/init/cloud-config.conf
root@lxd-xenial:~# dpkg-query --show cloud-init
cloud-init 0.7.9-256-g8555ddd-1~bddeb

[Chad Smith (chad.smith)]

review: Needs Information

[Scott Moser (smoser)]

I dont see the issues you're seeing . the etc/init/ files are not
present for me. See below. There was the empty etc/init/ directory
but I have a fix for that too i think.

$ git describe
0.7.9-257-g91675024

$ ./packages/bddeb -S -d
Creating a temporary tarball using the 'make-tarball' helper
Extracting temporary tarball 'cloud-init_0.7.9-257-g91675024.orig.tar.gz'
Creating a debian/ folder in '/tmp/tmpe5frrdol/cloud-init-0.7.9-257-g91675024'
Running 'debuild -S -d -us -uc' in '/tmp/tmpe5frrdol/cloud-init-0.7.9-257-g91675024'
Wrote 'cloud-init_0.7.9-257-g91675024-1~bddeb.debian.tar.xz'
Wrote 'cloud-init_0.7.9-257-g91675024-1~bddeb_source.build'
Wrote 'cloud-init_0.7.9-257-g91675024-1~bddeb_source.buildinfo'
Wrote 'cloud-init_0.7.9-257-g91675024-1~bddeb_source.changes'
Wrote 'cloud-init_0.7.9-257-g91675024.orig.tar.gz'
Wrote 'cloud-init_0.7.9-257-g91675024-1~bddeb.dsc'
Linked 'cloud-init_0.7.9-257-g91675024-1~bddeb.dsc' to 'cloud-init.dsc'

$ sbuild --dist=artful --arch=amd64 --arch-all cloud-init_0.7.9-257-g91675024-1~bddeb.dsc

$ pastebinit cloud-init_0.7.9-257-g91675024-1~bddeb_amd64.build
http://paste.ubuntu.com/25435409/

$ dpkg -c cloud-init_0.7.9-257-g91675024-1~bddeb_all.deb | grep etc/init
drwxr-xr-x root/root 0 2011-12-16 11:50 ./etc/init/

$ dpkg -c cloud-init_0.7.9-257-g91675024-1~bddeb_all.deb | pastebinit
http://paste.ubuntu.com/25435415/

8d180ec... by Scott Moser on 2017-08-31

do not include empty etc/init directory

the entry in debian/dirs was never actually needed.

[Scott Moser (smoser)]

$ dpkg -c /home/smoser-public/src/cloud-init/build/../cloud-init_0.7.9-258-g8d180ece-1~bddeb_all.deb | grep etc/init || echo none there
nonoe there.

$ dpkg -c ../cloud-init_0.7.9-258-g8d180ece-1~bddeb_all.deb | pastebinit
http://paste.ubuntu.com/25435440/

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:8d180ecee1bc71747e4145bf8e20e0aec3050acb
https://jenkins.ubuntu.com/server/job/cloud-init-ci/238/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/238/rebuild

review: Approve (continuous-integration)

bc9d1b4... by Scott Moser on 2017-08-31

fix 'bddeb' to default to only systemd

[Chad Smith (chad.smith)]

+1 once unit tests complete here. Thanks for the fix. This is the issue I was seeing on my side

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:bc9d1b4806c8125ad2a4f954338955fa6f3e75aa
https://jenkins.ubuntu.com/server/job/cloud-init-ci/240/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/240/rebuild

review: Approve (continuous-integration)

diff --git a/cloudinit/config/cc_ubuntu_init_switch.py b/cloudinit/config/cc_ubuntu_init_switch.py
deleted file mode 100644
index 5dd2690..0000000
--- a/cloudinit/config/cc_ubuntu_init_switch.py
+++ /dev/null
@@ -1,160 +0,0 @@
-# Copyright (C) 2014 Canonical Ltd.
-#
-# Author: Scott Moser <[e-mail]>
-#
-# This file is part of cloud-init. See LICENSE file for license information.
-
-"""
-Ubuntu Init Switch
-------------------
-**Summary:** reboot system into another init.
-
-This module provides a way for the user to boot with systemd even if the image
-is set to boot with upstart. It should be run as one of the first
-``cloud_init_modules``, and will switch the init system and then issue a
-reboot. The next boot will come up in the target init system and no action
-will be taken. This should be inert on non-ubuntu systems, and also
-exit quickly.
-
-.. note::
-    best effort is made, but it's possible this system will break, and probably
-    won't interact well with any other mechanism you've used to switch the init
-    system.
-
-**Internal name:** ``cc_ubuntu_init_switch``
-
-**Module frequency:** once per instance
-
-**Supported distros:** ubuntu
-
-**Config keys**::
-
-    init_switch:
-      target: systemd (can be 'systemd' or 'upstart')
-      reboot: true (reboot if a change was made, or false to not reboot)
-"""
-
-from cloudinit.distros import ubuntu
-from cloudinit import log as logging
-from cloudinit.settings import PER_INSTANCE
-from cloudinit import util
-
-import os
-import time
-
-frequency = PER_INSTANCE
-REBOOT_CMD = ["/sbin/reboot", "--force"]
-
-DEFAULT_CONFIG = {
-    'init_switch': {'target': None, 'reboot': True}
-}
-
-SWITCH_INIT = """
-#!/bin/sh
-# switch_init: [upstart | systemd]
-
-is_systemd() {
-   [ "$(dpkg-divert --listpackage /sbin/init)" = "systemd-sysv" ]
-}
-debug() { echo "$@" 1>&2; }
-fail() { echo "$@" 1>&2; exit 1; }
-
-if [ "$1" = "systemd" ]; then
-   if is_systemd; then
-      debug "already systemd, nothing to do"
-   else
-      [ -f /lib/systemd/systemd ] || fail "no systemd available";
-      dpkg-divert --package systemd-sysv --divert /sbin/init.diverted \\
-          --rename /sbin/init
-   fi
-   [ -f /sbin/init ] || ln /lib/systemd/systemd /sbin/init
-elif [ "$1" = "upstart" ]; then
-   if is_systemd; then
-      rm -f /sbin/init
-      dpkg-divert --package systemd-sysv --rename --remove /sbin/init
-   else
-      debug "already upstart, nothing to do."
-   fi
-else
-  fail "Error. expect 'upstart' or 'systemd'"
-fi
-"""
-
-distros = ['ubuntu']
-
-
-def handle(name, cfg, cloud, log, args):
-    """Handler method activated by cloud-init."""
-
-    if not isinstance(cloud.distro, ubuntu.Distro):
-        log.debug("%s: distro is '%s', not ubuntu. returning",
-                  name, cloud.distro.__class__)
-        return
-
-    cfg = util.mergemanydict([cfg, DEFAULT_CONFIG])
-    target = cfg['init_switch']['target']
-    reboot = cfg['init_switch']['reboot']
-
-    if len(args) != 0:
-        target = args[0]
-        if len(args) > 1:
-            reboot = util.is_true(args[1])
-
-    if not target:
-        log.debug("%s: target=%s. nothing to do", name, target)
-        return
-
-    if not util.which('dpkg'):
-        log.warn("%s: 'dpkg' not available. Assuming not ubuntu", name)
-        return
-
-    supported = ('upstart', 'systemd')
-    if target not in supported:
-        log.warn("%s: target set to %s, expected one of: %s",
-                 name, target, str(supported))
-
-    if os.path.exists("/run/systemd/system"):
-        current = "systemd"
-    else:
-        current = "upstart"
-
-    if current == target:
-        log.debug("%s: current = target = %s. nothing to do", name, target)
-        return
-
-    try:
-        util.subp(['sh', '-s', target], data=SWITCH_INIT)
-    except util.ProcessExecutionError as e:
-        log.warn("%s: Failed to switch to init '%s'. %s", name, target, e)
-        return
-
-    if util.is_false(reboot):
-        log.info("%s: switched '%s' to '%s'. reboot=false, not rebooting.",
-                 name, current, target)
-        return
-
-    try:
-        log.warn("%s: switched '%s' to '%s'. rebooting.",
-                 name, current, target)
-        logging.flushLoggers(log)
-        _fire_reboot(log, wait_attempts=4, initial_sleep=4)
-    except Exception as e:
-        util.logexc(log, "Requested reboot did not happen!")
-        raise
-
-
-def _fire_reboot(log, wait_attempts=6, initial_sleep=1, backoff=2):
-    util.subp(REBOOT_CMD)
-    start = time.time()
-    wait_time = initial_sleep
-    for _i in range(0, wait_attempts):
-        time.sleep(wait_time)
-        wait_time *= backoff
-        elapsed = time.time() - start
-        log.debug("Rebooted, but still running after %s seconds", int(elapsed))
-    # If we got here, not good
-    elapsed = time.time() - start
-    raise RuntimeError(("Reboot did not happen"
-                        " after %s seconds!") % (int(elapsed)))
-
-# vi: ts=4 expandtab
diff --git a/config/cloud.cfg.tmpl b/config/cloud.cfg.tmpl
index f4b9069..a537d65 100644
--- a/config/cloud.cfg.tmpl
+++ b/config/cloud.cfg.tmpl
@@ -45,9 +45,6 @@ datasource_list: ['ConfigDrive', 'Azure', 'OpenStack', 'Ec2']
 # The modules that run in the 'init' stage
 cloud_init_modules:
  - migrator
-{% if variant in ["ubuntu", "unknown", "debian"] %}
- - ubuntu-init-switch
-{% endif %}
  - seed_random
  - bootcmd
  - write-files
diff --git a/doc/rtd/topics/modules.rst b/doc/rtd/topics/modules.rst
index c963c09..cdb0f41 100644
--- a/doc/rtd/topics/modules.rst
+++ b/doc/rtd/topics/modules.rst
@@ -50,7 +50,6 @@ Modules
 .. automodule:: cloudinit.config.cc_ssh_authkey_fingerprints
 .. automodule:: cloudinit.config.cc_ssh_import_id
 .. automodule:: cloudinit.config.cc_timezone
-.. automodule:: cloudinit.config.cc_ubuntu_init_switch
 .. automodule:: cloudinit.config.cc_update_etc_hosts
 .. automodule:: cloudinit.config.cc_update_hostname
 .. automodule:: cloudinit.config.cc_users_groups
diff --git a/packages/bddeb b/packages/bddeb
index 609a94f..7c12354 100755
--- a/packages/bddeb
+++ b/packages/bddeb
@@ -112,8 +112,7 @@ def get_parser():
     parser.add_argument("--init-system", dest="init_system",
                         help=("build deb with INIT_SYSTEM=xxx"
                               " (default: %(default)s"),
-                        default=os.environ.get("INIT_SYSTEM",
-                                               "upstart,systemd"))
+                        default=os.environ.get("INIT_SYSTEM", "systemd"))
 
     parser.add_argument("--release", dest="release",
                         help=("build with changelog referencing RELEASE"),
diff --git a/packages/debian/dirs b/packages/debian/dirs
index 9a633c6..1315cf8 100644
--- a/packages/debian/dirs
+++ b/packages/debian/dirs
@@ -1,6 +1,5 @@
 var/lib/cloud
 usr/bin
-etc/init
 usr/share/doc/cloud
 etc/cloud
 lib/udev/rules.d
diff --git a/packages/debian/rules.in b/packages/debian/rules.in
index 053b764..b87a5e8 100755
--- a/packages/debian/rules.in
+++ b/packages/debian/rules.in
@@ -1,6 +1,6 @@
 ## template:basic
 #!/usr/bin/make -f
-INIT_SYSTEM ?= upstart,systemd
+INIT_SYSTEM ?= systemd
 export PYBUILD_INSTALL_ARGS=--init-system=$(INIT_SYSTEM)
 PYVER ?= python${pyver}
 
diff --git a/setup.py b/setup.py
index 5c65c7f..7662bd8 100755
--- a/setup.py
+++ b/setup.py
@@ -191,6 +191,8 @@ class InitsysInstallData(install):
             datakeys = [k for k in INITSYS_ROOTS
                         if k.partition(".")[0] == system]
             for k in datakeys:
+                if not INITSYS_FILES[k]:
+                    continue
                 self.distribution.data_files.append(
                     (INITSYS_ROOTS[k], INITSYS_FILES[k]))
         # Force that command to reinitalize (with new file list)
diff --git a/tests/cloud_tests/configs/modules/TODO.md b/tests/cloud_tests/configs/modules/TODO.md
index d496da9..0b933b3 100644
--- a/tests/cloud_tests/configs/modules/TODO.md
+++ b/tests/cloud_tests/configs/modules/TODO.md
@@ -89,8 +89,6 @@ Not applicable to write a test for this as it specifies when something should be
 ## ssh authkey fingerprints
 The authkey_hash key does not appear to work. In fact the default claims to be md5, however syslog only shows sha256
 
-## ubuntu init switch
-
 ## update etc hosts
 2016-11-17: Issues with changing /etc/hosts and lxc backend.