Merge lp:~cloud-init-dev/cloud-init/trunk into lp:~d-jj/cloud-init/chef-12-updates

[JJ Asghar]

This is to update the Chef examples and changes for the newer/est ways
Chef is converting to. Omnibus is now the default installation method, but
there are also more up-to-date package repos at packagecloud.io if the user
still wants to install the deb package.

I have also added the encrypted_data_bag option to the client.rb being that
it is a useful option to have for more secure environments.

[lp:~cloud-init-dev/cloud-init/trunk]

1072. By Scott Moser on 2015-03-03

packages/debian: move software-properties-common to recommends, add eatmydata

1073. By Scott Moser on 2015-03-04

Convert dmidecode values to sysfs names before looking for them.

dmidecode and /sys/class/dmi/id/* use different names for the same
information. This modified the logic in util.read_dmi_data to map from
dmidecode names to sysfs names before looking in sysfs.

1074. By Scott Moser on 2015-03-04

pull in 'snappy' support

This allows config to disable some of the config modules that were
failing and logging WARN on snapy. Also adds the snappy module
and changes the syslog perms to take a list of user:groups rather
than just a single.

1075. By Scott Moser on 2015-03-04

Add util.message_from_string to wrap email.message_from_string.

This is to work-around the fact that email.message_from_string uses
cStringIO in Python 2.6, which can't handle Unicode.

1076. By Scott Moser on 2015-03-04

Fix hang caused by HTTPretty on Python 3.4.2.

HTTPretty can causes hangs on Python 3.4.2 (and maybe Python 3.4.1), due
to a Python bug (fixed in Python 3.4.3). This works around the problem
in the appropriate Python versions.

See https://github.com/gabrielfalcao/HTTPretty/pull/193 and
https://github.com/gabrielfalcao/HTTPretty/issues/221 for details.

1077. By Scott Moser on 2015-03-05

DataSourceMAAS: adjust local timestamp in case of clock skew

This functionality has been introduced to fix LP: #978127, but was lost
while migrating cloud-init to python3.

1078. By Scott Moser on 2015-03-05

snappy: disable by default

this does 2 things actually
a.) disables snappy by default, and adds checks to filesystem to enable it
    this way it runs on snappy systems, but not on others.
b.) removes the 'render2env' that was mostly spike code.

1079. By Scott Moser on 2015-03-10

DataSourceMAAS: remove debug statement

1080. By Scott Moser on 2015-03-10

DataSourceMAAS: fix timestamp error in oauthlib

oddly enough, the timestamp you pass into oauthlib must be a None
or a string. If not, raises ValueError:
  Only unicode objects are escapable. Got 1426021488 of type <class 'int'>

1081. By Oleg Strikov on 2015-03-11

userdata-handlers: python3-related fixes on do-not-process-this-part path

Cloud-init crashed when received multipart userdata object with
'application/octet-stream' part or some other 'application/*' part
except archived ones (x-gzip and friends). These parts are not
processed by cloud-init and result only in a message in the log.
We used some non-python3-friendly techniques while generating
this log message which was a reason for the crash.

1082. By Scott Moser on 2015-03-16

emit_upstart: fix use of undeclared variable

1083. By Scott Moser on 2015-03-17

SmartOS: fixes for python3 reading from serial device.

We were hitting exceptions when writing to the SmartOS serial console and, once
that was fixed, we were hanging permanently waiting for b"." == "." to be true.

This fixes both of those issues.

1084. By Scott Moser on 2015-03-17

systemd: update config and final to run even if init jobs fail

1085. By Scott Moser on 2015-03-25

Update SmartOS data source to use v2 metadata.

v2 metadata is described at
 http://eng.joyent.com/mdata/protocol.html

1086. By Scott Moser on 2015-03-27

NoCloud: the local portion of NoCloud incorrectly claimed datasources

The intent has always been for the local datasource (NoCloud) to require
the provider of metadata to provide 'dsmode=local'. If that wasn't found,
then the default 'dsmode' would be 'net', and the NoCloudNet datasource
would then find the data.

The bug here was that the default 'net' wasn't being set when
data was found on a local source.

1087. By Scott Moser on 2015-03-27

fix snappy package installation.

Some tests added here to make sure this works.
The install '--config' will only start to work with the next
version of snappy-go.

1088. By Scott Moser on 2015-03-31

adjust cc_snappy for snappy install package with config.

It was believed that to install a package with config the command was:
  snappy install --config=config-file <package>
Instead, what was implemented in snappy was:
  snappy install <package> [<config-file>]

This modifies cloud-init to invoke the latter and changes the tests
appropriately.

1089. By Scott Moser on 2015-04-09

systemd: use network-online instead of network.target (LP: #1440180)

don't declare a Wants/Requires on network.target; this is a passive target
that should only be pulled in by implementors of the networking service.
The requirement for network needs to be expressed as a dependency on
network-online.target.

1090. By Scott Moser on 2015-04-16

pep8 fixes

1091. By Scott Moser on 2015-04-16

fix cloud-config-archive handling

handling of cloud-config-archive input would fail in fully_decoded_payload.
part.get_charset() would return a Charset object, but
get_charset.input_codec is a string suitable for passing to decode.

This handles that correctly, and is more careful about binary data inside
input.

The test added verifies that cloud-config inside a cloud-config-archive
is handled correctly and also that binary data there is ignored without
exceptions raised.

1092. By Joshua Harlow on 2015-04-21

Add functionality to fixate the uid of a newly added user.

1093. By Joshua Harlow on 2015-04-21

Don't overwrite the hostname if the user has changed it after we set it

1094. By Joshua Harlow on 2015-04-21

Update changelog with previously merged branches

1095. By Joshua Harlow on 2015-04-21

Add start of/improved CloudStack documentation

1096. By Joshua Harlow on 2015-04-21

Fix +tests for GCE datasource not handling per-instance SSH keys

1097. By Scott Moser on 2015-04-21

sysvinit: make cloud-init-local run before networking is brought up

While booting a VM from a snapshot image, network service use to find the
residual network config scripts and use to configure the old IPs.
cloud-init-local use to run later and populate the config scripts with latest
information, which does not override the previous configuration.

To solve this, we are ensuring that cloud-init-local runs before network.

1098. By Scott Moser on 2015-05-01

Azure: if user has set hostname, do not overwrite it

Due to the way the azure boot works, where we identify the hostname to the
fabric, we were inadvertently overwriting hostname that the user had set.

1099. By Scott Moser on 2015-05-01

fix 'Make pyflakes'

1100. By Scott Moser on 2015-05-01

Update is_disk_used for changed enumerate_disk output.

1101. By Scott Moser on 2015-05-01

Fix exception when running with no arguments on Python 3

1102. By Scott Moser on 2015-05-14

read_seeded: fix reed_seeded after regression

read_seeded was assuming a Response object back from load_tfile_or_url
but load_tfile_or_url was returning string.

since the only other user of this was a test, move load_tfile_or_url to
a test, and just do the right thing in read_seeded.

1103. By Scott Moser on 2015-05-15

detect that CentOS 7 uses systemd, write previous-hostname in that case.

Also, on RHEL-type systems using systemd,
/var/lib/cloud/data/previous-hostname would never get created (because
rather then write to files, it was executing hostnamectl)

1104. By Scott Moser on 2015-05-15

Azure: remove dependency on walinux-agent

This takes away our dependency on walinux-agent, by providing a builtin
path for doing cloud-init had delegated to it.

Currently the default is to still use the old path, but adding this code
in will allow us to move to the new code path with more confidence.

1105. By Scott Moser on 2015-05-15

pep8 fixes

1106. By Scott Moser on 2015-05-15

temporarily disasble test if no bin/cloud-init

the tests of bin/cloud-init would fail in a package build environment.
so, temporariliy skip them in that environment.

1107. By Scott Moser on 2015-05-19

packages/brpm: fix for oauth library name

brpm was broken when change from oauth to oauthlib.

1108. By Scott Moser on 2015-05-19

EC2: be aware of eu-central-1 availability zone

eu-central-1 means that 'central' is a direction to update the
regular expression to understand.

1109. By Scott Moser on 2015-05-22

Azure: Redact on-disk user password in /var/lib/ovf-env.xml

The fabric provides the user password in plain text via the CDROM,
and cloud-init has previously wrote the ovf-env.xml in /var/lib/waagent
with the password in plain text. This change redacts the password.

1110. By Scott Moser on 2015-06-01

Doc: include information on vendor-data in OpenStack

1111. By Scott Moser on 2015-06-05

packages/debian/control.in: mention recommends of gdisk

some of the partitioning code in 'disk_setup' module needs sgdisk.
In the future that could move to using sfdisk also but for now we
do need sgdisk for this.

1112. By Scott Moser on 2015-06-10

check for systemd using sd_booted() semantics

The existing code determines if systemd is in use by looking at the
distribution name and version. This is prone to error because:
- RHEL derivatives other than CentOS (e.g., Scientific Linux) will fail
  this test
- Distributions that are not derived from RHEL also use systemd

This patch makes cloud-init use the same logic that is used in systemd's
sd_booted() method
 http://www.freedesktop.org/software/systemd/man/sd_booted.html

1113. By Dan Watkins on 2015-06-10

Add a cloud-init plugin for helping users register and subscribe their RHEL-based systems.

This patch adds a cloud-init plugin for helping users register and subscribe
their RHEL based systems. As inputs, it can take:

- user and password OR activation key and org | requires on of the two pair
- auto-attach: True or False | optional
- service-level: <string> | optional
- add-pool [list, of, pool, ids] | optional
- enable-repos [list, of, yum, repos, to, enable] | optional
- disable-repos [list, of, yum, repos, to, disable] | optional

You can also pass the following to influence your registration via rhsm.conf:

- rhsm-baseurl | optional
- server-hostname | optional

1114. By Dan Watkins on 2015-06-10

Add ChangeLog entry for last merge.

1115. By Scott Moser on 2015-06-15

cc_rh_subscription: fixes for python3

1116. By Scott Moser on 2015-06-15

apt_configure: fix importing of apt gpg keys under in python3

1117. By Scott Moser on 2015-06-16

growpart: fix specification of 'devices' list.

given config:
 {'growpart': {'devices': ["/"]}}

the 'devices' was ignored, it was incorrectly read from the top
level non-namespaced location.

1118. By Scott Moser on 2015-06-16

Use wget to fetch CloudStack passwords.

Different versions of the CloudStack password server respond differently;
wget handles these nicely for us, so it's easier to just use wget.

1119. By Scott Moser on 2015-07-10

Fix regressed logic causing rightscale userdata to no longer work.

Change the logic for user data handling back to the original intent. A style
change also modified the logic of the user data handling leading to user data
being ignored even if it exists.

1120. By Scott Moser on 2015-07-16

CloudSigma: encode/decode data before communicating over the serial channel

this fixes the cloudsigma datasource when used with python3.

1121. By Dan Watkins on 2015-07-20

Return a sensible value for DataSourceGCE.availability_zone.

1122. By Scott Moser on 2015-07-21

tests: fix TestHandlerHandlePart tests

these tests were previously passing, but doing so erroneously.
I believe that an update to mock caused them to start failing.

I've updated the tests now. The simple change is
replacing 'assert_called_with_once' with 'assert_called_once_with'.

The second set of changes is seemingly a correction of the following tests
expectations:
 test_normal_version_2 : was not expecting to get frequency passed into
    handle_part, but should have been.
 test_no_handle_when_modfreq_once: was expecting to have handle_part
    called even though the test implies otherwise.
 test_exception_is_caught: this test just looked broken.
    Now, we're testing that the part handler is called and that no
    exception is raised past handle_part

1123. By Scott Moser on 2015-07-21

fix 'make pyflakes'

1124. By Scott Moser on 2015-07-22

packages/debian/control.in: add depends on iproute2

tests (specifically DataSourceOpenNebula) runs 'ip' which comes from
iproute2.

1125. By Scott Moser on 2015-07-22

fixes bug: https://launchpad.net/bugs/1461242
ssh: generate ed25519 host keys if supported

now we attempt to generate ed25519 host keys.
If ssh-keygen does not support it, a debug log message will be written.

1126. By Scott Moser on 2015-07-22

provide data source to mirror selection code for region-specific mirrors.

This makes the full data source available to the mirror selection code,
which means we can implement region logic on the data source for
everything to use rather than it being mirror-selection-specific.

It also implements that region logic for EC2 and GCE.

1127. By Scott Moser on 2015-07-22

mounts: support reliably detecting and using Azure ephemeral disks

Azure's ephemeral disks are not guaranteed to be assigned the same name by
the kernel every boot. This causes problems on ~2% of Azure instances, and
can be fixed by using udev rules to give us a deterministic path to mount;
this patch introduces those udev rules and modifies the Azure data source
to use them.

Changes to a couple of config modules were also required. In some places,
they just needed to learn to dereference symlinks. In cc_mounts this
wasn't sufficient because the dereferenced device would have been put in
/etc/fstab (rather defeating the point of using the udev rules in the
first place). A fairly hefty refactor was required to separate "is this a
valid block device?" from "what shall I put in fstab?".

1128. By Scott Moser on 2015-07-28

_read_dmi_syspath: fix bad log message causing unintended exception

1129. By Scott Moser on 2015-07-28

rsyslog: add new format of config

new format of config is namespaced under top level 'rsyslog'
rather than having 3 tokens in the top level namespace.

Additionally added here are:
 * tests
 * support for legacy config
 * support for 'remotes' configuration allowing simple config like
   rsyslog: remotes: {'myremote': "192.168.1.1"}

1130. By Scott Moser on 2015-07-28

rsyslog: skip empty or None in remotes format

This allows user to specify the following to overwrite a previously
declared entry without warnings.
 rsyslog: {'remotes': {'foo': None}}

1131. By Scott Moser on 2015-07-31

status_wrapper in main: fix use of print_exc when handling exception

1132. By Scott Moser on 2015-07-31

status_wrapper in main: correct fix use of print_exc when handling exception

1133. By Scott Moser on 2015-08-05

pep8 fixes

1134. By Scott Moser on 2015-08-07

Add initial reporting module and events

1135. By Scott Moser on 2015-08-07

reporting: fix logging reproter and tests

1136. By Scott Moser on 2015-08-10

reporting: remove unused variable, actually print in PrintHandler

1137. By Scott Moser on 2015-08-12

swap: create swap with fallocate if possible

fallocate is much faster than 'dd' for creating and initializing a
swap file.

1138. By Scott Moser on 2015-08-25

MAAS: fixes to data source and OauthUrlHelper

the previous version was broken. The vital fixes here are:
 * adding parsedate and oauth1 imports to url_helper
 * fix skew_data usage intending to use self.skew_data

Additionally:
 * reorder imports in url_helper
 * fixes to python3 -m cloudinit.sources.DataSourceMaas

1139. By Scott Moser on 2015-09-01

Handle symlink mount points in mount_cb.

The Azure data source now uses a /dev/disk symlink to identify devices,
but the dereferenced version of this appears in the mount table.
mount_cb therefore doesn't identify when a disk is already mounted, and
attempts to mount it a second time (which fails with NTFS).

1140. By Scott Moser on 2015-09-02

sync curtin reporting changes back to cloud-init

this pulls in the reporting framework that we'd been working on
in curtin back to cloud-init.

1141. By Scott Moser on 2015-09-03

reporting/events: catch a final remaining incorrect use of 'reporting'

1142. By Scott Moser on 2015-09-08

NoCloud: fix consumption of vendor-data

the content of vendordata was was being assigned to vendordata,
rather than vendordata_raw. The result was that it is not processed
for includes or part handlers or other things as it is in other
datasources.

1143. By Scott Moser on 2015-09-08

  power_state: support 'condition' argument

  if 'condition' is provided to config in power_state, then
  consult it before powering off.

  This allows the user to shut down only if a condition is met, and
  leave the system in a debuggable state otherwise.

  An example is as simple as:
   power_state:
     mode: poweroff
     condition: ['sh', '-c', '[ -f /disable-poweroff ]']

1144. By Scott Moser on 2015-09-11

snappy: enable ssh on snappy if ssh keys are provided or password auth

If the user provides ssh keys or requests password auth in their
user-data then we will enable ssh by default.

1145. By Ben Howard on 2015-09-15

Change Snappy SSH enabled default from false to 'auto' (LP: #1494816)

1146. By Scott Moser on 2015-09-17

webhook: report with json data

the handler was passing a dictionary to readurl
which was then passing that on to requests.request as 'data'.
the requests library would urlencode that, but we want the
json data posted instead.

1147. By Scott Moser on 2015-09-29

MAAS: fix issues with url_helper and oauth module

This would cause problems in the event that we actually had a bad
clock. We add a retry in the main (for test) also, to ensure that
the oauth timestamp fix gets in place.

1148. By Scott Moser on 2015-10-09

support configuring and installing the Ubuntu fan driver

  #cloud-config
  fan:
    config: |
      # fan 240
      10.0.0.0/8 eth0/16 dhcp
      10.0.0.0/8 eth1/16 dhcp off
      # fan 241
      241.0.0.0/8 eth0/16 dhcp
    config_path: /etc/network/fan

1149. By Scott Moser on 2015-10-15

azure: support extracting SSH key values from ovf-env.xml

Azure has or will be offering shortly the ability to directly define the SSH
key value instead of a fingerprint in the ovf-env.xml file. This patch
favors defined SSH keys over the fingerprint method (LP: #1506244).

1150. By Dan Watkins on 2015-10-20

AltCloud: Remove --quiet option from udevadm call

1151. By Ben Howard on 2015-11-02

Enable backports source pockets for Ubuntu archives (LP: #1177432).

1152. By Ben Howard on 2015-11-02

Fixed per MP review

1153. By Scott Moser on 2015-11-04

Ubuntu templates: make sources.list consistent with ISO installs.

This modifies a designed decision to have trimmer sources.list than
one would find on a server iso install. Specifically, here we now
enable: restricted and multiverse and backports.

1154. By Scott Moser on 2015-11-05

ubuntu templates: enable backports

Previous commit intended to enable backports, but failed to do so.

1155. By Scott Moser on 2015-11-10

cc_mounts: use 'nofail' if system uses systemd. (LP: #1514485)

'nofail' is the systemd equivalent of the mountall/upstart 'nobootwait'.
So, if a system uses systemd, then use 'nofail' in /etc/fstab for
ephemeral mounts.

1156. By Scott Moser on 2015-11-18

Azure: get instance id from dmi instead of SharedConfig

Replace the use of SharedConfig.xml in both the walinuxagent case,
and the case where we communicate with the Azure fabric ourselves.

The instance id present in the dmi data is unfortunately different
that that in the SharedConfig. This means that something needs
to handle migration so that a reboot after newer version is installed
will not re-run first instance things.

In Ubuntu this is being handled in packaging.

1157. By Scott Moser on 2015-11-30

systemd/power_state: fix power_state when cloud-final exited failure

if a runcmd exited failure, then power_state would not work.
This was because systemd was killing off subprocesses, and cloud-init
implemented power off with a subprocess that waited for the parent
to exit.

1158. By Scott Moser on 2016-02-04

tox: use test-requirements.txt and requirements.txt

This just allows stops us from repeating ourselves in tox.ini
from what is in test-requirements and requirements.txt.

1159. By Scott Moser on 2016-02-04

SmartOS: Add support for Joyent LX-Brand Zones

LX-brand zones on Joyent's SmartOS use a different metadata source
(socket file) than the KVM-based SmartOS virtualization (serial port).
This patch adds support for recognizing the different flavors of
virtualization on SmartOS and setting up a metadata source file object.
After the file object is created, the rest of the code for the datasource

1160. By Scott Moser on 2016-02-04

systemd: support using systemd-detect-virt to detect container

running-in-container is a wrapper around systemd-detect-virt -c,
and was removed from ubuntu. This adjusts to run if only running-in-container
is available.

1161. By Dan Watkins on 2016-02-15

Correct lock_passwd in docs

1162. By Scott Moser on 2016-02-16

Handle escaped quotes in WALinuxAgentShim.find_endpoint

1163. By Scott Moser on 2016-02-25

lxd: add support for setting up lxd using 'lxd init'

If lxd key is present in cfg, then run 'lxd init' with values from the
'init' entry in lxd configuration as flags.

1164. By Scott Moser on 2016-03-01

tox.ini: only specify py3 not specific py34

This makes tox work on xenial where python3 is python3.5
and on older (trusty) where python3 is python3.4.

1165. By Scott Moser on 2016-03-01

lxd: general fix after testing

A few changes:
 a.) change to using '--name=value' rather than '--name' 'value'
 b.) make sure only strings are passed to command
     (useful for storage_create_loop: which is likely an integer)
 c.) document simple working example
 d.) support installing zfs if not present and storage_backedn has it.

1166. By Scott Moser on 2016-03-01

Add Image Customization Parser for VMware vSphere Hypervisor Support.

This is the first changeset submitted as a part of project to
add cloud-init support for VMware vSphere Hypervisor. This changeset
contains _only_ the changes for a simple python parser for a
Image Customization Specification file pushed by VMware vSphere
hypervisor into the guest VMs. In a later changeset, will be submitting
another patch to actually detect the underlying VMware vSphere hypervisor
and do the necessary customization.

1167. By Scott Moser on 2016-03-01

timezone: use a symlink when updating /etc/localtime

Unless /etc/localtime is an existing file and not a symlink,
then we will symlink instead of copying the tz_file to /etc/localtime.

The copy was how older versions of Ubuntu handled this.
Those versions should have a file /etc/localtime . When the symlink
is preferred, then it should already exist in the image.

1168. By Scott Moser on 2016-03-03

fix pyflakes reported issues, and run it during package build and tox.

1169. By Scott Moser on 2016-03-03

lxd fix bug and only run if enabled.

The lxd module would run lxd init even if no lxd/init config was provided.

1170. By Scott Moser on 2016-03-04

packages/debian: make trunk packaging closer to ubuntu

The big difference is using:
 ${python3:Depends} or ${python:Depends}
rather than explicitly listing the dependencies (via template ${requires}).

which means we get paths of
  /usr/lib/python3/dist-packages/..
rather than
  /usr/lib/python3.5/dist-packages/..
when built on xenial.

Additionally it seems we no longer need this strange line.
  # Because setup tools didn't copy data...

1171. By Scott Moser on 2016-03-04

systemd: support disabling cloud-init via file or kernel cmdline

This adds a systemd generator for a 'cloud-init.target'. That target will
be WantedBy multi-user.target in the default case. If there is a file
/etc/cloud/cloud-init.disabled or the kernel command line contains
'cloud-init=disabled' then cloud-init will not affect boot at all.

There are some packages/debian changes to affect this:
 * postinst, preinst: these are necessary to remove some
   old target files for multi-user.target (LP: #1552999)
 * changes to include these files in the debian source package.
 * rules.in: supports DEB_BUILD_OPTIONS=nocheck to not run check

setup.py: mostly changes to support installing the generator
          but also pep8 fixes along the way

systemd/*: make each of the services 'WantedBy=cloud-init.target'
           rather than being wanted by multi-user.target

1172. By Scott Moser on 2016-03-04

Apply pep8, pyflakes fixes for python2 and 3

Update make check target to run pep8 and run pyflakes or pyflakes3
depending on the value of 'PYVER'. This way the python3 build
environment does not need python2 and vice versa.

Also have make check run the 'yaml' test.

tox: have tox run pep8 in the pyflakes

1173. By Scott Moser on 2016-03-04

Add a kill switch for customization on VMware platform.

The customization is set to False by default and is triggered only
when the option disable_vmware_customization is set to false in
/etc/cloud/cloud.cfg.

1174. By Dan Watkins on 2016-03-04

Enable password changing via a hashed string

1175. By Dan Watkins on 2016-03-04

Added Bigstep datasource

1176. By Scott Moser on 2016-03-04

pep8: update formatting to pass pep8 1.4.6 (trusty) and 1.6.2 (xenial)

make check fails in a trusty sbuild due to different rules on older pep8.
Fix formatting to pass in older and newer pep8.

1177. By Scott Moser on 2016-03-08

BigStep: enable datasource in default settings

This enables BigStep in the default settings, and also
changes the datasource to not raise exception if we are not
on BigStep.

1178. By Scott Moser on 2016-03-08

No longer run pollinate by default in seed_random

The user can still choose to run pollinate here to seed their
random data. And in an environment with network datasource, that
would be expected to work. However, we do not want to run it any
more from cloud-init because
a.) pollinate's own init system jobs should get it ran before ssh,
    which is the primary purpose of wanting cloud-init to run it.
b.) with a local datasource, there is no network guarantee when
    init_modules run, so pollinate -q would often cause issues then.
c.) cloud-init would run pollinate and log the failure causing
    many cloud-init specific failures that it could do nothing about.

Additionally, add documentation for the seed_random config module.

1179. By Scott Moser on 2016-03-10

add default user to 'lxd' group and create groups when necessary.

This add 'lxd' to the list of groups that the default user is added to.
It also changes behavior to create any necessary groups that are listed
for the user rather than failing to add the user.

Theres also a fix for usage of logexc that I found along the way.

1180. By Scott Moser on 2016-03-10

dmi data: fix failure of reading dmi data for unset dmi values

it is not uncommon to find dmi data in /sys full of 'ff'. utf-8
decoding of those would fail, causing warning and stacktrace.

Return '.' instead of \xff. This is what dmidecode would return.

$ dmidecode --string system-product-name

1181. By Scott Moser on 2016-03-14

change return value for dmi data of all \xff to be ""

Previously we returned a string of "." the same length as the dmi field.
That seems confusing to the user as "." would seem like a valid response
when in fact this value should not be considered valid.

So now, in this case, return empty string.

1182. By Scott Moser on 2016-03-14

some systemd cleanups

 generator:
  * write to directory /run/cloud-init/ rather than /run
  * read from /proc/1/cmdline for the command line if inside a container

 cloud-init-local.service: now run fully before any networking comes up.
 cloud-init.service: run before network-online.target and after
    networking.service. This ends up meaning other things that require
    network-online.target will run after cloud-init.service, but
    cloud-init.service will still have networking.

    The 'networking.service' is ifupdown specific. Further changes would
    be needed for other networking systems (networkd for example)

1183. By Scott Moser on 2016-03-14

doc: document that volume label must be 'cidata' NocCloud datasource.

1184. By Scott Moser on 2016-03-14

doc/ add new logo to rtd (read the docs) docs.

1185. By Scott Moser on 2016-03-14

fix ssh_pwauth behavior to function as documented.

Add option checking for ssh_pwauth to bring behavior inline with the
description cloud-config.txt example.

Previously, setting 'ssh_pwauth' to 'unchanged' or '' would result
in an empty value for PasswordAuthentication when it should have
simply not been modified.

1186. By Scott Moser on 2016-03-17

Send proper SUCCESS / FAILED events to the underlying VMware hypervisor.

Added code to customize timezone.
Added few utility functions to send events to the VMware hypervisor.
Re-factored the code little bit.
Added code to send SUCCESS event when customization succeeds.
Added code to send FAILED event if any error occurs during customization.

1187. By Scott Moser on 2016-03-18

debian packaging: adjust build-depends for xenial

python3 support was moved out of pyflakes into python3-pyflakes.
Adjust the package to build on trusty where python3-pyflakes was
not present and also on xenial where it is.

Note, this does mean that sbuild now requires '--resolve-alternatives'.
That is how it is used on launchpad but is not the default in sbuild.

1188. By Scott Moser on 2016-03-22

quickly check to see if the previous instance id is still valid

This adds a check in cloud-init to see if the existing (cached)
datasource is still valid. It relies on support from the Datasource
to implement 'check_instance_id'. That method should quickly determine
(if possible) if the instance id found in the datasource is still valid.

This means that we can still notice new instance ids without
depending on a network datasource on every boot.

I've also implemented check_instance_id for the superclass and for
sources:
  DataSourceAzure (check dmi data)
  DataSourceConfigDrive (check dmi data)
  DataSourceNocloud (check the seeded data or kernel command line)
  DataSourceOpenstack (check dmi data)

1189. By Scott Moser on 2016-03-24

support network configuration in cloud-init --local

this allows 'cloud-init --local' to fully run before networking comes up.
By doing so, we can now cleanly apply networking to the new system.

This adds support for reading ConfigDrive network configuration
and also from NoCloud. The support is only present for ubuntu/debian
at the current time. Other distros will follow.

Also ability to specify network configuration on kernel command line
via either ip= or network-config=<base64>.

1190. By Scott Moser on 2016-03-29

improve how cloud-init-wait waits

Instead of sleep and check loop, use 'udevadm settle' to wait.
since we run from a udev event, this is sufficient.
udev settle will exit when either of
   a.) the file exists
   b.) the udev event queue has all been processed.
   c.) timeout is reached.

Since cloud-init-wait is being run as a udev event, 'b' cannot
be satisfied until it finishes. Thus, this essentially becomes a
inotify based wait for the file /run/cloud-init/network-config-ready
and no loops are needed.

1191. By Scott Moser on 2016-03-29

apply_network_config improvements

3 things here:
 a.) do not raise exception, only warn when trying to apply a network
     config for a distro that does not have an implementation.
     This is important since debian/ubuntu is the only one *with* an
     implementation at the moment
 b.) apply network config in 'cloud-init --local' even if there is
     no datasource found. This means that the fallback datasource has
     to get things right.
 c.) do not write 70-persistent-net.rules
     the code was writing both 70-persistent-net.rules and
     /etc/systemd/network/50-cloud-init-*.link files
     that would just be confusing.

1192. By Scott Moser on 2016-03-29

Misc fixes for VMware Support.

 - Modified the code to look for customization specification file in
   /var/run/vmware-imc/ directory instead of /tmp
 - Fixed the 'seed file' issue. There was a regression in DataSourceOVF.py
   file. Fixed it.

1193. By Scott Moser on 2016-04-04

FreeBSD: improvements for packages, setting password and timezone

 - Implement set_passwd
 - Implement set_timezone
 - support installing packages
 - Use /bin/tcsh as default user shell (FreeBSD default)
 - Change default username to freebsd
 - Enable set-passwords, package-update-upgrade-install and timezone modules

1194. By Scott Moser on 2016-04-04

DataSource: set ds_cfg to be a dictionary

if the Datasource does not have an entry in config, then
set it to be a empty dictionary rather than None.
Also remove places that did this elsewhere.

1195. By Scott Moser on 2016-04-04

locale: list unsupported environment settings in warning

Now if you log in with unsupported locale, you'll see:
 The unknown environment variables are:
   LC_CTYPE=en_GB.utf-8 LC_MESSAGES=en_GB.utf-8 LC_ALL=en_GB.utf-8

1196. By Scott Moser on 2016-04-04

disk_setup: correctly send --force to mkfs on block devices

Send the --force flag to mkfs or other filesystems when target
is a block device. This fixes a general code flow issue where
we were setting the --force flag.

1197. By Scott Moser on 2016-04-04

chef: fix chef installation from gems

Installation from gems was previously always broken. This
fixes the order or parameters calling install_chef_from_gems.

1198. By Scott Moser on 2016-04-04

systemd: do not specify After of obsolete syslog.target (LP: #1536964)

syslog.target is obsolete in debian, this fixes a lintian warning.

1199. By Scott Moser on 2016-04-06

write_files: fix decompression of content

When provided with gzipped data, an exception would be raised
because of a conversion to string.

This fixes the issue and adds a test for write_files.

1200. By Scott Moser on 2016-04-06

fix adding of users when no group is specified

revision 1179 regressed adding a user that did not have a 'groups'
entry present in cloud-config.
This handles that correctly, making 'add_user' able to take:
  a.) groups="group1,group2"
  b.) groups=["group1", "group2"]
  c.) groups=None
  d.) no groups parameter

Additionally, if a primary group is specified it will also be created.

End result is that this is functional:
 #cloud-config
 groups: ["sudo"]
 users:
   - name: sysop
     primary-group: sysop
     groups: "sudo,adm"
     shell: /bin/bash
   - name: user1
     primary-group: users
     groups: sudo
   - name: foo1
   - name: bar
     gecos: Bar
     groups: ["bargroup"]

Resulting in:
 $ groups sysop
 sysop : sysop adm sudo
 $ groups user1
 user1 : users sudo
 $ groups foo1
 foo1 : foo1
 $ groups bar
 bar : bar bargroup

1201. By Scott Moser on 2016-04-06

Ensure that a resolve conf object is written as a string.

Instead of passing the raw object and expecting the write_file to work
automatically make sure we explicitly pass the string version of it so
that the write_file routine can correctly encode/decode it as needed.

1202. By Scott Moser on 2016-04-11

lxd: adds basic support for dpkg based lxd-bridge configuration.

It exposes the most useful debconf keys as cloud-init configuration keys.

1203. By Scott Moser on 2016-04-12

DataSourceNoCloud: fix check_instance_id when upgraded

A system that had booted, upgraded, and then rebooted would show
a cloud-init stack trace as it attempted to run new code
with the old pickled object. The old object would not have the
seed_dirs attribute. So we check and fallback correctly if that
is not present.

1204. By Scott Moser on 2016-04-12

SmartOS, CloudSigma: fix error when dmi data is not availble

In Cloudsigma, the datasource would warn if no product id was availble.
SmartOS would log exception. This fixes both of those, changing
the warning to a debug message.

1205. By Scott Moser on 2016-04-12

rh_subscription: only check subscription if configured

The rh_subscription config module would attempt to connect to the RHN servers
even when no config is provided.
Now, instead check to make sure that valid config is provided first.
That consists of username and password or a activation key.

1206. By Scott Moser on 2016-04-13

skip bridges when generating fallback networking

It does not make sense to consider bridges when searching for fallback
networking. If the system is configured with a bridge, then its probably
for some purpose other than to get to a metadata service.

Considering the bridge could make cloud-init pick the wrong device on reboot.

1207. By Scott Moser on 2016-04-13

chef: straighten out validation_cert and validation_key

Now, validation_key is always a path to a file, as it is in
chef's client.rb syntax.

validation_cert is always the *content* of that file that should
be written. However, if validation_cert is the string "system",
then we do not write that value, but rather assume the file exists.

1208. By Scott Moser on 2016-04-13

phone_home: allow usage of fqdn

This simply allows the phone_home template to pass the systems fully
qualified domain name.

1209. By Scott Moser on 2016-04-14

fallback net config: do not consider devices starting with 'veth'

Just skip devices that are named veth*.
The fix here is to ignore lxd created devices, but any other veth
device that is created at this point in boot is probably not the
right interface to dhcp on.

1210. By Scott Moser on 2016-04-15

networking: no longer delete eth0.cfg on debian/ubuntu

Ubuntu cloud images in created a file during build that
would interfere with cloud-init's discovered or rendered networking.
To avoid the issues, cloud-init was deleting
/etc/network/interfaces.d/eth0.cfg .

The build process no longer creates this file.
However, to address any existing files cloud-init will still remove
the file if it has known content and warn otherwise.

1211. By Scott Moser on 2016-04-15

apply networking less often

Do not apply networking configuration whenever a previous datasource
has been loaded from disk and found to be valid (via positive
 return 'check_instance_id' or user configuration of manual_cache_clean).

This effectively means that we apply fallback networking only once
per instance rather than every boot on any datasource with
'check_instance_id' implemented.

1212. By Scott Moser on 2016-04-15

network: do not write interface as 'auto' from ip= on command line.

When ip= on the kernel command line defines the networking, set
those network devices to be manually controlled, instead of 'auto'.

The reason for this is that if they're marked as 'auto':
 a.) a second attempt will be made to ifup them.
 b.) they'll be brought down on shutdown

'b' is problematic on network root filesystem.

Also this picks up 2 changes from curtin's net module:
  - Cleanup newline logic so we always have a clean '\n\n' between stanza
  - Add a unittest to validate bonding network config render, specifically
    when to emit auto $iface for dependent bond slaves.

1213. By Dan Watkins on 2016-04-29

cloudstack: Only use DHCPv4 lease files as a datasource

1214. By Scott Moser on 2016-04-29

Config Drive: fix check_instance_id signature.

After reboot cloud-init would fail as the previously pickled object
would have a check_instance_id signature but it didn't match expected

1215. By Scott Moser on 2016-04-29

Paths: fix instance path if datasource's id has a '/'.

If the datasource's instance id contained a '/' then the instance_id path
would not be as expected under /var/lib/cloud/instances/instance_id.

1216. By Scott Moser on 2016-05-06

fix timestamp in reporting events.

If no timestamp was passed into a ReportingEvent, then the default was
used. That default was 'time.time()' which was evaluated once only at
import time.

1217. By Joshua Harlow on 2016-05-12

Enable flake8 and fix a large amount of reported issues

1218. By Matt Fischer on 2016-05-17

Document improvements for runcmd/bootcmd

Note that runcmd runs only on first boot.
Note that strings need to be quoted, not escaped.
Switch bootcmd list text to use - not * like everything else.

1219. By Scott Moser on 2016-05-24

fix up tests that take too long due to retries and timeouts

1220. By Scott Moser on 2016-05-25

fix logic error in ec2 get_instance_userdata and slow tests

The change to get_instance_userdata is to fix an issue that
was causing retry in the test when it was not desired.
if user_data returned 404 it means "there was no user-data", so
dont bother retrying. However, _skip_retry_on_codes was returning
False indicating that readurl should retry.

test_merging was creating 2500 random tests, shrink that down to 100.
test_seed_runs is still on my system the slowest test, but
taking < .5 seconds where it was taking > 3.

1221. By Scott Moser on 2016-05-26

kernel command line: override all local settings

settings on the kernel command line (cc:) were documented to override
all local settings, but a bug in implementation meant they would only
override those that are in /etc/cloud/cloud.cfg, not any found in
/etc/cloud/cloud.cfg.d.

1222. By Scott Moser on 2016-05-26

packages/bddeb: update to know package mappings for flake8 and hacking

packages/bddeb failed to work after flake8 and hacking were added to
test-requirements.txt. The necessary fix is just to know about the
debian package names for those pypi packages.

1223. By Dan Watkins on 2016-06-02

Improve merging documentation

1224. By Scott Moser on 2016-06-03

Apt sources configuration improvements

- keyid-only (no source statement)
- key only (no source statement)
- custom source.list template
- support long gpg key fingerprints with spaces
- fix issue with key's that were already in the local gpg keyring
- allowing a new format to specify apt_sources in a dictionary instead of a
  list to allow merging of configurations

1225. By Scott Moser on 2016-06-03

improve network configuration

This branch accomplishes several things:
 - centrally handle 'dsmode' to be 'local' or 'net.
   This allows local data sources to run before networking
   but still have user-data read by default when networking is available.

 - support networking information being read on dreamcompute
   dreamcompute's openstack declares networking via the
   /etc/network/interfaces style 'network_config' format.

 - support reading and applying networking information on SmartOS

 - improve reading networking from openstack network_data.json (LP: #1577982)
   add support for mtu and routes and many miscellaneous fixes.

 - support for renaming devices in a container (LP: #1579130).
   Also rename network devices as instructed by the host on
   every boot where cloud-init networking is enabled. This is required
   because a.) containers do not get systemd.link files applied
   as they do not have udev. b.) if the initramfs is out of date
   then we need to apply them.

 - remove blocking of udev rules (LP: #1577844, LP: #1571761)

1226. By Scott Moser on 2016-06-04

tests: fix apt tests to run inside ubuntu build environment

This just mocks out use of lsb_release as it is not available
in a build environment.
Additionally mocks out use of getkeybyid. This admittedly
makes the test for a long key fingerprint not useful as it was
broken only inside getkeybyid.

Also fix 'make yaml' for cloud-config.txt

1227. By Scott Moser on 2016-06-05

skip test_apt_source_list_debian_mirrorfail for now

I've opened bug 1589174 with the intent to fix these tests
that I quickly fixed in the last 2 commits. Those were done
in haste so that we could get a functional trunk build again.

1228. By Scott Moser on 2016-06-07

make networking config provided in system config override datasource.

while datasource provided networking is more dynamic in most cases,
preference should still be given to networking configuration provided
in the system.

This is because the user of the image should be ultimately in control
of the networking configuration if they so choose.

1229. By Scott Moser on 2016-06-10

Change missing Cheetah log warning to debug [Andrew Jorgensen]

In the absence of cheetah, which is a fairly heavy templating engine, and
not strictly needed by anything in cloud-init, the only warning we saw in
the logs was this one from the templater. Degrading this to a debug
message makes any other warnings more relevant.

1230. By Scott Moser on 2016-06-10

Fix apt configure unittests to run in more environments

As well as some improvements that were found along testing them and due to
the fact that we review some of that code again in the scope of curtin
currently.

Tests:
 - add a test for an alternate keyserver
 - harden mirrorfail tests to detect and skip if no network is available
 - improve apt_source related tests to work on CentOS7

Changes:
 - gpg key handling is now in python instead of a shell blob and moved
   to its own module.
 - packages/bddeb has an option to sign as someone else than smoser
 - make exception handling of apt_source features more specific
   (do not catch broad 'Exception')
 - rename some functions to reflect better what they actually do
 - capture some helper subp calls output to avoid spilling into stdout when
   not intended

1231. By Scott Moser on 2016-06-10

clean up temp files made in tests

After a 'tox' run, now there are no tmpdirs left in /tmp.

1232. By Joshua Harlow on 2016-06-10

Refactor a large part of the networking code.

Splits off distro specific code into specific files so that
other kinds of networking configuration can be written by the
various distro(s) that cloud-init supports.

It also isolates some of the cloudinit.net code so that it can
be more easily used on its own (and incorporated into other
projects such as curtin).

During this process it adds tests so that the net process can
be tested (to some level) so that the format conversion processes
can be tested going forward.

1233. By Joshua Harlow on 2016-06-10

Fix the broken import and 'parse_net_config_data' function usage

1234. By Dan Watkins on 2016-06-13

Remove trailing dot from GCE metadata URL (LP: #1581200) [Phil Roche]

1235. By Scott Moser on 2016-06-14

[Revert] Remove trailing dot from GCE metadata URL

This change broke tox tests.

1236. By Scott Moser on 2016-06-14

fix pep8 failure introduced in recent commit.

The commit 1232 (Refactor a large part of the networking code) broke pep8.

1237. By Dan Watkins on 2016-06-15

Re-apply "Remove trailing dot from GCE metadata URL (LP: #1581200) [Phil Roche]"

This commit includes the content of that commit, plus a fix for the tests
(provided by Phil).

1238. By Scott Moser on 2016-06-16

move 'main' into cloudinit/cmd/ for easier testing

This moves bin/cloud-init's content into cloudinit/cmd/main.py,
and then fixes the pep8/flake8 issues with that.

The end result is easier testing of main.

1239. By Scott Moser on 2016-06-16

fix some errors reported by pylint

pylint --errors-only found several errors. Some of the changes
here represent real errors, others just code that pylint did
not like.

1240. By Scott Moser on 2016-06-16

fix usage of OSError.message that will not work in python3

python3's OSError does not have a .message attribute.

1241. By Scott Moser on 2016-06-16

DataSourceNoCloud: fix stack trace on reboot, default to dsmode=net

On reboot (loading module from obj.pkl) we would hit a AttributeError
when trying to access cmdline_id.
Addtionally, dsmode was inadvertantly defaulting to local for
DataSourceNoCloud.

1242. By Scott Moser on 2016-06-16

support network rendering to sysconfig (for centos and RHEL)

This intends to add support for rendering of network data under sysconfig
distributions (centos and rhel). The end result will be support for
network configuration via ConfigDrive or NoCloud on these OS.

1243. By Scott Moser on 2016-06-20

do not render systemd.link files

When fixing bug 1579130, we made cloud-init rename devices itself,
rather than relying on the systemd.link files to do that.
That was necessary to
 - rename devices in a container
 - rename devices on first boot or in any situation when the
   link files in the initramfs were stale.

However, cloud-init was still writing .link files like:
 /etc/systemd/network/50-cloud-init-ens2.link

That leads to just a confusing situation as cloud-init will trump
any renaming systemd does in all cases.

Also added here is a header into the rendered ENI file:
 /etc/network/interfaces.d/50-cloud-init.cfg

that describes how to disable cloud-init networking.

1244. By Scott Moser on 2016-06-21

write_files: if no permissions are given, just use default without warn.

if no permissions were given in a write_files stanza, then
a warning would be emitted.

The fix here is just to special case handling of None.

1245. By Scott Moser on 2016-06-22

user_data: fix error when user-data is not utf-8 decodable

when user-data was not decodable, cloud-init would raise exception.
This also changes the signature of user_data.convert_string.
The 'headers' argument was never used, and woudl have been broken
if it was, as it was expected to be a dictionary but then was
passed in with *headers.

1246. By Scott Moser on 2016-06-27

fix restoring from a datasource that did not have dsmode

On upgrade and reboot, if datasource restored from obj.pkl did not have
a dsmode attribute, then 'init --local' would fail due to stack trace.

1247. By Scott Moser on 2016-06-28

distros/debian.py: fix calling of eni renderer to not render link files

Under revno 1243 a failed attempt was made to not render systemd.link
files into /etc/systemd/network/ . The 'config' that was passed in was
incorrect though, and resulted in link files still getting rendered.

(original bug was LP: #1594546).

1248. By Joshua Harlow on 2016-07-13

Dict comprehensions don't work in 2.6

This fixes a small case of a leftover
dict comprehension being found that stops cloud-init
from working on centos6/rhel6 (which still use py2.6)

1249. By Joshua Harlow on 2016-07-13

Fixes missing/unpacked rpm files

There are a few new files that are missing from being
included in the rpm specification file (which if missing
causes rpmbuild to die) so make sure we add them in.

1250. By Joshua Harlow on 2016-07-13

Remove another stray dict comprehension

1251. By Joshua Harlow on 2016-07-13

Another stray occurence of a dict comprehension being removed

1252. By Joshua Harlow on 2016-07-13

String format requires positions on python 2.6

1253. By Joshua Harlow on 2016-07-13

Fix SmartOS datasource usage of dict comprehensions

1254. By Scott Moser on 2016-07-14

Fix mcollective module with python3

fixes mcollective when used with python3 and also adds a unit test.

1255. By Scott Moser on 2016-07-14

ConfigDrive: fix writing of 'injected' files and legacy networking

Previous commit inadvertently disabled the consumption of 'injected' files
in configdrive (openstack server boot --file=/target/file=local-file)
unless the datasource was in 'pass' mode. The default mode is 'net' so
that was not likely to happen.

Also here are:
a.) some comments to apply_network_config

b.) add backwards compatibility for distros that do not yet implement
    apply_network_config by converting the network config into ENI format
    and calling apply_network.

    This is required because prior to the previous commit, those distros
    would have had 'apply_network' called with the openstack provided
    ENI file. But after this change they will have apply_network_config
    called by cloudinit's main.

c.) add network_state_to_eni for converting net config to eni
    it supports the not-actually-correct 'hwaddress' field in ENI

1256. By Scott Moser on 2016-07-14

improvements to eni rendering

Some improvements here, and some bug fixes.
 - bring curtin revno 394's to support post-up for interface aliases.
 - sort attributes per interface for nicer order and consistent rendering
 - use arrays for each 'section' rather than content += . This allows
   better separation of the sections and also will perform better as long
   strings with += are slow.
 - improve how 'lo' is handled. If a network state that was being rendered
   had an entry for 'lo', then the rendered ENI would have 2 'lo'
   sections.
 - no longer skip 'lo' sections when loading an ENI in parse_deb_config
 - fix inet value for subnets, don't add interface attributes to alias
   (LP: #1588547)

Also add some tests of reading yaml and rendering ENI.

1257. By Joshua Harlow on 2016-07-19

Avoid depending on argparse in 2.7 or greater

At least (currently) for rhel7 the argparse package does
not get installed (even though rpm say it is installed by
the python core package) and this causes things that mention
argparse in there requirements to not believe that argparse
is installed (even though it is) so to avoid this whole mess
we can just avoid depending on argparse in python versions
where we don't need to (since it was included in the stdlib
in python 2.7+)

1258. By Scott Moser on 2016-07-19

mcollective: add tests, cleanups and bug fix when no config in /etc.

Things here:
 - restart rather than 'start' the service, to pick up a config change
   that we would have written.
 - update the config and write cert files whether or not the file
   existed on the system. Previously it would only write the cert
   files if /etc/mcollective/server.cfg already existed.
 - improve test coverage

1259. By Scott Moser on 2016-08-10

README: indicate move to git.

cloud-init development has moved its revision control to git.
It is available at
  https://code.launchpad.net/cloud-init

Clone with
  git clone https://git.launchpad.net/cloud-init
or
  git clone git+ssh://git.launchpad.net/cloud-ini

For more information see
  https://git.launchpad.net/cloud-init/tree/HACKING.rst

Unmerged revisions

1259. By Scott Moser on 2016-08-10

README: indicate move to git.

cloud-init development has moved its revision control to git.
It is available at
  https://code.launchpad.net/cloud-init

Clone with
  git clone https://git.launchpad.net/cloud-init
or
  git clone git+ssh://git.launchpad.net/cloud-ini

For more information see
  https://git.launchpad.net/cloud-init/tree/HACKING.rst

1258. By Scott Moser on 2016-07-19

mcollective: add tests, cleanups and bug fix when no config in /etc.

Things here:
 - restart rather than 'start' the service, to pick up a config change
   that we would have written.
 - update the config and write cert files whether or not the file
   existed on the system. Previously it would only write the cert
   files if /etc/mcollective/server.cfg already existed.
 - improve test coverage

1257. By Joshua Harlow on 2016-07-19

Avoid depending on argparse in 2.7 or greater

At least (currently) for rhel7 the argparse package does
not get installed (even though rpm say it is installed by
the python core package) and this causes things that mention
argparse in there requirements to not believe that argparse
is installed (even though it is) so to avoid this whole mess
we can just avoid depending on argparse in python versions
where we don't need to (since it was included in the stdlib
in python 2.7+)

1256. By Scott Moser on 2016-07-14

improvements to eni rendering

Some improvements here, and some bug fixes.
 - bring curtin revno 394's to support post-up for interface aliases.
 - sort attributes per interface for nicer order and consistent rendering
 - use arrays for each 'section' rather than content += . This allows
   better separation of the sections and also will perform better as long
   strings with += are slow.
 - improve how 'lo' is handled. If a network state that was being rendered
   had an entry for 'lo', then the rendered ENI would have 2 'lo'
   sections.
 - no longer skip 'lo' sections when loading an ENI in parse_deb_config
 - fix inet value for subnets, don't add interface attributes to alias
   (LP: #1588547)

Also add some tests of reading yaml and rendering ENI.

1255. By Scott Moser on 2016-07-14

ConfigDrive: fix writing of 'injected' files and legacy networking

Previous commit inadvertently disabled the consumption of 'injected' files
in configdrive (openstack server boot --file=/target/file=local-file)
unless the datasource was in 'pass' mode. The default mode is 'net' so
that was not likely to happen.

Also here are:
a.) some comments to apply_network_config

b.) add backwards compatibility for distros that do not yet implement
    apply_network_config by converting the network config into ENI format
    and calling apply_network.

    This is required because prior to the previous commit, those distros
    would have had 'apply_network' called with the openstack provided
    ENI file. But after this change they will have apply_network_config
    called by cloudinit's main.

c.) add network_state_to_eni for converting net config to eni
    it supports the not-actually-correct 'hwaddress' field in ENI

1254. By Scott Moser on 2016-07-14

Fix mcollective module with python3

fixes mcollective when used with python3 and also adds a unit test.

1253. By Joshua Harlow on 2016-07-13

Fix SmartOS datasource usage of dict comprehensions

1252. By Joshua Harlow on 2016-07-13

String format requires positions on python 2.6

1251. By Joshua Harlow on 2016-07-13

Another stray occurence of a dict comprehension being removed

1250. By Joshua Harlow on 2016-07-13

Remove another stray dict comprehension