lp:~smoser/cloud-init

Author: Scott Moser
Author Date: 2017-12-21 19:27:50 UTC

address chad's feedback

Author: Chad Smith
Author Date: 2017-12-15 17:12:05 UTC

releasing package cloud-init version 17.2-1-g4089e20c-0ubuntu1

Author: Chad Smith
Author Date: 2017-12-15 05:06:29 UTC

cli: Fix error in cloud-init modules --mode=init.

The cli help docs and argument parser allow the 'init' mode value
which caused a traceback.

Fix the cli to support 'init', 'config' and 'final' modes for the
cloud-init modules subcommand.

Add a check in the cli to raise a ValueError if a new
subcommand ends up allowing an unsupported/unimplemented modes.

Drive by unit test additions for a bit better coverage of error
handling.

LP: #1736600

Author: Scott Moser
Author Date: 2017-12-14 17:33:52 UTC

release 17.2

Bump the version in cloudinit/version.py to be 17.2 and update ChangeLog.

Author: Scott Moser
Author Date: 2017-12-12 19:30:40 UTC

update changelog

Author: Scott Moser
Author Date: 2017-12-12 19:30:40 UTC

update changelog

Author: Scott Moser
Author Date: 2017-12-12 15:21:22 UTC

ds-identify: failure in NoCloud due to unset variable usage.

The previous OVF datasource change added a debug message that
referenced an un-used variable. The failure path would be triggered
if an image was booted with a iso9660 filesystem attached to a device
that was not a cdrom.

A unit test is added for the specific failure found.

Additional safety to avoid 'cidata' labels is also added to the OVF
checker.

LP: #1737704

Author: Scott Moser
Author Date: 2017-12-11 17:57:44 UTC

tests: move to using tox 1.7.5

When we moved to having tox run on tests/ and tools/ we bumped tox
to version 1.7.1. That was an error on my part as I just checked the
version I had locally and didn't go looking for what the newest upstream
release was.

The current version as of today is 1.7.5 and 'tox -e pylint' works with
this version

Author: Scott Moser
Author Date: 2017-12-11 14:48:14 UTC

tests: remove leaked tmp files in config drive tests.

Make sure that some temporary files used by the config drive tests get
cleaned up properly.

Author: Scott Moser
Author Date: 2017-12-01 20:08:44 UTC

setup.py: Do not include rendered files in SOURCES.txt

cloud-init renders template files during its run of setup.py.
Those rendered files were in a temp dir and were making their
way into the SOURCES.txt file. That caused problems for SuSE's build
system that desired for reproducible builds.
  https://reproducible-builds.org/

We do not want to include those, so here we explicitly prune them out.

The issue of SOURCES.txt containing tmp files was reproducible with:
  $ rm -Rf cloud_init.egg-info
  $ git clean --force
  $ rm -Rf ../root.d; python3 setup.py install --root=../root.d
  $ grep ^tmp cloud_init.egg-info/SOURCES.txt

goo: http://bugzilla.opensuse.org/show_bug.cgi?id=1069635

Author: chad smith
Author Date: 2017-12-07 00:26:52 UTC

OVF: improve ds-identify to support finding OVF iso transport.

Previously the OVF transport would not be identified except for when
config files set 'ovf_vmware_guest_customization'. It would also
return DS_MAYBE almost always.

The change here is to add support to ds-identify for storing the
iso9660 filesystems that it finds (ISO9660_DEVS). Then the OVF check
will check that the iso9660 filesystem has ovf-env.xml on it. The least
wonderful part of this is that the check is done by 'grep' for case
insensitive ovf-env.xml.

Future improvement would be to identify VMware's OVF by label or UUID
so we could avoid the grep.

LP: #1731868

Author: Scott Moser
Author Date: 2017-12-07 19:20:43 UTC

do not provide trailing / on pylint.

this is actually a differen thing:
 pylint tools/
 pylint tools

I only know this because I ran 'pylint tools/' before fixing tools/
and did not see any errors. When running 'pylint tools' would show
errors in tools/*.py files.

Author: Scott Moser
Author Date: 2017-12-05 21:33:56 UTC

citest: In NoCloudKVM provide keys via metadata not userdata.

The NoCloudKVM platform was inserting ssh keys via user-data
rather than through meta-data like it is done on other platforms.
This way we are not forced to change the user-data provided.

Also, provide meta-data including a uuid as the instance-id.

Author: Chad Smith
Author Date: 2017-12-01 17:02:33 UTC

releasing package cloud-init version 17.1-46-g7acc9e68-0ubuntu1~17.04.1

Author: Chad Smith
Author Date: 2017-12-01 16:58:53 UTC

releasing package cloud-init version 17.1-46-g7acc9e68-0ubuntu1~17.10.1

Author: Scott Moser
Author Date: 2017-11-27 20:20:37 UTC

better error messages on multiple images found

Author: Scott Moser
Author Date: 2017-11-21 19:23:36 UTC

tests: Use apt-get to install a deb so that depends get resolved.

Instead of using 'dpkg -i' to install a package and then running
apt-get -f install, to hope that it would install needed dependencies
we can just use 'apt-get' directly to do the install.

The 'dpkg/apt-get -f' path was a problem if the installed deb was
older than the available deb. In that case it would get replaced.

Author: Scott Moser
Author Date: 2017-11-20 15:40:46 UTC

integration test: replace curtin test ppa with cloud-init test ppa.

Cloud-init integration tests should not depend on a curtin test ppa.
We already had a cloud-init test ppa for explicitly this purpose.
Just use it instead.

Author: Scott Moser
Author Date: 2017-11-17 21:44:23 UTC

show a warning if no fallback nic found

Author: Scott Moser
Author Date: 2017-11-17 19:14:48 UTC

update test

Author: Scott Moser
Author Date: 2017-11-17 02:59:35 UTC

tools/run-centos: support taking a commitish to run with.

By default, use 'tree', the current working state of your tree.

Author: Scott Moser
Author Date: 2017-11-08 02:35:23 UTC

Replace the temporary i9n.brickies.net with i9n.cloud-init.io.

We had used some dns records in i9n.brickies.net (my personal domain)
as a temporary solution until we got names registered in the cloud-init.io
namespace.

We now have CNAME records for:
  ubuntu.i9n.cloud-init.io
  cloudinit1.cloud-init.io
  cloudinit2.cloud-init.io

Author: Scott Moser
Author Date: 2017-11-06 15:49:33 UTC

re-use ssh connection.

Rather than creating a new ssh connection for each sub-command, re-use
the connection and create a new channel.

In my very un-scientific test with a sample set of 1, and entries in
hosts' /etc/hosts, i found:
 before: 6 commands took 1.277
 after : 6 commands took 0.469

Author: Scott Moser
Author Date: 2017-11-03 16:44:31 UTC

somewhat functional.

server run with:
 python3 -m tests.cloud_tests.jsoncket --port 9799

Then run a client like:
 python -m tests.cloud_tests.jsoncket.client localhost 9799 ls

Author: Scott Moser
Author Date: 2017-10-28 02:26:58 UTC

debian/cloud-init.templates: Fix capitilazation in 'AliYun' name.

The AliYun datasource is spelled 'AliYun' (capital Y) in the datasource
name. The templatel had a lower case 'Y', causing it to never
be found.

LP: #1728186

Author: Scott Moser
Author Date: 2017-10-23 18:02:39 UTC

my-reliable-test

Author: Scott Moser
Author Date: 2017-10-20 17:25:38 UTC

citest: fix remaining warnings raised by integration tests.

There was fallout in a full integration test run from my adding of
 test_no_warnings_in_log
which asserted that there could not be a WARNING found in the
/var/log/cloud-init.log

This fixes 2 of the cases:
 * TestCommandOutputSimple had a valid WARNING written, so adjust its
   test case to allow for that.
 * TestLxdDir had a valid config in the test but the module would
   log a WARNING, so fix the module.

Also updates lxd unit tests to look for WARN themselves.

Author: Scott Moser
Author Date: 2017-10-20 18:58:24 UTC

citest: show the class actual class name in results.

Tests are currently run by creating a temporary subclass of each class
and then executing it (in get_suites). When running the tests suite
the output would contain the temporary class name. That was less than
useful, and made batch runs almost impossible to identify which
test case had an error.

This change goes from output of:
    FAIL: test_no_warnings_in_log \
        (tests.cloud_tests.testcases.get_suite.<locals>.tmp)
To
    FAIL: test_no_warnings_in_log \
       (tests.cloud_tests.testcases.modules.ntp.TestNtp)

Author: Scott Moser
Author Date: 2017-10-17 20:12:59 UTC

schema: Log debug instead of warning when jsonschema is not available.

When operating in expected path, cloud-init should avoid logging with
warning. That causes 'WARNING' messages in /var/log/cloud-init.log.
By default, warnings also go to the console.

Since jsonschema is a optional dependency, and not present on xenial
and zesty, cloud-init should not warn there.

Also here:
* Add a test to integration tests to assert that there are no
  warnings in /var/log/cloud-init.log.
* Update one integration test that did show warning and the related
  documentation and examples.

LP: #1724354

Author: Scott Moser
Author Date: 2017-10-10 18:26:47 UTC

add a test and always call update_repos from handle.

this just gets the disabled-repos=None path called in tests, so we can
verify it functional.

Author: Scott Moser
Author Date: 2017-10-05 18:25:59 UTC

debian/new-upstream-snapshot: use --abbrev=8 for git-describe.

The tools that use "git describe" were just assuming a consisent
number of characters in the hash. It seems ubuntu 16.04 would use 7
and later versions use 8. To avoid that discrepency in developer
environments, set it to 8.

Author: Scott Moser
Author Date: 2017-10-05 18:21:00 UTC

tools: Give specific --abbrev=8 to "git describe"

The tools that use "git describe" were just assuming a consisent
number of characters in the hash. It seems ubuntu 16.04 would use 7
and later versions use 8. To avoid that discrepency in developer
environments, set it to 8.

Author: Scott Moser
Author Date: 2017-10-03 14:55:58 UTC

releasing package cloud-init version 17.1-13-g7fd04255-0ubuntu1

Author: Dimitri John Ledkov
Author Date: 2017-09-30 04:00:18 UTC

Azure, CloudStack: Support reading dhcp options from systemd-networkd.

Systems that used systemd-networkd's dhcp client would not be able to get
information on the Azure endpoint (placed in Option 245) or the CloudStack
server (in 'server_address').

The change here supports reading these files in /run/systemd/netif/leases.
The files declare that "This is private data. Do not parse.", but at this
point we do not have another option.

LP: #1718029

Author: Scott Moser
Author Date: 2017-10-02 19:20:10 UTC

packages/debian/copyright: remove mention of boto and MIT license

boto_utils.py had been removed some time ago, and the current
cloudinit/ec2_utils.py is not based on what was in boto_utils.

We just failed to remove the mention of it from the upstream
debian/copyright. And then put it back in everywhere in recent changes
to get upstream and ubuntu in sync.

Author: Scott Moser
Author Date: 2017-09-25 18:29:13 UTC

AltCloud: Trust PATH for udevadm and modprobe.

Previously we had hard coded paths in /sbin for the udevadm and modprobe
programs invoked by AltCloud. Its more flexible to expect the PATH to
be set correctly.

Debian: #852564

Author: Scott Moser
Author Date: 2017-09-21 20:59:59 UTC

tests: remove a temp file used in bootcmd tests.

The bootcmd test was leaving files in the tmpdir named
 ci-FakeExtendedTempFile.XXXXXX.

This cleans those up.

Author: Scott Moser
Author Date: 2017-09-21 18:59:07 UTC

release 17.1

Bump the version in cloudinit/version.py to be 17.1 and update
ChangeLog.

Author: Scott Moser
Author Date: 2017-09-20 15:32:00 UTC

some updates for run-opensuse networking checking

fix checking for networking

Author: Scott Moser
Author Date: 2017-09-19 18:28:39 UTC

tests: fix ds-identify unit tests to set EC2_STRICT_ID_DEFAULT.

The variable DI_EC2_STRICT_ID_DEFAULT was not being set in unit tests
so when 16.04 built, which changed that setting in patches the tests
would unexpectedly fail.

Author: Scott Moser
Author Date: 2017-09-19 00:47:53 UTC

Makefile: No longer look for yaml files in obsolete ./bin/.

The bin/ dir was deleted some time ago, but the Makefile was still
searching for files down it. This didn't cause any problems other
Than a wierd looking error message in a build log.

Author: Scott Moser
Author Date: 2017-09-16 02:50:21 UTC

GCE: Fix usage of user-data.

This regressed in the rework of GCE datasource to have a main.
The fix really just stores the user-data that was read in
self.userdata_raw, rather than self.userdata. That is consistent
with other datasources and ulitimately how it was before the refactor.

The main is updated to address the fact that user-data is binary data
and may not be able to be printed.

LP: #1717598

Author: Scott Moser
Author Date: 2017-09-18 01:13:05 UTC

add a debian/patches/series.clean to clean cpick

Author: Scott Moser
Author Date: 2017-09-18 01:10:34 UTC

debian/rules: in clean target, run debian/patches/series.clean if present.

This will allow recipe based builds of trunk to strip out the 'cpick'
patches which would not apply (because they're already in trunk).

Author: Scott Moser
Author Date: 2017-09-15 19:08:30 UTC

update changelog

Author: Scott Moser
Author Date: 2017-09-15 18:49:56 UTC

address chad's feedback.

Author: Scott Moser
Author Date: 2017-09-08 21:08:23 UTC

tests: execute: support command as a string, change default env parm.

If a string is passed to execute, then invoke 'sh', '-c', 'string'.
That allows the less verbose execution of simple commands:
  image.execute("ls /run")
compared to the more explicit but longer winded:
  image.execute(["ls", "/run"])

If 'env' was ever modified in execute or a method that it called,
then the next invocation's default value would be changed. Instead
use None and then set to a new empty dict in the method.

Author: Scott Moser
Author Date: 2017-09-07 18:23:22 UTC

docstring fix.

Author: Scott Moser
Author Date: 2017-09-07 18:16:45 UTC

add chad's test.

Author: Scott Moser
Author Date: 2017-09-07 14:30:59 UTC

address feedback: check that it returned maybe.

Author: Scott Moser
Author Date: 2017-08-31 16:39:27 UTC

fix 'bddeb' to default to only systemd

Author: Scott Moser
Author Date: 2017-08-30 14:35:30 UTC

tox: make xenial environment run with python3.6

The pinned versions of python packages in xenial do not work with
python3.6. Currently, the failure can be seen with:
 $ tox -e xenial tests/unittests/test_merging.py
which ends up failing with in /usr/lib/python3.6/inspect.py with:
  ValueError: Function has keyword-only parameters or annotations, use
  getfullargspec() API which can support them

Instead of setting 'basepython' to 3.5 for the 'xenial', we just update
the one package that does not run correctly with python3.6. That allows
the developer to have either python3.5 or python3.6 installed and have
tox work as expected.

Author: Scott Moser
Author Date: 2017-08-29 18:15:38 UTC

log timing on gce crawl.

Author: Scott Moser
Author Date: 2017-08-04 21:31:47 UTC

initial import from rharper's cloudinit-analyze

this works to show usage now:
   python3 -m cloudinit.analyze

Author: Scott Moser
Author Date: 2017-08-03 20:38:09 UTC

net: Reduce duplicate code. Have get_interfaces_by_mac use get_interfaces.

get_interfaces_by_mac and get_interfaces just looked much alike.
This makes get_interfaces_by_mac call get_interfaces.

Author: Scott Moser
Author Date: 2017-07-24 21:18:22 UTC

archlinux: Fix bug with empty dns, do not render 'lo' devices.

If no dns nameservers were provided a stack trace would occur.
The changes here add some unit tests for the arch distro.

Also avoids rendering an 'lo' interface.

LP: #1663045
LP: #1706593

Author: Scott Moser
Author Date: 2017-07-26 13:08:14 UTC

tests: adjust locale integration test to parse default locale.

The locale integration test started failing with commit 0ef61b28.
This was just because the test's expectations on the content/formatting
of /etc/default/locale were too strict. The change here is to read
the file as a set of shell variables and assert that the values are
set correctly.

Author: Scott Moser
Author Date: 2017-07-26 14:25:52 UTC

tests: remove 'yakkety' from releases as it is EOL.

This is not strictly necessary, but since yakkety is no longer supported
we will remove it from the releases.yaml file.

Author: Scott Moser
Author Date: 2017-07-26 14:51:20 UTC

centos: do not package systemd-fsck drop-in.

The change here fixes the build of CentOS 7 rpm.

The systemd 'drop-in' file systemd-fsck@.service.d/cloud-init.conf is
not expected to work on CentOS at the moment. There, the
cloud-init-local.service currently runs without DefaultDependencies=no.
That would likely cause a dependency loop.

The drop-in was added for bug 1691489. It may well be needed at a later
date in CentOS also.

Author: Scott Moser
Author Date: 2016-12-08 12:27:53 UTC

systemd: make systemd-fsck run after cloud-init.service

cloud-init.service may write filesystems (fs_setup) or re-partition
(disk_setup) disks.

If systemd-fsck is running on a device while that is occuring
then the partitioning or mkfs might fail due to the device being busy.
Alternatively, the fsck might fail and cause subsequent mount to fail.

LP: #1691489

Author: Scott Moser
Author Date: 2017-07-25 16:20:53 UTC

fix Chad's suggestions in review.

Author: Scott Moser
Author Date: 2017-06-08 20:42:12 UTC

locale: Do not re-run locale-gen if provided locale is system default.

If the system configure default in /etc/default/locale is set to the same
value that is provided for cloud-init's "locale" setting, then do not
re-run locale-gen. This allows images built with a locale already
generated to not re-run locale-gen (which can be very heavy).

Also here is a fix to invoke update-locale correctly and remove the
internal writing of /etc/default/locale. We were calling
  update-locale <locale>
This ends up having no affect. The more correct invocation is:
  update-locale LANG=<locale>

Also added some support here should we ever want to change setting
LANG to setting LC_ALL (or any other key).

Lastly, a test change to allow us to use assert_not_called from mock.
Versions of mock in CentOS 6 do not have assert_not_called.

Author: Scott Moser
Author Date: 2017-07-21 20:50:27 UTC

sysconfig: support subnet type of 'manual'.

The subnet type 'manual' was used as a way to declare a device
and set an MTU for it but not assign network addresses.

This updates the manual example config to handle that case and
provides expected rendered output for sysconfig, eni, and netplan.

Author: Scott Moser
Author Date: 2017-07-21 17:34:05 UTC

tools/run-centos: make running with no argument show help.

If you ran tools/run-centos without an argument it would fail due
to 'set -u' like:
  ./tools/run-centos: line 266: 1: unbound variable

Author: Ryan Harper
Author Date: 2017-07-20 19:46:30 UTC

sysconfig: use MACADDR on bonds/bridges to configure mac_address

Previously, sysconfig rendered HWADDR for all interface types, but
that value is only used to identify physical devices. Instead use
MACADDR to configure the MAC on virtual devices, like bonds and
bridges.

- Sort bond slave list to ensure consistent ordering in sysconfig
  rendered files.
- Add unittests for sysconfig rendering of bonds/bridges with
  mac_address

LP: #1701417

Author: Ryan Harper
Author Date: 2017-06-28 21:29:18 UTC

net: eni route rendering missed ipv6 default route config

In some network configurations a network value of '::' and a
netmask value of '::' were used to indicate a default IPV6 gateway.
Commit d00da2d5 removed ipv6 'netmask' attributes and calculate
a prefix length value instead. The eni route rendering failed to update
the check to use prefix value of 0 to indicate the presence of an IPV6
default route.

A broken ipv6 default route rendered like:

 post-up route add -net :: netmask :: gw 2001:4800:78ff:1b::1 || true

And with this patch, it now renders like:

 post-up route add -A inet6 default gw 2001:4800:78ff:1b::1 || true

LP: #1701097

Author: Ryan Harper
Author Date: 2017-06-15 18:12:03 UTC

sysconfig: handle manual type subnets

Implement manual control for sysconfig by using ONBOOT=N. This
allows an interface to be configured but not brought up.

Note that ONBOOT is per-interface not per address.

LP: #1687725

Author: Ryan Harper
Author Date: 2017-06-15 16:51:54 UTC

sysconfig: fix ipv6 gateway routes

Currently only the subnet is checked for 'ipv6' setting, however, the
routes array may include a mix of v4 or v6 configurations, in particular,
the gateway in a route may be ipv6, and if so, should export the value via
IPV6_DEFAULTGW in the ifcfg-XXXX file.

Additionally, if the route is v6, it should rendering a routes6-XXXX file;
this is present but missing the 'dev <interface>' scoping.

LP: #1694801

Author: Ryan Harper
Author Date: 2017-06-14 17:58:40 UTC

sysconfig: fix rendering of bond, bridge and vlan types.

Previously, virtual types (bond, bridge, vlan) were almost completely
broken. They would not get any network configuration (ip addresses or
dhcp config) and or routes rendered. This fixes those issues.

For bonds we now correctly render BONDING_SLAVE entries.

Also add tests for simple bond, bridge and vlan.

LP: #1695092

Author: Ryan Harper
Author Date: 2017-06-13 14:06:41 UTC

Templatize systemd unit files for cross distro deltas.

Under el7, cloud-init systemd files need some unit tweaks to ensure
they run at the right time. Pull in current el7 downstream systemd unit
changes.

Author: Scott Moser
Author Date: 2017-07-19 18:47:37 UTC

fixup! Use prefix for ipv6 and set v4 and v6 gateway keys, use DEFROUTE

sysconfig: ipv6 and default gateway fixes.

With this change, entries in IPV6ADDR and IPV6ADDR_SECONDARIES will now
always be in format addr/prefix. When a subnet has a gateway will be
written. If the gateway is ipv6, use the key IPV6_DEFAULTGW rather than
GATEWAY.

LP: #1704872

Author: Scott Moser
Author Date: 2017-07-19 13:28:52 UTC

net: fix renaming of nics to support mac addresses written in upper case.

The network device renaming code previously required the case of
the mac address input to match that of the data read from the system.
For example, if user provided network config with mac address
in upper case, then cloud-init would not rename the device correctly
as /sys/class/net/address stores lower case values.

The fix here is to always compare lower case mac addresses.

LP: #1705147

Author: Scott Moser
Author Date: 2017-07-13 00:16:35 UTC

tests: fixes for issues uncovered when moving to python 3.6.

This includes a few fixes found when testing with python 3.6.
 - fix eni renderer when target is None
   This just uses the util.target_path() in the event that target is None.
 - change test cases to not rely on the cached result of
   util.get_cmdline() and other cached globals. Update the base TestCase
   to unset that cache.
 - mock calls to system_is_snappy from the create_users test cases.
 - drop unused _pp_root in test_simple_run.py

LP: #1703697

Author: Scott Moser
Author Date: 2017-07-17 14:20:40 UTC

Support comments in content read by load_shell_content.

load_shell_content previously would not allow shell comment characters
in the content being parsed. If comments=True is not passed then an
exception would previously be raised as the line would not be guaranteed to
have an '=' in it.

Author: Scott Moser
Author Date: 2017-07-13 16:59:16 UTC

tests: fix usage of mock in GCE test.

The usage of mock in this test was simply invalid and only worked by
happenstance.

Author: Scott Moser
Author Date: 2017-07-03 20:37:09 UTC

run at local time frame wit ipv4ll

Author: Scott Moser
Author Date: 2017-06-29 17:32:15 UTC

read_dmi_data: always return None when inside a container.

This fixes stacktrace and warning message that would be printed
to the log if running inside a container and read_dmi_data tried
to access a key that was not present.

In a container, the /sys/class/dmi/id data is not relevant to the
but to the host. Additionally an unpriviledged container might see
strange behavior:
   # cd /sys/class/dmi/id/
   # id -u
   0
   # ls -l chassis_serial
   -r-------- 1 nobody nogroup 4096 Jun 29 16:49 chassis_serial
   # cat chassis_serial
   cat: /sys/class/dmi/id/chassis_serial: Permission denied

The solution here is to just always return None when running in a
container.

LP: #1701325

Author: Scott Moser
Author Date: 2017-06-28 18:11:27 UTC

update changelog

Author: Ryan Harper
Author Date: 2017-06-20 22:06:43 UTC

Azure: Add network-config, Refactor net layer to handle duplicate macs.

On systems with network devices with duplicate mac addresses, cloud-init
will fail to rename the devices according to the specified network
configuration. Refactor net layer to search by device driver and device
id if available. Azure systems may have duplicate mac addresses by
design.

Update Azure datasource to run at init-local time and let Azure datasource
generate a fallback networking config to handle advanced networking
configurations.

Lastly, add a 'setup' method to the datasources that is called before
userdata/vendordata is processed but after networking is up. That is
used here on Azure to interact with the 'fabric'.

Author: Scott Moser
Author Date: 2017-06-15 14:03:45 UTC

FreeBSD: Make freebsd a variant, fix unittests and tools/build-on-freebsd.

 - Simplify the logic of 'variant' in util.system_info
   much of the data from
   https://github.com/hpcugent/easybuild/wiki/OS_flavor_name_version
 - fix get_resource_disk_on_freebsd when running on a system without
   an Azure resource disk.
 - fix tools/build-on-freebsd to replace oauth with oauthlib and add
   bash which is a dependency for tests.
 - update a fiew places that were checking for freebsd but not using
   the util.is_FreeBSD()

Author: Scott Moser
Author Date: 2017-06-15 20:39:50 UTC

FreeBSD: fix test failure

The previous commit caused test failure.
This separates out _check_freebsd_cdrom and mocks it in a test
rather than patching open.

Author: Scott Moser
Author Date: 2017-06-14 13:33:54 UTC

tools/run-centos: cleanups and move to using read-dependencies

These changes are all in an effort to get tools/run-centos using
read-dependencies rather than the 'setup-centos' script with a separate
set of dependencies listed.

 - tools/read-dependencies: support taking multiple --requirements
   options. This allows run-centos to get both test and build
   dependencies. Ultimately, I think it might be nicer for
   read-dependencies to take a list of "goals" (build, test, run or
   test-tox) rather than having the caller need to know to provide
   multiple --requirements.

 - packages/pkg-deps.json: drop the version on the sudo package.
   centos 6 has newer (1.8.6p3) version than listed, so its not a problem.

 - test_handler_disk_setup.py: a test case here was using assertLogs
   which is not present in the version of unittest2 that is available in
   centos 6 epel. We just adjust it to use with_logs = True.

 - tools/run-cents:
   - improve usage with example
   - add 'inside_as_cd' to provide the dir you want to cd first to.
   - avoid the intermediate tarball on disk in the container.
   - add 'prep' subcommand and use it to install pre-dependencies.
   - use read-dependencies.

Author: Scott Moser
Author Date: 2017-06-12 18:10:58 UTC

systemd: make cloud-final.service run before apt daily services.

This changes all cloud-init systemd units to run 'Before' the apt processes
that run daily and may cause a lock on the apt database.

apt-daily-upgrade.service contains 'After=apt-daily.service'.
Thus following order is enforced, so we can just be 'Before' the first.
   apt-daily.service
   apt-daily-upgrade.service

Note that this means only that apt-daily* will not run until
cloud-init has entirely finished. Any other processes running apt-get
operations are still affected by the global lock.

LP: #1693361

Author: Scott Moser
Author Date: 2017-05-25 01:10:50 UTC

net: normalize data in network_state object

The network_state object's network and route keys would have different
information depending upon how the network_state object was populated.

This change cleans that up. Now:
  * address will always contain an IP address.
  * prefix will always include an integer value that is the
    network_prefix for the address.
  * netmask will be present only if the address is ipv4, and its
    value will always correlate to the 'prefix'.

Author: Scott Moser
Author Date: 2017-06-07 15:32:56 UTC

rhel/centos spec cleanups.

Many changes here to get us able to build rpms on CentOS 5 or 6 and RHEL.

 * add 'Requires' as 'BuildRequires' also.
   This allows us to run cloud-init tools in the build environment, and
   also will allow us to run tests in the build process.
 * build for both systemd and upstart (centos 5) init systems.
 * Add 'centos' as a variant
   Adding the variant means we can use the 'centos' user as default on centos
   rather than a 'fedora' or 'rhel'.
 * drop argparse from the requirements.
   On any system other than python 2.6, having a 'requirements' that mentions
   argparse just causes problems. Instead we add that Requires to the spec
   directly.
 * list dependency on dmidecode (as redhat distro spec had)
 * remove duplicate line in files section ({_unitdir}/cloud-*)
 * Use rpm macros for init-system chunks and drop use
   of init_system variable template
 * Add el6 only build-req on python-argparse
 * python-cheetah is not required in the build environment as the
   the spec is already rendered. (We will soon move the spec to jinja).

Author: Scott Moser
Author Date: 2016-09-28 20:20:55 UTC

cloud.cfg: move to a template. setup.py changes along the way.

Here we move the config/cloud.cfg to be rendered as a template.
That allows us to maintain deltas between distros in one place.
Currently we use 'variant' variable to make decisions.
A tools/render-cloudcfg is provided to render the file.

There were changes to setup.py, MANIFEST.in to allow us to put all
files into a virtual env installation and to render the cloud-config
file in 'install' or 'bdist' targets.

We have also included some config changes that were found in the
redhat distro spec.
 * include some config changes from the redhat distro spec.

The rendered cloud.cfg has some differences.
Ubuntu: white space and comment changes only.
Freebsd:
 - whitespace changes and comment changes
 - datasource_list definition moved to be closer to 'datasource'.
 - enable modules: migrator, write_files
 - move package-update-upgrade-install to final.

The initial work was done by Josh Harlow.

Author: Scott Moser
Author Date: 2017-06-01 14:39:34 UTC

tools/net-convert.py: support old cloudinit versions by using kwargs.

Older cloud-init versions have a bug in the signature of the
render_network_state method for netplan (bug 1685944).
The old had:
  render_network_state(target, network_state)
The fix was to change netplan's so it had the correct signature:
  render_network_state(network_state, target)

This just changes our caller to use kwargs style when invoking that
method so that it works with either the broken form or correct form.

Author: Scott Moser
Author Date: 2017-05-31 15:03:31 UTC

add warning on unexpected situation strict_id=true and platform unknown.

Author: Junjie.Wang
Author Date: 2017-04-21 12:06:09 UTC

AliYun: Enable platform identification and enable by default.

AliYun cloud platform is now identifying themselves by setting the dmi
product id to the well known value "Alibaba Cloud ECS". The changes here
identify that properly in tools/ds-identify and in the DataSourceAliYun.

Since the 'get_data' for AliYun now identifies itself correctly, we can
enable AliYun by default.

LP: #1638931

Author: Scott Moser
Author Date: 2017-05-25 17:06:08 UTC

disk_setup: udev settle before attempting partitioning or fs creation.

This attempts to use udevadm settle to wait until devices have been
fully "realized". If a device exists, there may still be events in
the udev queue that would create its partition table entries.
We need to wait until those have been processed also.

LP: #1692093

Author: Scott Moser
Author Date: 2017-05-18 17:19:14 UTC

fix tools/ds-identify to not write None twice.

If the user configured:
  datasource_list: ["Ec2", "None"]
then ds-identify would write
  datasource_list: ["Ec2", "None", "None"]
which would break the logic to avoid warning.

Author: Scott Moser
Author Date: 2017-05-23 17:09:26 UTC

flake8: move the pinned version of flake8 up to 3.3.0

This just moves flake8 and related tools up to newer versions and fixes
the complaints associated with that.

We added to the list of flake8 ignores:
 H102: do not put vim info in source files
 H304: no relative imports

Also updates and pins the following in the flake8 environment:
  pep8: 1.7.0 => drop (although hacking still pulls it in).
  pyflakes 1.1.0 => 1.5.0
  hacking 0.10.2 => 0.13.0
  flake8 2.5.4 => 3.3.0
  pycodestyle none => 2.3.1

Author: Scott Moser
Author Date: 2017-05-19 18:57:04 UTC

disk_setup: fix several issues with gpt disk partitions.

This fixes several shortcomings of disk_setup with gpt disks.
 * 'sgdisk -p' was being used to determine the size of a disk.
   this can fail if it believes there is a bad gpt partition table.
   Instead we just use blockdev now for both mbr or gpt disks.
 * parsing of sgdisk -p output assumed that the 'name' of the partition
   type would not have any spaces (Microsoft basic data)
 * interaction with sgdisk did not realize that sgdisk wants input
   of '8300' rather than '83' and will output the same.

LP: #1692087

Author: Scott Moser
Author Date: 2017-05-22 14:17:38 UTC

tox: move pylint target to 1.7.1

The motivation for this is to make tip-pylint target green.
It does 2 things:
 a.) silence a warning that is generated in pylint 1.7.1, but not
     other versions of pylint. This bug in pylint is filed at
     https://github.com/PyCQA/pylint/issues/1444
 b.) move tox -e pylint to use pylint 1.7.1

Author: Scott Moser
Author Date: 2017-04-28 13:23:25 UTC

Azure: fix reformatting of ephemeral disks on resize to large types.

Large instance types have a different disk format on the newly
partitioned ephemeral drive. So we have to adjust the logic in the
Azure datasource to recognize that a disk with 2 partitions and
an empty ntfs filesystem on the second one is acceptable.

This also adjusts the datasources's builtin fs_setup config to remove
the 'replace_fs' entry. This entry was previously ignored, and confusing.
I've clarified the doc on that also.

LP: #1686514

Author: Scott Moser
Author Date: 2017-05-04 03:36:39 UTC

requirements.txt: drop change to comment out argparse

Lets leave this in right now, we can investigate further dropping it
later. I think that centos6 needs it, and other environments
do not seem bothered by it. (argparse is builtin in python >= 2.7).

Author: Scott Moser
Author Date: 2017-04-26 19:42:54 UTC

unittests: fix unittests run on centos.

Some unit tests were broken when running on centos.
This fixes the unit test, with a small re-work of apt_configure.

Author: Scott Moser
Author Date: 2017-04-05 16:07:53 UTC

HACKING.rst: more info on filling out contributors agreement.

When signing the Contributors agreement, the user is prompted to provide
a 'Project contact' or 'Canonical Project Manager'.
Just update the HACKING.rst document to tell them what to put there.

Author: Scott Moser
Author Date: 2017-03-30 18:42:19 UTC

ds-identify: fix detecting of maas datasource.

The reading of MAAS datasource configuration was simply broken.
it was looking in /etc/cloud/*maas*.cfg rather than
/etc/cloud/cloud.cfg.d/*maas*.cfg.

along side here there is also:
 * doc improvement on check_config
 * remove the path restrictions when searching for values in both
   maas and ovf_vmware_guest_customization. that was done to improve
   performance as check_config's parsing is slow.
 * change to maas to search all config files rather than restricting
   to a subset as it tried before. that was done for
 * better variable names.
    - rename path_cloud_confd to path_etc_cloud
    - PATH_ETC_CLOUD: /etc/cloud
    - PATH_ETC_CI_CFG: /etc/cloud/cloud.cfg
    - PATH_ETC_CI_CFG_D: /etc/cloud/cloud.cfg.d

LP: #1677710

Author: Scott Moser
Author Date: 2017-03-21 13:50:28 UTC

Fix unit test when running in a system deployed with cloud-init.

test_netconfig.py:test_apply_network_config_eni_ub would attempt to
remove any .link files that cloud-init had written. This was just
a failure to mock out all of its interaction with the host.