Last commit made on 2017-09-01
Get this branch:
git clone -b bug/ec2-tests https://git.launchpad.net/~larsks/cloud-init
Only Lars Kellogg-Stedman can upload to this branch. If you are Lars Kellogg-Stedman please log in for upload directions.

Branch merges

Branch information


Recent commits

0443657... by Lars Kellogg-Stedman

test_ec2: metadata tests were mocking wrong urls

The ec2 metadata tests were only mocking one version of the metadata
api, but requests were made against both. This fixes _setup_ds to
register mock data at both versions of the API.

29025c3... by Lars Kellogg-Stedman

url_helper: fail gracefully if oauthlib is not available

We are unable to ship python-oauthlib in RHEL. This commit allows
imports of url_helper to succeed even when oauthlib is unavailable
and OauthUrlHelper.oauth_headers to raise a NotImplementedException
when called.

f831a87... by Chad Smith

cc_landscape & cc_puppet: Fix six.StringIO use in writing configs

Both landscape and puppet modules had issues with the way they wrote
/etc/landscape/client.conf or /etc/puppet/puppet.conf in either python3 or
python2. This branch adds initial unit tests for both modules which will
get better exercise under both python2 and python3.

The unit tests shed light on a few issues:
   - In the cc_landscape module py3 can't provide six.StringIO content to
     ConfigParser.write, so we need to use six.BytesIO instead
   - In the cc_puppet module, python <= 2.7 doesn't support using
     six.StringIO as a context manager, so we drop the context manager
     fanciness and directly set outputstream = StringIO().
   - The docstring in cc_puppet is fixed to document the 'conf'
     sub-key requiring valid puppet section names for each
     key-value list.

LP: #1699282
LP: #1710932

cc9762a... by Chad Smith

schema cli: Add schema subcommand to cloud-init cli and cc_runcmd schema

This branch does a few things:
  - Add 'schema' subcommand to cloud-init CLI for validating
    cloud-config files against strict module jsonschema definitions
  - Add --annotate parameter to 'cloud-init schema' to annotate
    existing cloud-config file content with validation errors
  - Add jsonschema definition to cc_runcmd
  - Add unit test coverage for cc_runcmd
  - Update CLI capabilities documentation

This branch only imports development (and analyze) subparsers when the
specific subcommand is provided on the CLI to avoid adding costly unused
file imports during cloud-init system boot.

The schema command allows a person to quickly validate a cloud-config text
file against cloud-init's known module schemas to avoid costly roundtrips
deploying instances in their cloud of choice. As of this branch, only
cc_ntp and cc_runcmd cloud-config modules define schemas. Schema
validation will ignore all undefined config keys until all modules define
a strict schema.

To perform validation of runcmd and ntp sections of a cloud-config file:
$ cat > cloud.cfg <<EOF
runcmd: bogus
$ python -m cloudinit.cmd.main schema --config-file cloud.cfg

$ python -m cloudinit.cmd.main schema --config-file cloud.cfg \

Once jsonschema is defined for all ~55 cc modules, we will move this
schema subcommand up as a proper subcommand of the cloud-init CLI.

3395a33... by =?utf-8?q?Joonas_Kylm=C3=A4l=C3=A4?= <email address hidden>

Debian: Remove non-free repositories from apt sources template.

The Debian GNU/Linux distribution doesn't come offically with the
non-free repositories enabled. Therefore, we want to disable those in
the cloud-init template.

LP: #1700091

e74d775... by Chad Smith

tools: Add tooling for basic cloud-init performance analysis.

This branch adds cloudinit-analyze into cloud-init proper. It adds an
"analyze" subcommand to the cloud-init command line utility for quick
performance assessment of cloud-init stages and events.

On a cloud-init configured instance, running "cloud-init analyze blame"
will now report which cloud-init events cost the most wall time. This
allows for quick assessment of the most costly stages of cloud-init.

This functionality is pulled from Ryan Harper's analyze work.

The cloudinit-analyze main script itself has been refactored a bit for
inclusion as a subcommand of cloud-init CLI. There will be a followup
branch at some point which will optionally instrument detailed strace
profiling, but that approach needs a bit more discussion first.

This branch also adds:
 * additional debugging topic to the sphinx-generated docs describing
   cloud-init analyze, dump and show as well as cloud-init single usage.
 * Updates the Makefile unittests target to include cloudinit directory
   because we now have unittests within that package.

LP: #1709761

dc2bd79... by Ryan Harper

network: add v2 passthrough and fix parsing v2 config with bonds/bridge params

If the network-config sent to cloud-init is in version: 2 format then
when rendering netplan, we can pass the content through and avoid
consuming network_state elements. This removes the need for trying to
map many v2 features onto network state where other renderers won't be
able to use anyhow (for example match parameters for multi-interface
configuration and wifi configuration support).

Additionally ensure we retain bond/bridge v2 configuration in network
state so when rendering to eni or sysconfig we don't lose the configuration

- Drop the NotImplemented wifi exception, log a warning that it works for
  netplan only
- Adjust unittests to new code path and output
- Fix issue with v2 macaddress values getting dropped
- Add unittests for consuming/validating v2 configurations

LP: #1709180

385d1ca... by Ryan Harper

doc: update capabilities with features available, link doc reference, cli example

1f8183f... by Maitreyee Saikia

vcloud directory: Guest Customization support for passwords

This feature enables the following VMware VCloud Director functionality:
1. Setting admin password
2. Expire password.
3. Set admin password and expire.

Password configuration is triggered only as part of a full
recustomization, that happens either on first power on or when
"poweron and full recustomization" is selected. Full customization
flow is determined by marker files. Unique marker ids are
generated when full recustomization is requested. And marker file based
on these marker ids help to determine if we need to execute the above

d5f855d... by Chad Smith

ec2: Allow Ec2 to run in init-local using dhclient in a sandbox.

This branch is a prerequisite for IPv6 support in AWS by allowing Ec2
datasource to query the metadata source version 2016-09-02 about whether
or not it needs to configure IPv6 on interfaces. If version 2016-09-02
is not present, fallback to the min_metadata_version of 2009-04-04. The
DataSourceEc2Local not run on FreeBSD because dhclient in doesn't
support the -sf flag allowing us to run dhclient without filesystem

To query AWS' metadata address @, the instance must have
a dhcp-allocated address configured. Configuring IPv4 link-local
addresses result in timeouts from the metadata service. We introduced a
DataSourceEc2Local subclass which will perform a sandboxed dhclient
discovery which obtains an authorized IP address on eth0 and crawl
metadata about full instance network configuration.

Since ec2 IPv6 metadata is not sufficient in itself to tell us all the
ipv6 knownledge we need, it only be used as a boolean to tell us which
nics need IPv6. Cloud-init will then configure desired interfaces to
DHCPv6 versus DHCPv4.

Performance side note: Shifting the dhcp work into init-local for Ec2
actually gets us 1 second faster deployments by skipping init-network
phase of alternate datasource checks because Ec2Local is configured in
an ealier boot stage. In 3 test runs prior to this change: cloud-init
runs were 5.5 seconds, with the change we now average 4.6 seconds.

This efficiency could be even further improved if we avoiding dhcp
discovery in order to talk to the metadata service from an AWS
authorized dhcp address if there were some way to advertize the dhcp
configuration via DMI/SMBIOS or system environment variables.

Inspecting time costs of the dhclient setup/teardown in 3 live runs the
time cost for the dhcp setup round trip on AWS is:
test 1: 76 milliseconds
         dhcp discovery + metadata: 0.347 seconds
         metadata alone: 0.271 seconds
test 2: 88 milliseconds
         dhcp discovery + metadata: 0.388 seconds
         metadata alone: 0.300 seconds
test 3: 75 milliseconds
         dhcp discovery + metadata: 0.366 seconds
         metadata alone: 0.291 seconds

LP: #1709772