cloud-init

Merge ~chad.smith/cloud-init:feature/snap-module into cloud-init:master

  1. Git
  2. lp:~chad.smith/cloud-init
  3. feature/snap-module
  4. Merge into master
Proposed by Chad Smith
Status: Merged
Approved by: Chad Smith
Approved revision: 962f98fe12f4215e8d1efd11f4c18a02c238ac88
Merge reported by: Chad Smith
Merged at revision: a1f678f8ebc080d4737f32275f42947b84ae025a
Proposed branch: ~chad.smith/cloud-init:feature/snap-module
Merge into: cloud-init:master
Diff against target: 1441 lines (+1098/-31)
20 files modified
cloudinit/config/cc_puppet.py (+4/-4)
cloudinit/config/cc_snap.py (+273/-0)
cloudinit/config/cc_snap_config.py (+7/-0)
cloudinit/config/cc_snappy.py (+8/-0)
cloudinit/config/tests/test_snap.py (+533/-0)
cloudinit/util.py (+12/-1)
config/cloud.cfg.tmpl (+3/-2)
doc/rtd/conf.py (+1/-0)
doc/rtd/topics/modules.rst (+1/-0)
tests/cloud_tests/releases.yaml (+3/-0)
tests/cloud_tests/testcases.yaml (+3/-0)
tests/cloud_tests/testcases/__init__.py (+3/-0)
tests/cloud_tests/testcases/base.py (+168/-5)
tests/cloud_tests/testcases/main/command_output_simple.py (+2/-15)
tests/cloud_tests/testcases/modules/snap.py (+16/-0)
tests/cloud_tests/testcases/modules/snap.yaml (+18/-0)
tests/cloud_tests/testcases/modules/snappy.py (+2/-0)
tests/cloud_tests/verify.py (+7/-4)
tests/unittests/test_handler/test_schema.py (+1/-0)
tests/unittests/test_util.py (+33/-0)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
Simon Poirier (community) Approve
Joshua Powers Pending
cloud-init Commiters Pending
Review via email: mp+338366@code.launchpad.net

Commit message

cc_snap: Add new module to install and configure snapd and snap packages.

Support installing and configuring snaps on ubuntu systems. Now,
cloud-config files can provide a list or dictionary of snap:assertions
which will be allow configuration of snapd on a system via 'snap ack'
calls. The snap:commands configuration option supports arbitrary system
commands intended to interact with snappy's cli. This allows users to run
arbitrary snappy commands to create users, download, install and
configure snap packages and snapd.

This branch also deprecates old snappy and snap_config modules leaving
warnings in documentation and runtime for consumers of these modules.
Deprecated snap* modules will be dropped in cloud-init v.18.2 release.

Description of the change

see commit message.

to test:
1. make a deb of this branch
make deb;
2. create a container and install the deb
lxc launch ubuntu-daily/bionic myb1;
lxc file push cloud-init_18*deb myb1/;
lxc exec myb1 -- dpkg -i /cloud-init*deb;

3. install snap user-data cloud-config
cat > snap.yaml <<EOF#cloud-config
snap:
  commands:
    - install hello-world
EOF

lxc file push snap.yaml myb1/var/lib/cloud/seed/nocloud-net/user-data;

4. clean boot the container so cloud-init runs 'fresh'
lxc exec myb1 -- cloud-init clean --reboot --logs
5. validate
lxc exec myb1 bash

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:921fb06b0988d217d3e88f45db6b30e6e8acc5e6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/776/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/776/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
fc2c72c... by Chad Smith

automatically prepend snap command if unspecified as a command of type list. Add unit tests for non-snap warnings and prepending operation. Update docs

445bab8... by Chad Smith

allow for a subclass to specify expected_warnings class attribute on which CloudTestCase.test_no_warnings_in_log will validate

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:9c782d4ef006225e40a0da8c375d9951c6a50ce7
https://jenkins.ubuntu.com/server/job/cloud-init-ci/822/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/822/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

some comments

~chad.smith/cloud-init:feature/snap-module updated
41a29dc... by Chad Smith

first round of review comments

 - use module LOG instead passing log parameter into so many functions
 - fix prepend logic to check where command item 0 != SNAP_CMD

Revision history for this message
Joshua Powers (powersj) wrote :

See inline

~chad.smith/cloud-init:feature/snap-module updated
3e0e20d... by Chad Smith

address comments round 2:

 - Add deprecated comments to cloud.cfg.tmpl for snappy and snap_config modules
 - drop bug reference and squashfuse warning as this will be fixed shortly

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:ac9bb942577352b6e42c0cfd2c9571955fd295a8
https://jenkins.ubuntu.com/server/job/cloud-init-ci/825/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/825/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:d68942b90ac4c18e9763563c288d436deed18456
https://jenkins.ubuntu.com/server/job/cloud-init-ci/826/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/826/rebuild

review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

My comments....
there is a lot here, as you said.
 * generally the test harness changes could be separated.
 * I think access to the platform and os info in the test cases is good.
 * I dont think the validation of /run/cloud-init/instance-data.json needs to be Snap specific at all, we should really always be doing that.

~chad.smith/cloud-init:feature/snap-module updated
f275585... by Chad Smith

run each snap command individually, emitting status messages to stderr. Add status_cb parameter to subp

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:c54c5378e3a6174533fa58de1ce9de2a745daa6d
https://jenkins.ubuntu.com/server/job/cloud-init-ci/831/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/831/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
035fc6a... by Chad Smith

fix unit tests

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:a59dc135625a7d41abd36564302ea4485bd6a7d6
https://jenkins.ubuntu.com/server/job/cloud-init-ci/833/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/833/rebuild

review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:7ce2f80f60efdd9eedd63ee0672ac14bbc1dcc2b
https://jenkins.ubuntu.com/server/job/cloud-init-ci/835/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/835/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
dbd0456... by Chad Smith

add integration test utility methods is_distro and os_version_cmp to limit integration tests on specific series >= X.Y

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:9fbe196e746382391fd74e9d8fd3fa4fb40332c8
https://jenkins.ubuntu.com/server/job/cloud-init-ci/837/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/837/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

two small things, interested in your thoughts.

Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Joshua Powers (powersj) wrote :

I like the comparisons on versions, that helps a lot on future tests.

In addition to the unit and style tests, the integration tests need fixing:
https://paste.ubuntu.com/p/MWzFZjZYwS/

Revision history for this message
Scott Moser (smoser) :
~chad.smith/cloud-init:feature/snap-module updated
c45721f... by Chad Smith

fixup dict versus list documentation for snap commands

785603b... by Chad Smith

cc_snap will pass a single element list to util.subp if a given command is a string and shell=True. lint fixes

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:532a608e5a466cf9aab6214d255a62e9a8a2ac9c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/846/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/846/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
5b2c1f8... by Chad Smith

util.subp should allow us to pass in a string instead of a list

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:70537d8c5989b99c647d8ee9b2c9f85bd25ba53f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/848/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/848/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:743df1cd81eb55d0fc2670fa480a44d9445b3f5a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/853/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/853/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
b9e1b2b... by Chad Smith

check basic command not found content in a flexible manner that passes on py26, py27 and py3

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:88980f346d0c926559ce2be207acc3da2f9c8e98
https://jenkins.ubuntu.com/server/job/cloud-init-ci/855/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    FAILED: MAAS Compatability Testing

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/855/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
f6cde54... by Chad Smith

use regular expression to make command not found messages

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:357e9a9919e45b7455259f55db157f50859b0b28
https://jenkins.ubuntu.com/server/job/cloud-init-ci/857/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/857/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
6cebec7... by Chad Smith

flake

Revision history for this message
Simon Poirier (simpoir) :
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:fd11c422178edb75572ba0258369af3a074f6b3a
https://jenkins.ubuntu.com/server/job/cloud-init-ci/858/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/858/rebuild

review: Approve (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
049c009... by Chad Smith

address simpoir's review comments. fix integration tests with compound comparison on ubuntu & >= bionic

Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:a158a134a000178b180e016ba3dafd0633d5789f
https://jenkins.ubuntu.com/server/job/cloud-init-ci/861/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/861/rebuild

review: Approve (continuous-integration)
Revision history for this message
Simon Poirier (simpoir) wrote :

+1 LGTM

review: Approve
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:72777189f54c86f9c393fb9355382ab10cbacb79
https://jenkins.ubuntu.com/server/job/cloud-init-ci/862/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/862/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
d12719b... by Chad Smith

move instance-data.json integration tests to base class

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:db4f161207d8b7cac4dc2a8337b7adcf625d27d9
https://jenkins.ubuntu.com/server/job/cloud-init-ci/864/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/864/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

We can probably just ditch the squashfuse stuff now.
that is slated to be fixed "real soon now" under bug 1756173.

Its not our problem, and this all works as long as snapd works.

~chad.smith/cloud-init:feature/snap-module updated
7c6007d... by Chad Smith

update deprecation messages and add RELEASE_BLOCKER comments for snap, snappy, snap_config

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:962f98fe12f4215e8d1efd11f4c18a02c238ac88
https://jenkins.ubuntu.com/server/job/cloud-init-ci/865/
Executed test runs:
    SUCCESS: Checkout
    FAILED: Unit & Style Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/865/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
5a4cd9e... by Chad Smith

lints/flakes

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:5a4cd9e4aa874445bad3ac0af69bc583eeef8f96
https://jenkins.ubuntu.com/server/job/cloud-init-ci/868/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    FAILED: Ubuntu LTS: Integration

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/868/rebuild

review: Needs Fixing (continuous-integration)
~chad.smith/cloud-init:feature/snap-module updated
eef890a... by Chad Smith

sort keys in assertions to avoid intermittent failures on dict key ordering

Revision history for this message
Server Team CI bot (server-team-bot) wrote :

PASSED: Continuous integration, rev:eef890ab7bc4022e807b7f01505bb78c03487b0c
https://jenkins.ubuntu.com/server/job/cloud-init-ci/869/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Unit & Style Tests
    SUCCESS: Ubuntu LTS: Build
    SUCCESS: Ubuntu LTS: Integration
    SUCCESS: MAAS Compatability Testing
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/cloud-init-ci/869/rebuild

review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

An upstream commit landed for this bug.

To view that commit see the following URL:
https://git.launchpad.net/cloud-init/commit/?id=a1f678f8

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/cloudinit/config/cc_puppet.py b/cloudinit/config/cc_puppet.py
2index 57a170f..297e072 100644
3--- a/cloudinit/config/cc_puppet.py
4+++ b/cloudinit/config/cc_puppet.py
5@@ -140,8 +140,9 @@ def handle(name, cfg, cloud, log, _args):
6 # (TODO(harlowja) is this really needed??)
7 cleaned_lines = [i.lstrip() for i in contents.splitlines()]
8 cleaned_contents = '\n'.join(cleaned_lines)
9- puppet_config.readfp(StringIO(cleaned_contents),
10- filename=p_constants.conf_path)
11+ puppet_config.readfp( # pylint: disable=W1505
12+ StringIO(cleaned_contents),
13+ filename=p_constants.conf_path)
14 for (cfg_name, cfg) in puppet_cfg['conf'].items():
15 # Cert configuration is a special case
16 # Dump the puppet master ca certificate in the correct place
17@@ -149,8 +150,7 @@ def handle(name, cfg, cloud, log, _args):
18 # Puppet ssl sub-directory isn't created yet
19 # Create it with the proper permissions and ownership
20 util.ensure_dir(p_constants.ssl_dir, 0o771)
21- util.chownbyname(p_constants.ssl_dir, 'puppet', 'root')
22- util.ensure_dir(p_constants.ssl_cert_dir)
23+
24 util.chownbyname(p_constants.ssl_cert_dir, 'puppet', 'root')
25 util.write_file(p_constants.ssl_cert_path, cfg)
26 util.chownbyname(p_constants.ssl_cert_path, 'puppet', 'root')
27diff --git a/cloudinit/config/cc_snap.py b/cloudinit/config/cc_snap.py
28new file mode 100644
29index 0000000..db96529
30--- /dev/null
31+++ b/cloudinit/config/cc_snap.py
32@@ -0,0 +1,273 @@
33+# Copyright (C) 2018 Canonical Ltd.
34+#
35+# This file is part of cloud-init. See LICENSE file for license information.
36+
37+"""Snap: Install, configure and manage snapd and snap packages."""
38+
39+import sys
40+from textwrap import dedent
41+
42+from cloudinit import log as logging
43+from cloudinit.config.schema import (
44+ get_schema_doc, validate_cloudconfig_schema)
45+from cloudinit.settings import PER_INSTANCE
46+from cloudinit import util
47+
48+
49+distros = ['ubuntu']
50+frequency = PER_INSTANCE
51+
52+LOG = logging.getLogger(__name__)
53+
54+schema = {
55+ 'id': 'cc_snap',
56+ 'name': 'Snap',
57+ 'title': 'Install, configure and manage snapd and snap packages',
58+ 'description': dedent("""\
59+ This module provides a simple configuration namespace in cloud-init to
60+ both setup snapd and install snaps.
61+
62+ .. note::
63+ Both ``assertions`` and ``commands`` values can be either a
64+ dictionary or a list. If these configs are provided as a
65+ dictionary, the keys are only used to order the execution of the
66+ assertions or commands and the dictionary is merged with any
67+ vendor-data snap configuration provided. If a list is provided by
68+ the user instead of a dict, any vendor-data snap configuration is
69+ ignored.
70+
71+ The ``assertions`` configuration option is a dictionary or list of
72+ properly-signed snap assertions which will run before any snap
73+ ``commands``. They will be added to snapd's assertion database by
74+ invoking ``snap ack <aggregate_assertion_file>``.
75+
76+ Snap ``commands`` is a dictionary or list of individual snap
77+ commands to run on the target system. These commands can be used to
78+ create snap users, install snaps and provide snap configuration.
79+
80+ .. note::
81+ If 'side-loading' private/unpublished snaps on an instance, it is
82+ best to create a snap seed directory and seed.yaml manifest in
83+ **/var/lib/snapd/seed/** which snapd automatically installs on
84+ startup.
85+
86+ **Development only**: The ``squashfuse_in_container`` boolean can be
87+ set true to install squashfuse package when in a container to enable
88+ snap installs. Default is false.
89+ """),
90+ 'distros': distros,
91+ 'examples': [dedent("""\
92+ snap:
93+ assertions:
94+ 00: |
95+ signed_assertion_blob_here
96+ 02: |
97+ signed_assertion_blob_here
98+ commands:
99+ 00: snap create-user --sudoer --known <snap-user>@mydomain.com
100+ 01: snap install canonical-livepatch
101+ 02: canonical-livepatch enable <AUTH_TOKEN>
102+ """), dedent("""\
103+ # LXC-based containers require squashfuse before snaps can be installed
104+ snap:
105+ commands:
106+ 00: apt-get install squashfuse -y
107+ 11: snap install emoj
108+
109+ """), dedent("""\
110+ # Convenience: the snap command can be omitted when specifying commands
111+ # as a list and 'snap' will automatically be prepended.
112+ # The following commands are equivalent:
113+ snap:
114+ commands:
115+ 00: ['install', 'vlc']
116+ 01: ['snap', 'install', 'vlc']
117+ 02: snap install vlc
118+ 03: 'snap install vlc'
119+ """)],
120+ 'frequency': PER_INSTANCE,
121+ 'type': 'object',
122+ 'properties': {
123+ 'snap': {
124+ 'type': 'object',
125+ 'properties': {
126+ 'assertions': {
127+ 'type': ['object', 'array'], # Array of strings or dict
128+ 'items': {'type': 'string'},
129+ 'additionalItems': False, # Reject items non-string
130+ 'minItems': 1,
131+ 'minProperties': 1,
132+ 'uniqueItems': True
133+ },
134+ 'commands': {
135+ 'type': ['object', 'array'], # Array of strings or dict
136+ 'items': {
137+ 'oneOf': [
138+ {'type': 'array', 'items': {'type': 'string'}},
139+ {'type': 'string'}]
140+ },
141+ 'additionalItems': False, # Reject non-string & non-list
142+ 'minItems': 1,
143+ 'minProperties': 1,
144+ 'uniqueItems': True
145+ },
146+ 'squashfuse_in_container': {
147+ 'type': 'boolean'
148+ }
149+ },
150+ 'additionalProperties': False, # Reject keys not in schema
151+ 'required': [],
152+ 'minProperties': 1
153+ }
154+ }
155+}
156+
157+# TODO schema for 'assertions' and 'commands' are too permissive at the moment.
158+# Once python-jsonschema supports schema draft 6 add support for arbitrary
159+# object keys with 'patternProperties' constraint to validate string values.
160+
161+__doc__ = get_schema_doc(schema) # Supplement python help()
162+
163+SNAP_CMD = "snap"
164+ASSERTIONS_FILE = "/var/lib/cloud/instance/snapd.assertions"
165+
166+
167+def add_assertions(assertions):
168+ """Import list of assertions.
169+
170+ Import assertions by concatenating each assertion into a
171+ string separated by a '\n'. Write this string to a instance file and
172+ then invoke `snap ack /path/to/file` and check for errors.
173+ If snap exits 0, then all assertions are imported.
174+ """
175+ if not assertions:
176+ return
177+ LOG.debug('Importing user-provided snap assertions')
178+ if isinstance(assertions, dict):
179+ assertions = assertions.values()
180+ elif not isinstance(assertions, list):
181+ raise TypeError(
182+ 'assertion parameter was not a list or dict: {assertions}'.format(
183+ assertions=assertions))
184+
185+ snap_cmd = [SNAP_CMD, 'ack']
186+ combined = "\n".join(assertions)
187+
188+ for asrt in assertions:
189+ LOG.debug('Snap acking: %s', asrt.split('\n')[0:2])
190+
191+ util.write_file(ASSERTIONS_FILE, combined.encode('utf-8'))
192+ util.subp(snap_cmd + [ASSERTIONS_FILE], capture=True)
193+
194+
195+def prepend_snap_commands(commands):
196+ """Ensure user-provided commands start with SNAP_CMD, warn otherwise.
197+
198+ Each command is either a list or string. Perform the following:
199+ - When the command is a list, pop the first element if it is None
200+ - When the command is a list, insert SNAP_CMD as the first element if
201+ not present.
202+ - When the command is a string containing a non-snap command, warn.
203+
204+ Support cut-n-paste snap command sets from public snappy documentation.
205+ Allow flexibility to provide non-snap environment/config setup if needed.
206+
207+ @commands: List of commands. Each command element is a list or string.
208+
209+ @return: List of 'fixed up' snap commands.
210+ @raise: TypeError on invalid config item type.
211+ """
212+ warnings = []
213+ errors = []
214+ fixed_commands = []
215+ for command in commands:
216+ if isinstance(command, list):
217+ if command[0] is None: # Avoid warnings by specifying None
218+ command = command[1:]
219+ elif command[0] != SNAP_CMD: # Automatically prepend SNAP_CMD
220+ command.insert(0, SNAP_CMD)
221+ elif isinstance(command, str):
222+ if not command.startswith('%s ' % SNAP_CMD):
223+ warnings.append(command)
224+ else:
225+ errors.append(str(command))
226+ continue
227+ fixed_commands.append(command)
228+
229+ if warnings:
230+ LOG.warning(
231+ 'Non-snap commands in snap config:\n%s', '\n'.join(warnings))
232+ if errors:
233+ raise TypeError(
234+ 'Invalid snap config.'
235+ ' These commands are not a string or list:\n' + '\n'.join(errors))
236+ return fixed_commands
237+
238+
239+def run_commands(commands):
240+ """Run the provided commands provided in snap:commands configuration.
241+
242+ Commands are run individually. Any errors are collected and reported
243+ after attempting all commands.
244+
245+ @param commands: A list or dict containing commands to run. Keys of a
246+ dict will be used to order the commands provided as dict values.
247+ """
248+ if not commands:
249+ return
250+ LOG.debug('Running user-provided snap commands')
251+ if isinstance(commands, dict):
252+ # Sort commands based on dictionary key
253+ commands = [v for _, v in sorted(commands.items())]
254+ elif not isinstance(commands, list):
255+ raise TypeError(
256+ 'commands parameter was not a list or dict: {commands}'.format(
257+ commands=commands))
258+
259+ fixed_snap_commands = prepend_snap_commands(commands)
260+
261+ cmd_failures = []
262+ for command in fixed_snap_commands:
263+ shell = isinstance(command, str)
264+ try:
265+ util.subp(command, shell=shell, status_cb=sys.stderr.write)
266+ except util.ProcessExecutionError as e:
267+ cmd_failures.append(str(e))
268+ if cmd_failures:
269+ msg = 'Failures running snap commands:\n{cmd_failures}'.format(
270+ cmd_failures=cmd_failures)
271+ util.logexc(LOG, msg)
272+ raise RuntimeError(msg)
273+
274+
275+# RELEASE_BLOCKER: Once LP: #1628289 is released on xenial, drop this function.
276+def maybe_install_squashfuse(cloud):
277+ """Install squashfuse if we are in a container."""
278+ if not util.is_container():
279+ return
280+ try:
281+ cloud.distro.update_package_sources()
282+ except Exception as e:
283+ util.logexc(LOG, "Package update failed")
284+ raise
285+ try:
286+ cloud.distro.install_packages(['squashfuse'])
287+ except Exception as e:
288+ util.logexc(LOG, "Failed to install squashfuse")
289+ raise
290+
291+
292+def handle(name, cfg, cloud, log, args):
293+ cfgin = cfg.get('snap', {})
294+ if not cfgin:
295+ LOG.debug(("Skipping module named %s,"
296+ " no 'snap' key in configuration"), name)
297+ return
298+
299+ validate_cloudconfig_schema(cfg, schema)
300+ if util.is_true(cfgin.get('squashfuse_in_container', False)):
301+ maybe_install_squashfuse(cloud)
302+ add_assertions(cfgin.get('assertions', []))
303+ run_commands(cfgin.get('commands', []))
304+
305+# vi: ts=4 expandtab
306diff --git a/cloudinit/config/cc_snap_config.py b/cloudinit/config/cc_snap_config.py
307index e82c081..afe297e 100644
308--- a/cloudinit/config/cc_snap_config.py
309+++ b/cloudinit/config/cc_snap_config.py
310@@ -4,11 +4,15 @@
311 #
312 # This file is part of cloud-init. See LICENSE file for license information.
313
314+# RELEASE_BLOCKER: Remove this deprecated module in 18.3
315 """
316 Snap Config
317 -----------
318 **Summary:** snap_config modules allows configuration of snapd.
319
320+**Deprecated**: Use :ref:`snap` module instead. This module will not exist
321+in cloud-init 18.3.
322+
323 This module uses the same ``snappy`` namespace for configuration but
324 acts only only a subset of the configuration.
325
326@@ -154,6 +158,9 @@ def handle(name, cfg, cloud, log, args):
327 LOG.debug('No snappy config provided, skipping')
328 return
329
330+ log.warning(
331+ 'DEPRECATION: snap_config module will be dropped in 18.3 release.'
332+ ' Use snap module instead')
333 if not(util.system_is_snappy()):
334 LOG.debug("%s: system not snappy", name)
335 return
336diff --git a/cloudinit/config/cc_snappy.py b/cloudinit/config/cc_snappy.py
337index eecb817..bab80bb 100644
338--- a/cloudinit/config/cc_snappy.py
339+++ b/cloudinit/config/cc_snappy.py
340@@ -1,10 +1,14 @@
341 # This file is part of cloud-init. See LICENSE file for license information.
342
343+# RELEASE_BLOCKER: Remove this deprecated module in 18.3
344 """
345 Snappy
346 ------
347 **Summary:** snappy modules allows configuration of snappy.
348
349+**Deprecated**: Use :ref:`snap` module instead. This module will not exist
350+in cloud-init 18.3.
351+
352 The below example config config would install ``etcd``, and then install
353 ``pkg2.smoser`` with a ``<config-file>`` argument where ``config-file`` has
354 ``config-blob`` inside it. If ``pkgname`` is installed already, then
355@@ -271,6 +275,10 @@ def handle(name, cfg, cloud, log, args):
356 LOG.debug("%s: 'auto' mode, and system not snappy", name)
357 return
358
359+ log.warning(
360+ 'DEPRECATION: snappy module will be dropped in 18.3 release.'
361+ ' Use snap module instead')
362+
363 set_snappy_command()
364
365 pkg_ops = get_package_ops(packages=mycfg['packages'],
366diff --git a/cloudinit/config/tests/test_snap.py b/cloudinit/config/tests/test_snap.py
367new file mode 100644
368index 0000000..c2dd6af
369--- /dev/null
370+++ b/cloudinit/config/tests/test_snap.py
371@@ -0,0 +1,533 @@
372+# This file is part of cloud-init. See LICENSE file for license information.
373+
374+import re
375+from six import StringIO
376+
377+from cloudinit.config.cc_snap import (
378+ ASSERTIONS_FILE, add_assertions, handle, prepend_snap_commands,
379+ maybe_install_squashfuse, run_commands, schema)
380+from cloudinit.config.schema import validate_cloudconfig_schema
381+from cloudinit import util
382+from cloudinit.tests.helpers import CiTestCase, mock, wrap_and_call
383+
384+
385+SYSTEM_USER_ASSERTION = """\
386+type: system-user
387+authority-id: LqvZQdfyfGlYvtep4W6Oj6pFXP9t1Ksp
388+brand-id: LqvZQdfyfGlYvtep4W6Oj6pFXP9t1Ksp
389+email: foo@bar.com
390+password: $6$E5YiAuMIPAwX58jG$miomhVNui/vf7f/3ctB/f0RWSKFxG0YXzrJ9rtJ1ikvzt
391+series:
392+- 16
393+since: 2016-09-10T16:34:00+03:00
394+until: 2017-11-10T16:34:00+03:00
395+username: baz
396+sign-key-sha3-384: RuVvnp4n52GilycjfbbTCI3_L8Y6QlIE75wxMc0KzGV3AUQqVd9GuXoj
397+
398+AcLBXAQAAQoABgUCV/UU1wAKCRBKnlMoJQLkZVeLD/9/+hIeVywtzsDA3oxl+P+u9D13y9s6svP
399+Jd6Wnf4FTw6sq1GjBE4ZA7lrwSaRCUJ9Vcsvf2q9OGPY7mOb2TBxaDe0PbUMjrSrqllSSQwhpNI
400+zG+NxkkKuxsUmLzFa+k9m6cyojNbw5LFhQZBQCGlr3JYqC0tIREq/UsZxj+90TUC87lDJwkU8GF
401+s4CR+rejZj4itIcDcVxCSnJH6hv6j2JrJskJmvObqTnoOlcab+JXdamXqbldSP3UIhWoyVjqzkj
402++to7mXgx+cCUA9+ngNCcfUG+1huGGTWXPCYkZ78HvErcRlIdeo4d3xwtz1cl/w3vYnq9og1XwsP
403+Yfetr3boig2qs1Y+j/LpsfYBYncgWjeDfAB9ZZaqQz/oc8n87tIPZDJHrusTlBfop8CqcM4xsKS
404+d+wnEY8e/F24mdSOYmS1vQCIDiRU3MKb6x138Ud6oHXFlRBbBJqMMctPqWDunWzb5QJ7YR0I39q
405+BrnEqv5NE0G7w6HOJ1LSPG5Hae3P4T2ea+ATgkb03RPr3KnXnzXg4TtBbW1nytdlgoNc/BafE1H
406+f3NThcq9gwX4xWZ2PAWnqVPYdDMyCtzW3Ck+o6sIzx+dh4gDLPHIi/6TPe/pUuMop9CBpWwez7V
407+v1z+1+URx6Xlq3Jq18y5pZ6fY3IDJ6km2nQPMzcm4Q=="""
408+
409+ACCOUNT_ASSERTION = """\
410+type: account-key
411+authority-id: canonical
412+revision: 2
413+public-key-sha3-384: BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0
414+account-id: canonical
415+name: store
416+since: 2016-04-01T00:00:00.0Z
417+body-length: 717
418+sign-key-sha3-384: -CvQKAwRQ5h3Ffn10FILJoEZUXOv6km9FwA80-Rcj-f-6jadQ89VRswH
419+
420+AcbBTQRWhcGAARAA0KKYYQWuHOrsFVi4p4l7ZzSvX7kLgJFFeFgOkzdWKBTHEnsMKjl5mefFe9j
421+qe8NlmJdfY7BenP7XeBtwKp700H/t9lLrZbpTNAPHXYxEWFJp5bPqIcJYBZ+29oLVLN1Tc5X482
422+vCiDqL8+pPYqBrK2fNlyPlNNSum9wI70rDDL4r6FVvr+osTnGejibdV8JphWX+lrSQDnRSdM8KJ
423+UM43vTgLGTi9W54oRhsA2OFexRfRksTrnqGoonCjqX5wO3OFSaMDzMsO2MJ/hPfLgDqw53qjzuK
424+Iec9OL3k5basvu2cj5u9tKwVFDsCKK2GbKUsWWpx2KTpOifmhmiAbzkTHbH9KaoMS7p0kJwhTQG
425+o9aJ9VMTWHJc/NCBx7eu451u6d46sBPCXS/OMUh2766fQmoRtO1OwCTxsRKG2kkjbMn54UdFULl
426+VfzvyghMNRKIezsEkmM8wueTqGUGZWa6CEZqZKwhe/PROxOPYzqtDH18XZknbU1n5lNb7vNfem9
427+2ai+3+JyFnW9UhfvpVF7gzAgdyCqNli4C6BIN43uwoS8HkykocZS/+Gv52aUQ/NZ8BKOHLw+7an
428+Q0o8W9ltSLZbEMxFIPSN0stiZlkXAp6DLyvh1Y4wXSynDjUondTpej2fSvSlCz/W5v5V7qA4nIc
429+vUvV7RjVzv17ut0AEQEAAQ==
430+
431+AcLDXAQAAQoABgUCV83k9QAKCRDUpVvql9g3IBT8IACKZ7XpiBZ3W4lqbPssY6On81WmxQLtvsM
432+WTp6zZpl/wWOSt2vMNUk9pvcmrNq1jG9CuhDfWFLGXEjcrrmVkN3YuCOajMSPFCGrxsIBLSRt/b
433+nrKykdLAAzMfG8rP1d82bjFFiIieE+urQ0Kcv09Jtdvavq3JT1Tek5mFyyfhHNlQEKOzWqmRWiL
434+3c3VOZUs1ZD8TSlnuq/x+5T0X0YtOyGjSlVxk7UybbyMNd6MZfNaMpIG4x+mxD3KHFtBAC7O6kL
435+eX3i6j5nCY5UABfA3DZEAkWP4zlmdBEOvZ9t293NaDdOpzsUHRkoi0Zez/9BHQ/kwx/uNc2WqrY
436+inCmu16JGNeXqsyinnLl7Ghn2RwhvDMlLxF6RTx8xdx1yk6p3PBTwhZMUvuZGjUtN/AG8BmVJQ1
437+rsGSRkkSywvnhVJRB2sudnrMBmNS2goJbzSbmJnOlBrd2WsV0T9SgNMWZBiov3LvU4o2SmAb6b+
438+rYwh8H5QHcuuYJuxDjFhPswIp6Wes5T6hUicf3SWtObcDS4HSkVS4ImBjjX9YgCuFy7QdnooOWE
439+aPvkRw3XCVeYq0K6w9GRsk1YFErD4XmXXZjDYY650MX9v42Sz5MmphHV8jdIY5ssbadwFSe2rCQ
440+6UX08zy7RsIb19hTndE6ncvSNDChUR9eEnCm73eYaWTWTnq1cxdVP/s52r8uss++OYOkPWqh5nO
441+haRn7INjH/yZX4qXjNXlTjo0PnHH0q08vNKDwLhxS+D9du+70FeacXFyLIbcWllSbJ7DmbumGpF
442+yYbtj3FDDPzachFQdIG3lSt+cSUGeyfSs6wVtc3cIPka/2Urx7RprfmoWSI6+a5NcLdj0u2z8O9
443+HxeIgxDpg/3gT8ZIuFKePMcLDM19Fh/p0ysCsX+84B9chNWtsMSmIaE57V+959MVtsLu7SLb9gi
444+skrju0pQCwsu2wHMLTNd1f3PTHmrr49hxetTus07HSQUApMtAGKzQilF5zqFjbyaTd4xgQbd+PK
445+CjFyzQTDOcUhXpuUGt/IzlqiFfsCsmbj2K4KdSNYMlqIgZ3Azu8KvZLIhsyN7v5vNIZSPfEbjde
446+ClU9r0VRiJmtYBUjcSghD9LWn+yRLwOxhfQVjm0cBwIt5R/yPF/qC76yIVuWUtM5Y2/zJR1J8OF
447+qWchvlImHtvDzS9FQeLyzJAOjvZ2CnWp2gILgUz0WQdOk1Dq8ax7KS9BQ42zxw9EZAEPw3PEFqR
448+IQsRTONp+iVS8YxSmoYZjDlCgRMWUmawez/Fv5b9Fb/XkO5Eq4e+KfrpUujXItaipb+tV8h5v3t
449+oG3Ie3WOHrVjCLXIdYslpL1O4nadqR6Xv58pHj6k"""
450+
451+
452+class FakeCloud(object):
453+ def __init__(self, distro):
454+ self.distro = distro
455+
456+
457+class TestAddAssertions(CiTestCase):
458+
459+ with_logs = True
460+
461+ def setUp(self):
462+ super(TestAddAssertions, self).setUp()
463+ self.tmp = self.tmp_dir()
464+
465+ @mock.patch('cloudinit.config.cc_snap.util.subp')
466+ def test_add_assertions_on_empty_list(self, m_subp):
467+ """When provided with an empty list, add_assertions does nothing."""
468+ add_assertions([])
469+ self.assertEqual('', self.logs.getvalue())
470+ m_subp.assert_not_called()
471+
472+ def test_add_assertions_on_non_list_or_dict(self):
473+ """When provided an invalid type, add_assertions raises an error."""
474+ with self.assertRaises(TypeError) as context_manager:
475+ add_assertions(assertions="I'm Not Valid")
476+ self.assertEqual(
477+ "assertion parameter was not a list or dict: I'm Not Valid",
478+ str(context_manager.exception))
479+
480+ @mock.patch('cloudinit.config.cc_snap.util.subp')
481+ def test_add_assertions_adds_assertions_as_list(self, m_subp):
482+ """When provided with a list, add_assertions adds all assertions."""
483+ self.assertEqual(
484+ ASSERTIONS_FILE, '/var/lib/cloud/instance/snapd.assertions')
485+ assert_file = self.tmp_path('snapd.assertions', dir=self.tmp)
486+ assertions = [SYSTEM_USER_ASSERTION, ACCOUNT_ASSERTION]
487+ wrap_and_call(
488+ 'cloudinit.config.cc_snap',
489+ {'ASSERTIONS_FILE': {'new': assert_file}},
490+ add_assertions, assertions)
491+ self.assertIn(
492+ 'Importing user-provided snap assertions', self.logs.getvalue())
493+ self.assertIn(
494+ 'sertions', self.logs.getvalue())
495+ self.assertEqual(
496+ [mock.call(['snap', 'ack', assert_file], capture=True)],
497+ m_subp.call_args_list)
498+ compare_file = self.tmp_path('comparison', dir=self.tmp)
499+ util.write_file(compare_file, '\n'.join(assertions).encode('utf-8'))
500+ self.assertEqual(
501+ util.load_file(compare_file), util.load_file(assert_file))
502+
503+ @mock.patch('cloudinit.config.cc_snap.util.subp')
504+ def test_add_assertions_adds_assertions_as_dict(self, m_subp):
505+ """When provided with a dict, add_assertions adds all assertions."""
506+ self.assertEqual(
507+ ASSERTIONS_FILE, '/var/lib/cloud/instance/snapd.assertions')
508+ assert_file = self.tmp_path('snapd.assertions', dir=self.tmp)
509+ assertions = {'00': SYSTEM_USER_ASSERTION, '01': ACCOUNT_ASSERTION}
510+ wrap_and_call(
511+ 'cloudinit.config.cc_snap',
512+ {'ASSERTIONS_FILE': {'new': assert_file}},
513+ add_assertions, assertions)
514+ self.assertIn(
515+ 'Importing user-provided snap assertions', self.logs.getvalue())
516+ self.assertIn(
517+ "DEBUG: Snap acking: ['type: system-user', 'authority-id: Lqv",
518+ self.logs.getvalue())
519+ self.assertIn(
520+ "DEBUG: Snap acking: ['type: account-key', 'authority-id: canonic",
521+ self.logs.getvalue())
522+ self.assertEqual(
523+ [mock.call(['snap', 'ack', assert_file], capture=True)],
524+ m_subp.call_args_list)
525+ compare_file = self.tmp_path('comparison', dir=self.tmp)
526+ combined = '\n'.join(assertions.values())
527+ util.write_file(compare_file, combined.encode('utf-8'))
528+ self.assertEqual(
529+ util.load_file(compare_file), util.load_file(assert_file))
530+
531+
532+class TestPrepentSnapCommands(CiTestCase):
533+
534+ with_logs = True
535+
536+ def test_prepend_snap_commands_errors_on_neither_string_nor_list(self):
537+ """Raise an error for each command which is not a string or list."""
538+ orig_commands = ['ls', 1, {'not': 'gonna work'}, ['snap', 'list']]
539+ with self.assertRaises(TypeError) as context_manager:
540+ prepend_snap_commands(orig_commands)
541+ self.assertEqual(
542+ "Invalid snap config. These commands are not a string or list:\n"
543+ "1\n{'not': 'gonna work'}",
544+ str(context_manager.exception))
545+
546+ def test_prepend_snap_commands_warns_on_non_snap_string_commands(self):
547+ """Warn on each non-snap for commands of type string."""
548+ orig_commands = ['ls', 'snap list', 'touch /blah', 'snap install x']
549+ fixed_commands = prepend_snap_commands(orig_commands)
550+ self.assertEqual(
551+ 'WARNING: Non-snap commands in snap config:\n'
552+ 'ls\ntouch /blah\n',
553+ self.logs.getvalue())
554+ self.assertEqual(orig_commands, fixed_commands)
555+
556+ def test_prepend_snap_commands_prepends_on_non_snap_list_commands(self):
557+ """Prepend 'snap' for each non-snap command of type list."""
558+ orig_commands = [['ls'], ['snap', 'list'], ['snapa', '/blah'],
559+ ['snap', 'install', 'x']]
560+ expected = [['snap', 'ls'], ['snap', 'list'],
561+ ['snap', 'snapa', '/blah'],
562+ ['snap', 'install', 'x']]
563+ fixed_commands = prepend_snap_commands(orig_commands)
564+ self.assertEqual('', self.logs.getvalue())
565+ self.assertEqual(expected, fixed_commands)
566+
567+ def test_prepend_snap_commands_removes_first_item_when_none(self):
568+ """Remove the first element of a non-snap command when it is None."""
569+ orig_commands = [[None, 'ls'], ['snap', 'list'],
570+ [None, 'touch', '/blah'],
571+ ['snap', 'install', 'x']]
572+ expected = [['ls'], ['snap', 'list'],
573+ ['touch', '/blah'],
574+ ['snap', 'install', 'x']]
575+ fixed_commands = prepend_snap_commands(orig_commands)
576+ self.assertEqual('', self.logs.getvalue())
577+ self.assertEqual(expected, fixed_commands)
578+
579+
580+class TestRunCommands(CiTestCase):
581+
582+ with_logs = True
583+
584+ def setUp(self):
585+ super(TestRunCommands, self).setUp()
586+ self.tmp = self.tmp_dir()
587+
588+ @mock.patch('cloudinit.config.cc_snap.util.subp')
589+ def test_run_commands_on_empty_list(self, m_subp):
590+ """When provided with an empty list, run_commands does nothing."""
591+ run_commands([])
592+ self.assertEqual('', self.logs.getvalue())
593+ m_subp.assert_not_called()
594+
595+ def test_run_commands_on_non_list_or_dict(self):
596+ """When provided an invalid type, run_commands raises an error."""
597+ with self.assertRaises(TypeError) as context_manager:
598+ run_commands(commands="I'm Not Valid")
599+ self.assertEqual(
600+ "commands parameter was not a list or dict: I'm Not Valid",
601+ str(context_manager.exception))
602+
603+ def test_run_command_logs_commands_and_exit_codes_to_stderr(self):
604+ """All exit codes are logged to stderr."""
605+ outfile = self.tmp_path('output.log', dir=self.tmp)
606+
607+ cmd1 = 'echo "HI" >> %s' % outfile
608+ cmd2 = 'bogus command'
609+ cmd3 = 'echo "MOM" >> %s' % outfile
610+ commands = [cmd1, cmd2, cmd3]
611+
612+ mock_path = 'cloudinit.config.cc_snap.sys.stderr'
613+ with mock.patch(mock_path, new_callable=StringIO) as m_stderr:
614+ with self.assertRaises(RuntimeError) as context_manager:
615+ run_commands(commands=commands)
616+
617+ self.assertIsNotNone(
618+ re.search(r'bogus: (command )?not found',
619+ str(context_manager.exception)),
620+ msg='Expected bogus command not found')
621+ expected_stderr_log = '\n'.join([
622+ 'Begin run command: {cmd}'.format(cmd=cmd1),
623+ 'End run command: exit(0)',
624+ 'Begin run command: {cmd}'.format(cmd=cmd2),
625+ 'ERROR: End run command: exit(127)',
626+ 'Begin run command: {cmd}'.format(cmd=cmd3),
627+ 'End run command: exit(0)\n'])
628+ self.assertEqual(expected_stderr_log, m_stderr.getvalue())
629+
630+ def test_run_command_as_lists(self):
631+ """When commands are specified as a list, run them in order."""
632+ outfile = self.tmp_path('output.log', dir=self.tmp)
633+
634+ cmd1 = 'echo "HI" >> %s' % outfile
635+ cmd2 = 'echo "MOM" >> %s' % outfile
636+ commands = [cmd1, cmd2]
637+ mock_path = 'cloudinit.config.cc_snap.sys.stderr'
638+ with mock.patch(mock_path, new_callable=StringIO):
639+ run_commands(commands=commands)
640+
641+ self.assertIn(
642+ 'DEBUG: Running user-provided snap commands',
643+ self.logs.getvalue())
644+ self.assertEqual('HI\nMOM\n', util.load_file(outfile))
645+ self.assertIn(
646+ 'WARNING: Non-snap commands in snap config:', self.logs.getvalue())
647+
648+ def test_run_command_dict_sorted_as_command_script(self):
649+ """When commands are a dict, sort them and run."""
650+ outfile = self.tmp_path('output.log', dir=self.tmp)
651+ cmd1 = 'echo "HI" >> %s' % outfile
652+ cmd2 = 'echo "MOM" >> %s' % outfile
653+ commands = {'02': cmd1, '01': cmd2}
654+ mock_path = 'cloudinit.config.cc_snap.sys.stderr'
655+ with mock.patch(mock_path, new_callable=StringIO):
656+ run_commands(commands=commands)
657+
658+ expected_messages = [
659+ 'DEBUG: Running user-provided snap commands']
660+ for message in expected_messages:
661+ self.assertIn(message, self.logs.getvalue())
662+ self.assertEqual('MOM\nHI\n', util.load_file(outfile))
663+
664+
665+class TestSchema(CiTestCase):
666+
667+ with_logs = True
668+
669+ def test_schema_warns_on_snap_not_as_dict(self):
670+ """If the snap configuration is not a dict, emit a warning."""
671+ validate_cloudconfig_schema({'snap': 'wrong type'}, schema)
672+ self.assertEqual(
673+ "WARNING: Invalid config:\nsnap: 'wrong type' is not of type"
674+ " 'object'\n",
675+ self.logs.getvalue())
676+
677+ @mock.patch('cloudinit.config.cc_snap.run_commands')
678+ def test_schema_disallows_unknown_keys(self, _):
679+ """Unknown keys in the snap configuration emit warnings."""
680+ validate_cloudconfig_schema(
681+ {'snap': {'commands': ['ls'], 'invalid-key': ''}}, schema)
682+ self.assertIn(
683+ 'WARNING: Invalid config:\nsnap: Additional properties are not'
684+ " allowed ('invalid-key' was unexpected)",
685+ self.logs.getvalue())
686+
687+ def test_warn_schema_requires_either_commands_or_assertions(self):
688+ """Warn when snap configuration lacks both commands and assertions."""
689+ validate_cloudconfig_schema(
690+ {'snap': {}}, schema)
691+ self.assertIn(
692+ 'WARNING: Invalid config:\nsnap: {} does not have enough'
693+ ' properties',
694+ self.logs.getvalue())
695+
696+ @mock.patch('cloudinit.config.cc_snap.run_commands')
697+ def test_warn_schema_commands_is_not_list_or_dict(self, _):
698+ """Warn when snap:commands config is not a list or dict."""
699+ validate_cloudconfig_schema(
700+ {'snap': {'commands': 'broken'}}, schema)
701+ self.assertEqual(
702+ "WARNING: Invalid config:\nsnap.commands: 'broken' is not of type"
703+ " 'object', 'array'\n",
704+ self.logs.getvalue())
705+
706+ @mock.patch('cloudinit.config.cc_snap.run_commands')
707+ def test_warn_schema_when_commands_is_empty(self, _):
708+ """Emit warnings when snap:commands is an empty list or dict."""
709+ validate_cloudconfig_schema(
710+ {'snap': {'commands': []}}, schema)
711+ validate_cloudconfig_schema(
712+ {'snap': {'commands': {}}}, schema)
713+ self.assertEqual(
714+ "WARNING: Invalid config:\nsnap.commands: [] is too short\n"
715+ "WARNING: Invalid config:\nsnap.commands: {} does not have enough"
716+ " properties\n",
717+ self.logs.getvalue())
718+
719+ @mock.patch('cloudinit.config.cc_snap.run_commands')
720+ def test_schema_when_commands_are_list_or_dict(self, _):
721+ """No warnings when snap:commands are either a list or dict."""
722+ validate_cloudconfig_schema(
723+ {'snap': {'commands': ['valid']}}, schema)
724+ validate_cloudconfig_schema(
725+ {'snap': {'commands': {'01': 'also valid'}}}, schema)
726+ self.assertEqual('', self.logs.getvalue())
727+
728+ @mock.patch('cloudinit.config.cc_snap.add_assertions')
729+ def test_warn_schema_assertions_is_not_list_or_dict(self, _):
730+ """Warn when snap:assertions config is not a list or dict."""
731+ validate_cloudconfig_schema(
732+ {'snap': {'assertions': 'broken'}}, schema)
733+ self.assertEqual(
734+ "WARNING: Invalid config:\nsnap.assertions: 'broken' is not of"
735+ " type 'object', 'array'\n",
736+ self.logs.getvalue())
737+
738+ @mock.patch('cloudinit.config.cc_snap.add_assertions')
739+ def test_warn_schema_when_assertions_is_empty(self, _):
740+ """Emit warnings when snap:assertions is an empty list or dict."""
741+ validate_cloudconfig_schema(
742+ {'snap': {'assertions': []}}, schema)
743+ validate_cloudconfig_schema(
744+ {'snap': {'assertions': {}}}, schema)
745+ self.assertEqual(
746+ "WARNING: Invalid config:\nsnap.assertions: [] is too short\n"
747+ "WARNING: Invalid config:\nsnap.assertions: {} does not have"
748+ " enough properties\n",
749+ self.logs.getvalue())
750+
751+ @mock.patch('cloudinit.config.cc_snap.add_assertions')
752+ def test_schema_when_assertions_are_list_or_dict(self, _):
753+ """No warnings when snap:assertions are a list or dict."""
754+ validate_cloudconfig_schema(
755+ {'snap': {'assertions': ['valid']}}, schema)
756+ validate_cloudconfig_schema(
757+ {'snap': {'assertions': {'01': 'also valid'}}}, schema)
758+ self.assertEqual('', self.logs.getvalue())
759+
760+
761+class TestHandle(CiTestCase):
762+
763+ with_logs = True
764+
765+ def setUp(self):
766+ super(TestHandle, self).setUp()
767+ self.tmp = self.tmp_dir()
768+
769+ @mock.patch('cloudinit.config.cc_snap.run_commands')
770+ @mock.patch('cloudinit.config.cc_snap.add_assertions')
771+ @mock.patch('cloudinit.config.cc_snap.validate_cloudconfig_schema')
772+ def test_handle_no_config(self, m_schema, m_add, m_run):
773+ """When no snap-related configuration is provided, nothing happens."""
774+ cfg = {}
775+ handle('snap', cfg=cfg, cloud=None, log=self.logger, args=None)
776+ self.assertIn(
777+ "DEBUG: Skipping module named snap, no 'snap' key in config",
778+ self.logs.getvalue())
779+ m_schema.assert_not_called()
780+ m_add.assert_not_called()
781+ m_run.assert_not_called()
782+
783+ @mock.patch('cloudinit.config.cc_snap.run_commands')
784+ @mock.patch('cloudinit.config.cc_snap.add_assertions')
785+ @mock.patch('cloudinit.config.cc_snap.maybe_install_squashfuse')
786+ def test_handle_skips_squashfuse_when_unconfigured(self, m_squash, m_add,
787+ m_run):
788+ """When squashfuse_in_container is unset, don't attempt to install."""
789+ handle(
790+ 'snap', cfg={'snap': {}}, cloud=None, log=self.logger, args=None)
791+ handle(
792+ 'snap', cfg={'snap': {'squashfuse_in_container': None}},
793+ cloud=None, log=self.logger, args=None)
794+ handle(
795+ 'snap', cfg={'snap': {'squashfuse_in_container': False}},
796+ cloud=None, log=self.logger, args=None)
797+ self.assertEqual([], m_squash.call_args_list) # No calls
798+ # snap configuration missing assertions and commands will default to []
799+ self.assertIn(mock.call([]), m_add.call_args_list)
800+ self.assertIn(mock.call([]), m_run.call_args_list)
801+
802+ @mock.patch('cloudinit.config.cc_snap.maybe_install_squashfuse')
803+ def test_handle_tries_to_install_squashfuse(self, m_squash):
804+ """If squashfuse_in_container is True, try installing squashfuse."""
805+ cfg = {'snap': {'squashfuse_in_container': True}}
806+ mycloud = FakeCloud(None)
807+ handle('snap', cfg=cfg, cloud=mycloud, log=self.logger, args=None)
808+ self.assertEqual(
809+ [mock.call(mycloud)], m_squash.call_args_list)
810+
811+ def test_handle_runs_commands_provided(self):
812+ """If commands are specified as a list, run them."""
813+ outfile = self.tmp_path('output.log', dir=self.tmp)
814+
815+ cfg = {
816+ 'snap': {'commands': ['echo "HI" >> %s' % outfile,
817+ 'echo "MOM" >> %s' % outfile]}}
818+ handle('snap', cfg=cfg, cloud=None, log=self.logger, args=None)
819+ self.assertEqual('HI\nMOM\n', util.load_file(outfile))
820+
821+ @mock.patch('cloudinit.config.cc_snap.util.subp')
822+ def test_handle_adds_assertions(self, m_subp):
823+ """Any configured snap assertions are provided to add_assertions."""
824+ assert_file = self.tmp_path('snapd.assertions', dir=self.tmp)
825+ compare_file = self.tmp_path('comparison', dir=self.tmp)
826+ cfg = {
827+ 'snap': {'assertions': [SYSTEM_USER_ASSERTION, ACCOUNT_ASSERTION]}}
828+ wrap_and_call(
829+ 'cloudinit.config.cc_snap',
830+ {'ASSERTIONS_FILE': {'new': assert_file}},
831+ handle, 'snap', cfg=cfg, cloud=None, log=self.logger, args=None)
832+ content = '\n'.join(cfg['snap']['assertions'])
833+ util.write_file(compare_file, content.encode('utf-8'))
834+ self.assertEqual(
835+ util.load_file(compare_file), util.load_file(assert_file))
836+
837+ @mock.patch('cloudinit.config.cc_snap.util.subp')
838+ def test_handle_validates_schema(self, m_subp):
839+ """Any provided configuration is runs validate_cloudconfig_schema."""
840+ assert_file = self.tmp_path('snapd.assertions', dir=self.tmp)
841+ cfg = {'snap': {'invalid': ''}} # Generates schema warning
842+ wrap_and_call(
843+ 'cloudinit.config.cc_snap',
844+ {'ASSERTIONS_FILE': {'new': assert_file}},
845+ handle, 'snap', cfg=cfg, cloud=None, log=self.logger, args=None)
846+ self.assertEqual(
847+ "WARNING: Invalid config:\nsnap: Additional properties are not"
848+ " allowed ('invalid' was unexpected)\n",
849+ self.logs.getvalue())
850+
851+
852+class TestMaybeInstallSquashFuse(CiTestCase):
853+
854+ with_logs = True
855+
856+ def setUp(self):
857+ super(TestMaybeInstallSquashFuse, self).setUp()
858+ self.tmp = self.tmp_dir()
859+
860+ @mock.patch('cloudinit.config.cc_snap.util.is_container')
861+ def test_maybe_install_squashfuse_skips_non_containers(self, m_container):
862+ """maybe_install_squashfuse does nothing when not on a container."""
863+ m_container.return_value = False
864+ maybe_install_squashfuse(cloud=FakeCloud(None))
865+ self.assertEqual([mock.call()], m_container.call_args_list)
866+ self.assertEqual('', self.logs.getvalue())
867+
868+ @mock.patch('cloudinit.config.cc_snap.util.is_container')
869+ def test_maybe_install_squashfuse_raises_install_errors(self, m_container):
870+ """maybe_install_squashfuse logs and raises package install errors."""
871+ m_container.return_value = True
872+ distro = mock.MagicMock()
873+ distro.update_package_sources.side_effect = RuntimeError(
874+ 'Some apt error')
875+ with self.assertRaises(RuntimeError) as context_manager:
876+ maybe_install_squashfuse(cloud=FakeCloud(distro))
877+ self.assertEqual('Some apt error', str(context_manager.exception))
878+ self.assertIn('Package update failed\nTraceback', self.logs.getvalue())
879+
880+ @mock.patch('cloudinit.config.cc_snap.util.is_container')
881+ def test_maybe_install_squashfuse_raises_update_errors(self, m_container):
882+ """maybe_install_squashfuse logs and raises package update errors."""
883+ m_container.return_value = True
884+ distro = mock.MagicMock()
885+ distro.update_package_sources.side_effect = RuntimeError(
886+ 'Some apt error')
887+ with self.assertRaises(RuntimeError) as context_manager:
888+ maybe_install_squashfuse(cloud=FakeCloud(distro))
889+ self.assertEqual('Some apt error', str(context_manager.exception))
890+ self.assertIn('Package update failed\nTraceback', self.logs.getvalue())
891+
892+ @mock.patch('cloudinit.config.cc_snap.util.is_container')
893+ def test_maybe_install_squashfuse_happy_path(self, m_container):
894+ """maybe_install_squashfuse logs and raises package install errors."""
895+ m_container.return_value = True
896+ distro = mock.MagicMock() # No errors raised
897+ maybe_install_squashfuse(cloud=FakeCloud(distro))
898+ self.assertEqual(
899+ [mock.call()], distro.update_package_sources.call_args_list)
900+ self.assertEqual(
901+ [mock.call(['squashfuse'])],
902+ distro.install_packages.call_args_list)
903+
904+# vi: ts=4 expandtab
905diff --git a/cloudinit/util.py b/cloudinit/util.py
906index 823d80b..cae8b19 100644
907--- a/cloudinit/util.py
908+++ b/cloudinit/util.py
909@@ -1827,7 +1827,8 @@ def subp_blob_in_tempfile(blob, *args, **kwargs):
910
911
912 def subp(args, data=None, rcs=None, env=None, capture=True, shell=False,
913- logstring=False, decode="replace", target=None, update_env=None):
914+ logstring=False, decode="replace", target=None, update_env=None,
915+ status_cb=None):
916
917 # not supported in cloud-init (yet), for now kept in the call signature
918 # to ease maintaining code shared between cloud-init and curtin
919@@ -1848,6 +1849,9 @@ def subp(args, data=None, rcs=None, env=None, capture=True, shell=False,
920 if target_path(target) != "/":
921 args = ['chroot', target] + list(args)
922
923+ if status_cb:
924+ command = ' '.join(args) if isinstance(args, list) else args
925+ status_cb('Begin run command: {command}\n'.format(command=command))
926 if not logstring:
927 LOG.debug(("Running command %s with allowed return codes %s"
928 " (shell=%s, capture=%s)"), args, rcs, shell, capture)
929@@ -1888,6 +1892,8 @@ def subp(args, data=None, rcs=None, env=None, capture=True, shell=False,
930 env=env, shell=shell)
931 (out, err) = sp.communicate(data)
932 except OSError as e:
933+ if status_cb:
934+ status_cb('ERROR: End run command: invalid command provided\n')
935 raise ProcessExecutionError(
936 cmd=args, reason=e, errno=e.errno,
937 stdout="-" if decode else b"-",
938@@ -1912,9 +1918,14 @@ def subp(args, data=None, rcs=None, env=None, capture=True, shell=False,
939
940 rc = sp.returncode
941 if rc not in rcs:
942+ if status_cb:
943+ status_cb(
944+ 'ERROR: End run command: exit({code})\n'.format(code=rc))
945 raise ProcessExecutionError(stdout=out, stderr=err,
946 exit_code=rc,
947 cmd=args)
948+ if status_cb:
949+ status_cb('End run command: exit({code})\n'.format(code=rc))
950 return (out, err)
951
952
953diff --git a/config/cloud.cfg.tmpl b/config/cloud.cfg.tmpl
954index cf2e240..56a34fa 100644
955--- a/config/cloud.cfg.tmpl
956+++ b/config/cloud.cfg.tmpl
957@@ -72,7 +72,8 @@ cloud_config_modules:
958 # Emit the cloud config ready event
959 # this can be used by upstart jobs for 'start on cloud-config'.
960 - emit_upstart
961- - snap_config
962+ - snap
963+ - snap_config # DEPRECATED- Drop in version 18.2
964 {% endif %}
965 - ssh-import-id
966 - locale
967@@ -102,7 +103,7 @@ cloud_config_modules:
968 # The modules that run in the 'final' stage
969 cloud_final_modules:
970 {% if variant in ["ubuntu", "unknown", "debian"] %}
971- - snappy
972+ - snappy # DEPRECATED- Drop in version 18.2
973 {% endif %}
974 - package-update-upgrade-install
975 {% if variant in ["ubuntu", "unknown", "debian"] %}
976diff --git a/doc/rtd/conf.py b/doc/rtd/conf.py
977index 0ea3b6b..50eb05c 100644
978--- a/doc/rtd/conf.py
979+++ b/doc/rtd/conf.py
980@@ -29,6 +29,7 @@ project = 'Cloud-Init'
981 extensions = [
982 'sphinx.ext.intersphinx',
983 'sphinx.ext.autodoc',
984+ 'sphinx.ext.autosectionlabel',
985 'sphinx.ext.viewcode',
986 ]
987
988diff --git a/doc/rtd/topics/modules.rst b/doc/rtd/topics/modules.rst
989index 7b14675..a0f6812 100644
990--- a/doc/rtd/topics/modules.rst
991+++ b/doc/rtd/topics/modules.rst
992@@ -45,6 +45,7 @@ Modules
993 .. automodule:: cloudinit.config.cc_seed_random
994 .. automodule:: cloudinit.config.cc_set_hostname
995 .. automodule:: cloudinit.config.cc_set_passwords
996+.. automodule:: cloudinit.config.cc_snap
997 .. automodule:: cloudinit.config.cc_snappy
998 .. automodule:: cloudinit.config.cc_snap_config
999 .. automodule:: cloudinit.config.cc_spacewalk
1000diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
1001index d8bc170..c7dcbe8 100644
1002--- a/tests/cloud_tests/releases.yaml
1003+++ b/tests/cloud_tests/releases.yaml
1004@@ -30,6 +30,9 @@ default_release_config:
1005 mirror_url: https://cloud-images.ubuntu.com/daily
1006 mirror_dir: '/srv/citest/images'
1007 keyring: /usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
1008+ # The OS version formatted as Major.Minor is used to compare releases
1009+ version: null # Each release needs to define this, for example 16.04
1010+
1011 ec2:
1012 # Choose from: [ebs, instance-store]
1013 root-store: ebs
1014diff --git a/tests/cloud_tests/testcases.yaml b/tests/cloud_tests/testcases.yaml
1015index 8e0fb62..a3e2990 100644
1016--- a/tests/cloud_tests/testcases.yaml
1017+++ b/tests/cloud_tests/testcases.yaml
1018@@ -15,6 +15,9 @@ base_test_data:
1019 instance-id: |
1020 #!/bin/sh
1021 cat /run/cloud-init/.instance-id
1022+ instance-data.json: |
1023+ #!/bin/sh
1024+ cat /run/cloud-init/instance-data.json
1025 result.json: |
1026 #!/bin/sh
1027 cat /run/cloud-init/result.json
1028diff --git a/tests/cloud_tests/testcases/__init__.py b/tests/cloud_tests/testcases/__init__.py
1029index a29a092..bd548f5 100644
1030--- a/tests/cloud_tests/testcases/__init__.py
1031+++ b/tests/cloud_tests/testcases/__init__.py
1032@@ -7,6 +7,8 @@ import inspect
1033 import unittest
1034 from unittest.util import strclass
1035
1036+from cloudinit.util import read_conf
1037+
1038 from tests.cloud_tests import config
1039 from tests.cloud_tests.testcases.base import CloudTestCase as base_test
1040
1041@@ -48,6 +50,7 @@ def get_suite(test_name, data, conf):
1042 def setUpClass(cls):
1043 cls.data = data
1044 cls.conf = conf
1045+ cls.release_conf = read_conf(config.RELEASES_CONF)['releases']
1046
1047 suite.addTest(unittest.defaultTestLoader.loadTestsFromTestCase(tmp))
1048
1049diff --git a/tests/cloud_tests/testcases/base.py b/tests/cloud_tests/testcases/base.py
1050index 20e9595..324c7c9 100644
1051--- a/tests/cloud_tests/testcases/base.py
1052+++ b/tests/cloud_tests/testcases/base.py
1053@@ -4,10 +4,14 @@
1054
1055 import crypt
1056 import json
1057+import re
1058 import unittest
1059
1060+
1061 from cloudinit import util as c_util
1062
1063+SkipTest = unittest.SkipTest
1064+
1065
1066 class CloudTestCase(unittest.TestCase):
1067 """Base test class for verifiers."""
1068@@ -16,6 +20,43 @@ class CloudTestCase(unittest.TestCase):
1069 data = {}
1070 conf = None
1071 _cloud_config = None
1072+ release_conf = {} # The platform's os release configuration
1073+
1074+ expected_warnings = () # Subclasses set to ignore expected WARN logs
1075+
1076+ @property
1077+ def os_cfg(self):
1078+ return self.release_conf[self.os_name]['default']
1079+
1080+ def is_distro(self, distro_name):
1081+ return self.os_cfg['os'] == distro_name
1082+
1083+ def os_version_cmp(self, cmp_version):
1084+ """Compare the version of the test to comparison_version.
1085+
1086+ @param: cmp_version: Either a float or a string representing
1087+ a release os from releases.yaml (e.g. centos66)
1088+
1089+ @return: -1 when version < cmp_version, 0 when version=cmp_version and
1090+ 1 when version > cmp_version.
1091+ """
1092+ version = self.release_conf[self.os_name]['default']['version']
1093+ if isinstance(cmp_version, str):
1094+ cmp_version = self.release_conf[cmp_version]['default']['version']
1095+ if version < cmp_version:
1096+ return -1
1097+ elif version == cmp_version:
1098+ return 0
1099+ else:
1100+ return 1
1101+
1102+ @property
1103+ def os_name(self):
1104+ return self.data.get('os_name', 'UNKNOWN')
1105+
1106+ @property
1107+ def platform(self):
1108+ return self.data.get('platform', 'UNKNOWN')
1109
1110 @property
1111 def cloud_config(self):
1112@@ -72,12 +113,134 @@ class CloudTestCase(unittest.TestCase):
1113 self.assertEqual(len(result['errors']), 0)
1114
1115 def test_no_warnings_in_log(self):
1116- """Warnings should not be found in the log."""
1117+ """Unexpected warnings should not be found in the log."""
1118+ warnings = [
1119+ l for l in self.get_data_file('cloud-init.log').splitlines()
1120+ if 'WARN' in l]
1121+ joined_warnings = '\n'.join(warnings)
1122+ for expected_warning in self.expected_warnings:
1123+ self.assertIn(
1124+ expected_warning, joined_warnings,
1125+ msg="Did not find %s in cloud-init.log" % expected_warning)
1126+ # Prune expected from discovered warnings
1127+ warnings = [w for w in warnings if expected_warning not in w]
1128+ self.assertEqual(
1129+ [], warnings, msg="'WARN' found inside cloud-init.log")
1130+
1131+ def test_instance_data_json_ec2(self):
1132+ """Validate instance-data.json content by ec2 platform.
1133+
1134+ This content is sourced by snapd when determining snapstore endpoints.
1135+ We validate expected values per cloud type to ensure we don't break
1136+ snapd.
1137+ """
1138+ if self.platform != 'ec2':
1139+ raise SkipTest(
1140+ 'Skipping ec2 instance-data.json on %s' % self.platform)
1141+ out = self.get_data_file('instance-data.json')
1142+ if not out:
1143+ if self.is_distro('ubuntu') and self.os_version_cmp('bionic') >= 0:
1144+ raise AssertionError(
1145+ 'No instance-data.json found on %s' % self.os_name)
1146+ raise SkipTest(
1147+ 'Skipping instance-data.json test.'
1148+ ' OS: %s not bionic or newer' % self.os_name)
1149+ instance_data = json.loads(out)
1150+ self.assertEqual(
1151+ ['ds/user-data'], instance_data['base64-encoded-keys'])
1152+ ds = instance_data.get('ds', {})
1153+ macs = ds.get('network', {}).get('interfaces', {}).get('macs', {})
1154+ if not macs:
1155+ raise AssertionError('No network data from EC2 meta-data')
1156+ # Check meta-data items we depend on
1157+ expected_net_keys = [
1158+ 'public-ipv4s', 'ipv4-associations', 'local-hostname',
1159+ 'public-hostname']
1160+ for mac, mac_data in macs.items():
1161+ for key in expected_net_keys:
1162+ self.assertIn(key, mac_data)
1163+ self.assertIsNotNone(
1164+ ds.get('placement', {}).get('availability-zone'),
1165+ 'Could not determine EC2 Availability zone placement')
1166+ ds = instance_data.get('ds', {})
1167+ v1_data = instance_data.get('v1', {})
1168+ self.assertIsNotNone(
1169+ v1_data['availability-zone'], 'expected ec2 availability-zone')
1170+ self.assertEqual('aws', v1_data['cloud-name'])
1171+ self.assertIn('i-', v1_data['instance-id'])
1172+ self.assertIn('ip-', v1_data['local-hostname'])
1173+ self.assertIsNotNone(v1_data['region'], 'expected ec2 region')
1174+
1175+ def test_instance_data_json_lxd(self):
1176+ """Validate instance-data.json content by lxd platform.
1177+
1178+ This content is sourced by snapd when determining snapstore endpoints.
1179+ We validate expected values per cloud type to ensure we don't break
1180+ snapd.
1181+ """
1182+ if self.platform != 'lxd':
1183+ raise SkipTest(
1184+ 'Skipping lxd instance-data.json on %s' % self.platform)
1185+ out = self.get_data_file('instance-data.json')
1186+ if not out:
1187+ if self.is_distro('ubuntu') and self.os_version_cmp('bionic') >= 0:
1188+ raise AssertionError(
1189+ 'No instance-data.json found on %s' % self.os_name)
1190+ raise SkipTest(
1191+ 'Skipping instance-data.json test.'
1192+ ' OS: %s not bionic or newer' % self.os_name)
1193+ instance_data = json.loads(out)
1194+ v1_data = instance_data.get('v1', {})
1195+ self.assertEqual(
1196+ ['ds/user-data', 'ds/vendor-data'],
1197+ sorted(instance_data['base64-encoded-keys']))
1198+ self.assertEqual('nocloud', v1_data['cloud-name'])
1199+ self.assertIsNone(
1200+ v1_data['availability-zone'],
1201+ 'found unexpected lxd availability-zone %s' %
1202+ v1_data['availability-zone'])
1203+ self.assertIn('cloud-test', v1_data['instance-id'])
1204+ self.assertIn('cloud-test', v1_data['local-hostname'])
1205+ self.assertIsNone(
1206+ v1_data['region'],
1207+ 'found unexpected lxd region %s' % v1_data['region'])
1208+
1209+ def test_instance_data_json_kvm(self):
1210+ """Validate instance-data.json content by nocloud-kvm platform.
1211+
1212+ This content is sourced by snapd when determining snapstore endpoints.
1213+ We validate expected values per cloud type to ensure we don't break
1214+ snapd.
1215+ """
1216+ if self.platform != 'nocloud-kvm':
1217+ raise SkipTest(
1218+ 'Skipping nocloud-kvm instance-data.json on %s' %
1219+ self.platform)
1220+ out = self.get_data_file('instance-data.json')
1221+ if not out:
1222+ if self.is_distro('ubuntu') and self.os_version_cmp('bionic') >= 0:
1223+ raise AssertionError(
1224+ 'No instance-data.json found on %s' % self.os_name)
1225+ raise SkipTest(
1226+ 'Skipping instance-data.json test.'
1227+ ' OS: %s not bionic or newer' % self.os_name)
1228+ instance_data = json.loads(out)
1229+ v1_data = instance_data.get('v1', {})
1230 self.assertEqual(
1231- [],
1232- [l for l in self.get_data_file('cloud-init.log').splitlines()
1233- if 'WARN' in l],
1234- msg="'WARN' found inside cloud-init.log")
1235+ ['ds/user-data'], instance_data['base64-encoded-keys'])
1236+ self.assertEqual('nocloud', v1_data['cloud-name'])
1237+ self.assertIsNone(
1238+ v1_data['availability-zone'],
1239+ 'found unexpected kvm availability-zone %s' %
1240+ v1_data['availability-zone'])
1241+ self.assertIsNotNone(
1242+ re.match('[\da-f]{8}(-[\da-f]{4}){3}-[\da-f]{12}',
1243+ v1_data['instance-id']),
1244+ 'kvm instance-id is not a UUID: %s' % v1_data['instance-id'])
1245+ self.assertIn('ubuntu', v1_data['local-hostname'])
1246+ self.assertIsNone(
1247+ v1_data['region'],
1248+ 'found unexpected lxd region %s' % v1_data['region'])
1249
1250
1251 class PasswordListTest(CloudTestCase):
1252diff --git a/tests/cloud_tests/testcases/main/command_output_simple.py b/tests/cloud_tests/testcases/main/command_output_simple.py
1253index 857881c..80a2c8d 100644
1254--- a/tests/cloud_tests/testcases/main/command_output_simple.py
1255+++ b/tests/cloud_tests/testcases/main/command_output_simple.py
1256@@ -7,6 +7,8 @@ from tests.cloud_tests.testcases import base
1257 class TestCommandOutputSimple(base.CloudTestCase):
1258 """Test functionality of simple output redirection."""
1259
1260+ expected_warnings = ('Stdout, stderr changing to',)
1261+
1262 def test_output_file(self):
1263 """Ensure that the output file is not empty and has all stages."""
1264 data = self.get_data_file('cloud-init-test-output')
1265@@ -15,20 +17,5 @@ class TestCommandOutputSimple(base.CloudTestCase):
1266 data.splitlines()[-1].strip())
1267 # TODO: need to test that all stages redirected here
1268
1269- def test_no_warnings_in_log(self):
1270- """Warnings should not be found in the log.
1271-
1272- This class redirected stderr and stdout, so it expects to find
1273- a warning in cloud-init.log to that effect."""
1274- redirect_msg = 'Stdout, stderr changing to'
1275- warnings = [
1276- l for l in self.get_data_file('cloud-init.log').splitlines()
1277- if 'WARN' in l]
1278- self.assertEqual(
1279- [], [w for w in warnings if redirect_msg not in w],
1280- msg="'WARN' found inside cloud-init.log")
1281- self.assertEqual(
1282- 1, len(warnings),
1283- msg="Did not find %s in cloud-init.log" % redirect_msg)
1284
1285 # vi: ts=4 expandtab
1286diff --git a/tests/cloud_tests/testcases/modules/snap.py b/tests/cloud_tests/testcases/modules/snap.py
1287new file mode 100644
1288index 0000000..ff68abb
1289--- /dev/null
1290+++ b/tests/cloud_tests/testcases/modules/snap.py
1291@@ -0,0 +1,16 @@
1292+# This file is part of cloud-init. See LICENSE file for license information.
1293+
1294+"""cloud-init Integration Test Verify Script"""
1295+from tests.cloud_tests.testcases import base
1296+
1297+
1298+class TestSnap(base.CloudTestCase):
1299+ """Test snap module"""
1300+
1301+ def test_snappy_version(self):
1302+ """Expect hello-world and core snaps are installed."""
1303+ out = self.get_data_file('snaplist')
1304+ self.assertIn('core', out)
1305+ self.assertIn('hello-world', out)
1306+
1307+# vi: ts=4 expandtab
1308diff --git a/tests/cloud_tests/testcases/modules/snap.yaml b/tests/cloud_tests/testcases/modules/snap.yaml
1309new file mode 100644
1310index 0000000..44043f3
1311--- /dev/null
1312+++ b/tests/cloud_tests/testcases/modules/snap.yaml
1313@@ -0,0 +1,18 @@
1314+#
1315+# Install snappy
1316+#
1317+required_features:
1318+ - snap
1319+cloud_config: |
1320+ #cloud-config
1321+ package_update: true
1322+ snap:
1323+ squashfuse_in_container: true
1324+ commands:
1325+ - snap install hello-world
1326+collect_scripts:
1327+ snaplist: |
1328+ #!/bin/bash
1329+ snap list
1330+
1331+# vi: ts=4 expandtab
1332diff --git a/tests/cloud_tests/testcases/modules/snappy.py b/tests/cloud_tests/testcases/modules/snappy.py
1333index b92271c..7d17fc5 100644
1334--- a/tests/cloud_tests/testcases/modules/snappy.py
1335+++ b/tests/cloud_tests/testcases/modules/snappy.py
1336@@ -7,6 +7,8 @@ from tests.cloud_tests.testcases import base
1337 class TestSnappy(base.CloudTestCase):
1338 """Test snappy module"""
1339
1340+ expected_warnings = ('DEPRECATION',)
1341+
1342 def test_snappy_version(self):
1343 """Test snappy version output"""
1344 out = self.get_data_file('snapd')
1345diff --git a/tests/cloud_tests/verify.py b/tests/cloud_tests/verify.py
1346index 2a9fd52..5a68a48 100644
1347--- a/tests/cloud_tests/verify.py
1348+++ b/tests/cloud_tests/verify.py
1349@@ -8,13 +8,16 @@ import unittest
1350 from tests.cloud_tests import (config, LOG, util, testcases)
1351
1352
1353-def verify_data(base_dir, tests):
1354+def verify_data(data_dir, platform, os_name, tests):
1355 """Verify test data is correct.
1356
1357- @param base_dir: base directory for data
1358+ @param data_dir: top level directory for all tests
1359+ @param platform: The platform name we for this test data (e.g. lxd)
1360+ @param os_name: The operating system under test (xenial, artful, etc.).
1361 @param tests: list of test names
1362 @return_value: {<test_name>: {passed: True/False, failures: []}}
1363 """
1364+ base_dir = os.sep.join((data_dir, platform, os_name))
1365 runner = unittest.TextTestRunner(verbosity=util.current_verbosity())
1366 res = {}
1367 for test_name in tests:
1368@@ -26,7 +29,7 @@ def verify_data(base_dir, tests):
1369 cloud_conf = test_conf['cloud_config']
1370
1371 # load script outputs
1372- data = {}
1373+ data = {'platform': platform, 'os_name': os_name}
1374 test_dir = os.path.join(base_dir, test_name)
1375 for script_name in os.listdir(test_dir):
1376 with open(os.path.join(test_dir, script_name), 'rb') as fp:
1377@@ -73,7 +76,7 @@ def verify(args):
1378
1379 # run test
1380 res[platform][os_name] = verify_data(
1381- os.sep.join((args.data_dir, platform, os_name)),
1382+ args.data_dir, platform, os_name,
1383 tests[platform][os_name])
1384
1385 # handle results
1386diff --git a/tests/unittests/test_handler/test_schema.py b/tests/unittests/test_handler/test_schema.py
1387index 1ecb6c6..9b50ee7 100644
1388--- a/tests/unittests/test_handler/test_schema.py
1389+++ b/tests/unittests/test_handler/test_schema.py
1390@@ -26,6 +26,7 @@ class GetSchemaTest(CiTestCase):
1391 'cc_ntp',
1392 'cc_resizefs',
1393 'cc_runcmd',
1394+ 'cc_snap',
1395 'cc_zypper_add_repo'
1396 ],
1397 [subschema['id'] for subschema in schema['allOf']])
1398diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
1399index 499e7c9..67d9607 100644
1400--- a/tests/unittests/test_util.py
1401+++ b/tests/unittests/test_util.py
1402@@ -785,6 +785,39 @@ class TestSubp(helpers.CiTestCase):
1403 decode=False)
1404 self.assertEqual(self.utf8_valid, out)
1405
1406+ def test_bogus_command_logs_status_messages(self):
1407+ """status_cb gets status messages logs on bogus commands provided."""
1408+ logs = []
1409+
1410+ def status_cb(log):
1411+ logs.append(log)
1412+
1413+ with self.assertRaises(util.ProcessExecutionError):
1414+ util.subp([self.bogus_command], status_cb=status_cb)
1415+
1416+ expected = [
1417+ 'Begin run command: {cmd}\n'.format(cmd=self.bogus_command),
1418+ 'ERROR: End run command: invalid command provided\n']
1419+ self.assertEqual(expected, logs)
1420+
1421+ def test_command_logs_exit_codes_to_status_cb(self):
1422+ """status_cb gets status messages containing command exit code."""
1423+ logs = []
1424+
1425+ def status_cb(log):
1426+ logs.append(log)
1427+
1428+ with self.assertRaises(util.ProcessExecutionError):
1429+ util.subp(['ls', '/I/dont/exist'], status_cb=status_cb)
1430+ util.subp(['ls'], status_cb=status_cb)
1431+
1432+ expected = [
1433+ 'Begin run command: ls /I/dont/exist\n',
1434+ 'ERROR: End run command: exit(2)\n',
1435+ 'Begin run command: ls\n',
1436+ 'End run command: exit(0)\n']
1437+ self.assertEqual(expected, logs)
1438+
1439
1440 class TestEncode(helpers.TestCase):
1441 """Test the encoding functions"""

Subscribers

People subscribed via source and target branches