Merge lp:~adam-collard/landscape-charm/install-sources-keys into lp:landscape-charm

Proposed by Adam Collard on 2019-01-17
Status: Merged
Approved by: Simon Poirier on 2019-01-17
Approved revision: 397
Merged at revision: 395
Proposed branch: lp:~adam-collard/landscape-charm/install-sources-keys
Merge into: lp:landscape-charm
Diff against target: 278 lines (+93/-17)
11 files modified
Makefile (+1/-1)
config.yaml (+24/-1)
dev/ubuntu-deps (+2/-1)
lib/apt.py (+18/-1)
lib/callbacks/tests/test_apt.py (+1/-1)
lib/tests/stubs.py (+8/-1)
lib/tests/test_apt.py (+34/-6)
lib/tests/test_install.py (+1/-1)
lib/tests/test_services.py (+1/-1)
lib/tests/test_upgrade.py (+1/-1)
tests/helpers.py (+2/-2)
To merge this branch: bzr merge lp:~adam-collard/landscape-charm/install-sources-keys
Reviewer Review Type Date Requested Status
Simon Poirier 2019-01-17 Approve on 2019-01-17
Landscape Builder test results Approve on 2019-01-17
Review via email: mp+361892@code.launchpad.net

Commit message

Add support for the harmonised configuration keys for setting sources.list: install_sources, install_keys.

Description of the change

Integration tests pass modulo a test for update_security_db.sh cron job which we can look into separately.

To post a comment you must log in.
review: Abstain (executing tests)
review: Approve (test results)
394. By Adam Collard on 2019-01-17

Noop change

395. By Adam Collard on 2019-01-17

Merge from trunk

396. By Adam Collard on 2019-01-17

392 alive again

397. By Adam Collard on 2019-01-17

393 alive again

review: Abstain (executing tests)
review: Approve (test results)
review: Abstain (executing tests)
review: Approve (test results)
Simon Poirier (simpoir) wrote :

+1 LGTM.

Also bundletester is a pain to get working nowadays

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'Makefile'
--- Makefile 2019-01-17 14:18:28 +0000
+++ Makefile 2019-01-17 14:24:17 +0000
@@ -45,7 +45,7 @@
45 fi45 fi
4646
47integration-test: test-depends47integration-test: test-depends
48 ~/.local/bin/bundletester -v -l DEBUG --skip-implicit -t .48 python2 ~/.local/bin/bundletester -v -l DEBUG --skip-implicit -t .
4949
50# Run integration tests using the LDS package from the lds-trunk PPA50# Run integration tests using the LDS package from the lds-trunk PPA
51integration-test-trunk: secrets51integration-test-trunk: secrets
5252
=== modified file 'config.yaml'
--- config.yaml 2019-01-17 14:18:28 +0000
+++ config.yaml 2019-01-17 14:24:17 +0000
@@ -1,8 +1,29 @@
1options:1options:
2 install_sources:
3 default: "['ppa:landscape/18.03']"
4 type: string
5 description: >
6 List of extra apt sources, per charm-helpers standard format (a
7 yaml list of strings encoded as a string). Each source may be either a line
8 that can be added directly to sources.list(5), or in the form ppa:<user>/<ppa-name>
9 for adding Personal Package Archives, or a distribution component to enable.
10 install_keys:
11 default: ""
12 type: string
13 description: >
14 List of signing keys for install_sources package sources, per
15 charmhelpers standard format (a yaml list of strings encoded as a string).
16 The keys should be the full ASCII armoured GPG public keys. While GPG key
17 ids are also supported and looked up on a keyserver, operators should be aware
18 that this mechanism is insecure. null can be used if a standard package signing
19 key is used that will already be installed on the machine, and for PPA sources
20 where the package signing key is securely retrieved from Launchpad.\n"
2 source:21 source:
3 default: "ppa:landscape/18.03"22 default: ""
4 type: string23 type: string
5 description: |24 description: |
25 *DEPRECATED*. See install_sources
26
6 The source from which to install the landscape packages. Possible27 The source from which to install the landscape packages. Possible
7 values are:28 values are:
8 - A version number: "16.06"29 - A version number: "16.06"
@@ -16,6 +37,8 @@
16 default: ""37 default: ""
17 type: string38 type: string
18 description: |39 description: |
40 *DEPRECATED*. See install_keys
41
19 Key ID to import to the apt keyring to support use with arbitary source42 Key ID to import to the apt keyring to support use with arbitary source
20 configuration from outside of Launchpad archives or PPA's.43 configuration from outside of Launchpad archives or PPA's.
21 In case multiple repositories are specified in the 'source' config key,44 In case multiple repositories are specified in the 'source' config key,
2245
=== modified file 'dev/ubuntu-deps'
--- dev/ubuntu-deps 2019-01-17 14:18:28 +0000
+++ dev/ubuntu-deps 2019-01-17 14:24:17 +0000
@@ -14,7 +14,8 @@
14 python3-yaml curl python3-zope.testrunner \14 python3-yaml curl python3-zope.testrunner \
15 python3-amulet python3-fixtures python3-jinja2 python3-fixtures \15 python3-amulet python3-fixtures python3-jinja2 python3-fixtures \
16 python3-psutil python3-twisted python3-flake8 \16 python3-psutil python3-twisted python3-flake8 \
17 python3-path python3-libcharmstore python3-zope.testrunner python3-six17 python3-path python3-libcharmstore python3-zope.testrunner python3-six \
18 python-twisted-core
1819
19sudo apt-get -y install python-fixtures python-six python-yaml dpkg-dev \20sudo apt-get -y install python-fixtures python-six python-yaml dpkg-dev \
20 pbuilder dh-make python-jinja2 python-psutil python-six21 pbuilder dh-make python-jinja2 python-psutil python-six
2122
=== modified file 'lib/apt.py'
--- lib/apt.py 2019-01-17 14:18:28 +0000
+++ lib/apt.py 2019-01-17 14:24:17 +0000
@@ -53,13 +53,22 @@
5353
5454
55class AptSourceAndKeyDontMatchError(CharmError):55class AptSourceAndKeyDontMatchError(CharmError):
56 """Provided config values for 'source' and 'key' do not match.."""56 """Provided config values for 'source' and 'key' do not match."""
5757
58 def __init__(self):58 def __init__(self):
59 message = "The 'source' and 'key' lists have different lengths"59 message = "The 'source' and 'key' lists have different lengths"
60 super(AptSourceAndKeyDontMatchError, self).__init__(message)60 super(AptSourceAndKeyDontMatchError, self).__init__(message)
6161
6262
63class SourceConflictError(CharmError):
64 """Config values for 'source' and 'install_sources' are present."""
65
66 def __init__(self, install_sources, source):
67 message = ("install_sources: {!r} and source: {!r} are "
68 "mutually exclusive.").format(install_sources, source)
69 super(SourceConflictError, self).__init__(message)
70
71
63class Apt(object):72class Apt(object):
64 """Perform APT-related tasks as setting sources and installing packages.73 """Perform APT-related tasks as setting sources and installing packages.
6574
@@ -130,6 +139,14 @@
130 """Set the remote APT repository to use, if new or changed."""139 """Set the remote APT repository to use, if new or changed."""
131 config = self._hookenv.config()140 config = self._hookenv.config()
132 source = config.get("source")141 source = config.get("source")
142
143 # New style install sources
144 install_sources = config.get("install_sources")
145 if install_sources and source:
146 raise SourceConflictError(install_sources, source)
147 if install_sources:
148 self._fetch.configure_sources()
149 return
133 if not source:150 if not source:
134 raise AptNoSourceConfigError()151 raise AptNoSourceConfigError()
135152
136153
=== modified file 'lib/callbacks/tests/test_apt.py'
--- lib/callbacks/tests/test_apt.py 2019-01-17 14:18:28 +0000
+++ lib/callbacks/tests/test_apt.py 2019-01-17 14:24:17 +0000
@@ -7,7 +7,7 @@
77
8 def setUp(self):8 def setUp(self):
9 super(SetAPTSourcesTest, self).setUp()9 super(SetAPTSourcesTest, self).setUp()
10 self.fetch = FetchStub()10 self.fetch = FetchStub(self.hookenv.config)
11 self.subprocess = SubprocessStub()11 self.subprocess = SubprocessStub()
12 self.callback = SetAPTSources(12 self.callback = SetAPTSources(
13 hookenv=self.hookenv, fetch=self.fetch, subprocess=self.subprocess)13 hookenv=self.hookenv, fetch=self.fetch, subprocess=self.subprocess)
1414
=== modified file 'lib/tests/stubs.py'
--- lib/tests/stubs.py 2019-01-17 14:18:28 +0000
+++ lib/tests/stubs.py 2019-01-17 14:24:17 +0000
@@ -107,7 +107,9 @@
107class FetchStub(object):107class FetchStub(object):
108 """Provide a testable stub for C{charmhelpers.fetch}."""108 """Provide a testable stub for C{charmhelpers.fetch}."""
109109
110 def __init__(self):110 def __init__(self, config):
111 self._config = config
112 self.configured_sources = []
111 self.sources = []113 self.sources = []
112 self.updates = []114 self.updates = []
113 self.filtered = []115 self.filtered = []
@@ -126,6 +128,11 @@
126 def apt_install(self, packages, options=None, fatal=False):128 def apt_install(self, packages, options=None, fatal=False):
127 self.installed.append((packages, options, fatal))129 self.installed.append((packages, options, fatal))
128130
131 def configure_sources(self):
132 install_sources = self._config().get('install_sources')
133 install_keys = self._config().get('install_keys')
134 self.configured_sources.append((install_sources, install_keys))
135
129136
130class HostStub(object):137class HostStub(object):
131 """Testable stub for C{charmhelpers.core.host}."""138 """Testable stub for C{charmhelpers.core.host}."""
132139
=== modified file 'lib/tests/test_apt.py'
--- lib/tests/test_apt.py 2019-01-17 14:18:28 +0000
+++ lib/tests/test_apt.py 2019-01-17 14:24:17 +0000
@@ -5,8 +5,8 @@
55
6from lib.apt import (6from lib.apt import (
7 Apt, AptNoSourceConfigError, AptSourceAndKeyDontMatchError,7 Apt, AptNoSourceConfigError, AptSourceAndKeyDontMatchError,
8 INSTALL_PACKAGES, DEFAULT_INSTALL_OPTIONS, SAMPLE_HASHIDS_PPA,8 SourceConflictError, INSTALL_PACKAGES, DEFAULT_INSTALL_OPTIONS,
9 SAMPLE_HASHIDS_KEY)9 SAMPLE_HASHIDS_PPA, SAMPLE_HASHIDS_KEY)
10from lib.tests.stubs import FetchStub, SubprocessStub10from lib.tests.stubs import FetchStub, SubprocessStub
11from lib.tests.helpers import HookenvTest11from lib.tests.helpers import HookenvTest
12from lib.tests.rootdir import RootDir12from lib.tests.rootdir import RootDir
@@ -16,7 +16,9 @@
1616
17 def setUp(self):17 def setUp(self):
18 super(AptTest, self).setUp()18 super(AptTest, self).setUp()
19 self.fetch = FetchStub()19 # charmhelpers.fetch doesn't support dependency injection, so
20 # we pass in our config to be able to inspect what was used
21 self.fetch = FetchStub(self.hookenv.config)
20 self.subprocess = SubprocessStub()22 self.subprocess = SubprocessStub()
21 self.root_dir = self.useFixture(RootDir())23 self.root_dir = self.useFixture(RootDir())
22 self.paths = self.root_dir.paths24 self.paths = self.root_dir.paths
@@ -64,6 +66,8 @@
64 self.apt.set_sources()66 self.apt.set_sources()
65 self.assertEqual([("ppa:landscape/14.10", None)], self.fetch.sources)67 self.assertEqual([("ppa:landscape/14.10", None)], self.fetch.sources)
66 self.assertEqual([True], self.fetch.updates)68 self.assertEqual([True], self.fetch.updates)
69 # With source set, fetch.configure_sources is not called
70 self.assertEqual(self.fetch.configured_sources, [])
6771
68 def test_set_shorthand_sources(self):72 def test_set_shorthand_sources(self):
69 """73 """
@@ -154,9 +158,33 @@
154 self.hookenv.config()["key"] = "xyz"158 self.hookenv.config()["key"] = "xyz"
155 with self.assertRaises(AptSourceAndKeyDontMatchError) as error:159 with self.assertRaises(AptSourceAndKeyDontMatchError) as error:
156 self.apt.set_sources()160 self.apt.set_sources()
157 self.assertEqual(161 self.assertEqual(
158 "The 'source' and 'key' lists have different lengths",162 "The 'source' and 'key' lists have different lengths",
159 str(error))163 str(error.exception))
164
165 def test_set_sources_and_install_sources(self):
166 """
167 If sources and install_sources are set set_sources complains.
168 """
169 self.hookenv.config()["source"] = "ppa:landscape/14.10"
170 self.hookenv.config()["install_sources"] = "anything"
171 with self.assertRaises(SourceConflictError) as error:
172 self.apt.set_sources()
173 self.assertEqual(
174 "install_sources: 'anything' and source: 'ppa:landscape/14.10' "
175 "are mutually exclusive.",
176 str(error.exception))
177
178 def test_install_sources(self):
179 """
180 If install_sources is set, it's passed to fetch.configure_sources.
181 """
182 self.hookenv.config()["install_sources"] = "ppa:landscape/19.01"
183 self.hookenv.config()["install_keys"] = "DEADBEEF"
184 self.apt.set_sources()
185 self.assertEqual(
186 self.fetch.configured_sources,
187 [('ppa:landscape/19.01', 'DEADBEEF')])
160188
161 def test_local_tarball(self):189 def test_local_tarball(self):
162 """190 """
163191
=== modified file 'lib/tests/test_install.py'
--- lib/tests/test_install.py 2019-01-17 14:18:28 +0000
+++ lib/tests/test_install.py 2019-01-17 14:24:17 +0000
@@ -9,7 +9,7 @@
99
10 def setUp(self):10 def setUp(self):
11 super(InstallHookTest, self).setUp()11 super(InstallHookTest, self).setUp()
12 self.fetch = FetchStub()12 self.fetch = FetchStub(self.hookenv.config)
13 self.subprocess = SubprocessStub()13 self.subprocess = SubprocessStub()
14 self.subprocess.add_fake_executable("apt-mark")14 self.subprocess.add_fake_executable("apt-mark")
15 self.hook = InstallHook(15 self.hook = InstallHook(
1616
=== modified file 'lib/tests/test_services.py'
--- lib/tests/test_services.py 2019-01-17 14:18:28 +0000
+++ lib/tests/test_services.py 2019-01-17 14:24:17 +0000
@@ -31,7 +31,7 @@
31 self.root_dir = self.useFixture(RootDir())31 self.root_dir = self.useFixture(RootDir())
32 self.paths = self.root_dir.paths32 self.paths = self.root_dir.paths
33 self.root_dir = self.useFixture(RootDir())33 self.root_dir = self.useFixture(RootDir())
34 self.fetch = FetchStub()34 self.fetch = FetchStub(self.hookenv.config)
35 self.psutil = PsutilStub(num_cpus=2, physical_memory=1*1024**3)35 self.psutil = PsutilStub(num_cpus=2, physical_memory=1*1024**3)
36 self.hook = ServicesHook(36 self.hook = ServicesHook(
37 hookenv=self.hookenv, host=self.host, subprocess=self.subprocess,37 hookenv=self.hookenv, host=self.host, subprocess=self.subprocess,
3838
=== modified file 'lib/tests/test_upgrade.py'
--- lib/tests/test_upgrade.py 2019-01-17 14:18:28 +0000
+++ lib/tests/test_upgrade.py 2019-01-17 14:24:17 +0000
@@ -11,7 +11,7 @@
1111
12 def setUp(self):12 def setUp(self):
13 super(UpgradeActionTest, self).setUp()13 super(UpgradeActionTest, self).setUp()
14 self.fetch = FetchStub()14 self.fetch = FetchStub(self.hookenv.config)
15 self.subprocess = SubprocessStub()15 self.subprocess = SubprocessStub()
16 self.subprocess.add_fake_executable("apt-mark")16 self.subprocess.add_fake_executable("apt-mark")
17 self.root_dir = self.useFixture(RootDir())17 self.root_dir = self.useFixture(RootDir())
1818
=== modified file 'tests/helpers.py'
--- tests/helpers.py 2019-01-17 14:18:28 +0000
+++ tests/helpers.py 2019-01-17 14:24:17 +0000
@@ -501,8 +501,8 @@
501 extra_config = yaml.safe_load(fd.read())501 extra_config = yaml.safe_load(fd.read())
502 else:502 else:
503 extra_config = {503 extra_config = {
504 "source": source,504 "install_sources": source,
505 "key": os.environ.get("LS_CHARM_KEY", "4652B4E6")505 "install_keys": os.environ.get("LS_CHARM_KEY", "4652B4E6")
506 }506 }
507 context["landscape"].update(extra_config)507 context["landscape"].update(extra_config)
508508

Subscribers

People subscribed via source and target branches