Merge ~kissiel/checkbox-ng:config-refactor-proposable into checkbox-ng:master

Proposed by Maciej Kisielewski
Status: Superseded
Proposed branch: ~kissiel/checkbox-ng:config-refactor-proposable
Merge into: checkbox-ng:master
Diff against target: 2638 lines (+761/-407)
16 files modified
checkbox_ng/config.py (+45/-38)
checkbox_ng/launcher/check_config.py (+27/-30)
checkbox_ng/launcher/checkbox_cli.py (+0/-2)
checkbox_ng/launcher/master.py (+15/-15)
checkbox_ng/launcher/stages.py (+88/-69)
checkbox_ng/launcher/subcommands.py (+41/-53)
dev/null (+0/-136)
plainbox/impl/applogic.py (+0/-18)
plainbox/impl/commands/__init__.py (+0/-11)
plainbox/impl/config.py (+383/-0)
plainbox/impl/ctrl.py (+0/-1)
plainbox/impl/runner.py (+0/-1)
plainbox/impl/session/assistant.py (+7/-12)
plainbox/impl/session/remote_assistant.py (+24/-19)
plainbox/impl/test_config.py (+130/-0)
plainbox/impl/unit/unit.py (+1/-2)
Reviewer Review Type Date Requested Status
Jonathan Cave (community) Approve
Devices Certification Bot Needs Fixing
Sylvain Pineau (community) Needs Fixing
Review via email: mp+407017@code.launchpad.net

This proposal has been superseded by a proposal from 2022-02-10.

Description of the change

This is a major refactor of configs handling in Checkbox.

To post a comment you must log in.
Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote (last edit ):

Too avoid confusion, what do you think about keeping only one dir to store the config file as opposed to:

search_dirs = [
+ '$SNAP_DATA',
+ '/etc/xdg/',
+ '~/.config/',
+ ]

/var/tmp/checkbox-ng/ already keeps the manifest.json and the sessions data.

AFAIK it would require packaging updates for:
1. https://git.launchpad.net/~checkbox-dev/plainbox-provider-certification-server/+git/packaging/tree/debian/plainbox-provider-certification-server.install#n1
2. https://git.launchpad.net/~oem-qa/oem-qa-checkbox/+git/oem-plainbox/tree/debian/plainbox-provider-oem-somerville.install#n1

Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote :

Unless someone frequently installs multiple checkbox-project snaps on a system I see no reason to keep $SNAP_DATA/checkbox.conf but that implies a rework of https://git.launchpad.net/checkbox-support/tree/checkbox_support/snap_utils/config.py#n57

Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote :

Is it doable to get rid of plainbox/impl/secure/config.py completely too?

review: Needs Information
Revision history for this message
Maciej Kisielewski (kissiel) wrote :

So the real shortstat of this branch is now:
 19 files changed, 732 insertions(+), 1263 deletions(-)

> Is it doable to get rid of plainbox/impl/secure/config.py completely too?
I did what I can to remove dependencies from that module in this refactor. But to completely remove it we would need to refactor provider_manager that heavily depends on it.

Revision history for this message
Jonathan Cave (jocave) wrote :

Fix needed below

review: Needs Fixing
Revision history for this message
Jonathan Cave (jocave) wrote :

From Sylvain:
> Too avoid confusion, what do you think about keeping only one dir to store the config file

I see this as a useful long term goal, but not something that should be a requirement for this refactor

Revision history for this message
Maciej Kisielewski (kissiel) wrote :

Added support for auxiliary configs.

Revision history for this message
Sylvain Pineau (sylvain-pineau) wrote :

Conflicts (at least in launcher/stages.py)

review: Needs Fixing
Revision history for this message
Jonathan Cave (jocave) wrote :

Thanks for restoring the conf file location functionality. The recursive function looks like it should work to me, although I'm somewhat worried someone might actually use it at some point!

+1, assuming the requested rebase is completed

review: Approve
Revision history for this message
Devices Certification Bot (ce-certification-qa) wrote :

The merge was fine but running tests failed.

"10.38.105.54"
"10.38.105.108"
"10.38.105.197"
[xenial] [13:15:40] starting container
[bionic] [13:15:41] starting container
[focal] [13:15:47] starting container
Device project added to xenial-testing
Device project added to bionic-testing
Device project added to focal-testing
"10.38.105.54"
[xenial] [13:15:58] provisioning container
"10.38.105.124"
[bionic] [13:16:01] provisioning container
"10.38.105.62"
[focal] [13:16:10] provisioning container
[xenial] [13:22:55] Starting tests...
[xenial] Found a test script: ./requirements/001-container-tests-plainbox-egg-info
[xenial] [13:22:55] 001-container-tests-plainbox-egg-info: PASS
[xenial] Found a test script: ./requirements/container-tests-checkbox-documentation
[xenial] [13:22:59] container-tests-checkbox-documentation: PASS
[xenial] Found a test script: ./requirements/container-tests-checkbox-ng-unit
[xenial] [13:23:03] container-tests-checkbox-ng-unit: FAIL
[xenial] output: https://paste.ubuntu.com/p/X5cGV8pHGQ/
[xenial] Found a test script: ./requirements/container-tests-providers-internal
[xenial] [13:23:08] container-tests-providers-internal: PASS
[xenial] [13:23:08] Fixing file permissions in source directory
[xenial] [13:23:08] Destroying container
[bionic] [13:23:24] Starting tests...
[bionic] Found a test script: ./requirements/001-container-tests-plainbox-egg-info
[bionic] [13:23:25] 001-container-tests-plainbox-egg-info: PASS
[bionic] Found a test script: ./requirements/container-tests-checkbox-documentation
[bionic] [13:23:30] container-tests-checkbox-documentation: PASS
[bionic] Found a test script: ./requirements/container-tests-checkbox-ng-unit
[bionic] [13:23:34] container-tests-checkbox-ng-unit: FAIL
[bionic] output: https://paste.ubuntu.com/p/sYDF6vSRdY/
[bionic] Found a test script: ./requirements/container-tests-providers-internal
[bionic] [13:23:39] container-tests-providers-internal: PASS
[bionic] [13:23:39] Fixing file permissions in source directory
[bionic] [13:23:40] Destroying container
[focal] [13:23:55] Starting tests...
[focal] Found a test script: ./requirements/001-container-tests-plainbox-egg-info
[focal] [13:23:56] 001-container-tests-plainbox-egg-info: PASS
[focal] Found a test script: ./requirements/container-tests-checkbox-documentation
[focal] [13:23:59] container-tests-checkbox-documentation: PASS
[focal] Found a test script: ./requirements/container-tests-checkbox-ng-unit
[focal] [13:24:04] container-tests-checkbox-ng-unit: FAIL
[focal] output: https://paste.ubuntu.com/p/YRchj97r34/
[focal] Found a test script: ./requirements/container-tests-providers-internal
[focal] [13:24:07] container-tests-providers-internal: PASS
[focal] [13:24:07] Fixing file permissions in source directory
[focal] [13:24:08] Destroying container

review: Needs Fixing
5a15e96... by Maciej Kisielewski

Remove: old plainbox API bits

Revision history for this message
Maciej Kisielewski (kissiel) wrote :

This is freshly rebased on top of all the removals that happened last year. Additionally it has one more small removal from old plainbox API. It passes all the tests. Including the problematic bionic and xenial ones.

Revision history for this message
Jonathan Cave (jocave) wrote :

I tested this out and found with a controller side running this code and the SUT side running current stable code.

Launcher file: https://pastebin.canonical.com/p/2BSjVz5nS5/

I found that the session got stuck in an endless auto-retry loop (note that auto-retry is "no" in the launcher). Switch the controller back to master resulted in expected no auto-retry behaviour.

review: Needs Fixing
Revision history for this message
Maciej Kisielewski (kissiel) wrote :

Those changes remote an RemoteAPI bump, so let's land this targeting RAPI-bump branch (stuff that should be on master, but are grouped together so only one bump is made).

Revision history for this message
Jonathan Cave (jocave) wrote :

Ok, sounds good to me. A coordinated landing would be good so that we can get the controller side containers rebuilt and try things out on edge

review: Approve

Unmerged commits

6a7c860... by Maciej Kisielewski

Add: support auxiliary config filenames

1e479bf... by Maciej Kisielewski

Change: refactor configs, add origins for values

This patch changes how configs (and launchers) are stored inside checkbox.
Apart from removing quite a lot of code, it also makes checkbox record the
place where particular config variable came from.

5a15e96... by Maciej Kisielewski

Remove: old plainbox API bits

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/checkbox_ng/config.py b/checkbox_ng/config.py
2index 5a4ba11..a8e8ad0 100644
3--- a/checkbox_ng/config.py
4+++ b/checkbox_ng/config.py
5@@ -23,55 +23,62 @@
6 =====================================================
7 """
8 import gettext
9-import itertools
10 import logging
11 import os
12
13-from plainbox.impl.launcher import DefaultLauncherDefinition
14-from plainbox.impl.launcher import LauncherDefinition
15+from plainbox.impl.config import Configuration
16
17
18 _ = gettext.gettext
19
20 _logger = logging.getLogger("config")
21
22+
23+SEARCH_DIRS = [
24+ '$SNAP_DATA',
25+ '/etc/xdg/',
26+ '~/.config/',
27+ ]
28+
29+
30 def expand_all(path):
31+ """Expand both: envvars and ~ in `path`."""
32 return os.path.expandvars(os.path.expanduser(path))
33
34-def load_configs(launcher_file=None):
35- # launcher can override the default name of config files to look for
36- # so first we need to establish the filename to look for
37- configs = []
38- config_filename = 'checkbox.conf'
39- launcher = DefaultLauncherDefinition()
40- if launcher_file:
41- configs.append(launcher_file)
42- generic_launcher = LauncherDefinition()
43- if not os.path.exists(launcher_file):
44- _logger.error(_(
45- "Unable to load launcher '%s'. File not found!"),
46- launcher_file)
47- raise SystemExit(1)
48- generic_launcher.read(launcher_file)
49- config_filename = os.path.expandvars(os.path.expanduser(
50- generic_launcher.config_filename))
51- launcher = generic_launcher.get_concrete_launcher()
52- if os.path.isabs(config_filename):
53- configs.append(config_filename)
54+
55+def load_configs(launcher_file=None, cfg=None):
56+ """
57+ Read a chain of configs/launchers.
58+
59+ In theory there can be a very long list of configs that are linked by
60+ specifying config_filename in each. Each time this happen we need to
61+ consider the new one and override all the values contained therein.
62+ And after this chain is exhausted the values in the launcher should
63+ take precedence over the previously read.
64+ Warning: some recursion ahead.
65+ """
66+ if not cfg:
67+ cfg = Configuration()
68+ previous_cfg_name = cfg.get_value('config', 'config_filename')
69+ if os.path.isabs(expand_all(previous_cfg_name)):
70+ cfg.update_from_another(
71+ Configuration.from_path(expand_all(previous_cfg_name)),
72+ 'config file: {}'.format(previous_cfg_name))
73 else:
74- search_dirs = [
75- '$SNAP_DATA',
76- '/etc/xdg/',
77- '~/.config/',
78- ]
79- for d in search_dirs:
80- config = expand_all(os.path.join(d, config_filename))
81+ for sdir in SEARCH_DIRS:
82+ config = expand_all(os.path.join(sdir, previous_cfg_name))
83 if os.path.exists(config):
84- configs.append(config)
85- launcher.read(configs)
86- if launcher.problem_list:
87- _logger.error(_("Unable to start launcher because of errors:"))
88- for problem in launcher.problem_list:
89- _logger.error("%s", str(problem))
90- raise SystemExit(1)
91- return launcher
92+ cfg.update_from_another(
93+ Configuration.from_path(config),
94+ 'config file: {}'.format(config))
95+ else:
96+ _logger.info(
97+ "Referenced config file doesn't exist: %s", config)
98+ new_cfg_filename = cfg.get_value('config', 'config_filename')
99+ if new_cfg_filename != previous_cfg_name:
100+ load_configs(launcher_file, cfg)
101+ if launcher_file:
102+ cfg.update_from_another(
103+ Configuration.from_path(launcher_file),
104+ 'Launcher file: {}'.format(launcher_file))
105+ return cfg
106diff --git a/checkbox_ng/launcher/check_config.py b/checkbox_ng/launcher/check_config.py
107index 359d3ca..0ac3088 100644
108--- a/checkbox_ng/launcher/check_config.py
109+++ b/checkbox_ng/launcher/check_config.py
110@@ -1,6 +1,6 @@
111 # This file is part of Checkbox.
112 #
113-# Copyright 2018-2019 Canonical Ltd.
114+# Copyright 2018-2021 Canonical Ltd.
115 # Written by:
116 # Maciej Kisielewski <maciej.kisielewski@canonical.com>
117 #
118@@ -15,41 +15,38 @@
119 #
120 # You should have received a copy of the GNU General Public License
121 # along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
122-from plainbox.impl.secure.config import ValidationError
123-from plainbox.i18n import gettext as _
124+"""This module contains the implementation of the `check-config` subcmd."""
125
126 from checkbox_ng.config import load_configs
127
128
129 class CheckConfig():
130- def invoked(self, ctx):
131+ """Implementation of the `check-config` sub-command."""
132+ @staticmethod
133+ def invoked(_):
134+ """Function that's run with `check-config` invocation."""
135 config = load_configs()
136- print(_("Configuration files:"))
137- for filename in config.filename_list:
138- print(" - {}".format(filename))
139- for variable in config.Meta.variable_list:
140- print(" [{0}]".format(variable.section))
141- print(" {0}={1}".format(
142- variable.name,
143- variable.__get__(config, config.__class__)))
144- for section in config.Meta.section_list:
145- print(" [{0}]".format(section.name))
146- section_value = section.__get__(config, config.__class__)
147- if section_value:
148- for key, value in sorted(section_value.items()):
149- print(" {0}={1}".format(key, value))
150- if config.problem_list:
151- print(_("Problems:"))
152- for problem in config.problem_list:
153- if isinstance(problem, ValidationError):
154- print(_(" - variable {0}: {1}").format(
155- problem.variable.name, problem.message))
156- else:
157- print(" - {0}".format(problem.message))
158- return 1
159- else:
160- print(_("No validation problems found"))
161+ print("Configuration files:")
162+ for source in config.sources:
163+ print(" - {}".format(source))
164+ for sect_name, section in config.sections.items():
165+ print(" [{0}]".format(sect_name))
166+ for var_name in section.keys():
167+ value = config.get_value(sect_name, var_name)
168+ if isinstance(value, list):
169+ value = ', '.join(value)
170+ origin = config.get_origin(sect_name, var_name)
171+ origin = "From {}".format(origin) if origin else "(Default)"
172+ key_val = "{}={}".format(var_name, value)
173+ print(" {0: <34} {1}".format(key_val, origin))
174+ problems = config.get_problems()
175+ if not problems:
176+ print("No problems with config(s) found!")
177 return 0
178+ print('Problems:')
179+ for problem in problems:
180+ print('- ', problem)
181+ return 1
182
183 def register_arguments(self, parser):
184- pass
185+ """Register extra args for this subcmd. No extra args ATM."""
186diff --git a/checkbox_ng/launcher/checkbox_cli.py b/checkbox_ng/launcher/checkbox_cli.py
187index b69da17..9740eb8 100644
188--- a/checkbox_ng/launcher/checkbox_cli.py
189+++ b/checkbox_ng/launcher/checkbox_cli.py
190@@ -27,8 +27,6 @@ import subprocess
191 import sys
192
193 from plainbox.impl.jobcache import ResourceJobCache
194-from plainbox.impl.launcher import DefaultLauncherDefinition
195-from plainbox.impl.launcher import LauncherDefinition
196 from plainbox.impl.session.assistant import SessionAssistant
197
198 from checkbox_ng.config import load_configs
199diff --git a/checkbox_ng/launcher/master.py b/checkbox_ng/launcher/master.py
200index 730fc02..2db2f7e 100644
201--- a/checkbox_ng/launcher/master.py
202+++ b/checkbox_ng/launcher/master.py
203@@ -37,8 +37,7 @@ from functools import partial
204 from tempfile import SpooledTemporaryFile
205
206 from plainbox.impl.color import Colorizer
207-from plainbox.impl.launcher import DefaultLauncherDefinition
208-from plainbox.impl.secure.config import Unset
209+from plainbox.impl.config import Configuration
210 from plainbox.impl.session.remote_assistant import RemoteSessionAssistant
211 from plainbox.vendor import rpyc
212 from checkbox_ng.urwid_ui import TestPlanBrowser
213@@ -111,7 +110,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
214
215 @property
216 def is_interactive(self):
217- return (self.launcher.ui_type == 'interactive' and
218+ return (self.launcher.get_value('ui', 'type') == 'interactive' and
219 sys.stdin.isatty() and sys.stdout.isatty())
220
221 @property
222@@ -129,7 +128,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
223 self._is_bootstrapping = False
224 self._target_host = ctx.args.host
225 self._normal_user = ''
226- self.launcher = DefaultLauncherDefinition()
227+ self.launcher = Configuration()
228 if ctx.args.launcher:
229 expanded_path = os.path.expanduser(ctx.args.launcher)
230 if not os.path.exists(expanded_path):
231@@ -137,7 +136,8 @@ class RemoteMaster(ReportsStage, MainLoopStage):
232 expanded_path))
233 with open(expanded_path, 'rt') as f:
234 self._launcher_text = f.read()
235- self.launcher.read_string(self._launcher_text)
236+ self.launcher = Configuration.from_text(
237+ self._launcher_text, 'Remote:{}'.format(expanded_path))
238 if ctx.args.user:
239 self._normal_user = ctx.args.user
240 timeout = 600
241@@ -267,8 +267,8 @@ class RemoteMaster(ReportsStage, MainLoopStage):
242 tps = self.sa.start_session(configuration)
243 except RuntimeError as exc:
244 raise SystemExit(exc.args[0]) from exc
245- if self.launcher.test_plan_forced:
246- self.select_tp(self.launcher.test_plan_default_selection)
247+ if self.launcher.get_value('test plan', 'forced'):
248+ self.select_tp(self.launcher.get_value('test plan', 'unit'))
249 self.select_jobs(self.jobs)
250 else:
251 self.interactively_choose_tp(tps)
252@@ -311,8 +311,8 @@ class RemoteMaster(ReportsStage, MainLoopStage):
253 return val.lower() in ('y', 'yes', 't', 'true', 'on', '1')
254
255 def select_jobs(self, all_jobs):
256- if self.launcher.test_selection_forced:
257- if self.launcher.manifest is not Unset:
258+ if self.launcher.get_value('test selection', 'forced'):
259+ if self.launcher.manifest:
260 self.sa.save_manifest(
261 {manifest_id:
262 self._strtobool(self.launcher.manifest[manifest_id]) for
263@@ -352,7 +352,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
264 Returns True if the remote should keep running.
265 And False if it should quit.
266 """
267- if self.launcher.ui_type == 'silent':
268+ if self.launcher.get_value('ui', 'type'):
269 self._sa.terminate()
270 return False
271 response = interrupt_dialog(self._target_host)
272@@ -372,7 +372,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
273
274 def finish_session(self):
275 print(self.C.header("Results"))
276- if self.launcher.local_submission:
277+ if self.launcher.get_value('launcher', 'local_submission'):
278 # Disable SIGINT while we save local results
279 with contextlib.ExitStack() as stack:
280 tmp_sig = signal.signal(signal.SIGINT, signal.SIG_IGN)
281@@ -390,7 +390,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
282 self.run_jobs()
283
284 def _handle_last_job_after_resume(self, resumed_session_info):
285- if self.launcher.ui_type == 'silent':
286+ if self.launcher.get_value('ui', 'type') == 'silent':
287 time.sleep(20)
288 else:
289 resume_dialog(10)
290@@ -420,11 +420,11 @@ class RemoteMaster(ReportsStage, MainLoopStage):
291 self._run_jobs(jobs_repr, total_num)
292 rerun_candidates = self.sa.get_rerun_candidates('manual')
293 if rerun_candidates:
294- if self.launcher.ui_type == 'interactive':
295+ if self.launcher.get_value('ui', 'type') == 'interactive':
296 while True:
297 if not self._maybe_manual_rerun_jobs():
298 break
299- if self.launcher.auto_retry:
300+ if self.launcher.get_value('ui', 'auto_retry'):
301 while True:
302 if not self._maybe_auto_rerun_jobs():
303 break
304@@ -510,7 +510,7 @@ class RemoteMaster(ReportsStage, MainLoopStage):
305 if not rerun_candidates:
306 return False
307 # we wait before retrying
308- delay = self.launcher.delay_before_retry
309+ delay = self.launcher.get_value('ui', 'delay_before_retry')
310 _logger.info(_("Waiting {} seconds before retrying failed"
311 " jobs...".format(delay)))
312 time.sleep(delay)
313diff --git a/checkbox_ng/launcher/stages.py b/checkbox_ng/launcher/stages.py
314index 84bdfe1..7c44561 100644
315--- a/checkbox_ng/launcher/stages.py
316+++ b/checkbox_ng/launcher/stages.py
317@@ -26,9 +26,11 @@ import json
318 import logging
319 import os
320 import time
321+import textwrap
322
323 from plainbox.abc import IJobResult
324 from plainbox.i18n import pgettext as C_
325+from plainbox.impl.config import Configuration
326 from plainbox.impl.result import JobResultBuilder
327 from plainbox.impl.result import tr_outcome
328 from plainbox.impl.transport import InvalidSecureIDError
329@@ -316,42 +318,44 @@ class ReportsStage(CheckboxUiStage):
330 self._export_fn = export_fn
331
332 def _prepare_stock_report(self, report):
333- # this is purposefully not using pythonic dict-keying for better
334- # readability
335- if not self.sa.config.transports:
336- self.sa.config.transports = dict()
337- if not self.sa.config.exporters:
338- self.sa.config.exporters = dict()
339- if not self.sa.config.reports:
340- self.sa.config.reports = dict()
341+
342+ new_origin = 'stock_reports'
343 if report == 'text':
344- self.sa.config.exporters['text'] = {
345- 'unit': 'com.canonical.plainbox::text'}
346- self.sa.config.transports['stdout'] = {
347- 'type': 'stream', 'stream': 'stdout'}
348- # '1_' prefix ensures ordering amongst other stock reports. This
349- # report name does not appear anywhere (because of forced: yes)
350- self.sa.config.reports['1_text_to_screen'] = {
351- 'transport': 'stdout', 'exporter': 'text', 'forced': 'yes'}
352+ additional_config = Configuration.from_text(textwrap.dedent("""
353+ [exporter:text]
354+ unit = com.canonical.plainbox::text
355+ [transport:stdout]
356+ stream = stdout
357+ type = stream
358+ [report:1_text_to_screen]
359+ exporter = text
360+ forced = yes
361+ transport = stdout
362+ """), new_origin)
363+ self.sa.config.update_from_another(additional_config, new_origin)
364 elif report == 'certification':
365- self.sa.config.exporters['tar'] = {
366- 'unit': 'com.canonical.plainbox::tar'}
367- self.sa.config.transports['c3'] = {
368- 'type': 'submission-service',
369- 'secure_id': self.sa.config.transports.get('c3', {}).get(
370- 'secure_id', None)}
371- self.sa.config.reports['upload to certification'] = {
372- 'transport': 'c3', 'exporter': 'tar'}
373+ additional_config = Configuration.from_text(textwrap.dedent("""
374+ [exporter:tar]
375+ unit = com.canonical.plainbox::tar
376+ [transport:c3]
377+ type = submission-service
378+ [report:upload to certification]
379+ exporter = tar
380+ transport = c3
381+ """), new_origin)
382+ self.sa.config.update_from_another(additional_config, new_origin)
383 elif report == 'certification-staging':
384- self.sa.config.exporters['tar'] = {
385- 'unit': 'com.canonical.plainbox::tar'}
386- self.sa.config.transports['c3-staging'] = {
387- 'type': 'submission-service',
388- 'secure_id': self.sa.config.transports.get('c3', {}).get(
389- 'secure_id', None),
390- 'staging': 'yes'}
391- self.sa.config.reports['upload to certification-staging'] = {
392- 'transport': 'c3-staging', 'exporter': 'tar'}
393+ additional_config = Configuration.from_text(textwrap.dedent("""
394+ [exporter:tar]
395+ unit = com.canonical.plainbox::tar
396+ [transport:c3]
397+ staging = yes
398+ type = submission-service
399+ [report:upload to certification-staging]
400+ exporter = tar
401+ transport = c3
402+ """), new_origin)
403+ self.sa.config.update_from_another(additional_config, new_origin)
404 elif report == 'submission_files':
405 # LP:1585326 maintain isoformat but removing ':' chars that cause
406 # issues when copying files.
407@@ -364,21 +368,21 @@ class ReportsStage(CheckboxUiStage):
408 ('tar', '.tar.xz')]:
409 path = os.path.join(self.base_dir, ''.join(
410 ['submission_', timestamp, file_ext]))
411- self.sa.config.transports['{}_file'.format(exporter)] = {
412- 'type': 'file',
413- 'path': path}
414- if exporter not in self.sa.config.exporters:
415- self.sa.config.exporters[exporter] = {
416- 'unit': 'com.canonical.plainbox::{}'.format(
417- exporter)}
418- if not self.sa.config.exporters[exporter].get('unit'):
419- unit = 'com.canonical.plainbox::{}'.format(exporter)
420- self.sa.config.exporters[exporter]['unit'] = unit
421- self.sa.config.reports['2_{}_file'.format(exporter)] = {
422- 'transport': '{}_file'.format(exporter),
423- 'exporter': '{}'.format(exporter),
424- 'forced': 'yes'
425- }
426+ template = textwrap.dedent("""
427+ [transport:{exporter}_file]
428+ path = {path}
429+ type = file
430+ [exporter:{exporter}]
431+ unit = com.canonical.plainbox::{exporter}
432+ [report:2_{exporter}_file]
433+ exporter = {exporter}
434+ forced = yes
435+ transport = {exporter}_file
436+ """)
437+ additional_config = Configuration.from_text(
438+ template.format(exporter=exporter, path=path), new_origin)
439+ self.sa.config.update_from_another(
440+ additional_config, new_origin)
441
442 def _prepare_transports(self):
443 self.base_dir = os.path.join(
444@@ -394,7 +398,9 @@ class ReportsStage(CheckboxUiStage):
445 # depending on the type of transport we need to pick variable that
446 # serves as the 'where' param for the transport. In case of
447 # certification site the URL is supplied here
448- tr_type = self.sa.config.transports[transport]['type']
449+ transport_cfg = self.sa.config.get_parametric_sections(
450+ 'transport')[transport]
451+ tr_type = transport_cfg['type']
452 if tr_type not in self._available_transports:
453 _logger.error(_("Unrecognized type '%s' of transport '%s'"),
454 tr_type, transport)
455@@ -402,14 +408,11 @@ class ReportsStage(CheckboxUiStage):
456 cls = self._available_transports[tr_type]
457 if tr_type == 'file':
458 self.transports[transport] = cls(
459- os.path.expanduser(
460- self.sa.config.transports[transport]['path']))
461+ os.path.expanduser(transport_cfg['path']))
462 elif tr_type == 'stream':
463- self.transports[transport] = cls(
464- self.sa.config.transports[transport]['stream'])
465+ self.transports[transport] = cls(transport_cfg['stream'])
466 elif tr_type == 'submission-service':
467- secure_id = self.sa.config.transports[transport].get(
468- 'secure_id', None)
469+ secure_id = transport_cfg.get('secure_id', None)
470 if self.is_interactive:
471 new_description = input(self.C.BLUE(_(
472 'Enter submission description (press Enter to skip): ')))
473@@ -425,7 +428,7 @@ class ReportsStage(CheckboxUiStage):
474 options = "secure_id={}".format(secure_id)
475 else:
476 options = ""
477- if self.sa.config.transports[transport].get('staging', False):
478+ if transport_cfg.get('staging', False):
479 url = ('https://certification.staging.canonical.com/'
480 'api/v1/submission/{}/'.format(secure_id))
481 elif os.getenv('C3_URL'):
482@@ -437,14 +440,15 @@ class ReportsStage(CheckboxUiStage):
483 self.transports[transport] = cls(url, options)
484
485 def _export_results(self):
486- if 'none' not in self.sa.config.stock_reports:
487- for report in self.sa.config.stock_reports:
488+ stock_reports = self.sa.config.get_value('launcher', 'stock_reports')
489+ if 'none' not in stock_reports:
490+ for report in stock_reports:
491 if report in ['certification', 'certification-staging']:
492 # skip stock c3 report if secure_id is not given from
493 # config files or launchers, and the UI is non-interactive
494 # (silent)
495- if ('c3' not in self.sa.config.transports and
496- not self.is_interactive):
497+ if ('transport:c3' not in self.sa.config.sections.keys()
498+ and not self.is_interactive):
499 continue
500 # don't generate stock c3 reports if sideloaded providers
501 # were in use, something that should only be done during
502@@ -457,7 +461,9 @@ class ReportsStage(CheckboxUiStage):
503 # reports are stored in an ordinary dict(), so sorting them ensures
504 # the same order of submitting them between runs, and if they
505 # share common prefix, they are next to each other
506- for name, params in sorted(self.sa.config.reports.items()):
507+ for name, params in sorted(
508+ self.sa.config.get_parametric_sections('report').items()):
509+
510 # don't generate stock c3 reports if sideloaded providers
511 # were in use, something that should only be done during
512 # development
513@@ -476,8 +482,10 @@ class ReportsStage(CheckboxUiStage):
514 cmd = 'y'
515 if cmd == 'n':
516 continue
517- exporter_id = self.sa.config.exporters[params['exporter']]['unit']
518- exporter_options = self.sa.config.exporters[
519+ all_exporters = self.sa.config.get_parametric_sections('exporter')
520+ exporter_id = self.sa.config.get_parametric_sections('exporter')[
521+ params['exporter']]['unit']
522+ exp_options = self.sa.config.get_parametric_sections('exporter')[
523 params['exporter']].get('options', '').split()
524 done_sending = False
525 while not done_sending:
526@@ -489,7 +497,7 @@ class ReportsStage(CheckboxUiStage):
527 else:
528 try:
529 result = self.sa.export_to_transport(
530- exporter_id, transport, exporter_options)
531+ exporter_id, transport, exp_options)
532 except ExporterError as exc:
533 _logger.warning(
534 _("Problem occured when preparing %s report:"
535@@ -512,14 +520,14 @@ class ReportsStage(CheckboxUiStage):
536 except InvalidSecureIDError:
537 _logger.warning(_("Invalid secure_id"))
538 if self._retry_dialog():
539- self.sa.config.transports['c3'].pop('secure_id')
540+ self.sa.config.sections['transports']['c3'].pop(
541+ 'secure_id')
542 continue
543- except Exception:
544+ except Exception as exc:
545 _logger.error(
546 _("Problem with a '%s' report using '%s' exporter "
547- "sent to '%s' transport."),
548- name, exporter_id, transport.url)
549- self._reset_auto_submission_retries()
550+ "sent to '%s' transport. Reason %s"),
551+ name, exporter_id, transport.url, exc)
552 done_sending = True
553
554 def _retry_dialog(self):
555@@ -541,3 +549,14 @@ class ReportsStage(CheckboxUiStage):
556 return True
557
558 return False
559+
560+template = textwrap.dedent("""
561+ [transport:{exporter}_file]
562+ type = file
563+ path = {path}
564+ [exporter:{exporter}]
565+ unit = com.canonical.plainbox::{exporter}
566+ [report:2_{exporter}_file]
567+ transport = {exporter}_file
568+ exporter = {exporter}
569+ forced = yes""")
570diff --git a/checkbox_ng/launcher/subcommands.py b/checkbox_ng/launcher/subcommands.py
571index b4f4350..1c51ac5 100644
572--- a/checkbox_ng/launcher/subcommands.py
573+++ b/checkbox_ng/launcher/subcommands.py
574@@ -42,7 +42,6 @@ from plainbox.impl.execution import UnifiedRunner
575 from plainbox.impl.highlevel import Explorer
576 from plainbox.impl.result import MemoryJobResult
577 from plainbox.impl.runner import slugify
578-from plainbox.impl.secure.config import Unset
579 from plainbox.impl.secure.sudo_broker import sudo_password_provider
580 from plainbox.impl.session.assistant import SA_RESTARTABLE
581 from plainbox.impl.session.restart import detect_restart_strategy
582@@ -168,10 +167,10 @@ class Launcher(MainLoopStage, ReportsStage):
583 return self._C
584
585 def get_sa_api_version(self):
586- return self.launcher.api_version
587+ return '0.99'
588
589 def get_sa_api_flags(self):
590- return self.launcher.api_flags
591+ return [SA_RESTARTABLE]
592
593 def invoked(self, ctx):
594 if ctx.args.version:
595@@ -184,12 +183,12 @@ class Launcher(MainLoopStage, ReportsStage):
596 # exited by now, so validation passed
597 print(_("Launcher seems valid."))
598 return
599- self.launcher = load_configs(ctx.args.launcher)
600+ self.configuration = load_configs(ctx.args.launcher)
601 logging_level = {
602 'normal': logging.WARNING,
603 'verbose': logging.INFO,
604 'debug': logging.DEBUG,
605- }[self.launcher.verbosity]
606+ }[self.configuration.get_value('ui', 'verbosity')]
607 if not ctx.args.verbose and not ctx.args.debug:
608 # Command line args take precendence
609 logging.basicConfig(level=logging_level)
610@@ -200,25 +199,28 @@ class Launcher(MainLoopStage, ReportsStage):
611 # replace the previously built SA with the defaults
612 self._configure_restart(ctx)
613 self._prepare_transports()
614- ctx.sa.use_alternate_configuration(self.launcher)
615+ ctx.sa.use_alternate_configuration(self.configuration)
616 if not self._maybe_resume_session():
617 self._start_new_session()
618 self._pick_jobs_to_run()
619 if not self.ctx.sa.get_static_todo_list():
620 return 0
621- if 'submission_files' in self.launcher.stock_reports:
622+ if 'submission_files' in self.configuration.get_value(
623+ 'launcher', 'stock_reports'):
624 print("Reports will be saved to: {}".format(self.base_dir))
625 # we initialize the nb of attempts for all the selected jobs...
626 for job_id in self.ctx.sa.get_dynamic_todo_list():
627 job_state = self.ctx.sa.get_job_state(job_id)
628- job_state.attempts = self.launcher.max_attempts
629+ job_state.attempts = self.configuration.get_value(
630+ 'ui', 'max_attempts')
631 # ... before running them
632 self._run_jobs(self.ctx.sa.get_dynamic_todo_list())
633- if self.is_interactive and not self.launcher.auto_retry:
634+ if self.is_interactive and not self.configuration.get_value(
635+ 'ui', 'auto_retry'):
636 while True:
637 if not self._maybe_rerun_jobs():
638 break
639- elif self.launcher.auto_retry:
640+ elif self.configuration.get_value('ui', 'auto_retry'):
641 while True:
642 if not self._maybe_auto_rerun_jobs():
643 break
644@@ -235,37 +237,14 @@ class Launcher(MainLoopStage, ReportsStage):
645
646 We can then interact with the user when we encounter OUTCOME_UNDECIDED.
647 """
648- return (self.launcher.ui_type == 'interactive' and
649- sys.stdin.isatty() and sys.stdout.isatty())
650+ return (self.configuration.get_value('ui', 'type') == 'interactive'
651+ and sys.stdin.isatty() and sys.stdout.isatty())
652
653 def _configure_restart(self, ctx):
654 if SA_RESTARTABLE not in self.get_sa_api_flags():
655 return
656- if self.launcher.restart_strategy:
657- try:
658- cls = get_strategy_by_name(
659- self.launcher.restart_strategy)
660- kwargs = copy.deepcopy(self.launcher.restart)
661- # [restart] section has the kwargs for the strategy initializer
662- # and the 'strategy' which is not one, let's pop it
663- kwargs.pop('strategy')
664- strategy = cls(**kwargs)
665- ctx.sa.use_alternate_restart_strategy(strategy)
666-
667- except KeyError:
668- _logger.warning(_('Unknown restart strategy: %s', (
669- self.launcher.restart_strategy)))
670- _logger.warning(_(
671- 'Using automatically detected restart strategy'))
672- try:
673- strategy = detect_restart_strategy()
674- except LookupError as exc:
675- _logger.warning(exc)
676- _logger.warning(_('Automatic restart disabled!'))
677- strategy = None
678- else:
679+ try:
680 strategy = detect_restart_strategy()
681- if strategy:
682 # gluing the command with pluses b/c the middle part
683 # (launcher path) is optional
684 snap_name = os.getenv('SNAP_NAME')
685@@ -282,6 +261,9 @@ class Launcher(MainLoopStage, ReportsStage):
686 respawn_cmd += '--resume {}' # interpolate with session_id
687 ctx.sa.configure_application_restart(
688 lambda session_id: [respawn_cmd.format(session_id)])
689+ except LookupError as exc:
690+ _logger.warning(exc)
691+ _logger.warning(_('Automatic restart disabled!'))
692
693 def _maybe_resume_session(self):
694 resume_candidates = list(self.ctx.sa.get_resumable_sessions())
695@@ -338,18 +320,21 @@ class Launcher(MainLoopStage, ReportsStage):
696
697 def _start_new_session(self):
698 print(_("Preparing..."))
699- title = self.ctx.args.title or self.launcher.session_title
700- title = title or self.launcher.app_id
701- if self.launcher.app_version:
702- title += ' {}'.format(self.launcher.app_version)
703+ title = self.ctx.args.title or self.configuration.get_value(
704+ 'launcher', 'session_title')
705+ title = title or self.configuration.get_value('launcher', 'app_id')
706+ if self.configuration.get_value('launcher', 'app_version'):
707+ title += ' {}'.format(self.configuration.get_value(
708+ 'launcher', 'app_version'))
709 runner_kwargs = {
710- 'normal_user_provider': lambda: self.launcher.normal_user,
711+ 'normal_user_provider': lambda: self.configuration.get_value(
712+ 'daemon', 'normal_user'),
713 'password_provider': sudo_password_provider.get_sudo_password,
714 'stdin': None,
715 }
716 self.ctx.sa.start_new_session(title, UnifiedRunner, runner_kwargs)
717- if self.launcher.test_plan_forced:
718- tp_id = self.launcher.test_plan_default_selection
719+ if self.configuration.get_value('test plan', 'forced'):
720+ tp_id = self.configuration.get_value('test plan', 'unit')
721 if tp_id not in self.ctx.sa.get_test_plans():
722 _logger.error(_(
723 'The test plan "%s" is not available!'), tp_id)
724@@ -365,7 +350,8 @@ class Launcher(MainLoopStage, ReportsStage):
725 if tp_id is None:
726 raise SystemExit(_("No test plan selected."))
727 self.ctx.sa.select_test_plan(tp_id)
728- description = self.ctx.args.message or self.launcher.session_desc
729+ description = self.ctx.args.message or self.configuration.get_value(
730+ 'launcher', 'session_desc')
731 self.ctx.sa.update_app_blob(json.dumps(
732 {'testplan_id': tp_id,
733 'description': description}).encode("UTF-8"))
734@@ -380,7 +366,7 @@ class Launcher(MainLoopStage, ReportsStage):
735 def _interactively_pick_test_plan(self):
736 test_plan_ids = self.ctx.sa.get_test_plans()
737 filtered_tp_ids = set()
738- for filter in self.launcher.test_plan_filters:
739+ for filter in self.configuration.get_value('test plan', 'filter'):
740 filtered_tp_ids.update(fnmatch.filter(test_plan_ids, filter))
741 tp_info_list = self._generate_tp_infos(filtered_tp_ids)
742 if not tp_info_list:
743@@ -388,19 +374,20 @@ class Launcher(MainLoopStage, ReportsStage):
744 return
745 selected_tp = TestPlanBrowser(
746 _("Select test plan"), tp_info_list,
747- self.launcher.test_plan_default_selection).run()
748+ self.configuration.get_value('test plan', 'unit')).run()
749 return selected_tp
750
751 def _strtobool(self, val):
752 return val.lower() in ('y', 'yes', 't', 'true', 'on', '1')
753
754 def _pick_jobs_to_run(self):
755- if self.launcher.test_selection_forced:
756- if self.launcher.manifest is not Unset:
757+ if self.configuration.get_value('test selection', 'forced'):
758+ if self.configuration.manifest:
759 self.ctx.sa.save_manifest(
760 {manifest_id:
761- self._strtobool(self.launcher.manifest[manifest_id]) for
762- manifest_id in self.launcher.manifest}
763+ self._strtobool(
764+ self.configuration.manifest[manifest_id]) for
765+ manifest_id in self.configuration.manifest}
766 )
767 # by default all tests are selected; so we're done here
768 return
769@@ -492,7 +479,7 @@ class Launcher(MainLoopStage, ReportsStage):
770 if not rerun_candidates:
771 return False
772 # we wait before retrying
773- delay = self.launcher.delay_before_retry
774+ delay = self.configuration.get_value('ui', 'delay_before_retry')
775 _logger.info(_("Waiting {} seconds before retrying failed"
776 " jobs...".format(delay)))
777 time.sleep(delay)
778@@ -525,10 +512,11 @@ class Launcher(MainLoopStage, ReportsStage):
779 def considering_job(self, job, job_state):
780 pass
781 show_out = True
782- if self.launcher.output == 'hide-resource-and-attachment':
783+ output = self.configuration.get_value('ui', 'output')
784+ if output == 'hide-resource-and-attachment':
785 if job.plugin in ('local', 'resource', 'attachment'):
786 show_out = False
787- elif self.launcher.output in ['hide', 'hide-automated']:
788+ elif output in ['hide', 'hide-automated']:
789 if job.plugin in ('shell', 'local', 'resource', 'attachment'):
790 show_out = False
791 if 'suppress-output' in job.get_flag_set():
792diff --git a/plainbox/impl/applogic.py b/plainbox/impl/applogic.py
793index 1d90dfd..84f9906 100644
794--- a/plainbox/impl/applogic.py
795+++ b/plainbox/impl/applogic.py
796@@ -31,7 +31,6 @@ import os
797 from plainbox.abc import IJobResult
798 from plainbox.i18n import gettext as _
799 from plainbox.impl.result import MemoryJobResult
800-from plainbox.impl.secure import config
801 from plainbox.impl.secure.qualifiers import select_jobs
802 from plainbox.impl.session import SessionManager
803 from plainbox.impl.session.jobs import InhibitionCause
804@@ -80,23 +79,6 @@ def run_job_if_possible(session, runner, config, job, update=True, ui=None):
805 return job_state, job_result
806
807
808-class PlainBoxConfig(config.Config):
809- """
810- Configuration for PlainBox itself
811- """
812-
813- environment = config.Section(
814- help_text=_("Environment variables for scripts and jobs"))
815-
816- class Meta:
817-
818- # TODO: properly depend on xdg and use real code that also handles
819- # XDG_CONFIG_HOME.
820- filename_list = [
821- '/etc/xdg/plainbox.conf',
822- os.path.expanduser('~/.config/plainbox.conf')]
823-
824-
825 def get_all_exporter_names():
826 """
827 Get the identifiers (names) of all the supported session state exporters.
828diff --git a/plainbox/impl/commands/__init__.py b/plainbox/impl/commands/__init__.py
829index 5a26853..efdb1f6 100644
830--- a/plainbox/impl/commands/__init__.py
831+++ b/plainbox/impl/commands/__init__.py
832@@ -47,7 +47,6 @@ class PlainBoxToolBase(ToolBase):
833 1. :meth:`get_exec_name()` -- to know how the command will be called
834 2. :meth:`get_exec_version()` -- to know how the version of the tool
835 3. :meth:`add_subcommands()` -- to add some actual commands to execute
836- 4. :meth:`get_config_cls()` -- to know which config to use
837
838 This class has some complex control flow to support important and
839 interesting use cases. There are some concerns to people that subclass this
840@@ -69,16 +68,6 @@ class PlainBoxToolBase(ToolBase):
841 known yet.
842 """
843
844- @classmethod
845- @abc.abstractmethod
846- def get_config_cls(cls):
847- """
848- Get the Config class that is used by this implementation.
849-
850- This can be overridden by subclasses to use a different config class
851- that is suitable for the particular application.
852- """
853-
854 def _load_config(self):
855 return self.get_config_cls().get()
856
857diff --git a/plainbox/impl/config.py b/plainbox/impl/config.py
858new file mode 100644
859index 0000000..05b4def
860--- /dev/null
861+++ b/plainbox/impl/config.py
862@@ -0,0 +1,383 @@
863+# This file is part of Checkbox.
864+#
865+# Copyright 2021-2022 Canonical Ltd.
866+# Written by:
867+# Maciej Kisielewski <maciej.kisielewski@canonical.com>
868+#
869+# Checkbox is free software: you can redistribute it and/or modify
870+# it under the terms of the GNU General Public License version 3,
871+# as published by the Free Software Foundation.
872+#
873+# Checkbox is distributed in the hope that it will be useful,
874+# but WITHOUT ANY WARRANTY; without even the implied warranty of
875+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
876+# GNU General Public License for more details.
877+#
878+# You should have received a copy of the GNU General Public License
879+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
880+"""
881+This module defines class for handling Checkbox configs.
882+
883+If we ever need to add validators to config variables, the addition should be
884+done in VarSpec (the fourth 'field').
885+"""
886+import io
887+import logging
888+import os
889+import shlex
890+
891+from configparser import ConfigParser
892+from collections import namedtuple, OrderedDict
893+
894+logger = logging.getLogger(__name__)
895+
896+
897+class Configuration:
898+ """
899+ Checkbox configuration storing objects.
900+
901+ Checkbox configs store various information on how to run the Checkbox.
902+ For instance what reports to generate, should the session be interactive,
903+ and many others. Look at CONFIG_SPEC for details.
904+ """
905+ def __init__(self, source=None):
906+ """Create a new configuration object filled with default values."""
907+ self.sections = OrderedDict()
908+ self._origins = dict()
909+ self._problems = []
910+ # sources is similar to origins, but instead of keeping an info on
911+ # each variable, we note what configs got read in general
912+ self._sources = [source] if source else []
913+ for section, contents in CONFIG_SPEC:
914+ if isinstance(contents, ParametricSection):
915+ # we don't know what the actual section name will be,
916+ # so let's wait with the creation until we know the full name
917+ continue
918+ if isinstance(contents, DynamicSection):
919+ self.sections[section] = DynamicSection()
920+ else:
921+ self.sections[section] = OrderedDict()
922+ self._origins[section] = dict()
923+ for name, spec in sorted(contents.items()):
924+ self.sections[section][name] = spec.default
925+ self._origins[section][name] = ''
926+
927+ @property
928+ def environment(self):
929+ """Return contents of the environment section."""
930+ return self.sections['environment']
931+
932+ @property
933+ def manifest(self):
934+ """Return contents of the manifest section."""
935+ return self.sections['manifest']
936+
937+ @property
938+ def sources(self):
939+ """Return list of sources for this configuration."""
940+ return self._sources
941+
942+ def notice_problem(self, problem):
943+ """ Record and log problem encountered when building configuration."""
944+ self._problems.append(problem)
945+ logger.warning(problem)
946+
947+ def get_problems(self):
948+ """Return a list of problem as strings."""
949+ return self._problems
950+
951+ def get_value(self, section, name):
952+ """Return a value of given `name` from given `section`,"""
953+ return self.sections[section][name]
954+
955+ def get_origin(self, section, name):
956+ """Return origin of the value."""
957+ return self._origins[section][name]
958+
959+ def update_from_another(self, configuration, origin):
960+ """
961+ Update this configuration with values from `configuration`.
962+
963+ Only the values that are not defaults from 'configuration` are taken
964+ into account.
965+ """
966+ for section, variables in configuration.sections.items():
967+ for name in variables.keys():
968+ new_origin = configuration.get_origin(section, name)
969+ if new_origin:
970+ if ':' in section and section not in self.sections.keys():
971+ self.sections[section] = OrderedDict()
972+ self._origins[section] = dict()
973+ self.sections[section][name] = configuration.get_value(
974+ section, name)
975+ self._origins[section][name] = origin or new_origin
976+ self._sources += configuration.sources
977+ self._problems += configuration.get_problems()
978+
979+ def dyn_set_value(self, section, name, value, origin):
980+ """Set a value of a var from a dynamic section."""
981+ if section == 'environment':
982+ name = name.upper()
983+ self.sections[section][name] = value
984+ self._origins[section][name] = origin
985+
986+ def set_value(self, section, name, value, origin):
987+ """Set a new value for variable and update its origin."""
988+ # we are kind off guaranteed that section will be found in the spec
989+ # but let's make linters happy
990+ if section in self._DYNAMIC_SECTIONS:
991+ self.dyn_set_value(section, name, value, origin)
992+ return
993+ parametrized = False
994+ if ':' in section:
995+ parametrized = True
996+ prefix, _ = section.split(':')
997+ if parametrized:
998+ # TODO: do the check here for typing
999+ pass
1000+
1001+ index = -1
1002+ for i, (sect_name, spec) in enumerate(CONFIG_SPEC):
1003+ if sect_name == section:
1004+ index = i
1005+ if isinstance(spec, ParametricSection):
1006+ if parametrized and sect_name == prefix:
1007+ if name not in spec:
1008+ problem = (
1009+ "Unexpected variable '{}' in section [{}] "
1010+ "Origin: {}").format(name, section, origin)
1011+ self.notice_problem(problem)
1012+ return
1013+ index = i
1014+ if index == -1:
1015+ # this should happen only for parametric sections
1016+ problem = "Unexpected section [{}]. Origin: {}".format(
1017+ section, origin)
1018+ self.notice_problem(problem)
1019+ return
1020+
1021+ assert index > -1
1022+ kind = CONFIG_SPEC[index][1][name].kind
1023+ try:
1024+ if kind == list:
1025+ value = shlex.split(value.replace(',', ' '))
1026+ else:
1027+ value = kind(value)
1028+ if parametrized:
1029+ # we couldn't have known the param names eariler (in __init__)
1030+ # but now we do know them, so let's create the dict to hold
1031+ # the values
1032+ if section not in self.sections.keys():
1033+ self.sections[section] = OrderedDict()
1034+ self._origins[section] = dict()
1035+ self.sections[section][name] = value
1036+ self._origins[section][name] = origin
1037+ except TypeError:
1038+ problem = (
1039+ "Problem with setting field {} in section [{}] "
1040+ "'{}' cannot be used as {}. Origin: {}").format(
1041+ name, section, value, kind, origin)
1042+ self.notice_problem(problem)
1043+
1044+ def get_parametric_sections(self, prefix):
1045+ """
1046+ Return a dict of parametrised section that share the same prefix.
1047+
1048+ The resulting dict is keyed by the parameter, the values are dicts
1049+ with the declared variables.
1050+
1051+ E.g.
1052+ If there's two sections: [report:myrep] and [report:other]
1053+ The resulting dict will have two keys: myrep and other.
1054+ """
1055+ result = dict()
1056+ # check if there is such section declared in the SPEC
1057+ for sect_name, section in CONFIG_SPEC:
1058+ if not isinstance(section, ParametricSection):
1059+ continue
1060+ if sect_name == prefix:
1061+ break
1062+ else:
1063+ raise ValueError("No such section in the spec ({}".format(prefix))
1064+ for sect_name, section in self.sections.items():
1065+ sect_prefix, _, sect_param = sect_name.partition(':')
1066+ if sect_prefix == prefix:
1067+ result[sect_param] = section
1068+ return result
1069+
1070+ @classmethod
1071+ def from_text(cls, text, origin):
1072+ """
1073+ Create a new configuration with values from the text.
1074+
1075+ Behaves just the same as the from_ini_file method, but accepts string
1076+ as the param.
1077+ """
1078+ return cls.from_ini_file(io.StringIO(text), origin)
1079+
1080+ @classmethod
1081+ def from_path(cls, path):
1082+ """Create a new configuration with values stored in a file at path."""
1083+ cfg = Configuration()
1084+ if not os.path.isfile(path):
1085+ cfg.notice_problem("{} file not found".format(path))
1086+ return cfg
1087+ with open(path, 'rt') as ini_file:
1088+ return cls.from_ini_file(ini_file, path)
1089+
1090+ @classmethod
1091+ def from_ini_file(cls, ini_file, origin):
1092+ """
1093+ Create a new configuration with values from the ini file.
1094+
1095+ ini_file should be a file object.
1096+
1097+ This function is designed not to fail (raise), so if some entry in the
1098+ ini file is misdefined then it should be ignored and the default value
1099+ should be kept. Each such problem is kept in the self._problems list.
1100+ """
1101+ cfg = Configuration(origin)
1102+ parser = ConfigParser(delimiters='=')
1103+ parser.read_string(ini_file.read())
1104+ for sect_name, section in parser.items():
1105+ if sect_name == 'DEFAULT':
1106+ for var_name in section:
1107+ problem = "[DEFAULT] section is not supported"
1108+ cfg.notice_problem(problem)
1109+ continue
1110+ if ':' in sect_name:
1111+ for var_name, var in section.items():
1112+ cfg.set_value(sect_name, var_name, var, origin)
1113+ continue
1114+ if sect_name not in cfg.sections:
1115+ problem = "Unexpected section [{}]. Origin: {}".format(
1116+ sect_name, origin)
1117+ cfg.notice_problem(problem)
1118+ continue
1119+ for var_name, var in section.items():
1120+ is_dyn = sect_name in cls._DYNAMIC_SECTIONS
1121+ if var_name not in cfg.sections[sect_name] and not is_dyn:
1122+ problem = (
1123+ "Unexpected variable '{}' in section [{}] "
1124+ "Origin: {}").format(var_name, sect_name, origin)
1125+ cfg.notice_problem(problem)
1126+ continue
1127+ cfg.set_value(sect_name, var_name, var, origin)
1128+ return cfg
1129+
1130+ _DYNAMIC_SECTIONS = ('environment', 'manifest')
1131+
1132+
1133+VarSpec = namedtuple('VarSpec', ['kind', 'default', 'help'])
1134+
1135+
1136+class ParametricSection(dict):
1137+ """ Dict for storing parametric section's contents."""
1138+
1139+
1140+class DynamicSection(dict):
1141+ """
1142+ Dict for storing dynamic section's contents.
1143+
1144+ This is an extra type to record the fact that this is a different section
1145+ compared to the predefined ones. It works and isn't very complex, but
1146+ a different way of storing this information might be more elegant.
1147+ """
1148+
1149+
1150+# in order to maintain the section order the CONFIG_SPEC is a list of pairs,
1151+# where the first value is the name of the section and the other is a dict
1152+# of variable specs.
1153+CONFIG_SPEC = [
1154+ ('config', {
1155+ 'config_filename': VarSpec(
1156+ str, 'checkbox.conf',
1157+ 'Name of the configuration file to look for.'),
1158+ }),
1159+ ('launcher', {
1160+ 'launcher_version': VarSpec(
1161+ int, 1, "Version of launcher to use"),
1162+ 'app_id': VarSpec(
1163+ str, 'checkbox-cli', "Identifier of the application"),
1164+ 'app_version': VarSpec(
1165+ str, '', "Version of the application"),
1166+ 'stock_reports': VarSpec(
1167+ list, ['text', 'certification', 'submission_files'],
1168+ "List of stock reports to use"),
1169+ 'local_submission': VarSpec(
1170+ bool, True, ("Send/generate submission report locally when using "
1171+ "checkbox remote")),
1172+ 'session_title': VarSpec(
1173+ str, 'session title',
1174+ ("A title to be applied to the sessions created using this "
1175+ "launcher that can be used in report generation")),
1176+ 'session_desc': VarSpec(
1177+ str, '', ("A string that can be applied to sessions created using "
1178+ "this launcher. Useful for storing some contextual "
1179+ "infomation about the session")),
1180+ }),
1181+ ('test plan', {
1182+ 'filter': VarSpec(
1183+ list, ['*'],
1184+ "Constrain interactive choice to test plans matching this glob"),
1185+ 'unit': VarSpec(str, '', "Select this test plan by default."),
1186+ 'forced': VarSpec(
1187+ bool, False, "Don't allow the user to change test plan."),
1188+ }),
1189+ ('test selection', {
1190+ 'forced': VarSpec(
1191+ bool, False, "Don't allow the user to alter test selection."),
1192+ 'exclude': VarSpec(
1193+ list, [], "Exclude test matching patterns from running."),
1194+ }),
1195+ ('ui', {
1196+ 'type': VarSpec(str, 'interactive', "Type of user interface to use."),
1197+ 'output': VarSpec(str, 'show', "Silence or restrict command output."),
1198+ 'dont_suppress_output': VarSpec(
1199+ bool, False,
1200+ "Don't suppress the output of certain job plugin types."),
1201+ 'verbosity': VarSpec(str, 'normal', "Verbosity level."),
1202+ 'auto_retry': VarSpec(
1203+ bool, False,
1204+ "Automatically retry failed jobs at the end of the session."),
1205+ 'max_attempts': VarSpec(
1206+ int, 3,
1207+ "Number of attempts to run a job when in auto-retry mode."),
1208+ 'delay_before_retry': VarSpec(
1209+ int, 1, ("Delay (in seconds) before "
1210+ "retrying failed jobs in auto-retry mode.")),
1211+ }),
1212+ ('daemon', {
1213+ 'normal_user': VarSpec(
1214+ str, '', "Username to use for jobs that don't specify user."),
1215+ }),
1216+ ('restart', {
1217+ 'strategy': VarSpec(str, '', "Use alternative restart strategy."),
1218+ }),
1219+ ('report', ParametricSection({
1220+ 'exporter': VarSpec(
1221+ str, '', "Name of the exporter to use"),
1222+ 'transport': VarSpec(
1223+ str, '', "Name of the transport to use"),
1224+ 'forced': VarSpec(
1225+ bool, False, "Don't ask the user if they want the report."),
1226+ })),
1227+ ('transport', ParametricSection({
1228+ 'type': VarSpec(
1229+ str, '', "Type of transport to use."),
1230+ 'stream': VarSpec(
1231+ str, 'stdout', "Stream to use - stdout or stderr."),
1232+ 'path': VarSpec(
1233+ str, '', "Path to where the report should be saved to."),
1234+ 'secure_id': VarSpec(
1235+ str, '', "Secure ID to use."),
1236+ 'staging': VarSpec(
1237+ bool, False, "Pushes to staging C3 instead of normal C3."),
1238+ })),
1239+ ('exporter', ParametricSection({
1240+ 'unit': VarSpec(str, '', "ID of the exporter to use."),
1241+ 'options': VarSpec(list, [], "Flags to forward to the exporter."),
1242+ })),
1243+ ('environment', DynamicSection()),
1244+ ('manifest', DynamicSection()),
1245+]
1246diff --git a/plainbox/impl/ctrl.py b/plainbox/impl/ctrl.py
1247index 5facd64..2242114 100644
1248--- a/plainbox/impl/ctrl.py
1249+++ b/plainbox/impl/ctrl.py
1250@@ -60,7 +60,6 @@ from plainbox.impl.resource import ExpressionCannotEvaluateError
1251 from plainbox.impl.resource import ExpressionFailedError
1252 from plainbox.impl.resource import ResourceProgramError
1253 from plainbox.impl.resource import Resource
1254-from plainbox.impl.secure.config import Unset
1255 from plainbox.impl.secure.origin import JobOutputTextSource
1256 from plainbox.impl.secure.providers.v1 import Provider1
1257 from plainbox.impl.secure.rfc822 import RFC822SyntaxError
1258diff --git a/plainbox/impl/launcher.py b/plainbox/impl/launcher.py
1259deleted file mode 100644
1260index 16b3059..0000000
1261--- a/plainbox/impl/launcher.py
1262+++ /dev/null
1263@@ -1,258 +0,0 @@
1264-# This file is part of Checkbox.
1265-#
1266-# Copyright 2014-2016 Canonical Ltd.
1267-# Written by:
1268-# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
1269-# Maciej Kisielewski <maciej.kisielewski@canonical.com>
1270-#
1271-# Checkbox is free software: you can redistribute it and/or modify
1272-# it under the terms of the GNU General Public License version 3,
1273-# as published by the Free Software Foundation.
1274-#
1275-# Checkbox is distributed in the hope that it will be useful,
1276-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1277-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1278-# GNU General Public License for more details.
1279-#
1280-# You should have received a copy of the GNU General Public License
1281-# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
1282-
1283-"""
1284-:mod:`plainbox.impl.launcher` -- launcher definition
1285-==================================================
1286-"""
1287-
1288-from gettext import gettext as _
1289-import logging
1290-
1291-from plainbox.impl.applogic import PlainBoxConfig
1292-from plainbox.impl.secure import config
1293-from plainbox.impl.session.assistant import SA_RESTARTABLE
1294-from plainbox.impl.session.assistant import get_all_sa_flags
1295-from plainbox.impl.session.assistant import get_known_sa_api_versions
1296-from plainbox.impl.transport import get_all_transports
1297-from plainbox.impl.transport import SECURE_ID_PATTERN
1298-
1299-
1300-logger = logging.getLogger("plainbox.launcher")
1301-
1302-
1303-class LauncherDefinition(PlainBoxConfig):
1304- """
1305- Launcher definition.
1306-
1307- Launchers are small executables using one of the available user interfaces
1308- as the interpreter. This class contains all the available options that can
1309- be set inside the launcher, that will affect the user interface at runtime.
1310- This generic launcher definition class helps to pick concrete version of
1311- the launcher definition.
1312- """
1313- launcher_version = config.Variable(
1314- section="launcher",
1315- help_text=_("Version of launcher to use"))
1316-
1317- config_filename = config.Variable(
1318- section="config",
1319- default="checkbox.conf",
1320- help_text=_("Name of custom configuration file"))
1321-
1322- def get_concrete_launcher(self):
1323- """Create appropriate LauncherDefinition instance.
1324-
1325- Depending on the value of launcher_version variable appropriate
1326- LauncherDefinition class is chosen and its instance returned.
1327-
1328- :returns: LauncherDefinition instance
1329- :raises KeyError: for unknown launcher_version values
1330- """
1331- return {'1': LauncherDefinition1}[self.launcher_version]()
1332-
1333-
1334-class LauncherDefinition1(LauncherDefinition):
1335- """
1336- Definition for launchers version 1.
1337-
1338- As specced in https://goo.gl/qJYtPX
1339- """
1340-
1341- def __init__(self):
1342- super().__init__()
1343-
1344- launcher_version = config.Variable(
1345- section="launcher",
1346- default='1',
1347- help_text=_("Version of launcher to use"))
1348-
1349- app_id = config.Variable(
1350- section='launcher',
1351- default='checkbox-cli',
1352- help_text=_('Identifier of the application'))
1353-
1354- app_version = config.Variable(
1355- section='launcher',
1356- help_text=_('Version of the application'))
1357-
1358- api_flags = config.Variable(
1359- section='launcher',
1360- kind=list,
1361- default=[SA_RESTARTABLE],
1362- validator_list=[config.SubsetValidator(get_all_sa_flags())],
1363- help_text=_('List of feature-flags the application requires'))
1364-
1365- api_version = config.Variable(
1366- section='launcher',
1367- default='0.99',
1368- validator_list=[config.ChoiceValidator(
1369- get_known_sa_api_versions())],
1370- help_text=_('Version of API the launcher uses'))
1371-
1372- stock_reports = config.Variable(
1373- section='launcher',
1374- kind=list,
1375- validator_list=[
1376- config.SubsetValidator({
1377- 'text', 'certification', 'certification-staging',
1378- 'submission_files', 'none'}),
1379- config.OneOrTheOtherValidator(
1380- {'none'}, {'text', 'certification', 'certification-staging',
1381- 'submission_files'}),
1382- ],
1383- default=['text', 'certification', 'submission_files'],
1384- help_text=_('List of stock reports to use'))
1385-
1386- local_submission = config.Variable(
1387- section='launcher',
1388- kind=bool,
1389- default=True,
1390- help_text=_("Send/generate submission report locally when using "
1391- "checkbox remote"))
1392-
1393- session_title = config.Variable(
1394- section='launcher',
1395- default='session title',
1396- help_text=_("A title to be applied to the sessions created using "
1397- "this launcher that can be used in report generation"))
1398-
1399- session_desc = config.Variable(
1400- section='launcher',
1401- default='',
1402- help_text=_("A string that can be applied to sessions created using "
1403- "this launcher. Useful for storing some contextual "
1404- "infomation about the session"))
1405-
1406- test_plan_filters = config.Variable(
1407- section='test plan',
1408- name='filter',
1409- default=['*'],
1410- kind=list,
1411- help_text=_('Constrain interactive choice to test plans matching this'
1412- 'glob'))
1413-
1414- test_plan_default_selection = config.Variable(
1415- section='test plan',
1416- name='unit',
1417- help_text=_('Select this test plan by default.'))
1418-
1419- test_plan_forced = config.Variable(
1420- section='test plan',
1421- name='forced',
1422- kind=bool,
1423- default=False,
1424- help_text=_("Don't allow the user to change test plan."))
1425-
1426- test_selection_forced = config.Variable(
1427- section='test selection',
1428- name='forced',
1429- kind=bool,
1430- default=False,
1431- help_text=_("Don't allow the user to alter test selection."))
1432-
1433- test_exclude = config.Variable(
1434- section='test selection',
1435- name='exclude',
1436- default=[],
1437- kind=list,
1438- help_text=_("Exclude test matching the patterns from running"))
1439-
1440- ui_type = config.Variable(
1441- section='ui',
1442- name='type',
1443- default='interactive',
1444- validator_list=[config.ChoiceValidator(
1445- ['interactive', 'silent'])],
1446- help_text=_('Type of stock user interface to use.'))
1447-
1448- output = config.Variable(
1449- section='ui',
1450- default='show',
1451- validator_list=[config.ChoiceValidator(
1452- ['show', 'hide', 'hide-resource-and-attachment',
1453- 'hide-automated'])],
1454- help_text=_('Silence or restrict command output'))
1455-
1456- dont_suppress_output = config.Variable(
1457- section="ui", kind=bool, default=False,
1458- help_text=_("Don't suppress the output of certain job plugin types."))
1459-
1460- verbosity = config.Variable(
1461- section="ui", validator_list=[config.ChoiceValidator(
1462- ['normal', 'verbose', 'debug'])], help_text=_('Verbosity level'),
1463- default='normal')
1464-
1465- auto_retry = config.Variable(
1466- section='ui',
1467- kind=bool,
1468- default=False,
1469- help_text=_("Automatically retry failed jobs at the end"
1470- " of the session."))
1471-
1472- max_attempts = config.Variable(
1473- section='ui',
1474- kind=int,
1475- default=3,
1476- help_text=_("Number of attempts to run a job when in auto-retry mode."))
1477-
1478- delay_before_retry = config.Variable(
1479- section='ui',
1480- kind=int,
1481- default=1,
1482- help_text=_("Delay (in seconds) before retrying failed jobs in"
1483- " auto-retry mode."))
1484-
1485- normal_user = config.Variable(
1486- section='daemon',
1487- kind=str,
1488- default='',
1489- help_text=_("Username to use for jobs that don't specify user"))
1490-
1491- restart_strategy = config.Variable(
1492- section='restart',
1493- name='strategy',
1494- help_text=_('Use alternative restart strategy'))
1495-
1496- restart = config.Section(
1497- help_text=_('Restart strategy parameters'))
1498-
1499- reports = config.ParametricSection(
1500- name='report',
1501- help_text=_('Report declaration'))
1502-
1503- exporters = config.ParametricSection(
1504- name='exporter',
1505- help_text=_('Exporter declaration'))
1506-
1507- transports = config.ParametricSection(
1508- name='transport',
1509- help_text=_('Transport declaration'))
1510-
1511- environment = config.Section(
1512- help_text=_('Environment variables to use'))
1513-
1514- daemon = config.Section(
1515- name='daemon',
1516- help_text=_('Daemon-specific configuration'))
1517-
1518- manifest = config.Section(
1519- help_text=_('Manifest entries to use'))
1520-
1521-DefaultLauncherDefinition = LauncherDefinition1
1522diff --git a/plainbox/impl/runner.py b/plainbox/impl/runner.py
1523index f46a99f..fae15fb 100644
1524--- a/plainbox/impl/runner.py
1525+++ b/plainbox/impl/runner.py
1526@@ -50,7 +50,6 @@ from plainbox.i18n import gettext as _
1527 from plainbox.impl.result import IOLogRecord
1528 from plainbox.impl.result import IOLogRecordWriter
1529 from plainbox.impl.result import JobResultBuilder
1530-from plainbox.impl.secure.config import Unset
1531 from plainbox.vendor import extcmd
1532 from plainbox.vendor import morris
1533
1534diff --git a/plainbox/impl/secure/test_config.py b/plainbox/impl/secure/test_config.py
1535deleted file mode 100644
1536index 1efab12..0000000
1537--- a/plainbox/impl/secure/test_config.py
1538+++ /dev/null
1539@@ -1,608 +0,0 @@
1540-# This file is part of Checkbox.
1541-#
1542-# Copyright 2013, 2014 Canonical Ltd.
1543-# Written by:
1544-# Zygmunt Krynicki <zygmunt.krynicki@canonical.com>
1545-#
1546-# Checkbox is free software: you can redistribute it and/or modify
1547-# it under the terms of the GNU General Public License version 3,
1548-# as published by the Free Software Foundation.
1549-
1550-#
1551-# Checkbox is distributed in the hope that it will be useful,
1552-# but WITHOUT ANY WARRANTY; without even the implied warranty of
1553-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
1554-# GNU General Public License for more details.
1555-#
1556-# You should have received a copy of the GNU General Public License
1557-# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
1558-
1559-"""
1560-plainbox.impl.secure.test_config
1561-================================
1562-
1563-Test definitions for plainbox.impl.secure.config module
1564-"""
1565-from io import StringIO
1566-from unittest import TestCase
1567-import configparser
1568-
1569-from plainbox.impl.secure.config import ChoiceValidator
1570-from plainbox.impl.secure.config import ConfigMetaData
1571-from plainbox.impl.secure.config import KindValidator
1572-from plainbox.impl.secure.config import NotEmptyValidator
1573-from plainbox.impl.secure.config import NotUnsetValidator
1574-from plainbox.impl.secure.config import OneOrTheOtherValidator
1575-from plainbox.impl.secure.config import PatternValidator
1576-from plainbox.impl.secure.config import ParametricSection
1577-from plainbox.impl.secure.config import PlainBoxConfigParser, Config
1578-from plainbox.impl.secure.config import ValidationError
1579-from plainbox.impl.secure.config import Variable, Section, Unset
1580-from plainbox.impl.secure.config import understands_Unset
1581-from plainbox.vendor import mock
1582-
1583-
1584-class UnsetTests(TestCase):
1585-
1586- def test_str(self):
1587- self.assertEqual(str(Unset), "unset")
1588-
1589- def test_repr(self):
1590- self.assertEqual(repr(Unset), "Unset")
1591-
1592- def test_bool(self):
1593- self.assertEqual(bool(Unset), False)
1594-
1595-
1596-class understands_Unset_Tests(TestCase):
1597-
1598- def test_func(self):
1599- @understands_Unset
1600- def func():
1601- pass
1602-
1603- self.assertTrue(hasattr(func, 'understands_Unset'))
1604- self.assertTrue(getattr(func, 'understands_Unset'))
1605-
1606- def test_cls(self):
1607- @understands_Unset
1608- class cls:
1609- pass
1610-
1611- self.assertTrue(hasattr(cls, 'understands_Unset'))
1612- self.assertTrue(getattr(cls, 'understands_Unset'))
1613-
1614-
1615-class VariableTests(TestCase):
1616-
1617- def test_name(self):
1618- v1 = Variable()
1619- self.assertIsNone(v1.name)
1620- v2 = Variable('var')
1621- self.assertEqual(v2.name, 'var')
1622- v3 = Variable(name='var')
1623- self.assertEqual(v3.name, 'var')
1624-
1625- def test_section(self):
1626- v1 = Variable()
1627- self.assertEqual(v1.section, 'DEFAULT')
1628- v2 = Variable(section='foo')
1629- self.assertEqual(v2.section, 'foo')
1630-
1631- def test_kind(self):
1632- v1 = Variable(kind=bool)
1633- self.assertIs(v1.kind, bool)
1634- v2 = Variable(kind=int)
1635- self.assertIs(v2.kind, int)
1636- v3 = Variable(kind=float)
1637- self.assertIs(v3.kind, float)
1638- v4 = Variable(kind=str)
1639- self.assertIs(v4.kind, str)
1640- v5 = Variable()
1641- self.assertIs(v5.kind, str)
1642- v6 = Variable(kind=list)
1643- self.assertIs(v6.kind, list)
1644- with self.assertRaises(ValueError):
1645- Variable(kind=dict)
1646-
1647- def test_validator_list__default(self):
1648- """
1649- verify that each Variable has a validator_list and that by default,
1650- that list contains a KindValidator as the first element
1651- """
1652- self.assertEqual(Variable().validator_list, [KindValidator])
1653-
1654- def test_validator_list__explicit(self):
1655- """
1656- verify that each Variable has a validator_list and that, if
1657- customized, the list contains the custom validators, preceded by
1658- the implicit KindValidator object
1659- """
1660- def DummyValidator(variable, new_value):
1661- """ Dummy validator for the test below"""
1662- pass
1663- var = Variable(validator_list=[DummyValidator])
1664- self.assertEqual(var.validator_list, [KindValidator, DummyValidator])
1665-
1666- def test_validator_list__with_NotUnsetValidator(self):
1667- """
1668- verify that each Variable has a validator_list and that, if
1669- customized, and if using NotUnsetValidator it will take precedence
1670- over all other validators, including the implicit KindValidator
1671- """
1672- var = Variable(validator_list=[NotUnsetValidator()])
1673- self.assertEqual(
1674- var.validator_list, [NotUnsetValidator(), KindValidator])
1675-
1676-
1677-class SectionTests(TestCase):
1678-
1679- def test_name(self):
1680- s1 = Section()
1681- self.assertIsNone(s1.name)
1682- s2 = Section('sec')
1683- self.assertEqual(s2.name, 'sec')
1684- s3 = Variable(name='sec')
1685- self.assertEqual(s3.name, 'sec')
1686-
1687-
1688-class ConfigTests(TestCase):
1689-
1690- def test_Meta_present(self):
1691- class TestConfig(Config):
1692- pass
1693- self.assertTrue(hasattr(TestConfig, 'Meta'))
1694-
1695- def test_Meta_base_cls(self):
1696- class TestConfig(Config):
1697- pass
1698- self.assertTrue(issubclass(TestConfig.Meta, ConfigMetaData))
1699-
1700- class HelperMeta:
1701- pass
1702-
1703- class TestConfigWMeta(Config):
1704- Meta = HelperMeta
1705- self.assertTrue(issubclass(TestConfigWMeta.Meta, ConfigMetaData))
1706- self.assertTrue(issubclass(TestConfigWMeta.Meta, HelperMeta))
1707-
1708- def test_Meta_variable_list(self):
1709- class TestConfig(Config):
1710- v1 = Variable()
1711- v2 = Variable()
1712- self.assertEqual(
1713- TestConfig.Meta.variable_list,
1714- [TestConfig.v1, TestConfig.v2])
1715-
1716- def test_variable_smoke(self):
1717- class TestConfig(Config):
1718- v = Variable()
1719- conf = TestConfig()
1720- self.assertIs(conf.v, Unset)
1721- conf.v = "value"
1722- self.assertEqual(conf.v, "value")
1723- del conf.v
1724- self.assertIs(conf.v, Unset)
1725-
1726- def _get_featureful_config(self):
1727- # define a featureful config class
1728- class TestConfig(Config):
1729- v1 = Variable()
1730- v2 = Variable(section="v23_section")
1731- v3 = Variable(section="v23_section")
1732- v_unset = Variable()
1733- v_bool = Variable(section="type_section", kind=bool)
1734- v_int = Variable(section="type_section", kind=int)
1735- v_float = Variable(section="type_section", kind=float)
1736- v_list = Variable(section="type_section", kind=list)
1737- v_str = Variable(section="type_section", kind=str)
1738- s = Section()
1739- ps = ParametricSection()
1740- conf = TestConfig()
1741- # assign value to each variable, except v3_unset
1742- conf.v1 = "v1 value"
1743- conf.v2 = "v2 value"
1744- conf.v3 = "v3 value"
1745- conf.v_bool = True
1746- conf.v_int = -7
1747- conf.v_float = 1.5
1748- conf.v_str = "hi"
1749- conf.v_list = ['foo', 'bar']
1750- # assign value to the section
1751- conf.s = {"a": 1, "b": 2}
1752- conf.ps = {"foo": {"c": 3, "d": 4}}
1753- return conf
1754-
1755- def test_get_parser_obj(self):
1756- """
1757- verify that Config.get_parser_obj() properly writes all the data to the
1758- ConfigParser object.
1759- """
1760- conf = self._get_featureful_config()
1761- parser = conf.get_parser_obj()
1762- # verify that section and section-less variables work
1763- self.assertEqual(parser.get("DEFAULT", "v1"), "v1 value")
1764- self.assertEqual(parser.get("v23_section", "v2"), "v2 value")
1765- self.assertEqual(parser.get("v23_section", "v3"), "v3 value")
1766- # verify that unset variable is not getting set to anything
1767- with self.assertRaises(configparser.Error):
1768- parser.get("DEFAULT", "v_unset")
1769- # verify that various types got converted correctly and still resolve
1770- # to correct typed values
1771- self.assertEqual(parser.get("type_section", "v_bool"), "True")
1772- self.assertEqual(parser.getboolean("type_section", "v_bool"), True)
1773- self.assertEqual(parser.get("type_section", "v_int"), "-7")
1774- self.assertEqual(parser.getint("type_section", "v_int"), -7)
1775- self.assertEqual(parser.get("type_section", "v_float"), "1.5")
1776- self.assertEqual(parser.getfloat("type_section", "v_float"), 1.5)
1777- self.assertEqual(parser.get("type_section", "v_str"), "hi")
1778- # verify that section work okay
1779- self.assertEqual(parser.get("s", "a"), "1")
1780- self.assertEqual(parser.get("s", "b"), "2")
1781- # verify that parametric section works okay
1782- self.assertEqual(parser.get("ps:foo", "c"), "3")
1783- self.assertEqual(parser.get("ps:foo", "d"), "4")
1784-
1785- def test_write(self):
1786- """
1787- verify that Config.write() works
1788- """
1789- conf = self._get_featureful_config()
1790- with StringIO() as stream:
1791- conf.write(stream)
1792- self.assertEqual(stream.getvalue(), (
1793- "[DEFAULT]\n"
1794- "v1 = v1 value\n"
1795- "\n"
1796- "[v23_section]\n"
1797- "v2 = v2 value\n"
1798- "v3 = v3 value\n"
1799- "\n"
1800- "[type_section]\n"
1801- "v_bool = True\n"
1802- "v_float = 1.5\n"
1803- "v_int = -7\n"
1804- "v_list = foo, bar\n"
1805- "v_str = hi\n"
1806- "\n"
1807- "[s]\n"
1808- "a = 1\n"
1809- "b = 2\n"
1810- "\n"
1811- "[ps:foo]\n"
1812- "c = 3\n"
1813- "d = 4\n"
1814- "\n"))
1815-
1816- def test_section_smoke(self):
1817- class TestConfig(Config):
1818- s = Section()
1819- conf = TestConfig()
1820- self.assertIs(conf.s, Unset)
1821- with self.assertRaises(TypeError):
1822- conf.s['key'] = "key-value"
1823- conf.s = {}
1824- self.assertEqual(conf.s, {})
1825- conf.s['key'] = "key-value"
1826- self.assertEqual(conf.s['key'], "key-value")
1827- del conf.s
1828- self.assertIs(conf.s, Unset)
1829-
1830- def test_read_string(self):
1831- class TestConfig(Config):
1832- v = Variable()
1833- conf = TestConfig()
1834- conf.read_string(
1835- "[DEFAULT]\n"
1836- "v = 1")
1837- self.assertEqual(conf.v, "1")
1838- self.assertEqual(len(conf.problem_list), 0)
1839-
1840- def test_read_list_with_spaces(self):
1841- class TestConfig(Config):
1842- l = Variable(kind=list)
1843- conf = TestConfig()
1844- conf.read_string('[DEFAULT]\nl = foo bar')
1845- self.assertEqual(conf.l, ['foo', 'bar'])
1846- self.assertEqual(len(conf.problem_list), 0)
1847-
1848- def test_read_list_with_commas(self):
1849- class TestConfig(Config):
1850- l = Variable(kind=list)
1851- conf = TestConfig()
1852- conf.read_string('[DEFAULT]\nl = foo,bar')
1853- self.assertEqual(conf.l, ['foo', 'bar'])
1854- self.assertEqual(len(conf.problem_list), 0)
1855-
1856- def test_read_list_quoted_strings(self):
1857- class TestConfig(Config):
1858- l = Variable(kind=list)
1859- conf = TestConfig()
1860- conf.read_string('[DEFAULT]\nl = foo "bar baz"')
1861- self.assertEqual(conf.l, ['foo', 'bar baz'])
1862- self.assertEqual(len(conf.problem_list), 0)
1863-
1864- def test_read_string_calls_validate_whole(self):
1865- """
1866- verify that Config.read_string() calls validate_whole()"
1867- """
1868- conf = Config()
1869- with mock.patch.object(conf, 'validate_whole') as mocked_validate:
1870- conf.read_string('')
1871- mocked_validate.assert_called_once_with()
1872-
1873- def test_read_calls_validate_whole(self):
1874- """
1875- verify that Config.read() calls validate_whole()"
1876- """
1877- conf = Config()
1878- with mock.patch.object(conf, 'validate_whole') as mocked_validate:
1879- conf.read([])
1880- mocked_validate.assert_called_once_with()
1881-
1882- def test_read__handles_errors_from_validate_whole(self):
1883- """
1884- verify that Config.read() collects errors from validate_whole()".
1885- """
1886- class TestConfig(Config):
1887- v = Variable()
1888-
1889- def validate_whole(self):
1890- raise ValidationError(TestConfig.v, self.v, "v is evil")
1891- conf = TestConfig()
1892- conf.read([])
1893- self.assertEqual(len(conf.problem_list), 1)
1894- self.assertEqual(conf.problem_list[0].variable, TestConfig.v)
1895- self.assertEqual(conf.problem_list[0].new_value, Unset)
1896- self.assertEqual(conf.problem_list[0].message, "v is evil")
1897-
1898- def test_read_string__does_not_ignore_nonmentioned_variables(self):
1899- class TestConfig(Config):
1900- v = Variable(validator_list=[NotUnsetValidator()])
1901- conf = TestConfig()
1902- conf.read_string("")
1903- # Because Unset is the default, sadly
1904- self.assertEqual(conf.v, Unset)
1905- # But there was a problem noticed
1906- self.assertEqual(len(conf.problem_list), 1)
1907- self.assertEqual(conf.problem_list[0].variable, TestConfig.v)
1908- self.assertEqual(conf.problem_list[0].new_value, Unset)
1909- self.assertEqual(conf.problem_list[0].message,
1910- "must be set to something")
1911-
1912- def test_read_string__handles_errors_from_validate_whole(self):
1913- """
1914- verify that Config.read_strig() collects errors from validate_whole()".
1915- """
1916- class TestConfig(Config):
1917- v = Variable()
1918-
1919- def validate_whole(self):
1920- raise ValidationError(TestConfig.v, self.v, "v is evil")
1921- conf = TestConfig()
1922- conf.read_string("")
1923- self.assertEqual(len(conf.problem_list), 1)
1924- self.assertEqual(conf.problem_list[0].variable, TestConfig.v)
1925- self.assertEqual(conf.problem_list[0].new_value, Unset)
1926- self.assertEqual(conf.problem_list[0].message, "v is evil")
1927-
1928-
1929-class ConfigMetaDataTests(TestCase):
1930-
1931- def test_filename_list(self):
1932- self.assertEqual(ConfigMetaData.filename_list, [])
1933-
1934- def test_variable_list(self):
1935- self.assertEqual(ConfigMetaData.variable_list, [])
1936-
1937- def test_section_list(self):
1938- self.assertEqual(ConfigMetaData.section_list, [])
1939-
1940- def test_parametric_section_list(self):
1941- self.assertEqual(ConfigMetaData.parametric_section_list, [])
1942-
1943-
1944-class PlainBoxConfigParserTest(TestCase):
1945-
1946- def test_parser(self):
1947- conf_file = StringIO("[testsection]\nlower = low\nUPPER = up")
1948- config = PlainBoxConfigParser()
1949- config.read_file(conf_file)
1950-
1951- self.assertEqual(['testsection'], config.sections())
1952- all_keys = list(config['testsection'].keys())
1953- self.assertTrue('lower' in all_keys)
1954- self.assertTrue('UPPER' in all_keys)
1955- self.assertFalse('upper' in all_keys)
1956-
1957- def test_parametric_sections_parsing(self):
1958- class TestConfig(Config):
1959- ps = ParametricSection()
1960- conf_str = "[ps:foo]\nval = baz\n[ps:bar]\nvar = biz"
1961- config = TestConfig()
1962- config.read_string(conf_str)
1963- self.assertEqual(
1964- config.ps,
1965- {'foo': {'val': 'baz'}, 'bar': {'var': 'biz'}})
1966-
1967-
1968-class KindValidatorTests(TestCase):
1969-
1970- class _Config(Config):
1971- var_bool = Variable(kind=bool)
1972- var_int = Variable(kind=int)
1973- var_float = Variable(kind=float)
1974- var_str = Variable(kind=str)
1975-
1976- def test_error_msg(self):
1977- """
1978- verify that KindValidator() has correct error message for each type
1979- """
1980- bad_value = object()
1981- self.assertEqual(
1982- KindValidator(self._Config.var_bool, bad_value),
1983- "expected a boolean")
1984- self.assertEqual(
1985- KindValidator(self._Config.var_int, bad_value),
1986- "expected an integer")
1987- self.assertEqual(
1988- KindValidator(self._Config.var_float, bad_value),
1989- "expected a floating point number")
1990- self.assertEqual(
1991- KindValidator(self._Config.var_str, bad_value),
1992- "expected a string")
1993-
1994-
1995-class PatternValidatorTests(TestCase):
1996-
1997- class _Config(Config):
1998- var = Variable()
1999-
2000- def test_smoke(self):
2001- """
2002- verify that PatternValidator works as intended
2003- """
2004- validator = PatternValidator("foo.+")
2005- self.assertEqual(validator(self._Config.var, "foobar"), None)
2006- self.assertEqual(
2007- validator(self._Config.var, "foo"),
2008- "does not match pattern: 'foo.+'")
2009-
2010- def test_comparison_works(self):
2011- self.assertTrue(PatternValidator('foo') == PatternValidator('foo'))
2012- self.assertTrue(PatternValidator('foo') != PatternValidator('bar'))
2013- self.assertTrue(PatternValidator('foo') != object())
2014-
2015-
2016-class ChoiceValidatorTests(TestCase):
2017-
2018- class _Config(Config):
2019- var = Variable()
2020-
2021- def test_smoke(self):
2022- """
2023- verify that ChoiceValidator works as intended
2024- """
2025- validator = ChoiceValidator(["foo", "bar"])
2026- self.assertEqual(validator(self._Config.var, "foo"), None)
2027- self.assertEqual(
2028- validator(self._Config.var, "omg"),
2029- "var must be one of foo, bar. Got 'omg'")
2030-
2031- def test_comparison_works(self):
2032- self.assertTrue(ChoiceValidator(["a"]) == ChoiceValidator(["a"]))
2033- self.assertTrue(ChoiceValidator(["a"]) != ChoiceValidator(["b"]))
2034- self.assertTrue(ChoiceValidator(["a"]) != object())
2035-
2036-
2037-class NotUnsetValidatorTests(TestCase):
2038- """
2039- Tests for the NotUnsetValidator class
2040- """
2041-
2042- class _Config(Config):
2043- var = Variable()
2044-
2045- def test_understands_Unset(self):
2046- """
2047- verify that Unset can be handled at all
2048- """
2049- self.assertTrue(getattr(NotUnsetValidator, "understands_Unset"))
2050-
2051- def test_rejects_unset_values(self):
2052- """
2053- verify that Unset variables are rejected
2054- """
2055- validator = NotUnsetValidator()
2056- self.assertEqual(
2057- validator(self._Config.var, Unset), "must be set to something")
2058-
2059- def test_accepts_other_values(self):
2060- """
2061- verify that other values are accepted
2062- """
2063- validator = NotUnsetValidator()
2064- self.assertIsNone(validator(self._Config.var, None))
2065- self.assertIsNone(validator(self._Config.var, "string"))
2066- self.assertIsNone(validator(self._Config.var, 15))
2067-
2068- def test_supports_custom_message(self):
2069- """
2070- verify that custom message is used
2071- """
2072- validator = NotUnsetValidator("value required!")
2073- self.assertEqual(
2074- validator(self._Config.var, Unset), "value required!")
2075-
2076- def test_comparison_works(self):
2077- """
2078- verify that comparison works as expected
2079- """
2080- self.assertTrue(NotUnsetValidator() == NotUnsetValidator())
2081- self.assertTrue(NotUnsetValidator("?") == NotUnsetValidator("?"))
2082- self.assertTrue(NotUnsetValidator() != NotUnsetValidator("?"))
2083- self.assertTrue(NotUnsetValidator() != object())
2084-
2085-
2086-class NotEmptyValidatorTests(TestCase):
2087-
2088- class _Config(Config):
2089- var = Variable()
2090-
2091- def test_rejects_empty_values(self):
2092- validator = NotEmptyValidator()
2093- self.assertEqual(validator(self._Config.var, ""), "cannot be empty")
2094-
2095- def test_supports_custom_message(self):
2096- validator = NotEmptyValidator("name required!")
2097- self.assertEqual(validator(self._Config.var, ""), "name required!")
2098-
2099- def test_isnt_broken(self):
2100- validator = NotEmptyValidator()
2101- self.assertEqual(validator(self._Config.var, "some value"), None)
2102-
2103- def test_comparison_works(self):
2104- self.assertTrue(NotEmptyValidator() == NotEmptyValidator())
2105- self.assertTrue(NotEmptyValidator("?") == NotEmptyValidator("?"))
2106- self.assertTrue(NotEmptyValidator() != NotEmptyValidator("?"))
2107- self.assertTrue(NotEmptyValidator() != object())
2108-
2109-
2110-class OneOrTheOtherValidatorTests(TestCase):
2111-
2112- class _Config(Config):
2113- var = Variable("The Name", kind=list)
2114-
2115- def test_pass_validation(self):
2116- validator = OneOrTheOtherValidator({'foo'}, {'bar'})
2117- value = ['foo']
2118- self.assertIsNone(validator(self._Config.var, value))
2119- value = ['bar']
2120- self.assertIsNone(validator(self._Config.var, value))
2121-
2122- def test_fail_validation(self):
2123- validator = OneOrTheOtherValidator({'foo'}, {'bar'})
2124- value = ['foo', 'bar']
2125- self.assertEquals(
2126- validator(self._Config.var, value),
2127- "The Name can only use values from {'foo'} or from {'bar'}")
2128-
2129- def test_pass_empty(self):
2130- validator = OneOrTheOtherValidator({'foo'}, {'bar'})
2131- value = []
2132- self.assertIsNone(validator(self._Config.var, value))
2133-
2134- def test_comparison_works(self):
2135- self.assertEqual(
2136- OneOrTheOtherValidator({'foo'}, {'bar'}),
2137- OneOrTheOtherValidator({'foo'}, {'bar'})
2138- )
2139- self.assertEqual(
2140- OneOrTheOtherValidator({1, 2}, {3, 4}),
2141- OneOrTheOtherValidator({2, 1}, {4, 3})
2142- )
2143- self.assertNotEqual(
2144- OneOrTheOtherValidator({1}, {2}),
2145- OneOrTheOtherValidator({1}, {'foo'})
2146- )
2147- self.assertNotEqual(OneOrTheOtherValidator({1}, {2}), object())
2148diff --git a/plainbox/impl/session/assistant.py b/plainbox/impl/session/assistant.py
2149index b2d6991..c560828 100644
2150--- a/plainbox/impl/session/assistant.py
2151+++ b/plainbox/impl/session/assistant.py
2152@@ -38,16 +38,15 @@ from plainbox.abc import IJobResult
2153 from plainbox.abc import IJobRunnerUI
2154 from plainbox.abc import ISessionStateTransport
2155 from plainbox.i18n import gettext as _
2156-from plainbox.impl.applogic import PlainBoxConfig
2157 from plainbox.impl.decorators import raises
2158 from plainbox.impl.developer import UnexpectedMethodCall
2159 from plainbox.impl.developer import UsageExpectation
2160 from plainbox.impl.execution import UnifiedRunner
2161+from plainbox.impl.config import Configuration
2162 from plainbox.impl.providers import get_providers
2163 from plainbox.impl.result import JobResultBuilder
2164 from plainbox.impl.result import MemoryJobResult
2165 from plainbox.impl.runner import JobRunnerUIDelegate
2166-from plainbox.impl.secure.config import Unset
2167 from plainbox.impl.secure.origin import Origin
2168 from plainbox.impl.secure.qualifiers import select_jobs
2169 from plainbox.impl.secure.qualifiers import FieldQualifier
2170@@ -162,7 +161,7 @@ class SessionAssistant:
2171 self._app_version = app_version
2172 self._api_version = api_version
2173 self._api_flags = api_flags
2174- self._config = PlainBoxConfig().get()
2175+ self._config = Configuration()
2176 Unit.config = self._config
2177 self._execution_ctrl_list = None # None is "default"
2178 self._ctrl_setup_list = []
2179@@ -297,8 +296,7 @@ class SessionAssistant:
2180 Use alternate configuration object.
2181
2182 :param config:
2183- A configuration object that implements a superset of the plainbox
2184- configuration.
2185+ A Checkbox configuration object.
2186 :raises UnexpectedMethodCall:
2187 If the call is made at an unexpected time. Do not catch this error.
2188 It is a bug in your program. The error message will indicate what
2189@@ -311,7 +309,7 @@ class SessionAssistant:
2190 UsageExpectation.of(self).enforce()
2191 self._config = config
2192 self._exclude_qualifiers = []
2193- for pattern in self._config.test_exclude:
2194+ for pattern in self._config.get_value('test selection', 'exclude'):
2195 self._exclude_qualifiers.append(
2196 RegExpJobQualifier(pattern, None, False))
2197 Unit.config = config
2198@@ -1199,7 +1197,7 @@ class SessionAssistant:
2199 if os.path.isfile(manifest):
2200 with open(manifest, 'rt', encoding='UTF-8') as stream:
2201 manifest_cache = json.load(stream)
2202- if self._config is not None and self._config.manifest is not Unset:
2203+ if self._config is not None and self._config.manifest:
2204 for manifest_id in self._config.manifest:
2205 manifest_cache.update(
2206 {manifest_id: self._config.manifest[manifest_id]})
2207@@ -1362,11 +1360,8 @@ class SessionAssistant:
2208 f.writelines(self._restart_cmd_callback(
2209 self.get_session_id()))
2210 if not native:
2211- if self._config.environment is Unset:
2212- result = self._runner.run_job(job, job_state, ui=ui)
2213- else:
2214- result = self._runner.run_job(job, job_state,
2215- self._config.environment, ui)
2216+ result = self._runner.run_job(job, job_state,
2217+ self._config.environment, ui)
2218 builder = result.get_builder()
2219 else:
2220 builder = JobResultBuilder(
2221diff --git a/plainbox/impl/session/remote_assistant.py b/plainbox/impl/session/remote_assistant.py
2222index a975fc9..7c02140 100644
2223--- a/plainbox/impl/session/remote_assistant.py
2224+++ b/plainbox/impl/session/remote_assistant.py
2225@@ -29,6 +29,7 @@ from tempfile import SpooledTemporaryFile
2226 from threading import Thread, Lock
2227 from subprocess import CalledProcessError, check_output
2228
2229+from plainbox.impl.config import Configuration
2230 from plainbox.impl.execution import UnifiedRunner
2231 from plainbox.impl.session.assistant import SessionAssistant
2232 from plainbox.impl.session.assistant import SA_RESTARTABLE
2233@@ -273,20 +274,20 @@ class RemoteSessionAssistant():
2234
2235 self._launcher = load_configs()
2236 if configuration['launcher']:
2237- self._launcher.read_string(configuration['launcher'], False)
2238- if self._launcher.session_title:
2239- session_title = self._launcher.session_title
2240- if self._launcher.session_desc:
2241- session_desc = self._launcher.session_desc
2242+ self._launcher = Configuration.from_text(
2243+ configuration['launcher'], 'Remote launcher')
2244+ session_title = self._launcher.get_value(
2245+ 'launcher', 'session_title') or session_title
2246+ session_desc = self._launcher.get_value(
2247+ 'launcher', 'session_desc') or session_desc
2248
2249 self._sa.use_alternate_configuration(self._launcher)
2250
2251 if configuration['normal_user']:
2252 self._normal_user = configuration['normal_user']
2253 else:
2254- self._normal_user = self._launcher.normal_user
2255- if not self._normal_user:
2256- self._normal_user = _guess_normal_user()
2257+ self._normal_user = self._launcher.get_value(
2258+ 'daemon', 'normal_user') or _guess_normal_user()
2259 runner_kwargs = {
2260 'normal_user_provider': lambda: self._normal_user,
2261 'stdin': self._pipe_to_subproc,
2262@@ -305,7 +306,7 @@ class RemoteSessionAssistant():
2263 self._session_id = self._sa.get_session_id()
2264 tps = self._sa.get_test_plans()
2265 filtered_tps = set()
2266- for filter in self._launcher.test_plan_filters:
2267+ for filter in self._launcher.get_value('test plan', 'filter'):
2268 filtered_tps.update(fnmatch.filter(tps, filter))
2269 filtered_tps = list(filtered_tps)
2270 response = zip(filtered_tps, [self._sa.get_test_plan(
2271@@ -335,10 +336,11 @@ class RemoteSessionAssistant():
2272 def finish_bootstrap(self):
2273 self._sa.finish_bootstrap()
2274 self._state = Bootstrapped
2275- if self._launcher.auto_retry:
2276+ if self._launcher.get_value('ui', 'auto_retry'):
2277 for job_id in self._sa.get_static_todo_list():
2278 job_state = self._sa.get_job_state(job_id)
2279- job_state.attempts = self._launcher.max_attempts
2280+ job_state.attempts = self._launcher.get_value(
2281+ 'ui', 'max_attempts')
2282 return self._sa.get_static_todo_list()
2283
2284 def get_manifest_repr(self):
2285@@ -363,10 +365,12 @@ class RemoteSessionAssistant():
2286
2287 def _get_ui_for_job(self, job):
2288 show_out = True
2289- if self._launcher.output == 'hide-resource-and-attachment':
2290+ if self._launcher.get_value(
2291+ 'ui', 'output') == 'hide-resource-and-attachment':
2292 if job.plugin in ('local', 'resource', 'attachment'):
2293 show_out = False
2294- elif self._launcher.output in ['hide', 'hide-automated']:
2295+ elif self._launcher.get_value(
2296+ 'ui', 'output') in ['hide', 'hide-automated']:
2297 if job.plugin in ('shell', 'local', 'resource', 'attachment'):
2298 show_out = False
2299 if 'suppress-output' in job.get_flag_set():
2300@@ -558,7 +562,7 @@ class RemoteSessionAssistant():
2301 if self._state != Bootstrapping:
2302 if not self._sa.get_dynamic_todo_list():
2303 if (
2304- self._launcher.auto_retry and
2305+ self._launcher.get_value('ui', 'auto_retry') and
2306 self.get_rerun_candidates('auto')
2307 ):
2308 self._state = TestsSelected
2309@@ -644,11 +648,12 @@ class RemoteSessionAssistant():
2310 meta = self._sa.resume_session(session_id, runner_kwargs=runner_kwargs)
2311 app_blob = json.loads(meta.app_blob.decode("UTF-8"))
2312 launcher = app_blob['launcher']
2313- self._launcher.read_string(launcher, False)
2314+ self._launcher = Configuration.from_text(launcher, 'Remote launcher')
2315 self._sa.use_alternate_configuration(self._launcher)
2316
2317 self._normal_user = app_blob.get(
2318- 'effective_normal_user', self._launcher.normal_user)
2319+ 'effective_normal_user', self._launcher.get_value(
2320+ 'daemon', 'normal_user'))
2321 _logger.info(
2322 "normal_user after loading metadata: %r", self._normal_user)
2323 test_plan_id = app_blob['testplan_id']
2324@@ -684,12 +689,12 @@ class RemoteSessionAssistant():
2325
2326 # some jobs have already been run, so we need to update the attempts
2327 # count for future auto-rerunning
2328- if self._launcher.auto_retry:
2329+ if self._launcher.get_value('ui', 'auto_retry'):
2330 for job_id in [
2331 job.id for job in self.get_rerun_candidates('auto')]:
2332 job_state = self._sa.get_job_state(job_id)
2333- job_state.attempts = self._launcher.max_attempts - len(
2334- job_state.result_history)
2335+ job_state.attempts = self._launcher.get_value(
2336+ 'ui', 'max_attempts') - len(job_state.result_history)
2337
2338 self._state = TestsSelected
2339
2340diff --git a/plainbox/impl/test_config.py b/plainbox/impl/test_config.py
2341new file mode 100644
2342index 0000000..fce0700
2343--- /dev/null
2344+++ b/plainbox/impl/test_config.py
2345@@ -0,0 +1,130 @@
2346+# This file is part of Checkbox.
2347+#
2348+# Copyright 2020 Canonical Ltd.
2349+# Written by:
2350+# Maciej Kisielewski <maciej.kisielewski@canonical.com>
2351+#
2352+# Checkbox is free software: you can redistribute it and/or modify
2353+# it under the terms of the GNU General Public License version 3,
2354+# as published by the Free Software Foundation.
2355+#
2356+# Checkbox is distributed in the hope that it will be useful,
2357+# but WITHOUT ANY WARRANTY; without even the implied warranty of
2358+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2359+# GNU General Public License for more details.
2360+#
2361+# You should have received a copy of the GNU General Public License
2362+# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
2363+"""
2364+This module contains tests for the new Checkbox Config module
2365+"""
2366+from contextlib import contextmanager
2367+import logging
2368+from unittest import TestCase
2369+from unittest.mock import mock_open, patch
2370+
2371+from plainbox.impl.config import Configuration
2372+
2373+
2374+@contextmanager
2375+def muted_logging():
2376+ """Disable logging so the test that use this have no output."""
2377+ saved_level = logging.root.getEffectiveLevel()
2378+ logging.root.setLevel(logging.CRITICAL)
2379+ yield
2380+ logging.root.setLevel(saved_level)
2381+
2382+
2383+class ConfigurationTests(TestCase):
2384+ """Tests for the Configuration class."""
2385+
2386+ def test_empty_file_yields_defaults(self):
2387+ """A default configuration instance should have default values."""
2388+ # let's check a few values from random sections
2389+ cfg = Configuration()
2390+ self.assertEqual(cfg.get_value('test plan', 'filter'), ['*'])
2391+ self.assertTrue(cfg.get_value('launcher', 'local_submission'))
2392+ self.assertEqual(cfg.get_value('daemon', 'normal_user'), '')
2393+
2394+ @patch('os.path.isfile', return_value=True)
2395+ def test_one_var_overwrites(self, _):
2396+ """
2397+ One variable properly shadows defaults.
2398+
2399+ Having one (good) value in config should yield a config with
2400+ defaults except the one var placed in the config file.
2401+ """
2402+ ini_data = """
2403+ [launcher]
2404+ stock_reports = text
2405+ """
2406+ with patch('builtins.open', mock_open(read_data=ini_data)):
2407+ cfg = Configuration.from_path('unit test')
2408+ self.assertEqual(cfg.get_value('test plan', 'filter'), ['*'])
2409+ self.assertTrue(cfg.get_value('launcher', 'local_submission'))
2410+ self.assertEqual(cfg.get_value('daemon', 'normal_user'), '')
2411+ self.assertEqual(cfg.get_origin('daemon', 'normal_user'), '')
2412+ self.assertEqual(cfg.get_value('launcher', 'stock_reports'), ['text'])
2413+ self.assertEqual(
2414+ cfg.get_origin('launcher', 'stock_reports'),
2415+ 'unit test')
2416+
2417+ @patch('os.path.isfile', return_value=True)
2418+ def test_string_list_distinction(self, _):
2419+ """
2420+ Parsing of lists and multi-word strings.
2421+
2422+ Depending on the config spec the field can be considered a string
2423+ (with spaces) or a list.
2424+ """
2425+ ini_data = """
2426+ [launcher]
2427+ launcher_version = 1
2428+ stock_reports = submission_files, text
2429+ session_title = A session title
2430+ """
2431+ with patch('builtins.open', mock_open(read_data=ini_data)):
2432+ cfg = Configuration.from_path('unit test')
2433+ self.assertEqual(
2434+ cfg.get_value('launcher', 'stock_reports'),
2435+ ['submission_files', 'text'])
2436+ self.assertEqual(
2437+ cfg.get_value('launcher', 'session_title'),
2438+ 'A session title')
2439+
2440+ @patch('os.path.isfile', return_value=True)
2441+ def test_unexpected_content(self, _):
2442+ """
2443+ Yield problems with extra data in configs.
2444+ """
2445+ ini_data = """
2446+ [launcher]
2447+ barfoo = 5
2448+ [foobar]
2449+ """
2450+ with muted_logging():
2451+ with patch('builtins.open', mock_open(read_data=ini_data)):
2452+ cfg = Configuration.from_path('unit test')
2453+ self.assertEqual(len(cfg.get_problems()), 2)
2454+
2455+ def test_default_vars_are_not_supported(self):
2456+ """
2457+ Yield a problem when variable is defined in the [DEFAULT] section.
2458+ """
2459+ ini_data = """
2460+ [DEFAULT]
2461+ badvar = 4
2462+ """
2463+ with muted_logging():
2464+ with patch('builtins.open', mock_open(read_data=ini_data)):
2465+ cfg = Configuration.from_path('unit test')
2466+ self.assertEqual(len(cfg.get_problems()), 1)
2467+
2468+ @patch('os.path.isfile', return_value=False)
2469+ def test_ini_not_found(self, _):
2470+ """
2471+ Yield a problem when an ini file cannot be opened.
2472+ """
2473+ with muted_logging():
2474+ cfg = Configuration.from_path('invalid path')
2475+ self.assertEqual(len(cfg.get_problems()), 1)
2476diff --git a/plainbox/impl/test_launcher.py b/plainbox/impl/test_launcher.py
2477deleted file mode 100644
2478index 47b47d1..0000000
2479--- a/plainbox/impl/test_launcher.py
2480+++ /dev/null
2481@@ -1,136 +0,0 @@
2482-# This file is part of Checkbox.
2483-#
2484-# Copyright 2016 Canonical Ltd.
2485-# Written by:
2486-# Maciej Kisielewski <maciej.kisielewski@canonical.com>
2487-#
2488-# Checkbox is free software: you can redistribute it and/or modify
2489-# it under the terms of the GNU General Public License version 3,
2490-# as published by the Free Software Foundation.
2491-#
2492-# Checkbox is distributed in the hope that it will be useful,
2493-# but WITHOUT ANY WARRANTY; without even the implied warranty of
2494-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
2495-# GNU General Public License for more details.
2496-#
2497-# You should have received a copy of the GNU General Public License
2498-# along with Checkbox. If not, see <http://www.gnu.org/licenses/>.
2499-
2500-"""
2501-plainbox.impl.test_launcher
2502-=========================
2503-
2504-Test definitions for plainbox.imlp.launcher module
2505-"""
2506-
2507-from unittest import TestCase
2508-from textwrap import dedent
2509-
2510-from plainbox.impl.secure.config import Unset
2511-
2512-from plainbox.impl.launcher import LauncherDefinition
2513-from plainbox.impl.launcher import LauncherDefinition1
2514-
2515-
2516-class LauncherDefinitionTests(TestCase):
2517- launcher_version_legacy = dedent("""
2518- [launcher]
2519- """)
2520- launcher_version_1 = dedent("""
2521- [launcher]
2522- launcher_version = 1
2523- """)
2524- launcher_version_future = dedent("""
2525- [launcher]
2526- launcher_version = 2
2527- """)
2528-
2529- def test_get_concrete_launcher_legacy(self):
2530- l = LauncherDefinition()
2531- l.read_string(self.launcher_version_legacy)
2532- with self.assertRaises(KeyError):
2533- l.get_concrete_launcher()
2534-
2535- def test_get_concrete_launcher_launcher1(self):
2536- l = LauncherDefinition()
2537- l.read_string(self.launcher_version_1)
2538- cls = l.get_concrete_launcher().__class__
2539- self.assertIs(cls, LauncherDefinition1)
2540-
2541- def test_get_concrete_launcher_future_raises(self):
2542- l = LauncherDefinition()
2543- l.read_string(self.launcher_version_future)
2544- with self.assertRaises(KeyError):
2545- l.get_concrete_launcher()
2546-
2547-
2548-class LauncherDefinition1Tests(TestCase):
2549-
2550- def test_defaults(self):
2551- empty_launcher = dedent("""
2552- [launcher]
2553- launcher_version = 1
2554- """)
2555- l = LauncherDefinition1()
2556- l.read_string(empty_launcher)
2557- self.assertEqual(l.api_version, '0.99')
2558- self.assertEqual(l.app_id, 'checkbox-cli')
2559- self.assertEqual(l.api_flags, ['restartable'])
2560- self.assertEqual(l.test_plan_filters, ['*'])
2561- self.assertEqual(l.test_plan_default_selection, Unset)
2562- self.assertEqual(l.test_plan_forced, False)
2563- self.assertEqual(l.test_selection_forced, False)
2564- self.assertEqual(l.ui_type, 'interactive')
2565- self.assertEqual(l.auto_retry, False)
2566- self.assertEqual(l.max_attempts, 3)
2567- self.assertEqual(l.delay_before_retry, 1)
2568- self.assertEqual(l.restart_strategy, Unset)
2569-
2570- def test_smoke(self):
2571- definition = dedent("""
2572- [launcher]
2573- launcher_version = 1
2574- api_version = 0.99
2575- api_flags = restartable
2576- app_id = FOOBAR
2577- [test plan]
2578- unit = 2000.the.chosen.one
2579- filter = 2000*, 3000* tp_foo*
2580- forced = yes
2581- [test selection]
2582- forced = yes
2583- [ui]
2584- type = silent
2585- auto_retry = yes
2586- max_attempts = 5
2587- delay_before_retry = 60
2588- [restart]
2589- strategy = magic
2590- [report:foo_report]
2591- exporter = bar_exporter
2592- transport = file
2593- [exporter:bar_exporter]
2594- unit = bar_exporter_unit
2595- [transport:file]
2596- path = /tmp/path
2597- """)
2598- l = LauncherDefinition1()
2599- l.read_string(definition)
2600- self.assertEqual(l.api_version, '0.99')
2601- self.assertEqual(l.app_id, 'FOOBAR')
2602- self.assertEqual(l.api_flags, ['restartable'])
2603- self.assertEqual(l.test_plan_filters, ['2000*', '3000*', 'tp_foo*'])
2604- self.assertEqual(l.test_plan_default_selection, '2000.the.chosen.one')
2605- self.assertEqual(l.test_plan_forced, True)
2606- self.assertEqual(l.test_selection_forced, True)
2607- self.assertEqual(l.ui_type, 'silent')
2608- self.assertEqual(l.auto_retry, True)
2609- self.assertEqual(l.max_attempts, 5)
2610- self.assertEqual(l.delay_before_retry, 60)
2611- self.assertEqual(l.restart_strategy, 'magic')
2612- self.assertEqual(l.reports, {
2613- 'foo_report': {'exporter': 'bar_exporter', 'transport': 'file'}})
2614- self.assertEqual(l.exporters, {
2615- 'bar_exporter': {'unit': 'bar_exporter_unit'}})
2616- self.assertEqual(l.transports, {
2617- 'file': {'path': '/tmp/path'}})
2618diff --git a/plainbox/impl/unit/unit.py b/plainbox/impl/unit/unit.py
2619index cd7b365..8041683 100644
2620--- a/plainbox/impl/unit/unit.py
2621+++ b/plainbox/impl/unit/unit.py
2622@@ -36,7 +36,6 @@ from jinja2 import Template
2623 from plainbox.i18n import gettext as _
2624 from plainbox.impl.decorators import cached_property
2625 from plainbox.impl.decorators import instance_method_lru_cache
2626-from plainbox.impl.secure.config import Unset
2627 from plainbox.impl.secure.origin import Origin
2628 from plainbox.impl.secure.rfc822 import normalize_rfc822_value
2629 from plainbox.impl.symbol import Symbol
2630@@ -609,7 +608,7 @@ class Unit(metaclass=UnitType):
2631
2632 @instance_method_lru_cache(maxsize=None)
2633 def _checkbox_env(self):
2634- if self.config is not None and self.config.environment is not Unset:
2635+ if self.config is not None and self.config.environment:
2636 return self.config.environment
2637 else:
2638 return {}

Subscribers

People subscribed via source and target branches