Merge ~sylvain-pineau/plainbox:xmas_diet into plainbox:master
- Git
- lp:~sylvain-pineau/plainbox
- xmas_diet
- Merge into master
Status: | Merged |
---|---|
Approved by: | Sylvain Pineau |
Approved revision: | b9e6cf83e09bc42c03ed9749c9dd149e7c86620c |
Merged at revision: | 638986c4a11114b188cbc61f754ebafebf78b36a |
Proposed branch: | ~sylvain-pineau/plainbox:xmas_diet |
Merge into: | plainbox:master |
Diff against target: |
19011 lines (+217/-3860) 78 files modified
MANIFEST.in (+0/-1) dev/null (+0/-459) docs/author/index.rst (+0/-2) docs/author/intro.rst (+4/-169) docs/author/provider-files.rst (+0/-6) docs/author/provider-namespaces.rst (+4/-4) docs/author/provider-template.rst (+7/-29) docs/author/providers.rst (+1/-1) docs/dev/old.rst (+0/-54) docs/dev/trusted-launcher.rst (+5/-9) docs/glossary.rst (+5/-6) docs/manpages/plainbox-dev-analyze.rst (+5/-12) docs/manpages/plainbox-exporter-units.rst (+1/-33) docs/manpages/plainbox-file-units.rst (+1/-4) docs/manpages/plainbox-job-units.rst (+0/-3) docs/manpages/plainbox-run.rst (+5/-75) docs/manpages/plainbox-test-plan-units.rst (+5/-6) docs/manpages/plainbox-trusted-launcher-1.rst (+1/-6) docs/usage.rst (+2/-13) plainbox/__init__.py (+2/-8) plainbox/abc.py (+4/-22) plainbox/impl/applogic.py (+0/-20) plainbox/impl/buildsystems.py (+0/-7) plainbox/impl/commands/__init__.py (+1/-1) plainbox/impl/commands/cmd_analyze.py (+0/-11) plainbox/impl/commands/cmd_checkbox.py (+0/-9) plainbox/impl/commands/inv_analyze.py (+0/-46) plainbox/impl/commands/inv_checkbox.py (+0/-52) plainbox/impl/commands/inv_run.py (+4/-9) plainbox/impl/commands/inv_special.py (+0/-3) plainbox/impl/commands/test_run.py (+3/-8) plainbox/impl/ctrl.py (+3/-115) plainbox/impl/depmgr.py (+1/-1) plainbox/impl/exporter/__init__.py (+0/-9) plainbox/impl/exporter/tar.py (+1/-2) plainbox/impl/exporter/test_html.py (+1/-1) plainbox/impl/exporter/test_init.py (+0/-2) plainbox/impl/exporter/xlsx.py (+2/-48) plainbox/impl/highlevel.py (+1/-6) plainbox/impl/launcher.py (+1/-87) plainbox/impl/providers/__init__.py (+1/-12) plainbox/impl/providers/exporters/data/checkbox.html (+1/-1) plainbox/impl/providers/exporters/data/checkbox.json (+1/-1) plainbox/impl/providers/exporters/units/exporter.pxu (+1/-7) plainbox/impl/providers/stubbox/units/jobs/representative.pxu (+0/-9) plainbox/impl/providers/stubbox/units/jobs/stub.pxu (+0/-24) plainbox/impl/result.py (+1/-25) plainbox/impl/runner.py (+3/-40) plainbox/impl/secure/launcher1.py (+5/-9) plainbox/impl/secure/providers/__init__.py (+1/-12) plainbox/impl/secure/providers/test_v1.py (+18/-94) plainbox/impl/secure/providers/v1.py (+16/-203) plainbox/impl/secure/qualifiers.py (+1/-223) plainbox/impl/secure/test_launcher1.py (+10/-37) plainbox/impl/secure/test_qualifiers.py (+0/-231) plainbox/impl/session/assistant.py (+5/-6) plainbox/impl/session/jobs.py (+1/-1) plainbox/impl/session/manager.py (+8/-29) plainbox/impl/session/state.py (+5/-16) plainbox/impl/session/storage.py (+38/-471) plainbox/impl/session/test_manager.py (+3/-3) plainbox/impl/session/test_resume.py (+1/-302) plainbox/impl/session/test_state.py (+3/-64) plainbox/impl/session/test_storage.py (+4/-71) plainbox/impl/session/test_suspend.py (+0/-217) plainbox/impl/test_box.py (+0/-67) plainbox/impl/test_ctrl.py (+4/-134) plainbox/impl/test_launcher.py (+2/-24) plainbox/impl/unit/concrete_validators.py (+0/-5) plainbox/impl/unit/file.py (+0/-1) plainbox/impl/unit/job.py (+2/-21) plainbox/impl/unit/test_job.py (+1/-12) plainbox/impl/unit/testplan.py (+3/-3) plainbox/impl/xparsers.py (+0/-98) plainbox/impl/xscanners.py (+1/-1) plainbox/provider_manager.py (+5/-19) plainbox/test_provider_manager.py (+1/-3) setup.py (+1/-5) |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Devices Certification Bot | Needs Fixing | ||
Maciej Kisielewski | Approve | ||
Review via email: mp+335580@code.launchpad.net |
Commit message
Description of the change
Cleanup of obsolete code (whitelists, local jobs, py3.2)
Devices Certification Bot (ce-certification-qa) wrote : | # |
The merge was fine but running tests failed.
[trusty] starting container
[trusty] (timing) 0.00user 0.00system 0:00.14elapsed 5%CPU (0avgtext+0avgdata 4960maxresident)k
[trusty] (timing) 432inputs+72outputs (1major+
[trusty] provisioning container
[trusty] (timing) 38.72user 12.08system 2:06.54elapsed 40%CPU (0avgtext+0avgdata 68076maxresident)k
[trusty] (timing) 591960inputs+
[trusty-testing] Starting tests...
Found a test script: ./requirements/
[trusty-testing] 001-container-
[trusty-testing] (timing) 0.29user 0.03system 0:00.34elapsed 94%CPU (0avgtext+0avgdata 22680maxresident)k
[trusty-testing] (timing) 760inputs+96outputs (17major+
Found a test script: ./requirements/
[trusty-testing] container-
[trusty-testing] (timing) 37.84user 2.28system 0:45.87elapsed 87%CPU (0avgtext+0avgdata 68532maxresident)k
[trusty-testing] (timing) 18768inputs+
Found a test script: ./requirements/
[trusty-testing] container-
[trusty-testing] stdout: https:/
[trusty-testing] stderr: https:/
[trusty-testing] (timing) Command exited with non-zero status 1
[trusty-testing] (timing) 1.40user 0.07system 0:01.55elapsed 94%CPU (0avgtext+0avgdata 52832maxresident)k
[trusty-testing] (timing) 11544inputs+
Found a test script: ./requirements/
[trusty-testing] container-
[trusty-testing] (timing) 0.91user 0.06system 0:01.02elapsed 95%CPU (0avgtext+0avgdata 39536maxresident)k
[trusty-testing] (timing) 56inputs+112outputs (0major+
Found a test script: ./requirements/
[trusty-testing] container-
[trusty-testing] (timing) 1.12user 0.05system 0:01.20elapsed 98%CPU (0avgtext+0avgdata 30060maxresident)k
[trusty-testing] (timing) 264inputs+8outputs (1major+
[trusty-testing] Fixing file permissions in source directory
[trusty-testing] Destroying container
Name: trusty-testing
State: STOPPED
[xenial] starting container
[xenial] (timing) 0.00user 0.00system 0:00.13elapsed 5%CPU (0avgtext+0avgdata 4984maxresident)k
[xenial] (timing) 496inputs+72outputs (1major+
[xenial] provisioning container
[xenial] (timing) 42.40user 7.98system 1:44.10elapsed 48%CPU (0avgtext+0avgdata 81308maxresident)k
[xenial] (timing) 725800inputs+
[xenial-testing] Starting tests...
Found a test script: ./requirements/
[xenial-testing] 001-container-
[xenial-testing] (timing) 0.34user 0.04system 0:00.40elapsed 94%CPU (0avgtext+0avgdata 23860ma...
Preview Diff
1 | diff --git a/MANIFEST.in b/MANIFEST.in |
2 | index e51f963..1eb7c3b 100644 |
3 | --- a/MANIFEST.in |
4 | +++ b/MANIFEST.in |
5 | @@ -35,7 +35,6 @@ include plainbox/impl/providers/stubbox/po/*.pot |
6 | include plainbox/impl/providers/stubbox/po/POTFILES.in |
7 | include plainbox/impl/providers/stubbox/units/jobs/*.pxu |
8 | include plainbox/impl/providers/stubbox/units/testplans/*.pxu |
9 | -include plainbox/impl/providers/stubbox/whitelists/*.whitelist |
10 | |
11 | include plainbox/vendor/argparse/py*-argparse.py |
12 | |
13 | diff --git a/daily-package-testing/README b/daily-package-testing/README |
14 | deleted file mode 100644 |
15 | index 27d3e8e..0000000 |
16 | --- a/daily-package-testing/README |
17 | +++ /dev/null |
18 | @@ -1,18 +0,0 @@ |
19 | -About |
20 | -===== |
21 | - |
22 | -This Vagrantfile can be used to see how packaged version of plainbox from the |
23 | -ppa looks like. |
24 | - |
25 | -The package is built with this recipe: |
26 | -https://code.launchpad.net/~checkbox-dev/+recipe/plainbox-daily |
27 | - |
28 | -Usage instructions |
29 | -================== |
30 | - |
31 | -$TARGET is either precise or quantal |
32 | - |
33 | -$ vagrant up $TARGET |
34 | -$ vagrant ssh $TARGET |
35 | -$ plainbox --help |
36 | -$ exit |
37 | diff --git a/daily-package-testing/Vagrantfile b/daily-package-testing/Vagrantfile |
38 | deleted file mode 100644 |
39 | index 28213a8..0000000 |
40 | --- a/daily-package-testing/Vagrantfile |
41 | +++ /dev/null |
42 | @@ -1,48 +0,0 @@ |
43 | -# -*- mode: ruby -*- |
44 | -# vi: set ft=ruby sw=2 ts=2 : |
45 | - |
46 | -Vagrant::Config.run do |config| |
47 | - |
48 | - config.ssh.timeout = 60 |
49 | - |
50 | - # Define a Ubuntu Server image (cloud) for the 12.04 release (precise) |
51 | - config.vm.define :precise do |precise_config| |
52 | - precise_config.vm.box = "precise-cloud-i386" |
53 | - precise_config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box" |
54 | - end |
55 | - |
56 | - # Define a Ubuntu Server image (cloud) for the 12.10 release (quantal) |
57 | - config.vm.define :quantal do |quantal_config| |
58 | - quantal_config.vm.box = "quantal-cloud-i386" |
59 | - quantal_config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/quantal/current/quantal-server-cloudimg-i386-vagrant-disk1.box" |
60 | - end |
61 | - |
62 | - # Define a Ubuntu Server image (cloud) for the 13.04 release (raring) |
63 | - config.vm.define :raring do |raring_config| |
64 | - raring_config.vm.box = "raring-cloud-i386" |
65 | - raring_config.vm.box_url = "http://cloud-images.ubuntu.com/vagrant/raring/current/raring-server-cloudimg-i386-vagrant-disk1.box" |
66 | - end |
67 | - |
68 | - # For debugging and later future GUI testing |
69 | - if ENV.key? "VAGRANT_GUI" |
70 | - config.vm.boot_mode = :gui |
71 | - end |
72 | - |
73 | - # Setup an apt cache if one is available |
74 | - if ENV.key? "VAGRANT_APT_CACHE" |
75 | - config.vm.provision :shell, :inline => "echo 'Acquire::http { Proxy \"#{ENV['VAGRANT_APT_CACHE']}\"; };' > /etc/apt/apt.conf" |
76 | - end |
77 | - |
78 | - # Update to have the latest packages, this is needed because the image comes |
79 | - # with an old (and no longer working) apt cache and links to many packages no |
80 | - # longer work. |
81 | - config.vm.provision :shell, :inline => "apt-get update && DEBIAN_FRONTEND=noninteractive apt-get dist-upgrade --yes" |
82 | - # Install package that provides add-apt-repository |
83 | - config.vm.provision :shell, :inline => "apt-get install --yes python-software-properties" |
84 | - # Add the checkbox-dev/ppa ppa with daily builds |
85 | - config.vm.provision :shell, :inline => "add-apt-repository ppa:checkbox-dev/ppa" |
86 | - # Update apt cache again |
87 | - config.vm.provision :shell, :inline => "apt-get update" |
88 | - # Install plainbox |
89 | - config.vm.provision :shell, :inline => "apt-get install --yes plainbox" |
90 | -end |
91 | diff --git a/daily-package-testing/test-in-vagrant.sh b/daily-package-testing/test-in-vagrant.sh |
92 | deleted file mode 100755 |
93 | index c21b00f..0000000 |
94 | --- a/daily-package-testing/test-in-vagrant.sh |
95 | +++ /dev/null |
96 | @@ -1,59 +0,0 @@ |
97 | -#!/bin/sh |
98 | -# Runs simple smoke tests on the current package in the daily ppa. |
99 | - |
100 | -mkdir -p vagrant-logs |
101 | - |
102 | -test -z $(which vagrant) && echo "You need to install vagrant first" && exit |
103 | - |
104 | -outcome=0 |
105 | -# XXX: this list needs to be in sync with plainbox/daily-pkg-testing/Vagrantfile |
106 | -target_list="precise quantal raring" |
107 | -for target in $target_list; do |
108 | - # Bring up target if needed |
109 | - if ! vagrant status $target | grep -q running; then |
110 | - echo "[$target] Bringing VM 'up'" |
111 | - if ! vagrant up $target >vagrant-logs/$target.startup.log 2>vagrant-logs/$target.startup.err; then |
112 | - outcome=1 |
113 | - echo "[$target] Unable to 'up' VM!" |
114 | - echo "[$target] stdout: $(pastebinit vagrant-logs/$target.startup.log)" |
115 | - echo "[$target] stderr: $(pastebinit vagrant-logs/$target.startup.err)" |
116 | - echo "[$target] NOTE: unable to execute tests, marked as failed" |
117 | - continue |
118 | - fi |
119 | - fi |
120 | - # Display something before the first test output |
121 | - echo "[$target] Starting tests..." |
122 | - # Test that plainbox --help works correctly |
123 | - if vagrant ssh $target -c 'plainbox --help' >vagrant-logs/$target.help.log 2>vagrant-logs/$target.help.err; then |
124 | - echo "[$target] packaged PlainBox plainbox --help: pass" |
125 | - else |
126 | - outcome=1 |
127 | - echo "[$target] packaged PlainBox plainbox --help: fail" |
128 | - echo "[$target] stdout: $(pastebinit vagrant-logs/$target.help.log)" |
129 | - echo "[$target] stderr: $(pastebinit vagrant-logs/$target.help.err)" |
130 | - fi |
131 | - case $VAGRANT_DONE_ACTION in |
132 | - suspend) |
133 | - # Suspend the target to conserve resources |
134 | - echo "[$target] Suspending VM" |
135 | - if ! vagrant suspend $target >vagrant-logs/$target.suspend.log 2>vagrant-logs/$target.suspend.err; then |
136 | - echo "[$target] Unable to suspend VM!" |
137 | - echo "[$target] stdout: $(pastebinit vagrant-logs/$target.suspend.log)" |
138 | - echo "[$target] stderr: $(pastebinit vagrant-logs/$target.suspend.err)" |
139 | - echo "[$target] You may need to manually 'vagrant destroy $target' to fix this" |
140 | - fi |
141 | - ;; |
142 | - destroy) |
143 | - # Destroy the target to work around virtualbox hostsf bug |
144 | - echo "[$target] Destroying VM" |
145 | - if ! vagrant destroy --force $target >vagrant-logs/$target.destroy.log 2>vagrant-logs/$target.destroy.err; then |
146 | - echo "[$target] Unable to destroy VM!" |
147 | - echo "[$target] stdout: $(pastebinit vagrant-logs/$target.suspend.log)" |
148 | - echo "[$target] stderr: $(pastebinit vagrant-logs/$target.suspend.err)" |
149 | - echo "[$target] You may need to manually 'vagrant destroy $target' to fix this" |
150 | - fi |
151 | - ;; |
152 | - esac |
153 | -done |
154 | -# Propagate failure code outside |
155 | -exit $outcome |
156 | diff --git a/docs/author/index.rst b/docs/author/index.rst |
157 | index 8e213ae..b22bb46 100644 |
158 | --- a/docs/author/index.rst |
159 | +++ b/docs/author/index.rst |
160 | @@ -8,10 +8,8 @@ core. |
161 | |
162 | .. toctree:: |
163 | intro.rst |
164 | - tutorial.rst |
165 | qml-job-tutorial.rst |
166 | providers.rst |
167 | - whitelists.rst |
168 | rfc822.rst |
169 | faq.rst |
170 | |
171 | diff --git a/docs/author/intro.rst b/docs/author/intro.rst |
172 | index 2bbabb7..2dae15d 100644 |
173 | --- a/docs/author/intro.rst |
174 | +++ b/docs/author/intro.rst |
175 | @@ -49,7 +49,7 @@ Terminology |
176 | In developing or using Plainbox, you'll run into several unfamiliar terms. |
177 | Check the :doc:`../glossary` to learn what they mean. In fact, you should |
178 | probably check it now. Pay particular attention to the terms *Checkbox*, |
179 | -*Plainbox*, *job*, *provider*, and *whitelist*. |
180 | +*Plainbox*, *job* and *provider*. |
181 | |
182 | Getting Started |
183 | --------------- |
184 | @@ -73,7 +73,7 @@ tests. First up is a welcome screen: |
185 | select tests. |
186 | |
187 | When you press the Enter key, ``checkbox-cli`` lets you select which |
188 | -whitelist to use: |
189 | +test plan to use: |
190 | |
191 | .. image:: cc2.png |
192 | :height: 343 |
193 | @@ -81,7 +81,7 @@ whitelist to use: |
194 | :scale: 100 |
195 | :alt: checkbox-cli enables you to select which test suite to run. |
196 | |
197 | -With a whitelist selected, you can choose the individual tests to run: |
198 | +With a test plan selected, you can choose the individual tests to run: |
199 | |
200 | .. image:: cc3.png |
201 | :height: 600 |
202 | @@ -117,7 +117,7 @@ A provider is described in a configuration file (stored in |
203 | ``/usr/share/plainbox-providers-1``). This file describes where to find all |
204 | the files from the provider. This file is usually managed automatically |
205 | (more on this later). A provider can ship jobs, binaries, data and |
206 | -whitelists. |
207 | +test plans. |
208 | |
209 | A **job** or **test** is the smallest unit or description that Plainbox |
210 | knows about. It describes a single test (historically they're called |
211 | @@ -140,10 +140,6 @@ types include (but are not limited to): |
212 | * ``shell`` -- An automated test that requires no user interaction; the |
213 | test is passed or failed on the basis of the return value of the script |
214 | or command. |
215 | - * ``local`` -- This type of job is similar to a ``shell`` test, but it |
216 | - supports creating multiple tests from a single definition (say, to test |
217 | - all the Ethernet ports on a computer). Jobs using the ``local`` plugin |
218 | - are run when Plainbox is initialized. |
219 | * ``user-interact`` -- A test that asks the user to perform some action |
220 | *before* the test is performed. The test then passes or fails |
221 | automatically based on the output of the test. An example is |
222 | @@ -162,170 +158,9 @@ types include (but are not limited to): |
223 | which probes the system to determine the maximum supported resolution |
224 | and then asks the user to confirm that the resolution is correct. |
225 | |
226 | -A fairly complex example definition is:: |
227 | - |
228 | - plugin: local |
229 | - _summary: Automated test to walk multiple network cards and test each one in sequence. |
230 | - id: ethernet/multi_nic |
231 | - requires: |
232 | - device.category == 'NETWORK' |
233 | - _description: Automated test to walk multiple network cards and test each one in sequence. |
234 | - command: |
235 | - cat <<'EOF' | run_templates -s 'udev_resource | filter_templates -w "category=NETWORK" | awk "/path: / { print \$2 }" | xargs -n 1 sh -c "for i in \``ls /sys\$0/net 2>/dev/null\``; do echo \$0 \$i; done"' |
236 | - plugin: shell |
237 | - id: ethernet/multi_nic_$2 |
238 | - requires: |
239 | - package.name == 'ethtool' |
240 | - package.name == 'nmap' |
241 | - device.path == "$1" |
242 | - user: root |
243 | - environ: TEST_TARGET_FTP TEST_TARGET_IPERF TEST_USER TEST_PASS |
244 | - command: network test -i $2 -t iperf --fail-threshold 80 |
245 | - estimated_duration: 330.0 |
246 | - description: |
247 | - Testing for NIC $2 |
248 | - EOF |
249 | - |
250 | -Key points to note include: |
251 | - |
252 | - * If a field name begins with an underscore, its value can be localized. |
253 | - * The values of fields can appear on the same line as their field names, |
254 | - as in ``plugin: local``; or they can appear on a subsequent line, which |
255 | - is indented, as in the preceding example's ``requires: device.category |
256 | - == 'NETWORK'``. |
257 | - * The ``requires`` field can be used to specify dependencies; if the |
258 | - specified condition is not met, the test does not run. |
259 | - * The ``command`` field specifies the command that's used to run the test. |
260 | - This can be a standard Linux command (or even a set of commands) or a |
261 | - Checkbox test script. In this example's ``local`` test definition, the |
262 | - first ``command`` line generates a list of network devices that is fed |
263 | - to an embedded test, which is defined beginning with the second |
264 | - ``plugin`` line immediately following the first ``command`` line. |
265 | - * In this example, the line that reads ``EOF`` ends the |
266 | - ``ethernet/ethtool_multi_nic_$2`` test's command; it's matched to the |
267 | - ``EOF`` that's part of ``cat << 'EOF'`` near the start of that command. |
268 | - |
269 | Each provider has a ``bin`` directory and all binaries there are available |
270 | in the path. |
271 | |
272 | -Whitelists |
273 | -`````````` |
274 | - |
275 | -In the job files we have a "universe" of known jobs. We don't normally want |
276 | -to run them all; rather we want to select a subset depending on what we're |
277 | -testing, and maybe give the user a way to fine-tune that selection. Also, |
278 | -we need a way to determine the order in which they will run, beyond what |
279 | -dependencies may provide. This is where the whitelist comes in; think of it |
280 | -as a mask or selection filter from the universe of jobs. Whitelists support |
281 | -regular expressions, and Plainbox will attempt to run tests in the order |
282 | -shown in the whitelist. Again, providers ship whitelists in a specific |
283 | -directory, and you can use ``plainbox`` to run a specific whitelist with |
284 | -the ``-w`` option. |
285 | - |
286 | -You can also use ``plainbox`` to run a test with the ``-i`` syntax. This is |
287 | -good for quickly running a job and ensuring it works well. |
288 | - |
289 | -Let's look at ``checkbox-cli`` for a moment. This is a "launcher"; it |
290 | -specifies a set of configuration options for a specific testing purpose. |
291 | -This enables us to create mini-clients for each testing purpose, without |
292 | -changing the core utility (``checkbox-launcher``). For instance, let's look |
293 | -at the launcher for ``canonical-certification-server``, which appears in |
294 | -``./providers/plainbox-provider-certification-server/launcher/canonical-certification-server`` |
295 | -in the Checkbox source tree:: |
296 | - |
297 | - #!/usr/bin/env checkbox-launcher |
298 | - [welcome] |
299 | - text = Welcome to System Certification! |
300 | - This application will gather information from your system. Then you will be |
301 | - asked manual tests to confirm that the system is working properly. Finally, |
302 | - you will be asked for the Secure ID of the computer to submit the |
303 | - information to the certification.canonical.com database. |
304 | - To learn how to create or locate the Secure ID, please see here: |
305 | - https://certification.canonical.com/ |
306 | - |
307 | - [suite] |
308 | - # Whitelist(s) displayed in the suite selection screen |
309 | - whitelist_filter = ^((network|storage|usb|virtualization)-only)|(server-(full|functional)-14.04)$ |
310 | - # Whitelist(s) pre-selected in the suite selection screen, default whitelist(s) |
311 | - whitelist_selection = ^server-full-14.04$ |
312 | - |
313 | - [transport] |
314 | - submit_to = certification |
315 | - |
316 | - [config] |
317 | - config_filename = canonical-certification.conf |
318 | - |
319 | -A launcher such as this sets up an environment that includes introductory |
320 | -text to be shown to users, a filter to determine what whitelists to present |
321 | -as options, information on where to (optionally) submit results, and a |
322 | -configuration filename. This allows each provider to ship a launcher or |
323 | -binary with which to launch its relevant tests. |
324 | - |
325 | -Developing Tests |
326 | -```````````````` |
327 | - |
328 | -One way to deliver tests via Plainbox is to start your own provider. To |
329 | -learn how to do that, see the :ref:`tutorial`. |
330 | - |
331 | -In other cases you want to add tests to the main Checkbox repository (which |
332 | -is also what we recommend to keep tests centralized, unless they're so |
333 | -purpose-specific that this makes no sense). |
334 | - |
335 | -This is a bit easier because the provider in question already exists. So |
336 | -let's get started by branching a copy of ``lp:checkbox``. In brief, you |
337 | -should change to your software development directory and type ``bzr branch |
338 | -lp:checkbox my-branch`` to create a copy of the ``checkbox`` Launchpad |
339 | -project in the ``my-branch`` subdirectory. You can then edit the files in |
340 | -that subdirectory, upload the results to your own Launchpad account, and |
341 | -request a merge. |
342 | - |
343 | -To begin, consider the files and subdirectories in the main Checkbox |
344 | -development directory (``my-branch`` if you used the preceding ``bzr`` |
345 | -command without change): |
346 | - |
347 | - * ``checkbox-gui`` -- Checkbox GUI components, used in desktop/laptop |
348 | - testing |
349 | - * ``checkbox-ng`` -- The Plainbox-based version of Checkbox |
350 | - * ``checkbox-support`` -- Support code for many providers |
351 | - * ``checkbox-touch`` -- A Checkbox frontend optimized for touch/tablet |
352 | - devices |
353 | - * ``mk-venv`` -- A symbolic link to a script used to set up an environment |
354 | - for testing Checkbox |
355 | - * ``plainbox`` -- A Python3 library and development tools at the heart of |
356 | - Plainbox |
357 | - * ``plainbox-client`` -- Unfinished Python3 interface for Checkbox |
358 | - * ``providers`` -- Provider definitions, including test scripts |
359 | - * ``README.md`` -- A file describing the contents of the subdirectory in |
360 | - greater detail |
361 | - * ``setup.py`` -- A setup script |
362 | - * ``support`` -- Support code that's not released |
363 | - * ``tarmac-verify`` -- A support script |
364 | - * ``test-in-lxc.sh`` -- A support script for testing in an LXC |
365 | - * ``test-in-vagrant.sh`` -- A support script for testing with Vagrant |
366 | - * ``test-with-coverage`` -- A link to a support script for testing with |
367 | - coverage |
368 | - * ``Vagrantfile`` -- A Vagrant configuration file |
369 | - |
370 | -Let's say I want to write a test to ensure that the ubuntu user exists in |
371 | -``/etc/passwd``. You need to remove any existing Checkbox provider |
372 | -packages, lest they interfere with your new or modified tests. The |
373 | -``setup.py`` script will set up a Plainbox development environment for you. |
374 | - |
375 | -We can write a simple job here, then add a requirement, perhaps a |
376 | -dependency, then a script in the directory. Note that scripts can be |
377 | -anything that's executable, we usually prefer either shell or Python but |
378 | -anything goes. |
379 | - |
380 | -Plainbox will supply two environment variables, ``PLAINBOX_PROVIDER_DATA`` |
381 | -and ``SHARE``, we usually try to use them in the job description only, not |
382 | -in the scripts, to keep the scripts Plainbox-agnostic if possible. |
383 | - |
384 | -Once the test is running correctly, we can create a whitelist with a few |
385 | -tests and name it. |
386 | - |
387 | -Once we get everything running correctly we can prepare and propose a merge |
388 | -request using ``bzr`` as usual. |
389 | - |
390 | Other Questions |
391 | --------------- |
392 | |
393 | diff --git a/docs/author/provider-files.rst b/docs/author/provider-files.rst |
394 | index 7a33346..34f21d4 100644 |
395 | --- a/docs/author/provider-files.rst |
396 | +++ b/docs/author/provider-files.rst |
397 | @@ -53,11 +53,6 @@ jobs_dir |
398 | Absolute pathname to a directory with :term:`job definitions <job>` |
399 | as individual ``.txt`` files using the :doc:`job file format <jobs>`. |
400 | |
401 | -whitelists_dir |
402 | - Absolute pathname to a directory with :term:`whitelists <whitelist>` |
403 | - as individual ``.whitelist`` files using the |
404 | - :doc:`whitelist format <whitelists>`. |
405 | - |
406 | bin_dir |
407 | Absolute pathname to a directory with additional executables required by |
408 | any of the job definitions. |
409 | @@ -80,7 +75,6 @@ location |
410 | Variable Default Value |
411 | ================ ===================== |
412 | jobs_dir $location/jobs |
413 | - whitelists_dir $location/whitelists |
414 | bin_dir $location/bin |
415 | data_dir $location/data |
416 | locale_dir $location/locale |
417 | diff --git a/docs/author/provider-namespaces.rst b/docs/author/provider-namespaces.rst |
418 | index 4459e50..e25036b 100644 |
419 | --- a/docs/author/provider-namespaces.rst |
420 | +++ b/docs/author/provider-namespaces.rst |
421 | @@ -120,7 +120,7 @@ The part of the provide name before the colon is used as the name-space. The |
422 | colon is *not* a part of the name-space. |
423 | |
424 | The implicit name-space is used to construct non-partial job definition names |
425 | -as well as to implicitly prefix each pattern inside :term:`whitelists <whitelist>`. |
426 | +as well as to implicitly prefix each pattern inside test plans. |
427 | |
428 | Using Explicit Name-Spaces |
429 | -------------------------- |
430 | @@ -133,15 +133,15 @@ Explicit name-spaces need to be used in two situations: |
431 | This is required as any partial ID may silently change the job it resolves |
432 | to and we didn't want to introduce that ambiguity. |
433 | |
434 | -2. When including a job from another name-space inside a whitelist, e.g.:: |
435 | +2. When including a job from another name-space inside a test plan, e.g.:: |
436 | |
437 | - ~/com.example.some:provider$ cat whitelists/cross.whitelist |
438 | + ~/com.example.some:provider$ cat units/test-plan.pxu |
439 | job-a |
440 | job-b |
441 | com\.example\.other::job-a |
442 | ~com.example.some:provider$ |
443 | |
444 | - Here the whitelist names three jobs: |
445 | + Here the test plan names three jobs: |
446 | |
447 | * com.example.some::job-a |
448 | * com.example.some::job-b |
449 | diff --git a/docs/author/provider-template.rst b/docs/author/provider-template.rst |
450 | index 91a1db3..ae5bb76 100644 |
451 | --- a/docs/author/provider-template.rst |
452 | +++ b/docs/author/provider-template.rst |
453 | @@ -23,17 +23,14 @@ The following files and directories are generated:: |
454 | ├── data |
455 | │  ├── example.dat |
456 | │  └── README.md |
457 | - ├── jobs |
458 | - │  ├── examples-intermediate.txt |
459 | - │  ├── examples-normal.txt |
460 | - │  └── examples-trivial.txt |
461 | ├── manage.py |
462 | ├── po |
463 | │  └── POTFILES.in |
464 | ├── README.md |
465 | - └── whitelists |
466 | - ├── normal.whitelist |
467 | - └── trivial.whitelist |
468 | + └── units |
469 | +   ├── examples-intermediate.txt |
470 | +   ├── examples-normal.txt |
471 | +   └── examples-trivial.txt |
472 | |
473 | Generated Content |
474 | ================= |
475 | @@ -59,10 +56,8 @@ README.md |
476 | Plainbox parlance, is the smallest piece of executable test code. Each |
477 | job has a name and a number of other attributes. |
478 | |
479 | - Jobs can be arranged in lists, test plans if you will that are known |
480 | - as "whitelists". Those are defined in the ``whitelists/`` directory, |
481 | - this time one per file. You can create as many whitelists as you need, |
482 | - referring to arbitrary subsets of your jobs. |
483 | + Jobs can be arranged in lists, test plans if you will. You can create as |
484 | + many test plans as you need, referring to arbitrary subsets of your jobs. |
485 | |
486 | Then there are the ``bin/`` and ``data/`` directories. Those are |
487 | entirely for custom content you may need. You can put arbitrary |
488 | @@ -340,7 +335,7 @@ jobs/examples-intermediate.txt |
489 | estimated_duration: 30 |
490 | |
491 | |
492 | -po/PORFILES.in |
493 | +po/POTFILES.in |
494 | -------------- |
495 | |
496 | :: |
497 | @@ -350,20 +345,3 @@ po/PORFILES.in |
498 | [type: gettext/rfc822deb] jobs/examples-normal.txt |
499 | [type: gettext/rfc822deb] jobs/examples-intermediate.txt |
500 | manage.py |
501 | - |
502 | -whitelists/trivial.whitelist |
503 | ----------------------------- |
504 | - |
505 | -:: |
506 | - |
507 | - # select two trivial jobs by directly selecting their names |
508 | - examples/trivial/always-pass |
509 | - examples/trivial/always-fail |
510 | - |
511 | -whitelists/normal.whitelist |
512 | ---------------------------- |
513 | - |
514 | -:: |
515 | - |
516 | - # use regular expression to select all normal jobs |
517 | - examples/normal/.* |
518 | diff --git a/docs/author/providers.rst b/docs/author/providers.rst |
519 | index 6640750..cc1bb6c 100644 |
520 | --- a/docs/author/providers.rst |
521 | +++ b/docs/author/providers.rst |
522 | @@ -5,7 +5,7 @@ Providers |
523 | Providers are a new feature introduced in Plainbox 0.5. They allow third party |
524 | developers to produce and maintain private and public test collections. |
525 | |
526 | -All :term:`jobs <job>` and :term:`whitelists <whitelist>` are now loaded from a provider. This |
527 | +All :term:`jobs <job>` and test plans are now loaded from a provider. This |
528 | also affects the :term:`Checkbox` project that now produces a custom user |
529 | interface and a number of providers for various purposes. |
530 | |
531 | diff --git a/docs/author/tutorial.rst b/docs/author/tutorial.rst |
532 | deleted file mode 100644 |
533 | index 892cb19..0000000 |
534 | --- a/docs/author/tutorial.rst |
535 | +++ /dev/null |
536 | @@ -1,177 +0,0 @@ |
537 | -.. _tutorial: |
538 | - |
539 | -======== |
540 | -Tutorial |
541 | -======== |
542 | - |
543 | -To best illustrate how providers work, we will walk through creating one |
544 | -step-by-step. At the end of this tutorial you will have a provider which adds |
545 | -a new :term:`whitelist`, several new jobs and the scripts and test data |
546 | -supporting those jobs. Before starting this tutorial you will need to have a |
547 | -running version of :term:`Plainbox` installed. You can either install it from |
548 | -the repositories of Debian or its derivatives by running ``apt-get install |
549 | -plainbox``, or if you prefer to work with the source, see :doc:`Getting |
550 | -started with development <../dev/intro>`. There is also a Launchpad PPA with |
551 | -the very latest development build for Ubuntu, which is `ppa:checkbox-dev/ppa`. |
552 | - |
553 | -#. To get started we create an initial template for our provider by running |
554 | - ``plainbox startprovider com.example:myprovider``. |
555 | - |
556 | -#. This will create a directory called ``com.example:myprovider``. |
557 | - Change to this directory and you will see that it contains:: |
558 | - |
559 | - /bin |
560 | - /data |
561 | - /integration-tests |
562 | - /jobs |
563 | - manage.py |
564 | - README.md |
565 | - /whitelists |
566 | - |
567 | - The ``manage.py`` script is a helper script for developing the provider. |
568 | - It provides a set of commands which assist in validating the correctness |
569 | - of the provider and making it ready for distribution. |
570 | - |
571 | -#. Let’s create some jobs first by changing to the jobs directory. It currently |
572 | - contains a file called category.txt which serves as an example of how |
573 | - jobs should look. Let’s delete it and instead create a file called |
574 | - ``myjobs.txt``. This can contain the following simple jobs:: |
575 | - |
576 | - plugin: shell |
577 | - name: myjobs/shell_command |
578 | - command: true |
579 | - _description: |
580 | - An example job that uses a command provided by the shell. |
581 | - |
582 | - plugin: shell |
583 | - name: myjobs/provider_command |
584 | - command: mycommand |
585 | - _description: |
586 | - An example job that uses a test command provided by this provider. |
587 | - |
588 | - At this point we can check that everything looks okay by running the command |
589 | - ``./manage.py info`` which displays some information about the provider. The |
590 | - output should be something like:: |
591 | - |
592 | - [Provider MetaData] |
593 | - name: com.example:myprovider |
594 | - version: 1.0 |
595 | - [Job Definitions] |
596 | - 'myjobs/builtin_command', from jobs/myjobs.txt:1-5 |
597 | - 'myjobs/provider_command', from jobs/myjobs.txt:7-11 |
598 | - [White Lists] |
599 | - 'category', from whitelists/category.whitelist:1-1 |
600 | - |
601 | - This shows all three jobs from the job file we added - great! |
602 | - |
603 | -#. Next we need to change directory to ``bin`` to add the command used by the |
604 | - job ``myjobs/this_provider_command``. We create a file there called |
605 | - ``mycommand`` which contains the following text:: |
606 | - |
607 | - #!/bin/sh |
608 | - test `cat $CHECKBOX_SHARE/data/testfile` = 'expected' |
609 | - |
610 | - This needs to be executable to be used in the job command so we need to run |
611 | - ``chmod a+x mycommand`` to make it executable. |
612 | - |
613 | - You'll notice the command uses a file in ``$CHECKBOX_SHARE/data`` - we'll |
614 | - add this file to our provider next. |
615 | - |
616 | -#. Because the command we’re using uses a file that we expect to be located in |
617 | - ``$CHECKBOX_SHARE/data``, we need to add this file to our provider so that |
618 | - after the provider is installed this file is available in that location. |
619 | - First we need to change to the directory called ``data``, then as indicated |
620 | - by the contents of the script we wrote in the previous step, we need to |
621 | - create a file there called ``testfile`` with the contents:: |
622 | - |
623 | - expected |
624 | - |
625 | - As simple as that! |
626 | - |
627 | -#. Lastly we need to add a :term:`whitelist` that utilizes the jobs we created |
628 | - earlier. We need to change to the directory called ``whitelists``. As with |
629 | - the ``jobs`` directory there is already an example file there called |
630 | - ``category.whitelist``. We can delete that and add a file called |
631 | - ``mywhitelist.whitelist``. The contents should be:: |
632 | - |
633 | - myjobs/shell_command |
634 | - myjobs/provider_command |
635 | - |
636 | - The ``miscellanea/submission_resources`` and ``graphics/glxgears`` jobs |
637 | - are from the default provider that is part of Plainbox. |
638 | - |
639 | - We can check that everything is correct with the whitelist by running the |
640 | - ``./manage.py info`` command again. The output should be like:: |
641 | - |
642 | - [Provider MetaData] |
643 | - name: com.example:myprovider |
644 | - version: 1.0 |
645 | - [Job Definitions] |
646 | - 'myjobs/builtin_command', from jobs/myjobs.txt:1-5 |
647 | - 'myjobs/provider_command', from jobs/myjobs.txt:7-11 |
648 | - [White Lists] |
649 | - 'mywhitelist', from whitelists/mywhitelist.whitelist:1-2 |
650 | - |
651 | - Our new :term:`whitelist` is listed there. |
652 | - |
653 | -#. Now we have a provider we need to test it to make sure everything is |
654 | - correct. The first thing to do is to install the provider so that it |
655 | - it visible to Plainbox. Run ``./manage.py develop`` then run |
656 | - ``plainbox dev list provider``. Your provider should be in the list |
657 | - that is displayed. |
658 | - |
659 | -#. We should also make sure the whole provider works end-to-end by running |
660 | - the :term:`whitelist` which it provides. Run the following command - |
661 | - ``plainbox run -w whitelists/mywhitelist.whitelist``. |
662 | - |
663 | -#. Assuming everything works okay, we can now package the provider for |
664 | - distribution. This involves creating a basic ``debian`` directory |
665 | - containing all of the files needed for packaging your provider. Create |
666 | - a directory called ``debian`` at the base of your provider, and then |
667 | - create the following files within it. |
668 | - |
669 | - ``compat``:: |
670 | - |
671 | - 9 |
672 | - |
673 | - ``control``:: |
674 | - |
675 | - Source: plainbox-myprovider |
676 | - Section: utils |
677 | - Priority: optional |
678 | - Maintainer: Brendan Donegan <brendan.donegan@canonical.com> |
679 | - Standards-Version: 3.9.3 |
680 | - X-Python3-Version: >= 3.2 |
681 | - Build-Depends: debhelper (>= 9.2), |
682 | - lsb-release, |
683 | - python3 (>= 3.2), |
684 | - python3-plainbox |
685 | - |
686 | - Package: plainbox-myprovider |
687 | - Architecture: all |
688 | - Depends: plainbox-provider-checkbox |
689 | - Description: My whitelist provider |
690 | - A provider for Plainbox. |
691 | - |
692 | - ``rules``:: |
693 | - |
694 | - #!/usr/bin/make -f |
695 | - %: |
696 | - dh "$@" |
697 | - |
698 | - override_dh_auto_build: |
699 | - $(CURDIR)/manage.py install |
700 | - |
701 | - Note that the ``rules`` file must be executable. Make it so with |
702 | - ``chmod a+x rules``. Also, be careful with the indentation in the |
703 | - file - all indents must be actual TAB characters, not four spaces |
704 | - for example. |
705 | - |
706 | - ``source/format``:: |
707 | - |
708 | - 3.0 (native) |
709 | - |
710 | - Finally we should create a ``changelog`` file. The easiest way to do this |
711 | - is to run the command ``dch --create 'Initial release.'``. You'll need to |
712 | - edit the field ``PACKAGE`` to the name of your provider and the field |
713 | - ``VERSION`` to something like ``0.1``. |
714 | diff --git a/docs/author/whitelists.rst b/docs/author/whitelists.rst |
715 | deleted file mode 100644 |
716 | index 7b5f162..0000000 |
717 | --- a/docs/author/whitelists.rst |
718 | +++ /dev/null |
719 | @@ -1,120 +0,0 @@ |
720 | -======================== |
721 | -Checkbox Whitelist Files |
722 | -======================== |
723 | - |
724 | -When creating a test suite for a particular purpose, it will be necessary to |
725 | -specify which tests to run and which order they should run in. For this purpose |
726 | -Checkbox provides the concept of Whitelists. |
727 | - |
728 | -Whitelist Format |
729 | -================ |
730 | - |
731 | -A whitelist is a text file containing a line-seperated sequence of patterns, |
732 | -each representing one or more 'jobs'. These patterns are in the Python regular |
733 | -expression syntax. Comments may be included in the file by starting the line |
734 | -with '#'. |
735 | - |
736 | -Minimal Whitelist File |
737 | -====================== |
738 | - |
739 | -In order to be useful a whitelist file needs to include a particular subset of |
740 | -jobs which provide Checkbox with all of the information it needs to run tests |
741 | -properly. These include jobs which attach hardware information and resource |
742 | -jobs which provide other jobs with information of the environment they a |
743 | -re running in (available hardware, available packages etc). To make this easy |
744 | -to do a single job exists whose purpose is to execute all of these other jobs:: |
745 | - |
746 | - miscellanea/submission-resources |
747 | - |
748 | -This should be included as the first job in any whitelist. |
749 | - |
750 | -Job Categories |
751 | -============== |
752 | - |
753 | -In order to allow Checkbox to display jobs by category in the UI it is |
754 | -necessary to include a particular local job which itself generates jobs which |
755 | -belong to that category. This job will normally look like ``__<category>__`` |
756 | -where <category> is the name of the job file which contains the job. This is |
757 | -indicated again by the prefix of the job (before the ``/`` in the job name). |
758 | -As a quick example, the job ``graphics/glxgears`` is contained in |
759 | -``graphics.txt``. Therefore we should include the ``__graphics__`` job so that |
760 | -the ``graphics/glxgears`` job shows correctly under the category. The |
761 | -``__graphics__`` job itself looks like:: |
762 | - |
763 | - name: __graphics__ |
764 | - plugin: local |
765 | - _description: Graphics tests |
766 | - command: |
767 | - shopt -s extglob |
768 | - cat $CHECKBOX_SHARE/jobs/graphics.txt?(.in) |
769 | - |
770 | -Checkbox will interpret this job as a request to display any job in |
771 | -``graphics.txt`` (or its untranslated version ``graphics.txt.in``) under the |
772 | -heading shown in the description of this job (in this case 'Graphics tests'). |
773 | - |
774 | -Tutorial |
775 | -======== |
776 | - |
777 | -To compound what we discussed before, below is a brief tutorial which walks |
778 | -through assembling a basic whitelist file. |
779 | - |
780 | -1. First we need to create a file, let's name it tutorial.whitelist. |
781 | -Whitelists don't have to end with the .whitelist suffix but this is the |
782 | -convention used to help identify them. |
783 | -2. We start by adding the one job that is required for all whitelists, as |
784 | -explained above in the section 'Minimal Whitelist File', so our whitelist file |
785 | -looks like:: |
786 | - |
787 | - miscellanea/submission-resources |
788 | - |
789 | -3. Next we should choose some jobs that we want to run. This all depends on |
790 | -your specific use-case of course, but I've selected a few jobs that will help |
791 | -clearly illustrate more of the concepts involved in whitelists. These jobs will |
792 | -give us a whitelist file that looks like:: |
793 | - |
794 | - miscellanea/submission-resources |
795 | - cpu/clocktest |
796 | - ethernet/multi_nic |
797 | - ethernet/multi_nic_eth0 |
798 | - graphics/glxgears |
799 | - |
800 | - If we run this whitelist now then all of these jobs will be executed and a |
801 | - valid test submission will be created, but we can still improve it in a couple |
802 | - of ways. |
803 | - |
804 | -4. The first way is by adding the necessary jobs to allow the Checkbox UI to |
805 | -group the jobs into specific categories. To do this we need to add a job with |
806 | -a name like ``__<category>__`` for each category. We have three categories in |
807 | -our whitelist file - cpu, ethernet and graphics. The category of the job is |
808 | -the prefix of the job name prior to the ``/``. So now our whitelist file looks |
809 | -like:: |
810 | - |
811 | - miscellanea/submission-resources |
812 | - __cpu__ |
813 | - __ethernet__ |
814 | - __graphics__ |
815 | - cpu/clocktest |
816 | - ethernet/multi_nic |
817 | - ethernet/multi_nic_eth0 |
818 | - graphics/glxgears |
819 | - |
820 | - Now the Checkbox UI will group the jobs into these categories. |
821 | - |
822 | -5. Although it's not immediately apparent there is another problem with this |
823 | -whitelist. The ``ethernet/multi_nic`` tests are only able to include one job |
824 | -for the ethernet port 'eth0'. It would be better if we included all of the |
825 | -jobs generated by 'ethernet/multi_nic', no matter how many ethernet ports are |
826 | -present on the system under test. The best way to do this is to write the |
827 | -pattern so that it matches all of the possible job names. We can take advantage |
828 | -of the Python regular expression syntax and use the ``\d`` special character |
829 | -to match any decimal number. After doing this the whitelist file will look |
830 | -like this:: |
831 | - |
832 | - miscellanea/submission-resources |
833 | - __cpu__ |
834 | - __ethernet__ |
835 | - __graphics__ |
836 | - cpu/clocktest |
837 | - ethernet/multi_nic |
838 | - ethernet/multi_nic_eth\d |
839 | - graphics/glxgears |
840 | diff --git a/docs/dev/old.rst b/docs/dev/old.rst |
841 | index 151f21c..7921f68 100644 |
842 | --- a/docs/dev/old.rst |
843 | +++ b/docs/dev/old.rst |
844 | @@ -144,8 +144,6 @@ dependencies) being added to the repository. |
845 | Implementation issues |
846 | --------------------- |
847 | |
848 | -There are two issues that are known at this time: |
849 | - |
850 | * There is too much checkbox-specific knowledge which really belongs |
851 | elsewhere. We are working to remove that so that non-checkbox jobs |
852 | can be introduced later. There is a branch in progress that entirely |
853 | @@ -155,15 +153,6 @@ There are two issues that are known at this time: |
854 | that was previously internal (most notably a way to add new jobs and |
855 | resources). |
856 | |
857 | -* The way jobs are currently selected is unfortunate because of local jobs |
858 | - that can add new jobs to the system. This causes considerable complexity |
859 | - at the application level where the application must check if each |
860 | - executed job is a 'local' job and re-compute the desired_job_list. This |
861 | - should be replaced by a matcher function that can be passed to |
862 | - SessionState once so that desired_job_list is re-evaluated internally |
863 | - whenever job_list changes. |
864 | - |
865 | - |
866 | :class:`~plainbox.impl.job.JobDefinition` |
867 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
868 | |
869 | @@ -211,49 +200,6 @@ plugin == "manual" |
870 | This value is used for fully manual jobs. It has no special handling in the core |
871 | apart from requiring a human-provided outcome (pass/fail classification) |
872 | |
873 | -.. _local: |
874 | - |
875 | -plugin == "local" |
876 | -################# |
877 | - |
878 | -This value is used for special job generator jobs. The output of such jobs is |
879 | -interpreted as additional jobs and is identical in effect to loading such jobs |
880 | -from a job definition file. |
881 | - |
882 | -There are two practical uses for such jobs: |
883 | - |
884 | -* Some local jobs are used to generate a number of jobs for each object. |
885 | - This is needed where the tested machine may have a number of such objects |
886 | - and each requires unique testing. A good example is a computer where all |
887 | - network tests are explicitly "instantiated" for each network card |
888 | - present. |
889 | - |
890 | - This is a valid use case but is rather unfortunate for architecture of |
891 | - Plainbox and there is a desire to replace it with equally-expressive |
892 | - pattern jobs. The advantage is that unlike local jobs (which cannot be |
893 | - "discovered" without enduring any potential side effects that may be |
894 | - caused by the job script command) pattern jobs would allow the core to |
895 | - determine the names of jobs that can be generated and, for example, |
896 | - automatically determine that a pattern job needs to be executed as a |
897 | - dependency of a phantom (yet undetermined) job with a given name. |
898 | - |
899 | - The solution with "pattern" jobs may be executed in future phases of |
900 | - Plainbox development. Currently there is no support for that at all. |
901 | - |
902 | - Currently Plainbox cannot determine job dependencies across local jobs. |
903 | - That is, unless a local job is explicitly requested (in the desired job |
904 | - list) Plainbox will not be able to run a job that is generated by a local |
905 | - job at all and will treat it as if that job never existed. |
906 | - |
907 | -* Some local jobs are used to create a form of informal "category". |
908 | - Typically all such jobs have a leading and trailing double underscore, |
909 | - for example '__audio__'. This is currently being used by Checkbox for |
910 | - building a hierarchical tree of tests that the user may select. |
911 | - |
912 | - Since this has the same flaws as described above (for pattern jobs) it |
913 | - will likely be replaced by an explicit category field that can be |
914 | - specified each job. |
915 | - |
916 | plugin == "resource" |
917 | #################### |
918 | |
919 | diff --git a/docs/dev/trusted-launcher.rst b/docs/dev/trusted-launcher.rst |
920 | index d74735e..a0cc565 100644 |
921 | --- a/docs/dev/trusted-launcher.rst |
922 | +++ b/docs/dev/trusted-launcher.rst |
923 | @@ -145,7 +145,7 @@ Usage |
924 | .. code-block:: text |
925 | |
926 | plainbox-trusted-launcher-1 [-h] (--hash HASH | --warmup) |
927 | - [--via LOCAL-JOB-HASH] |
928 | + [--via GENERATOR-JOB-HASH] |
929 | [NAME=VALUE [NAME=VALUE ...]] |
930 | |
931 | positional arguments: |
932 | @@ -156,7 +156,7 @@ Usage |
933 | --hash HASH job hash to match |
934 | --warmup Return immediately, only useful when used with |
935 | pkexec(1) |
936 | - --via LOCAL-JOB-HASH Local job hash to use to match the generated job |
937 | + --via GENERATOR-JOB-HASH Generator job hash to use to match the generated job |
938 | |
939 | .. note:: |
940 | |
941 | @@ -185,15 +185,11 @@ thanks to the installed policy file the authentication will be kept. |
942 | Special case of jobs using the Checkbox local plugin |
943 | ---------------------------------------------------- |
944 | |
945 | -For jobs generated from :ref:`local <local>` jobs (e.g. |
946 | +For jobs generated from resources jobs (e.g. |
947 | disk/read_performance.*) the trusted launcher is started with ``--via`` meaning |
948 | -that we have to first eval a local job to find a hash match. Once a match is |
949 | +that we have to first eval a generator job to find a hash match. Once a match is |
950 | found, the job command is executed. |
951 | |
952 | .. code-block:: bash |
953 | |
954 | - $ pkexec plainbox-trusted-launcher-1 --hash JOB-HASH --via LOCAL-JOB-HASH |
955 | - |
956 | -.. note:: |
957 | - |
958 | - it will obviously fail if any local job can ever generate another local job. |
959 | + $ pkexec plainbox-trusted-launcher-1 --hash JOB-HASH --via GENERATOR-JOB-HASH |
960 | diff --git a/docs/glossary.rst b/docs/glossary.rst |
961 | index 6174025..212d9a2 100644 |
962 | --- a/docs/glossary.rst |
963 | +++ b/docs/glossary.rst |
964 | @@ -46,12 +46,11 @@ Glossary |
965 | necessary for end-user work. ``plainbox`` is usually installed |
966 | explicitly if needed. |
967 | |
968 | - whitelist |
969 | + test plan |
970 | |
971 | - Whitelists are text files used by Checkbox to select jobs for |
972 | - execution. They can include simple regular expressions to match and |
973 | - pick many similar jobs at once. For more information see |
974 | - :doc:`Checkbox Whitelist Files <author/whitelists>` |
975 | + Test plans are text files used by Checkbox to select jobs for |
976 | + execution. They can include simple regular expressions to match and |
977 | + pick many similar jobs at once. |
978 | |
979 | job |
980 | |
981 | @@ -63,7 +62,7 @@ Glossary |
982 | |
983 | provider |
984 | |
985 | - A container for jobs, whitelists, private executables and data. |
986 | + A container for jobs, test plans, private executables and data. |
987 | Providers are the foundation of Plainbox as they *provide* all of the |
988 | content. Providers can be created and managed by any entity, separately |
989 | from the Checkbox project. |
990 | diff --git a/docs/manpages/plainbox-dev-analyze.rst b/docs/manpages/plainbox-dev-analyze.rst |
991 | index e5fe5dc..23fc999 100644 |
992 | --- a/docs/manpages/plainbox-dev-analyze.rst |
993 | +++ b/docs/manpages/plainbox-dev-analyze.rst |
994 | @@ -15,13 +15,6 @@ plainbox-dev-analyze (1) |
995 | and the command prints nothing at all) to inspect certain aspects of the |
996 | hypothetical session |
997 | |
998 | - The only exception to the rule above is the ``--run-local`` option. With that |
999 | - option all local jobs and their dependencies *are* started. This is |
1000 | - technically required to correctly emulate the behavior of ``plainbox run`` |
1001 | - that does so unconditionally. Still, local jobs can cause harm so don't run |
1002 | - untrusted code this way (the author of this man page recalls one local job |
1003 | - that ran ``sudo reboot`` to measure bootchart data) |
1004 | - |
1005 | Report Types |
1006 | ============ |
1007 | |
1008 | @@ -74,11 +67,11 @@ plainbox-dev-analyze (1) |
1009 | always includes additional jobs (such as resource jobs and other |
1010 | dependencies) |
1011 | |
1012 | - The run list is of great importance. Most of the time the test operator will |
1013 | - see tests in precisely this order. The only exception is that some test |
1014 | - applications choose to pre-run local jobs. Still, if your job ordering is |
1015 | - wrong in any way, inspecting the run list is the best way to debug the |
1016 | - problem. |
1017 | + The run list is of great importance. Most of the time the test operator |
1018 | + will see tests in precisely this order. The only exception is that some |
1019 | + test applications choose to pre-run generator jobs (resources). Still, if |
1020 | + your job ordering is wrong in any way, inspecting the run list is the best |
1021 | + way to debug the problem. |
1022 | |
1023 | See Also |
1024 | ======== |
1025 | diff --git a/docs/manpages/plainbox-exporter-units.rst b/docs/manpages/plainbox-exporter-units.rst |
1026 | index 7017071..df7172c 100644 |
1027 | --- a/docs/manpages/plainbox-exporter-units.rst |
1028 | +++ b/docs/manpages/plainbox-exporter-units.rst |
1029 | @@ -120,36 +120,4 @@ The provider shipping such unit can be as follow:: |
1030 |   └── exporters.pxu |
1031 | |
1032 | Note that exporters.pxu is not strictly needed to store the exporter units, but |
1033 | -keeping them in a dedidated file is a good practice. |
1034 | - |
1035 | -How to use exporter units? |
1036 | --------------------------- |
1037 | - |
1038 | -In order to call an exporter unit from provider foo, you just need to add the |
1039 | -unit id to the cli or the gui launcher in the exporter section: |
1040 | - |
1041 | -Example of a gui launcher: |
1042 | - |
1043 | - #!/usr/bin/checkbox-gui |
1044 | - |
1045 | - [welcome] |
1046 | - title = "Foo" |
1047 | - text = "bar" |
1048 | - |
1049 | - [exporter] |
1050 | - HTML = "com.foo.bar::my_html" |
1051 | - |
1052 | -Example of a cli launcher: |
1053 | - |
1054 | - #!/usr/bin/env checkbox-launcher |
1055 | - [welcome] |
1056 | - text = Foo |
1057 | - |
1058 | - [suite] |
1059 | - whitelist_filter = ^.*$ |
1060 | - whitelist_selection = ^default$ |
1061 | - |
1062 | - [exporter] |
1063 | - com.foo.bar::my_html |
1064 | - com.foo.bar::my_json |
1065 | - com.foo.baz::my_html |
1066 | +keeping them in a dedicated file is a good practice. |
1067 | diff --git a/docs/manpages/plainbox-file-units.rst b/docs/manpages/plainbox-file-units.rst |
1068 | index 2a61277..3c5ddbd 100644 |
1069 | --- a/docs/manpages/plainbox-file-units.rst |
1070 | +++ b/docs/manpages/plainbox-file-units.rst |
1071 | @@ -36,9 +36,6 @@ There are two fields that are used by the file unit: |
1072 | 'unit-source': |
1073 | The file is a source of unit definitions. Currently this is the only |
1074 | actually implemented value. |
1075 | - |
1076 | - 'legacy-whitelist': |
1077 | - This file is a legacy whitelist. |
1078 | |
1079 | 'script': |
1080 | This file is an architecture independent executable. |
1081 | @@ -60,4 +57,4 @@ There are two fields that are used by the file unit: |
1082 | This file contains copyright and licensing information. |
1083 | |
1084 | 'docs': |
1085 | - This file contains documentation. |
1086 | \ No newline at end of file |
1087 | + This file contains documentation. |
1088 | diff --git a/docs/manpages/plainbox-job-units.rst b/docs/manpages/plainbox-job-units.rst |
1089 | index be0c17f..08e8fd1 100644 |
1090 | --- a/docs/manpages/plainbox-job-units.rst |
1091 | +++ b/docs/manpages/plainbox-job-units.rst |
1092 | @@ -56,9 +56,6 @@ Following fields may be used by the job unit: |
1093 | test's outcome. This is essentially a manual job with a command. |
1094 | :attachment: jobs whose command output will be attached to the |
1095 | test report or submission. |
1096 | - :local: a job whose command output needs to be in Checkbox job |
1097 | - format. Jobs output by a local job will be added to the set of |
1098 | - available jobs to be run. |
1099 | :resource: A job whose command output results in a set of rfc822 |
1100 | records, containing key/value pairs, and that can be used in other |
1101 | jobs' ``requires`` expressions. |
1102 | diff --git a/docs/manpages/plainbox-run.rst b/docs/manpages/plainbox-run.rst |
1103 | index eaf5098..3363aa8 100644 |
1104 | --- a/docs/manpages/plainbox-run.rst |
1105 | +++ b/docs/manpages/plainbox-run.rst |
1106 | @@ -74,44 +74,6 @@ plainbox-run (1) |
1107 | |
1108 | plainbox run -i '.*::foo' -i '.*::bar' |
1109 | |
1110 | - Selecting jobs with whitelists |
1111 | - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
1112 | - |
1113 | - The second mechanism is the ``--whitelist WHITELIST`` command-line option. |
1114 | - WhiteLists (or test plans, which is somewhat easier to relate to). |
1115 | - Whitelists are simple text files composed of a list of regular expressions, |
1116 | - identical to those that may be passed with the ``-i`` option. |
1117 | - |
1118 | - Unlike the ``-i`` option though, there are two kinds of whitelists. |
1119 | - Standalone whitelists are not associated with any Plainbox Provider. Such |
1120 | - whitelists can be distributed entirely separately from any other component |
1121 | - and thus have no association with any namespace. |
1122 | - |
1123 | - Therefore, be fully qualified, each pattern must include both the namespace |
1124 | - and the partial identifier components. For example, this is a valid, fully |
1125 | - quallified whitelist:: |
1126 | - |
1127 | - com.canonical.plainbox::stub/.* |
1128 | - |
1129 | - It will unambiguously select some of the jobs from the special, internal |
1130 | - StubBox provider that is built into Plainbox. It can be saved under any |
1131 | - filename and stored in any directory and it will always select the same set |
1132 | - of jobs. |
1133 | - |
1134 | - In contrast, whitelists that are associated with a particular provider, by |
1135 | - being stored in the per-provider ``whitelists/`` directory, carry an |
1136 | - implicit namespace. Such whitelists are typically written without |
1137 | - mentioning the namespace component. |
1138 | - |
1139 | - For example, the same "stub/.*" pattern can be abbreviated to:: |
1140 | - |
1141 | - stub/.* |
1142 | - |
1143 | - Typically this syntax is used in all whitelists specific to a particular |
1144 | - provider unless the provider maintainer explicitly wants to include a job |
1145 | - from another namespace (for example, one of the well-known Checkbox job |
1146 | - definitions). |
1147 | - |
1148 | GENERATED JOBS |
1149 | ============== |
1150 | |
1151 | @@ -125,28 +87,7 @@ plainbox-run (1) |
1152 | storage devices) and then duplicate each of the store specific tests so |
1153 | that all devices are tested separately. |
1154 | |
1155 | - At this time jobs can be generated only from jobs using the plugin type |
1156 | - `local`. Jobs of this kind are expected to print fully conforming job |
1157 | - definitions on stdout. Generated jobs cause a few complexities and one |
1158 | - limitation that is currently enforced is that generated jobs cannot |
1159 | - generate additional jobs if any of the affected jobs need to run as another |
1160 | - user. |
1161 | - |
1162 | - Another limitation is that jobs cannot override existing definitions. |
1163 | - |
1164 | - Creating Parent-Child Association |
1165 | - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
1166 | - |
1167 | - A relatively niche and legacy feature of generated jobs is to print a |
1168 | - verbatim copy of existing job definitions from a ``local`` job definition |
1169 | - named afer a generic testing theme or category. For example the Checkbox |
1170 | - job definition ``__wireless__`` prints, with the help of ``cat`` (1), all |
1171 | - of the job definitions defined in the file ``wireless.txt``. |
1172 | - |
1173 | - This behavior is special-cased not to cause redefinition errors. Instead, |
1174 | - existing definitions gain the ``via`` attribute that links them to the |
1175 | - generator job. This feature is used by derivative application such as |
1176 | - Checkbox. Plainbox is not using it at this time. |
1177 | + One limitation is that jobs cannot override existing definitions. |
1178 | |
1179 | RESUMING |
1180 | ======== |
1181 | @@ -302,35 +243,24 @@ plainbox-run (1) |
1182 | Exported data will include comments added by the test operator to each |
1183 | job result that has them. |
1184 | |
1185 | - with-job-via: |
1186 | - Exported data will include the ``via`` attribute alongside each job |
1187 | - result. The via attribute contains the checksum of the job definition |
1188 | - that generated a particular job definition. This is useful for tracking |
1189 | - jobs generated by jobs with the plugin type `local`. |
1190 | - |
1191 | with-job-hash: |
1192 | Exported data will include the ``hash`` attribute alongside each job |
1193 | result. The hash attribute is the checksum of the job definition's |
1194 | - data. It can be useful alongside with `with-job-via`. |
1195 | + data. |
1196 | |
1197 | machine-json: |
1198 | The generated JSON document will be minimal (devoid of any optional |
1199 | whitespace). This option is best to be used if the result is not |
1200 | intended to be read by humans as it saves some space. |
1201 | |
1202 | - rfc822 |
1203 | + text |
1204 | ------ |
1205 | |
1206 | All of the options have the same meaning as for the `json` exporter: |
1207 | `with-io-log`, `squash-io-log`, `flatten-io-log`, `with-run-list`, |
1208 | `with-job-list`, `with-resource-map`, `with-job-defs`, `with-attachments`, |
1209 | - `with-comments`, `with-job-via`, `with-job-hash`. The only exception is |
1210 | - the `machine-json` option which doesn't exist for this exporter. |
1211 | - |
1212 | - text |
1213 | - ---- |
1214 | - |
1215 | - Same as with rfc822. |
1216 | + `with-comments`, `with-job-hash`. The only exception is the `machine-json` |
1217 | + option which doesn't exist for this exporter. |
1218 | |
1219 | xlsx |
1220 | ---- |
1221 | diff --git a/docs/manpages/plainbox-test-plan-units.rst b/docs/manpages/plainbox-test-plan-units.rst |
1222 | index 7d77e6f..6857100 100644 |
1223 | --- a/docs/manpages/plainbox-test-plan-units.rst |
1224 | +++ b/docs/manpages/plainbox-test-plan-units.rst |
1225 | @@ -99,11 +99,10 @@ copy such constructs when working on a new test plan from scratch |
1226 | common and most test plans used by Checkbox actually look like that. |
1227 | |
1228 | - You can use regular expressions to select many tests at the same time. |
1229 | - This is the only way to select generated jobs (created either by |
1230 | - template units or by job definitions using the legacy 'local' plugin |
1231 | - type). Please remember that the dot character has a special meaning |
1232 | - so unless you actually want to match *any character* escape the dot |
1233 | - with the backslash character (\\). |
1234 | + This is the only way to select generated jobs (created by template |
1235 | + units). Please remember that the dot character has a special meaning so |
1236 | + unless you actually want to match *any character* escape the dot with |
1237 | + the backslash character (\\). |
1238 | |
1239 | Regardless of if you use patterns or literal job identifiers you can use |
1240 | their fully qualified name (the one that includes the namespace they reside |
1241 | @@ -149,7 +148,7 @@ copy such constructs when working on a new test plan from scratch |
1242 | |
1243 | Note that each entry in the bootstrap_include section must be a valid job |
1244 | identifier and cannot be a regular expression pattern. |
1245 | - Also note that only local and resource jobs are allowed in this section. |
1246 | + Also note that only resource jobs are allowed in this section. |
1247 | |
1248 | ``exclude``: |
1249 | A multi-line list of job identifiers or patterns matching such identifiers |
1250 | diff --git a/docs/manpages/plainbox-trusted-launcher-1.rst b/docs/manpages/plainbox-trusted-launcher-1.rst |
1251 | index ba889e0..77cdfeb 100644 |
1252 | --- a/docs/manpages/plainbox-trusted-launcher-1.rst |
1253 | +++ b/docs/manpages/plainbox-trusted-launcher-1.rst |
1254 | @@ -77,13 +77,8 @@ The following environment variables *DO NOT* affect ``plainbox-trusted-launcher- |
1255 | Bugs |
1256 | ==== |
1257 | |
1258 | -Currently it is impossible to use ``plainbox-trusted-launcher-1`` with a |
1259 | -``local`` job needs to run as root, that generates another ``local`` job that |
1260 | -needs to run as root, to generate any additional jobs that also need to run as |
1261 | -root. In other words, only one-level job generation is supported. |
1262 | - |
1263 | The launcher is somewhat inefficient, in that it has to re-run all of the |
1264 | -dependencies of the ``local`` job over and over. Ideally those would be cached, |
1265 | +dependencies of the generator job over and over. Ideally those would be cached, |
1266 | per-session, but that would significantly increase the complexity of the code |
1267 | running as root. |
1268 | |
1269 | diff --git a/docs/usage.rst b/docs/usage.rst |
1270 | index e2037b9..e2f2f0a 100644 |
1271 | --- a/docs/usage.rst |
1272 | +++ b/docs/usage.rst |
1273 | @@ -47,17 +47,6 @@ To list all known jobs run: |
1274 | |
1275 | plainbox dev special --list-jobs |
1276 | |
1277 | -Running a white list |
1278 | -^^^^^^^^^^^^^^^^^^^^ |
1279 | - |
1280 | -To run a :term:`whitelist` pass the ``--whitelist`` or ``-w`` option. |
1281 | - |
1282 | -For example, to run the default white list run: |
1283 | - |
1284 | -.. code-block:: bash |
1285 | - |
1286 | - $ plainbox run -w /path/to/some/file.whitelist |
1287 | - |
1288 | Saving test results |
1289 | ^^^^^^^^^^^^^^^^^^^ |
1290 | |
1291 | @@ -72,7 +61,7 @@ To generate a JSON file with all of the internally available data (for storage, |
1292 | processing or other automation) you will need to pass three additional |
1293 | arguments to ``plainbox run``: |
1294 | |
1295 | -#. ``--output-format=com.canonical.plainbox::json`` |
1296 | +#. ``--output-format=com.canonical.com.canonical.plainbox::json`` |
1297 | #. ``--output-options=OPTION1,OPTION2`` where *OPTIONx* are option names. |
1298 | #. ``--output-file=NAME`` where *NAME* is a file name. |
1299 | |
1300 | @@ -81,7 +70,7 @@ exporter options can be specified, separated with commas. |
1301 | |
1302 | .. code-block:: bash |
1303 | |
1304 | - $ plainbox run --whitelist=/path/to/some/file.whitelist --output-format=com.canonical.plainbox::json --output-file=results.json |
1305 | + $ plainbox run -i com.canonical.certification::foo --output-format=com.canonical.plainbox::json --output-file=results.json |
1306 | |
1307 | Other Exporters |
1308 | --------------- |
1309 | diff --git a/plainbox/__init__.py b/plainbox/__init__.py |
1310 | index 2a7c4e9..336f833 100644 |
1311 | --- a/plainbox/__init__.py |
1312 | +++ b/plainbox/__init__.py |
1313 | @@ -1,6 +1,6 @@ |
1314 | # This file is part of Checkbox. |
1315 | # |
1316 | -# Copyright 2012-2014 Canonical Ltd. |
1317 | +# Copyright 2012-2018 Canonical Ltd. |
1318 | # Written by: |
1319 | # Zygmunt Krynicki <zygmunt.krynicki@canonical.com> |
1320 | # |
1321 | @@ -19,17 +19,11 @@ |
1322 | :mod:`plainbox` -- main package |
1323 | =============================== |
1324 | |
1325 | -Simple checkbox redesign, without the complex message passing |
1326 | +Simple checkbox (2008 version) redesign, without the complex message passing |
1327 | |
1328 | -All public API is in :mod:`plainbox.public`. |
1329 | All abstract base classes are in :mod:`plainbox.abc`. |
1330 | """ |
1331 | |
1332 | -import sys |
1333 | - |
1334 | -if sys.version_info[0:2] < (3, 2): |
1335 | - raise ImportError("plainbox requires python 3.2") # pragma: no cover |
1336 | - |
1337 | # PEP440 compliant version declaration. |
1338 | # |
1339 | # This is used by @public decorator to enforce our public API guarantees. |
1340 | diff --git a/plainbox/__main__.py b/plainbox/__main__.py |
1341 | deleted file mode 100644 |
1342 | index 41fa1ed..0000000 |
1343 | --- a/plainbox/__main__.py |
1344 | +++ /dev/null |
1345 | @@ -1,32 +0,0 @@ |
1346 | -# This file is part of Checkbox. |
1347 | -# |
1348 | -# Copyright 2013 Canonical Ltd. |
1349 | -# Written by: |
1350 | -# Zygmunt Krynicki <zygmunt.krynicki@canonical.com> |
1351 | -# |
1352 | -# Checkbox is free software: you can redistribute it and/or modify |
1353 | -# it under the terms of the GNU General Public License version 3, |
1354 | -# as published by the Free Software Foundation. |
1355 | -# |
1356 | -# Checkbox is distributed in the hope that it will be useful, |
1357 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1358 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1359 | -# GNU General Public License for more details. |
1360 | -# |
1361 | -# You should have received a copy of the GNU General Public License |
1362 | -# along with Checkbox. If not, see <http://www.gnu.org/licenses/>. |
1363 | - |
1364 | -""" |
1365 | -:mod:`plainbox.__main__` -- execute plainbox |
1366 | -============================================ |
1367 | - |
1368 | -This module allows plainbox to be executed with: |
1369 | - |
1370 | - python3 -m plainbox |
1371 | -""" |
1372 | - |
1373 | -from plainbox.public import main |
1374 | - |
1375 | - |
1376 | -if __name__ == '__main__': |
1377 | - main() |
1378 | diff --git a/plainbox/_lazymod.py b/plainbox/_lazymod.py |
1379 | deleted file mode 100644 |
1380 | index 0d697bb..0000000 |
1381 | --- a/plainbox/_lazymod.py |
1382 | +++ /dev/null |
1383 | @@ -1,138 +0,0 @@ |
1384 | -# This file is part of Checkbox. |
1385 | -# |
1386 | -# Copyright 2015 Canonical Ltd. |
1387 | -# Written by: |
1388 | -# Zygmunt Krynicki <zygmunt.krynicki@canonical.com> |
1389 | -# |
1390 | -# Checkbox is free software: you can redistribute it and/or modify |
1391 | -# it under the terms of the GNU General Public License version 3, |
1392 | -# as published by the Free Software Foundation. |
1393 | -# |
1394 | -# Checkbox is distributed in the hope that it will be useful, |
1395 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
1396 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
1397 | -# GNU General Public License for more details. |
1398 | -# |
1399 | -# You should have received a copy of the GNU General Public License |
1400 | -# along with Checkbox. If not, see <http://www.gnu.org/licenses/>. |
1401 | - |
1402 | -""" Implementation of a lazy module. """ |
1403 | - |
1404 | -import inspect |
1405 | -import sys |
1406 | -import types |
1407 | - |
1408 | - |
1409 | -class LazyModule(types.ModuleType): |
1410 | - |
1411 | - """ |
1412 | - A module subclass that imports things lazily on demand. |
1413 | - |
1414 | - There are some special provisions to make dir() and __all__ work better so |
1415 | - that pydoc is more informative. |
1416 | - |
1417 | - :ivar _lazy: |
1418 | - A mapping of 'name' to 'callable'. The callable is called only once and |
1419 | - defines the lazily loaded version of 'name'. |
1420 | - :ivar _all: |
1421 | - A set of all the "public" objects. This is exposed as the module's |
1422 | - __all__ property. It automatically collects all the objects reported |
1423 | - via :meth:`lazily()` and :meth:`immediate()`. |
1424 | - :ivar _old: |
1425 | - Reference to the old (original) module. This is kept around for python |
1426 | - 2.x compatibility. It also seems to help with implementing __dir__() |
1427 | - """ |
1428 | - |
1429 | - def __init__(self, name, doc, old): |
1430 | - """ Initialize a new lazy module. """ |
1431 | - super(LazyModule, self).__init__(name, doc) |
1432 | - self._lazy = {} |
1433 | - self._all = set() |
1434 | - self._old = old |
1435 | - |
1436 | - def __dir__(self): |
1437 | - """ Lazy-aware version of dir(). """ |
1438 | - if sys.version_info[0] == 3: |
1439 | - data = super(LazyModule, self).__dir__() |
1440 | - else: |
1441 | - data = self.__dict__.keys() |
1442 | - data = set(data) | self._all |
1443 | - return sorted(data) |
1444 | - |
1445 | - def __getattr__(self, name): |
1446 | - """ Lazy-aware version of getattr(). """ |
1447 | - try: |
1448 | - callable, args = self._lazy[name] |
1449 | - except KeyError: |
1450 | - raise AttributeError(name) |
1451 | - value = callable(*args) |
1452 | - del self._lazy[name] |
1453 | - setattr(self, name, value) |
1454 | - return value |
1455 | - |
1456 | - @classmethod |
1457 | - def shadow_normal_module(cls, mod_name=None): |
1458 | - """ |
1459 | - Shadow a module with an instance of LazyModule. |
1460 | - |
1461 | - :param mod_name: |
1462 | - Name of the module to shadow. By default this is the module that is |
1463 | - making the call into this method. This is not hard-coded as that |
1464 | - module might be called '__main__' if it is executed via 'python -m' |
1465 | - :returns: |
1466 | - A fresh instance of :class:`LazyModule`. |
1467 | - """ |
1468 | - if mod_name is None: |
1469 | - frame = inspect.currentframe() |
1470 | - try: |
1471 | - mod_name = frame.f_back.f_locals['__name__'] |
1472 | - finally: |
1473 | - del frame |
1474 | - orig_mod = sys.modules[mod_name] |
1475 | - lazy_mod = cls(orig_mod.__name__, orig_mod.__doc__, orig_mod) |
1476 | - for attr in dir(orig_mod): |
1477 | - setattr(lazy_mod, attr, getattr(orig_mod, attr)) |
1478 | - sys.modules[mod_name] = lazy_mod |
1479 | - return lazy_mod |
1480 | - |
1481 | - def lazily(self, name, callable, args): |
1482 | - """ Load something lazily. """ |
1483 | - self._lazy[name] = callable, args |
1484 | - self._all.add(name) |
1485 | - |
1486 | - def immediate(self, name, value): |
1487 | - """ Load something immediately. """ |
1488 | - setattr(self, name, value) |
1489 | - self._all.add(name) |
1490 | - |
1491 | - @property |
1492 | - def __all__(self): |
1493 | - """ |
1494 | - A lazy-aware version of __all__. |
1495 | - |
1496 | - In addition to exposing all of the original module's __all__ it also |
1497 | - contains all the (perhaps not yet loaded) objects defined via |
1498 | - :meth:`lazily()` |
1499 | - """ |
1500 | - return sorted(self._all) |
1501 | - |
1502 | - @__all__.setter |
1503 | - def __all__(self, value): |
1504 | - """ |
1505 | - Setter for __all__ that just updates the internal set :ivar:`_all`. |
1506 | - |
1507 | - This is used by :meth:`shadow_normal_module()` which copies (assigns) |
1508 | - all of the original module's attributes, which also assigns __all__. |
1509 | - """ |
1510 | - self._all.update(value) |
1511 | - |
1512 | - |
1513 | -def far(import_path): |
1514 | - """ Helper for lazy imports for :meth:`LazyModule.lazily()`. """ |
1515 | - module_name, func_name = import_path.split(":", 1) |
1516 | - module = __import__(module_name, fromlist=['']) |
1517 | - try: |
1518 | - return getattr(module, func_name) |
1519 | - except AttributeError: |
1520 | - raise NotImplementedError( |
1521 | - "%s.%s does not exist" % (module_name, func_name)) |
1522 | diff --git a/plainbox/abc.py b/plainbox/abc.py |
1523 | index 3315d9d..f879732 100644 |
1524 | --- a/plainbox/abc.py |
1525 | +++ b/plainbox/abc.py |
1526 | @@ -298,11 +298,11 @@ class IJobQualifier(metaclass=ABCMeta): |
1527 | can simply check if a qualifier designates a particular job (if it selects |
1528 | it and marks for subsequent execution). This API works fine for certain |
1529 | tasks but it was found that it is insufficient to implement so-called |
1530 | - whitelist ordering, where the order of jobs in a whitelist is preserved |
1531 | - when selecting that whitelist for execution. This spawned the second, |
1532 | + test plan ordering, where the order of jobs in a test plan is preserved |
1533 | + when selecting that test plan for execution. This spawned the second, |
1534 | lower-level API, that gives portable visibility into composite qualifiers |
1535 | - (such as a whitelist) and distinct select, deselect vote so that full range |
1536 | - of current expressiveness can be preserved. |
1537 | + and distinct select, deselect vote so that full range of current |
1538 | + expressiveness can be preserved. |
1539 | |
1540 | :attr VOTE_EXCLUDE: |
1541 | (0) vote indicating that a job should *not* be included for |
1542 | @@ -642,12 +642,6 @@ class IProviderBackend1(metaclass=ABCMeta): |
1543 | """ |
1544 | |
1545 | @abstractproperty |
1546 | - def whitelists_dir(self): |
1547 | - """ |
1548 | - Return an absolute path of the whitelist directory |
1549 | - """ |
1550 | - |
1551 | - @abstractproperty |
1552 | def data_dir(self): |
1553 | """ |
1554 | absolute path of the data directory |
1555 | @@ -758,18 +752,6 @@ class IProvider1(IProviderBackend1): |
1556 | """ |
1557 | |
1558 | @abstractproperty |
1559 | - def whitelist_list(self): |
1560 | - """ |
1561 | - List of loaded whitelists. |
1562 | - |
1563 | - .. warning:: |
1564 | - :class:`WhiteList` is currently deprecated. You should never need |
1565 | - to access them in any new code. They are entirely replaced by |
1566 | - :class:`TestPlan`. This property is provided for completeness and |
1567 | - it will be **removed** once whitelists classes are no longer used. |
1568 | - """ |
1569 | - |
1570 | - @abstractproperty |
1571 | def problem_list(self): |
1572 | """ |
1573 | list of problems encountered by the loading process |
1574 | diff --git a/plainbox/impl/applogic.py b/plainbox/impl/applogic.py |
1575 | index 6f56ed4..1d90dfd 100644 |
1576 | --- a/plainbox/impl/applogic.py |
1577 | +++ b/plainbox/impl/applogic.py |
1578 | @@ -48,21 +48,6 @@ def get_matching_job_list(job_list, qualifier): |
1579 | return select_jobs(job_list, [qualifier]) |
1580 | |
1581 | |
1582 | -def get_whitelist_by_name(provider_list, desired_whitelist): |
1583 | - """ |
1584 | - Get the first whitelist matching desired_whitelist from the loaded |
1585 | - providers |
1586 | - """ |
1587 | - for provider in provider_list: |
1588 | - for whitelist in provider.whitelist_list: |
1589 | - if whitelist.name == desired_whitelist: |
1590 | - return whitelist |
1591 | - else: |
1592 | - raise LookupError( |
1593 | - _("None of the providers had a whitelist " |
1594 | - "named '{}'").format(desired_whitelist)) |
1595 | - |
1596 | - |
1597 | def run_job_if_possible(session, runner, config, job, update=True, ui=None): |
1598 | """ |
1599 | Coupling point for session, runner, config and job |
1600 | @@ -103,11 +88,6 @@ class PlainBoxConfig(config.Config): |
1601 | environment = config.Section( |
1602 | help_text=_("Environment variables for scripts and jobs")) |
1603 | |
1604 | - extcmd = config.Variable( |
1605 | - section='FEATURE-FLAGS', kind=str, default="legacy", |
1606 | - validator_list=[config.ChoiceValidator(["legacy", "glibc"])], |
1607 | - help_text=_("Which implementation of extcmd to use")) |
1608 | - |
1609 | class Meta: |
1610 | |
1611 | # TODO: properly depend on xdg and use real code that also handles |
1612 | diff --git a/plainbox/impl/buildsystems.py b/plainbox/impl/buildsystems.py |
1613 | index 2e8d7ec..8eabeb6 100644 |
1614 | --- a/plainbox/impl/buildsystems.py |
1615 | +++ b/plainbox/impl/buildsystems.py |
1616 | @@ -29,13 +29,6 @@ from plainbox.abc import IBuildSystem |
1617 | from plainbox.impl.secure.plugins import PkgResourcesPlugInCollection |
1618 | |
1619 | |
1620 | -# python3.2 doesn't have shlex.quote |
1621 | -# so let's use the bundled copy here |
1622 | -if not hasattr(shlex, 'quote'): |
1623 | - from ._shlex import quote |
1624 | - shlex.quote = quote |
1625 | - |
1626 | - |
1627 | class MakefileBuildSystem(IBuildSystem): |
1628 | """ |
1629 | A build system for projects using classic makefiles |
1630 | diff --git a/plainbox/impl/commands/__init__.py b/plainbox/impl/commands/__init__.py |
1631 | index aa0c4de..5a26853 100644 |
1632 | --- a/plainbox/impl/commands/__init__.py |
1633 | +++ b/plainbox/impl/commands/__init__.py |
1634 | @@ -28,7 +28,7 @@ import logging |
1635 | |
1636 | from plainbox.impl.clitools import CommandBase |
1637 | from plainbox.impl.clitools import ToolBase |
1638 | -from plainbox.public import get_providers |
1639 | +from plainbox.impl.providers import get_providers |
1640 | |
1641 | |
1642 | logger = logging.getLogger("plainbox.commands") |
1643 | diff --git a/plainbox/impl/commands/cmd_analyze.py b/plainbox/impl/commands/cmd_analyze.py |
1644 | index 81c1f32..d9c040e 100644 |
1645 | --- a/plainbox/impl/commands/cmd_analyze.py |
1646 | +++ b/plainbox/impl/commands/cmd_analyze.py |
1647 | @@ -53,17 +53,6 @@ class AnalyzeCommand(PlainBoxCommand, CheckBoxCommandMixIn): |
1648 | parser = subparsers.add_parser( |
1649 | "analyze", help=_("analyze how selected jobs would be executed"), |
1650 | prog="plainbox dev analyze") |
1651 | - group = parser.add_mutually_exclusive_group() |
1652 | - group.add_argument( |
1653 | - '-l', '--run-local', |
1654 | - action='store_true', dest='run_local', |
1655 | - help=_('run all selected local jobs, required to see true data')) |
1656 | - group.add_argument( |
1657 | - '-L', '--skip-local', |
1658 | - action='store_false', dest='run_local', |
1659 | - # TRANSLATORS: please keep the word 'local' untranslated. |
1660 | - # It designates special type of jobs, not their location. |
1661 | - help=_('do not run local jobs')) |
1662 | group = parser.add_argument_group("reports") |
1663 | group.add_argument( |
1664 | '-s', '--print-stats', action='store_true', |
1665 | diff --git a/plainbox/impl/commands/cmd_checkbox.py b/plainbox/impl/commands/cmd_checkbox.py |
1666 | index 02fa6ca..5bc9e3c 100644 |
1667 | --- a/plainbox/impl/commands/cmd_checkbox.py |
1668 | +++ b/plainbox/impl/commands/cmd_checkbox.py |
1669 | @@ -62,12 +62,3 @@ class CheckBoxCommandMixIn: |
1670 | metavar=_("PATTERN"), default=[], dest='exclude_pattern_list', |
1671 | # TRANSLATORS: this is in imperative form |
1672 | help=_("exclude jobs matching the given regular expression")) |
1673 | - # TODO: Find a way to handle the encoding of the file |
1674 | - group.add_argument( |
1675 | - '-w', '--whitelist', |
1676 | - action="append", |
1677 | - metavar=_("WHITELIST"), |
1678 | - default=[], |
1679 | - type=FileType("rt"), |
1680 | - # TRANSLATORS: this is in imperative form |
1681 | - help=_("load whitelist containing run patterns")) |
1682 | diff --git a/plainbox/impl/commands/inv_analyze.py b/plainbox/impl/commands/inv_analyze.py |
1683 | index 71d99fe..194d9ef 100644 |
1684 | --- a/plainbox/impl/commands/inv_analyze.py |
1685 | +++ b/plainbox/impl/commands/inv_analyze.py |
1686 | @@ -56,12 +56,6 @@ class AnalyzeInvocation(CheckBoxInvocationMixIn): |
1687 | self.desired_job_list) |
1688 | |
1689 | def run(self): |
1690 | - if self.ns.run_local: |
1691 | - if self.ns.print_desired_job_list: |
1692 | - self._print_desired_job_list() |
1693 | - if self.ns.print_run_list: |
1694 | - self._print_run_list() |
1695 | - self._run_local_jobs() |
1696 | if self.ns.print_stats: |
1697 | self._print_general_stats() |
1698 | if self.ns.print_dependency_report: |
1699 | @@ -89,46 +83,6 @@ class AnalyzeInvocation(CheckBoxInvocationMixIn): |
1700 | for job in self.session.run_list: |
1701 | print("{}".format(job.id)) |
1702 | |
1703 | - def _run_local_jobs(self): |
1704 | - print(_("[Running Local Jobs]").center(80, '=')) |
1705 | - manager = SessionManager.create_with_state(self.session) |
1706 | - try: |
1707 | - manager.state.metadata.title = "plainbox dev analyze session" |
1708 | - manager.state.metadata.flags = [SessionMetaData.FLAG_INCOMPLETE] |
1709 | - manager.checkpoint() |
1710 | - runner = JobRunner( |
1711 | - manager.storage.location, self.provider_list, |
1712 | - os.path.join(manager.storage.location, 'io-logs'), |
1713 | - command_io_delegate=self) |
1714 | - again = True |
1715 | - while again: |
1716 | - for job in self.session.run_list: |
1717 | - if job.plugin == 'local': |
1718 | - job_state = self.session.job_state_map[job.id] |
1719 | - if job_state.result.outcome is None: |
1720 | - self._run_local_job(manager, runner, job, job_state) |
1721 | - break |
1722 | - else: |
1723 | - again = False |
1724 | - manager.state.metadata.flags = [] |
1725 | - manager.checkpoint() |
1726 | - finally: |
1727 | - manager.destroy() |
1728 | - |
1729 | - def _run_local_job(self, manager, runner, job, job_state): |
1730 | - print("{job}".format(job=job.id)) |
1731 | - manager.state.metadata.running_job_name = job.id |
1732 | - manager.checkpoint() |
1733 | - result = runner.run_job(job, job_state, self.config) |
1734 | - self.session.update_job_result(job, result) |
1735 | - new_desired_job_list = self._get_matching_job_list( |
1736 | - self.ns, self.session.job_list) |
1737 | - new_problem_list = self.session.update_desired_job_list( |
1738 | - new_desired_job_list) |
1739 | - if new_problem_list: |
1740 | - print(_("Problem list"), new_problem_list) |
1741 | - self.problem_list.extend(new_problem_list) |
1742 | - |
1743 | def _print_general_stats(self): |
1744 | print(_("[General Statistics]").center(80, '=')) |
1745 | print(_("Known jobs: {}").format(len(self.session.job_list))) |
1746 | diff --git a/plainbox/impl/commands/inv_checkbox.py b/plainbox/impl/commands/inv_checkbox.py |
1747 | index 5136c55..1321985 100644 |
1748 | --- a/plainbox/impl/commands/inv_checkbox.py |
1749 | +++ b/plainbox/impl/commands/inv_checkbox.py |
1750 | @@ -33,7 +33,6 @@ from plainbox.impl.secure.origin import CommandLineTextSource |
1751 | from plainbox.impl.secure.origin import Origin |
1752 | from plainbox.impl.secure.qualifiers import RegExpJobQualifier |
1753 | from plainbox.impl.secure.qualifiers import select_jobs |
1754 | -from plainbox.impl.secure.qualifiers import WhiteList |
1755 | from plainbox.impl.secure.rfc822 import FileTextSource |
1756 | |
1757 | logger = getLogger("plainbox.commands.checkbox") |
1758 | @@ -66,51 +65,6 @@ class CheckBoxInvocationMixIn: |
1759 | return list( |
1760 | itertools.chain(*[p.job_list for p in self.provider_list])) |
1761 | |
1762 | - def get_whitelist_from_file(self, filename, stream=None): |
1763 | - """ |
1764 | - Load a whitelist from a file, with special behavior. |
1765 | - |
1766 | - :param filename: |
1767 | - name of the file to load |
1768 | - :param stream: |
1769 | - (optional) pre-opened stream pointing at the whitelist |
1770 | - :returns: |
1771 | - The loaded whitelist or None if loading fails for any reason |
1772 | - |
1773 | - This function implements special loading behavior for whitelists that |
1774 | - makes them inherit the implicit namespace of the provider they may be a |
1775 | - part of. Before loading the whitelist directly from the file, all known |
1776 | - providers are interrogated to see if any of them has a whitelist that |
1777 | - was loaded from the same file (as indicated by os.path.realpath()) |
1778 | - |
1779 | - The stream argument can be provided if the caller already has an open |
1780 | - file object, which is typically the case when working with argparse. |
1781 | - """ |
1782 | - # Look up a whitelist with the same name in any of the providers |
1783 | - wanted_realpath = os.path.realpath(filename) |
1784 | - for provider in self.provider_list: |
1785 | - for whitelist in provider.whitelist_list: |
1786 | - if (whitelist.origin is not None |
1787 | - and whitelist.origin.source is not None |
1788 | - and isinstance(whitelist.origin.source, |
1789 | - FileTextSource) |
1790 | - and os.path.realpath( |
1791 | - whitelist.origin.source.filename) == |
1792 | - wanted_realpath): |
1793 | - logger.debug( |
1794 | - _("Using whitelist %r obtained from provider %r"), |
1795 | - whitelist.name, provider) |
1796 | - return whitelist |
1797 | - # Or load it directly |
1798 | - try: |
1799 | - if stream is not None: |
1800 | - return WhiteList.from_string(stream.read(), filename=filename) |
1801 | - else: |
1802 | - return WhiteList.from_file(filename) |
1803 | - except Exception as exc: |
1804 | - logger.warning( |
1805 | - _("Unable to load whitelist %r: %s"), filename, exc) |
1806 | - |
1807 | def _get_matching_job_list(self, ns, job_list): |
1808 | logger.debug("_get_matching_job_list(%r, %r)", ns, job_list) |
1809 | qualifier_list = [] |
1810 | @@ -126,12 +80,6 @@ class CheckBoxInvocationMixIn: |
1811 | break |
1812 | else: |
1813 | logger.debug(_("There is no test plan: %s"), ns.test_plan) |
1814 | - # Add whitelists |
1815 | - for whitelist_file in ns.whitelist: |
1816 | - qualifier = self.get_whitelist_from_file( |
1817 | - whitelist_file.name, whitelist_file) |
1818 | - if qualifier is not None: |
1819 | - qualifier_list.append(qualifier) |
1820 | # Add all the --include jobs |
1821 | for pattern in ns.include_pattern_list: |
1822 | origin = Origin(CommandLineTextSource('-i', pattern), None, None) |
1823 | diff --git a/plainbox/impl/commands/inv_run.py b/plainbox/impl/commands/inv_run.py |
1824 | index 1f0d5f8..c61c764 100644 |
1825 | --- a/plainbox/impl/commands/inv_run.py |
1826 | +++ b/plainbox/impl/commands/inv_run.py |
1827 | @@ -403,16 +403,11 @@ class RunInvocation(CheckBoxInvocationMixIn): |
1828 | self.metadata.flags.add(SessionMetaData.FLAG_INCOMPLETE) |
1829 | self.manager.checkpoint() |
1830 | # Select all the jobs that we are likely to run. This is the |
1831 | - # initial selection as we haven't started any jobs yet. Local jobs |
1832 | - # will cause that to happen again. |
1833 | + # initial selection as we haven't started any jobs yet. |
1834 | self.do_initial_job_selection() |
1835 | # Print out our estimates |
1836 | self.print_estimated_duration() |
1837 | - # Maybe ask the secure launcher to prompt for the password now. This is |
1838 | - # imperfect as we are going to run local jobs and we cannot see if they |
1839 | - # might need root or not. This cannot be fixed before template jobs are |
1840 | - # added and local jobs deprecated and removed (at least not being a |
1841 | - # part of the session we want to execute). |
1842 | + # Maybe ask the secure launcher to prompt for the password now. |
1843 | self.maybe_warm_up_authentication() |
1844 | # Iterate through the run list and run jobs if possible. This function |
1845 | # also implements backtrack to run new jobs that were added (and |
1846 | @@ -731,7 +726,7 @@ class RunInvocation(CheckBoxInvocationMixIn): |
1847 | estimated_time = 0 |
1848 | # gather jobs that we want to run and skip the jobs that already |
1849 | # have result, this is only needed when we run over the list of |
1850 | - # jobs again, after discovering new jobs via the local job output |
1851 | + # jobs again |
1852 | for job in self.state.run_list: |
1853 | job_state = self.state.job_state_map[job.id] |
1854 | if job_state.result.outcome is None: |
1855 | @@ -758,7 +753,7 @@ class RunInvocation(CheckBoxInvocationMixIn): |
1856 | |
1857 | def get_ui_for_job(self, job): |
1858 | if self.ns.dont_suppress_output is False and (job.plugin in ( |
1859 | - 'local', 'resource', 'attachment') or |
1860 | + 'resource', 'attachment') or |
1861 | 'suppress-output' in job.get_flag_set()): |
1862 | return NormalUI(self.C.c, show_cmd_output=False) |
1863 | else: |
1864 | diff --git a/plainbox/impl/commands/inv_special.py b/plainbox/impl/commands/inv_special.py |
1865 | index f93d0a7..bc82fa7 100644 |
1866 | --- a/plainbox/impl/commands/inv_special.py |
1867 | +++ b/plainbox/impl/commands/inv_special.py |
1868 | @@ -95,9 +95,6 @@ class SpecialInvocation(CheckBoxInvocationMixIn): |
1869 | print('\t"{}" [shape=ellipse,color=blue];'.format(job.id)) |
1870 | elif job.plugin == "attachment": |
1871 | print('\t"{}" [color=green];'.format(job.id)) |
1872 | - elif job.plugin == "local": |
1873 | - print('\t"{}" [shape=invtriangle,color=red];'.format( |
1874 | - job.id)) |
1875 | elif job.plugin == "shell": |
1876 | print('\t"{}" [];'.format(job.id)) |
1877 | elif job.plugin in ("manual", "user-verify", "user-interact"): |
1878 | diff --git a/plainbox/impl/commands/test_run.py b/plainbox/impl/commands/test_run.py |
1879 | index 19d7c9c..fe6f2a0 100644 |
1880 | --- a/plainbox/impl/commands/test_run.py |
1881 | +++ b/plainbox/impl/commands/test_run.py |
1882 | @@ -34,7 +34,6 @@ from inspect import cleandoc |
1883 | from unittest import TestCase |
1884 | |
1885 | from plainbox.impl.box import main |
1886 | -from plainbox.impl.exporter.rfc822 import RFC822SessionStateExporter |
1887 | from plainbox.impl.exporter.text import TextSessionStateExporter |
1888 | from plainbox.testing_utils.io import TestIO |
1889 | from plainbox.vendor.mock import patch, Mock |
1890 | @@ -63,7 +62,7 @@ class TestRun(TestCase): |
1891 | usage: plainbox run [-h] [--non-interactive] [-n] [--dont-suppress-output] |
1892 | [-f FORMAT] [-p OPTIONS] [-o FILE] [-t TRANSPORT] |
1893 | [--transport-where WHERE] [--transport-options OPTIONS] |
1894 | - [-T TEST-PLAN-ID] [-i PATTERN] [-x PATTERN] [-w WHITELIST] |
1895 | + [-T TEST-PLAN-ID] [-i PATTERN] [-x PATTERN] |
1896 | |
1897 | optional arguments: |
1898 | -h, --help show this help message and exit |
1899 | @@ -100,8 +99,6 @@ class TestRun(TestCase): |
1900 | include jobs matching the given regular expression |
1901 | -x PATTERN, --exclude-pattern PATTERN |
1902 | exclude jobs matching the given regular expression |
1903 | - -w WHITELIST, --whitelist WHITELIST |
1904 | - load whitelist containing run patterns |
1905 | """ |
1906 | self.assertEqual(io.combined, cleandoc(expected) + "\n") |
1907 | |
1908 | @@ -133,7 +130,6 @@ class TestRun(TestCase): |
1909 | Available output formats: |
1910 | com.canonical.plainbox::html - Generate a standalone HTML |
1911 | com.canonical.plainbox::json - Generate JSON output |
1912 | - com.canonical.plainbox::rfc822 - Generate RCF822 output |
1913 | com.canonical.plainbox::text - Generate plain text output |
1914 | com.canonical.plainbox::tar - Generate a tar.xz archive |
1915 | com.canonical.plainbox::xlsx - Generate an Excel 2007+ XLSX document |
1916 | @@ -150,10 +146,9 @@ class TestRun(TestCase): |
1917 | Each format may support a different set of options |
1918 | com.canonical.plainbox::html: |
1919 | com.canonical.plainbox::json: |
1920 | - com.canonical.plainbox::rfc822: with-io-log, squash-io-log, flatten-io-log, with-run-list, with-job-list, with-resource-map, with-job-defs, with-attachments, with-comments, with-job-via, with-job-hash, with-category-map, with-certification-status |
1921 | - com.canonical.plainbox::text: with-io-log, squash-io-log, flatten-io-log, with-run-list, with-job-list, with-resource-map, with-job-defs, with-attachments, with-comments, with-job-via, with-job-hash, with-category-map, with-certification-status |
1922 | + com.canonical.plainbox::text: with-io-log, squash-io-log, flatten-io-log, with-run-list, with-job-list, with-resource-map, with-job-defs, with-attachments, with-comments, with-job-hash, with-category-map, with-certification-status |
1923 | com.canonical.plainbox::tar: |
1924 | - com.canonical.plainbox::xlsx: with-sys-info, with-summary, with-job-description, with-text-attachments, with-unit-categories |
1925 | + com.canonical.plainbox::xlsx: with-sys-info, with-summary, with-job-description, with-text-attachments |
1926 | com.canonical.plainbox::global: |
1927 | """ |
1928 | self.assertIn(cleandoc(expected) + "\n", io.combined) |
1929 | diff --git a/plainbox/impl/ctrl.py b/plainbox/impl/ctrl.py |
1930 | index 1551c0e..056ee47 100644 |
1931 | --- a/plainbox/impl/ctrl.py |
1932 | +++ b/plainbox/impl/ctrl.py |
1933 | @@ -21,7 +21,7 @@ |
1934 | :mod:`plainbox.impl.ctrl` -- Controller Classes |
1935 | =============================================== |
1936 | |
1937 | -Session controller classes implement the glue between models (jobs, whitelists, |
1938 | +Session controller classes implement the glue between models (jobs, test plans, |
1939 | session state) and the rest of the application. They encapsulate knowledge that |
1940 | used to be special-cased and sprinkled around various parts of both plainbox |
1941 | and particular plainbox-using applications. |
1942 | @@ -102,9 +102,6 @@ class CheckBoxSessionStateController(ISessionStateController): |
1943 | true. This is expressed via the 'requires' attribute. A job will |
1944 | become inhibited if any of the requirement programs evaluates to |
1945 | value other than True. |
1946 | - * A job may have the attribute 'plugin' equal to "local" which will |
1947 | - cause the controller to interpret the stdout of the command as a set |
1948 | - of job definitions. |
1949 | * A job may have the attribute 'plugin' equal to "resource" which will |
1950 | cause the controller to interpret the stdout of the command as a set |
1951 | of resource definitions. |
1952 | @@ -243,8 +240,8 @@ class CheckBoxSessionStateController(ISessionStateController): |
1953 | Results also change the ready map (jobs that can run) because of |
1954 | dependency relations. |
1955 | |
1956 | - Some results have deeper meaning, those are results for local and |
1957 | - resource jobs. They are discussed in detail below: |
1958 | + Some results have deeper meaning, those are results for resource jobs. |
1959 | + They are discussed in detail below: |
1960 | |
1961 | Resource jobs produce resource records which are used as data to run |
1962 | requirement expressions against. Each time a result for a resource job |
1963 | @@ -252,11 +249,6 @@ class CheckBoxSessionStateController(ISessionStateController): |
1964 | records. A new entry is created in the resource map (entirely replacing |
1965 | any old entries), with a list of the resources that were parsed from |
1966 | the IO log. |
1967 | - |
1968 | - Local jobs produce more jobs. Like with resource jobs, their IO log is |
1969 | - parsed and interpreted as additional jobs. Unlike in resource jobs |
1970 | - local jobs don't replace anything. They cannot replace an existing job |
1971 | - with the same id. |
1972 | """ |
1973 | # Store the result in job_state_map |
1974 | session_state.job_state_map[job.id].result = result |
1975 | @@ -265,8 +257,6 @@ class CheckBoxSessionStateController(ISessionStateController): |
1976 | # Treat some jobs specially and interpret their output |
1977 | if job.plugin == "resource": |
1978 | self._process_resource_result(session_state, job, result) |
1979 | - elif job.plugin == "local": |
1980 | - self._process_local_result(session_state, job, result) |
1981 | |
1982 | def _process_resource_result(self, session_state, job, result): |
1983 | """ |
1984 | @@ -333,108 +323,6 @@ class CheckBoxSessionStateController(ISessionStateController): |
1985 | new_unit.id] |
1986 | job_state.via_job = job |
1987 | |
1988 | - def _process_local_result(self, session_state, job, result): |
1989 | - """ |
1990 | - Analyze a result of a CheckBox "local" job and generate |
1991 | - additional job definitions |
1992 | - """ |
1993 | - # First parse all records and create a list of new jobs (confusing |
1994 | - # name, not a new list of jobs) |
1995 | - new_job_list = [] |
1996 | - for record in gen_rfc822_records_from_io_log(job, result): |
1997 | - # Skip non-job units as the code below is wired to work with jobs |
1998 | - # Fixes: https://bugs.launchpad.net/plainbox/+bug/1443228 |
1999 | - if record.data.get('unit', 'job') != 'job': |
2000 | - continue |
2001 | - new_job = job.create_child_job_from_record(record) |
2002 | - check_result = new_job.check() |
2003 | - # Only ignore jobs for which check() returns an error |
2004 | - if [c for c in check_result if c.severity == Severity.error]: |
2005 | - logger.error(_("Ignoring invalid generated job %s"), |
2006 | - new_job.id) |
2007 | - else: |
2008 | - new_job_list.append(new_job) |
2009 | - # Then for each new job, add it to the job_list, unless it collides |
2010 | - # with another job with the same id. |
2011 | - for new_job in new_job_list: |
2012 | - try: |
2013 | - added_unit = session_state.add_unit(new_job, recompute=False) |
2014 | - except DependencyDuplicateError as exc: |
2015 | - # XXX: there should be a channel where such errors could be |
2016 | - # reported back to the UI layer. Perhaps update_job_result() |
2017 | - # could simply return a list of problems in a similar manner |
2018 | - # how update_desired_job_list() does. |
2019 | - logger.warning( |
2020 | - # TRANSLATORS: keep the word "local" untranslated. It is a |
2021 | - # special type of job that needs to be distinguished. |
2022 | - _("Local job %s produced job %s that collides with" |
2023 | - " an existing job %s (from %s), the new job was" |
2024 | - " discarded"), |
2025 | - job.id, exc.duplicate_job.id, exc.job.id, exc.job.origin) |
2026 | - else: |
2027 | - # Set the via_job attribute of the newly added job to point to |
2028 | - # the generator job. This way it can be traced back to the old |
2029 | - # __category__-style local jobs or to their corresponding |
2030 | - # generator job in general. |
2031 | - # |
2032 | - # NOTE: this is the only place where we assign via_job so as |
2033 | - # long as that holds true, we can detect and break via cycles. |
2034 | - # |
2035 | - # Via cycles occur whenever a job can reach itself again |
2036 | - # through via associations. Note that the chain may be longer |
2037 | - # than one link (A->A) and can include other jobs in the list |
2038 | - # (A->B->C->A) |
2039 | - # |
2040 | - # To detect a cycle we must iterate back the via chain (and we |
2041 | - # must do it here because we have access to job_state_map that |
2042 | - # allows this iteration to happen) and break the cycle if we |
2043 | - # see the job being added. |
2044 | - job_state_map = session_state.job_state_map |
2045 | - job_state_map[added_unit.id].via_job = job |
2046 | - via_cycle = get_via_cycle(job_state_map, added_unit) |
2047 | - if via_cycle: |
2048 | - logger.warning(_("Automatically breaking via-cycle: %s"), |
2049 | - ' -> '.join(str(cycle_job) |
2050 | - for cycle_job in via_cycle)) |
2051 | - job_state_map[added_unit.id].via_job = None |
2052 | - |
2053 | - |
2054 | -def get_via_cycle(job_state_map, job): |
2055 | - """ |
2056 | - Find a possible cycle including via_job. |
2057 | - |
2058 | - :param job_state_map: |
2059 | - A dictionary mapping job.id to a JobState object. |
2060 | - :param via_job: |
2061 | - Any job, start of a hypothetical via job cycle. |
2062 | - :raises KeyError: |
2063 | - If any of the encountered jobs are not present in job_state_map. |
2064 | - :return: |
2065 | - A list of jobs that represent the cycle or an empty tuple if no cycle |
2066 | - is present. The list has the property that item[0] is item[-1] |
2067 | - |
2068 | - A via cycle occurs if *job* is reachable through the *via_job* by |
2069 | - recursively following via_job connection until via_job becomes None. |
2070 | - """ |
2071 | - cycle = [] |
2072 | - seen = set() |
2073 | - while job is not None: |
2074 | - cycle.append(job) |
2075 | - seen.add(job) |
2076 | - next_job = job_state_map[job.id].via_job |
2077 | - if next_job in seen: |
2078 | - break |
2079 | - job = next_job |
2080 | - else: |
2081 | - return () |
2082 | - # Discard all the jobs leading to the cycle. |
2083 | - # cycle = cycle[cycle.index(next_job):] |
2084 | - # This is just to hold the promise of the return value so |
2085 | - # that processing is easier for the caller. |
2086 | - cycle.append(next_job) |
2087 | - # assert cycle[0] is cycle[-1] |
2088 | - return cycle |
2089 | - |
2090 | |
2091 | def gen_rfc822_records_from_io_log(job, result): |
2092 | """ |
2093 | diff --git a/plainbox/impl/depmgr.py b/plainbox/impl/depmgr.py |
2094 | index aa7fb50..f91bc2f 100644 |
2095 | --- a/plainbox/impl/depmgr.py |
2096 | +++ b/plainbox/impl/depmgr.py |
2097 | @@ -31,9 +31,9 @@ Job Dependency Solver. |
2098 | from abc import ABCMeta |
2099 | from abc import abstractproperty |
2100 | from logging import getLogger |
2101 | +import enum |
2102 | |
2103 | from plainbox.i18n import gettext as _ |
2104 | -from plainbox.vendor import enum |
2105 | |
2106 | |
2107 | logger = getLogger("plainbox.depmgr") |
2108 | diff --git a/plainbox/impl/exporter/__init__.py b/plainbox/impl/exporter/__init__.py |
2109 | index 703beca..4c86c5d 100644 |
2110 | --- a/plainbox/impl/exporter/__init__.py |
2111 | +++ b/plainbox/impl/exporter/__init__.py |
2112 | @@ -79,7 +79,6 @@ class SessionStateExporterBase(ISessionStateExporter): |
2113 | OPTION_WITH_JOB_DEFS = 'with-job-defs' |
2114 | OPTION_WITH_ATTACHMENTS = 'with-attachments' |
2115 | OPTION_WITH_COMMENTS = 'with-comments' |
2116 | - OPTION_WITH_JOB_VIA = 'with-job-via' |
2117 | OPTION_WITH_JOB_HASH = 'with-job-hash' |
2118 | OPTION_WITH_CATEGORY_MAP = 'with-category-map' |
2119 | OPTION_WITH_CERTIFICATION_STATUS = 'with-certification-status' |
2120 | @@ -94,7 +93,6 @@ class SessionStateExporterBase(ISessionStateExporter): |
2121 | OPTION_WITH_JOB_DEFS, |
2122 | OPTION_WITH_ATTACHMENTS, |
2123 | OPTION_WITH_COMMENTS, |
2124 | - OPTION_WITH_JOB_VIA, |
2125 | OPTION_WITH_JOB_HASH, |
2126 | OPTION_WITH_CATEGORY_MAP, |
2127 | OPTION_WITH_CERTIFICATION_STATUS, |
2128 | @@ -261,13 +259,6 @@ class SessionStateExporterBase(ISessionStateExporter): |
2129 | data['result_map'][job_id]['comments'] = \ |
2130 | job_state.result.comments |
2131 | |
2132 | - # Add Parent hash if requested |
2133 | - if self.OPTION_WITH_JOB_VIA in self._option_list: |
2134 | - data['result_map'][job_id]['via'] = ( |
2135 | - job_state.via_job.checksum |
2136 | - if job_state.via_job is not None else None |
2137 | - ) |
2138 | - |
2139 | # Add Job hash if requested |
2140 | if self.OPTION_WITH_JOB_HASH in self._option_list: |
2141 | data['result_map'][job_id]['hash'] = job_state.job.checksum |
2142 | diff --git a/plainbox/impl/exporter/rfc822.py b/plainbox/impl/exporter/rfc822.py |
2143 | deleted file mode 100644 |
2144 | index 8f3a185..0000000 |
2145 | --- a/plainbox/impl/exporter/rfc822.py |
2146 | +++ /dev/null |
2147 | @@ -1,48 +0,0 @@ |
2148 | -# This file is part of Checkbox. |
2149 | -# |
2150 | -# Copyright 2012 Canonical Ltd. |
2151 | -# Written by: |
2152 | -# Sylvain Pineau <sylvain.pineau@canonical.com> |
2153 | -# |
2154 | -# Checkbox is free software: you can redistribute it and/or modify |
2155 | -# it under the terms of the GNU General Public License version 3, |
2156 | -# as published by the Free Software Foundation. |
2157 | - |
2158 | -# |
2159 | -# Checkbox is distributed in the hope that it will be useful, |
2160 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2161 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2162 | -# GNU General Public License for more details. |
2163 | -# |
2164 | -# You should have received a copy of the GNU General Public License |
2165 | -# along with Checkbox. If not, see <http://www.gnu.org/licenses/>. |
2166 | - |
2167 | -""" |
2168 | -:mod:`plainbox.impl.exporter.rfc822` -- RFC822 exporter |
2169 | -======================================================= |
2170 | - |
2171 | -.. warning:: |
2172 | - |
2173 | - THIS MODULE DOES NOT HAVE STABLE PUBLIC API |
2174 | -""" |
2175 | - |
2176 | -from collections import OrderedDict |
2177 | -from io import StringIO |
2178 | - |
2179 | -from plainbox.impl.exporter import SessionStateExporterBase |
2180 | -from plainbox.impl.secure.rfc822 import RFC822Record |
2181 | - |
2182 | - |
2183 | -class RFC822SessionStateExporter(SessionStateExporterBase): |
2184 | - """ |
2185 | - Session state exporter creating rfc822 documents |
2186 | - """ |
2187 | - |
2188 | - def dump(self, data, stream): |
2189 | - entry = OrderedDict() |
2190 | - string_stream = StringIO() |
2191 | - for job_name, job_data in sorted(data['result_map'].items()): |
2192 | - entry['name'] = job_name |
2193 | - entry.update(job_data) |
2194 | - RFC822Record(entry).dump(string_stream) |
2195 | - stream.write(string_stream.getvalue().encode('UTF-8')) |
2196 | diff --git a/plainbox/impl/exporter/tar.py b/plainbox/impl/exporter/tar.py |
2197 | index 6fe8acb..6723e93 100644 |
2198 | --- a/plainbox/impl/exporter/tar.py |
2199 | +++ b/plainbox/impl/exporter/tar.py |
2200 | @@ -33,8 +33,8 @@ import time |
2201 | from plainbox.impl.exporter import SessionStateExporterBase |
2202 | from plainbox.impl.exporter.jinja2 import Jinja2SessionStateExporter |
2203 | from plainbox.impl.exporter.xlsx import XLSXSessionStateExporter |
2204 | +from plainbox.impl.providers import get_providers |
2205 | from plainbox.impl.unit.exporter import ExporterUnitSupport |
2206 | -from plainbox.public import get_providers |
2207 | |
2208 | |
2209 | class TARSessionStateExporter(SessionStateExporterBase): |
2210 | @@ -78,7 +78,6 @@ class TARSessionStateExporter(SessionStateExporterBase): |
2211 | XLSXSessionStateExporter.OPTION_WITH_SUMMARY, |
2212 | XLSXSessionStateExporter.OPTION_WITH_DESCRIPTION, |
2213 | XLSXSessionStateExporter.OPTION_WITH_TEXT_ATTACHMENTS, |
2214 | - XLSXSessionStateExporter.OPTION_WITH_UNIT_CATEGORIES |
2215 | ] |
2216 | xlsx_exporter = XLSXSessionStateExporter(options_list) |
2217 | xlsx_exporter.dump_from_session_manager(manager, xlsx_stream) |
2218 | diff --git a/plainbox/impl/exporter/test_html.py b/plainbox/impl/exporter/test_html.py |
2219 | index 9827a00..e763571 100644 |
2220 | --- a/plainbox/impl/exporter/test_html.py |
2221 | +++ b/plainbox/impl/exporter/test_html.py |
2222 | @@ -31,12 +31,12 @@ import io |
2223 | from plainbox.abc import IJobResult |
2224 | from plainbox.testing_utils import resource_string |
2225 | from plainbox.impl.exporter.jinja2 import Jinja2SessionStateExporter |
2226 | +from plainbox.impl.providers import get_providers |
2227 | from plainbox.impl.resource import Resource |
2228 | from plainbox.impl.result import MemoryJobResult |
2229 | from plainbox.impl.session import SessionManager |
2230 | from plainbox.impl.unit.exporter import ExporterUnitSupport |
2231 | from plainbox.impl.unit.job import JobDefinition |
2232 | -from plainbox.public import get_providers |
2233 | |
2234 | |
2235 | class HTMLExporterTests(TestCase): |
2236 | diff --git a/plainbox/impl/exporter/test_init.py b/plainbox/impl/exporter/test_init.py |
2237 | index e590ae4..8980390 100644 |
2238 | --- a/plainbox/impl/exporter/test_init.py |
2239 | +++ b/plainbox/impl/exporter/test_init.py |
2240 | @@ -219,7 +219,6 @@ class SessionStateExporterBaseTests(TestCase): |
2241 | 'uncategorised')), |
2242 | ('outcome', 'pass'), |
2243 | ('comments', None), |
2244 | - ('via', None), |
2245 | ('hash', '2def0c995e1b6d934c5a91286ba164' |
2246 | '18845da26d057bc992a2b5dfeae2e2fe91'), |
2247 | ('plugin', 'shell'), |
2248 | @@ -234,7 +233,6 @@ class SessionStateExporterBaseTests(TestCase): |
2249 | 'uncategorised')), |
2250 | ('outcome', 'pass'), |
2251 | ('comments', 'foo'), |
2252 | - ('via', None), |
2253 | ('hash', 'ed19ba54624864a7c622ff7d1e8ed5' |
2254 | '96b1a0fddc4b78c8fb780fe41e55250e6f'), |
2255 | ('plugin', 'resource'), |
2256 | diff --git a/plainbox/impl/exporter/test_rfc822.py b/plainbox/impl/exporter/test_rfc822.py |
2257 | deleted file mode 100644 |
2258 | index b858b61..0000000 |
2259 | --- a/plainbox/impl/exporter/test_rfc822.py |
2260 | +++ /dev/null |
2261 | @@ -1,47 +0,0 @@ |
2262 | -# This file is part of Checkbox. |
2263 | -# |
2264 | -# Copyright 2012 Canonical Ltd. |
2265 | -# Written by: |
2266 | -# Zygmunt Krynicki <zygmunt.krynicki@canonical.com> |
2267 | -# Daniel Manrique <roadmr@ubuntu.com> |
2268 | -# |
2269 | -# Checkbox is free software: you can redistribute it and/or modify |
2270 | -# it under the terms of the GNU General Public License version 3, |
2271 | -# as published by the Free Software Foundation. |
2272 | - |
2273 | -# |
2274 | -# Checkbox is distributed in the hope that it will be useful, |
2275 | -# but WITHOUT ANY WARRANTY; without even the implied warranty of |
2276 | -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the |
2277 | -# GNU General Public License for more details. |
2278 | -# |
2279 | -# You should have received a copy of the GNU General Public License |
2280 | -# along with Checkbox. If not, see <http://www.gnu.org/licenses/>. |
2281 | - |
2282 | -""" |
2283 | -plainbox.impl.exporter.test_rfc822 |
2284 | -================================== |
2285 | - |
2286 | -Test definitions for plainbox.impl.exporter.rfc822 module |
2287 | -""" |
2288 | - |
2289 | -from io import BytesIO |
2290 | -from unittest import TestCase |
2291 | - |
2292 | -from plainbox.impl.exporter.rfc822 import RFC822SessionStateExporter |
2293 | - |
2294 | - |
2295 | -class RFC822SessionStateExporterTests(TestCase): |
2296 | - |
2297 | - def test_dump(self): |
2298 | - exporter = RFC822SessionStateExporter() |
2299 | - # exporter expects this data format |
2300 | - data = {'result_map': {'job_name': {'outcome': 'fail'}}} |
2301 | - stream = BytesIO() |
2302 | - exporter.dump(data, stream) |
2303 | - expected_bytes = ( |
2304 | - "name: job_name\n" |
2305 | - "outcome: fail\n" |
2306 | - "\n" |
2307 | - ).encode('UTF-8') |
2308 | - self.assertEqual(stream.getvalue(), expected_bytes) |
2309 | diff --git a/plainbox/impl/exporter/xlsx.py b/plainbox/impl/exporter/xlsx.py |
2310 | index 7de2e8a..e51b193 100644 |
2311 | --- a/plainbox/impl/exporter/xlsx.py |
2312 | +++ b/plainbox/impl/exporter/xlsx.py |
2313 | @@ -63,14 +63,12 @@ class XLSXSessionStateExporter(SessionStateExporterBase): |
2314 | OPTION_WITH_SUMMARY = 'with-summary' |
2315 | OPTION_WITH_DESCRIPTION = 'with-job-description' |
2316 | OPTION_WITH_TEXT_ATTACHMENTS = 'with-text-attachments' |
2317 | - OPTION_WITH_UNIT_CATEGORIES = 'with-unit-categories' |
2318 | |
2319 | SUPPORTED_OPTION_LIST = ( |
2320 | OPTION_WITH_SYSTEM_INFO, |
2321 | OPTION_WITH_SUMMARY, |
2322 | OPTION_WITH_DESCRIPTION, |
2323 | OPTION_WITH_TEXT_ATTACHMENTS, |
2324 | - OPTION_WITH_UNIT_CATEGORIES, |
2325 | ) |
2326 | |
2327 | def __init__(self, option_list=None, exporter_unit=None): |
2328 | @@ -96,8 +94,6 @@ class XLSXSessionStateExporter(SessionStateExporterBase): |
2329 | SessionStateExporterBase.OPTION_FLATTEN_IO_LOG, |
2330 | SessionStateExporterBase.OPTION_WITH_COMMENTS, |
2331 | SessionStateExporterBase.OPTION_WITH_JOB_DEFS, |
2332 | - SessionStateExporterBase.OPTION_WITH_JOB_VIA, |
2333 | - SessionStateExporterBase.OPTION_WITH_JOB_HASH, |
2334 | SessionStateExporterBase.OPTION_WITH_RESOURCE_MAP, |
2335 | SessionStateExporterBase.OPTION_WITH_ATTACHMENTS, |
2336 | SessionStateExporterBase.OPTION_WITH_CATEGORY_MAP, |
2337 | @@ -443,30 +439,6 @@ class XLSXSessionStateExporter(SessionStateExporterBase): |
2338 | 'x_offset': 0, 'y_offset': 10, 'x_scale': 0.50, 'y_scale': 0.50 |
2339 | }) |
2340 | |
2341 | - def _set_category_status(self, result_map, via, child): |
2342 | - for parent in [j for j in result_map if result_map[j]['hash'] == via]: |
2343 | - if 'category_status' not in result_map[parent]: |
2344 | - result_map[parent]['category_status'] = None |
2345 | - child_status = result_map[child]['outcome'] |
2346 | - if 'category_status' in result_map[child]: |
2347 | - child_status = result_map[child]['category_status'] |
2348 | - # Ignore categories without any child |
2349 | - elif result_map[child]['plugin'] == 'local': |
2350 | - continue |
2351 | - if child_status == IJobResult.OUTCOME_FAIL: |
2352 | - result_map[parent]['category_status'] = IJobResult.OUTCOME_FAIL |
2353 | - elif ( |
2354 | - child_status == IJobResult.OUTCOME_PASS and |
2355 | - result_map[parent]['category_status'] != |
2356 | - IJobResult.OUTCOME_FAIL |
2357 | - ): |
2358 | - result_map[parent]['category_status'] = IJobResult.OUTCOME_PASS |
2359 | - elif ( |
2360 | - result_map[parent]['category_status'] not in |
2361 | - (IJobResult.OUTCOME_PASS, IJobResult.OUTCOME_FAIL) |
2362 | - ): |
2363 | - result_map[parent]['category_status'] = IJobResult.OUTCOME_SKIP |
2364 | - |
2365 | def _tree(self, result_map, category_map): |
2366 | res = {} |
2367 | tmp_result_map = {} |
2368 | @@ -500,21 +472,6 @@ class XLSXSessionStateExporter(SessionStateExporterBase): |
2369 | result_map.update(tmp_result_map) |
2370 | return res, 2 |
2371 | |
2372 | - def _legacy_tree(self, result_map, via=None, level=0, max_level=0): |
2373 | - res = {} |
2374 | - for job_name in [j for j in result_map if result_map[j]['via'] == via]: |
2375 | - level += 1 |
2376 | - # Find the maximum depth of the test tree |
2377 | - if level > max_level: |
2378 | - max_level = level |
2379 | - res[job_name], max_level = self._legacy_tree( |
2380 | - result_map, result_map[job_name]['hash'], level, max_level) |
2381 | - # Generate parent categories status |
2382 | - if via is not None: |
2383 | - self._set_category_status(result_map, via, job_name) |
2384 | - level -= 1 |
2385 | - return res, max_level |
2386 | - |
2387 | def _write_job(self, tree, result_map, max_level, level=0): |
2388 | for job, children in OrderedDict( |
2389 | sorted( |
2390 | @@ -635,11 +592,8 @@ class XLSXSessionStateExporter(SessionStateExporterBase): |
2391 | {'hidden': True}) |
2392 | |
2393 | def write_results(self, data): |
2394 | - if self.OPTION_WITH_UNIT_CATEGORIES in self._option_list: |
2395 | - tree, max_level = self._tree( |
2396 | - data['result_map'], data['category_map']) |
2397 | - else: |
2398 | - tree, max_level = self._legacy_tree(data['result_map']) |
2399 | + tree, max_level = self._tree( |
2400 | + data['result_map'], data['category_map']) |
2401 | self.worksheet3.write(3, 1, _('Tests Performed'), self.format03) |
2402 | self.worksheet3.freeze_panes(6, 0) |
2403 | self.worksheet3.set_tab_color('#DC4C00') # Orange |
2404 | diff --git a/plainbox/impl/highlevel.py b/plainbox/impl/highlevel.py |
2405 | index 4c36ae8..eeed23e 100644 |
2406 | --- a/plainbox/impl/highlevel.py |
2407 | +++ b/plainbox/impl/highlevel.py |
2408 | @@ -123,9 +123,6 @@ class PlainBoxObject: |
2409 | return self._attrs |
2410 | |
2411 | |
2412 | -# NOTE: This should merge with the service object below but I didn't want |
2413 | -# to do it right away as that would have to alter Service.__init__() and |
2414 | -# I want to get Explorer API right first. |
2415 | class Explorer: |
2416 | """ |
2417 | Class simplifying discovery of various PlainBox objects. |
2418 | @@ -164,7 +161,6 @@ class Explorer: |
2419 | the explorer itself |
2420 | - all providers |
2421 | - all jobs |
2422 | - - all whitelists |
2423 | - all executables |
2424 | - all repositories |
2425 | - all storages |
2426 | @@ -173,7 +169,7 @@ class Explorer: |
2427 | self, |
2428 | name='service object', |
2429 | group="service") |
2430 | - # Milk each provider for jobs and whitelists |
2431 | + # Milk each provider for jobs and test plans |
2432 | for provider in self.provider_list: |
2433 | provider_obj = PlainBoxObject( |
2434 | provider, |
2435 | @@ -189,7 +185,6 @@ class Explorer: |
2436 | ('tr_description', provider.tr_description()), |
2437 | ('jobs_dir', provider.jobs_dir), |
2438 | ('units_dir', provider.units_dir), |
2439 | - ('whitelists_dir', provider.whitelists_dir), |
2440 | ('data_dir', provider.data_dir), |
2441 | ('locale_dir', provider.locale_dir), |
2442 | ('gettext_domain', provider.gettext_domain), |
2443 | diff --git a/plainbox/impl/launcher.py b/plainbox/impl/launcher.py |
2444 | index c535aef..897d80c 100644 |
2445 | --- a/plainbox/impl/launcher.py |
2446 | +++ b/plainbox/impl/launcher.py |
2447 | @@ -65,93 +65,7 @@ class LauncherDefinition(PlainBoxConfig): |
2448 | :returns: LauncherDefinition instance |
2449 | :raises KeyError: for unknown launcher_version values |
2450 | """ |
2451 | - return { |
2452 | - '1': LauncherDefinition1, |
2453 | - config.Unset: LauncherDefinitionLegacy |
2454 | - }[self.launcher_version]() |
2455 | - |
2456 | - |
2457 | -class LauncherDefinitionLegacy(LauncherDefinition): |
2458 | - """ |
2459 | - Launcher definition class for 'unversioned' launchers. |
2460 | - |
2461 | - This definition is used for launchers that do not specify |
2462 | - 'launcher_version' in their [launcher] section. |
2463 | - """ |
2464 | - |
2465 | - api_flags = config.Variable( |
2466 | - section='launcher', |
2467 | - kind=list, |
2468 | - default=[], |
2469 | - help_text=_('List of feature-flags the application requires')) |
2470 | - |
2471 | - api_version = config.Variable( |
2472 | - section='launcher', |
2473 | - default='0.99', |
2474 | - help_text=_('Version of API the launcher uses')) |
2475 | - |
2476 | - title = config.Variable( |
2477 | - section="welcome", |
2478 | - help_text=_("Application Title")) |
2479 | - |
2480 | - text = config.Variable( |
2481 | - section="welcome", |
2482 | - help_text=_("Welcome Message")) |
2483 | - |
2484 | - dont_suppress_output = config.Variable( |
2485 | - section="ui", kind=bool, default=False, |
2486 | - help_text=_("Don't suppress the output of certain job plugin types.")) |
2487 | - |
2488 | - whitelist_filter = config.Variable( |
2489 | - section="suite", |
2490 | - # TODO: valid regexp text validator |
2491 | - help_text=_("Pattern that whitelists need to match to be displayed")) |
2492 | - |
2493 | - whitelist_selection = config.Variable( |
2494 | - section="suite", |
2495 | - # TODO: valid regexp text validator |
2496 | - help_text=_("Pattern that whitelists need to match to be selected")) |
2497 | - |
2498 | - skip_whitelist_selection = config.Variable( |
2499 | - section="suite", |
2500 | - kind=bool, |
2501 | - default=False, |
2502 | - help_text=_("If enabled then suite selection screen is not displayed")) |
2503 | - |
2504 | - skip_test_selection = config.Variable( |
2505 | - section="suite", |
2506 | - kind=bool, |
2507 | - default=False, |
2508 | - help_text=_("If enabled then test selection screen is not displayed")) |
2509 | - |
2510 | - input_type = config.Variable( |
2511 | - section="submission", |
2512 | - # TODO: probably a choice validator |
2513 | - help_text=_("Type of the input field?")) |
2514 | - |
2515 | - ok_btn_text = config.Variable( |
2516 | - section="submission", |
2517 | - help_text=_("Label on the 'send' button")) |
2518 | - |
2519 | - submit_to = config.Variable( |
2520 | - section="transport", |
2521 | - validator_list=[config.ChoiceValidator(get_all_transports().keys())], |
2522 | - help_text=_("Where to submit the test results to")) |
2523 | - |
2524 | - # TODO: Add a validator to ensure it looks like a valid URL |
2525 | - submit_url = config.Variable( |
2526 | - section="transport", |
2527 | - help_text=_("HTTP endpoint to submit data to, using the" |
2528 | - " transport specified with submit_to.")) |
2529 | - |
2530 | - secure_id = config.Variable( |
2531 | - section="submission", |
2532 | - validator_list=[config.PatternValidator(SECURE_ID_PATTERN)], |
2533 | - help_text=_("Secure ID to identify the system this" |
2534 | - " submission belongs to.")) |
2535 | - |
2536 | - exporter = config.Section( |
2537 | - help_text=_("Section with only exported unit ids as keys (no values)")) |
2538 | + return {'1': LauncherDefinition1}[self.launcher_version]() |
2539 | |
2540 | |
2541 | class LauncherDefinition1(LauncherDefinition): |
2542 | diff --git a/plainbox/impl/providers/__init__.py b/plainbox/impl/providers/__init__.py |
2543 | index a6a510b..8a6b121 100644 |
2544 | --- a/plainbox/impl/providers/__init__.py |
2545 | +++ b/plainbox/impl/providers/__init__.py |
2546 | @@ -22,18 +22,10 @@ APIs for working with providers. |
2547 | :mod:`plainbox.impl.providers` -- providers package |
2548 | =================================================== |
2549 | |
2550 | -Providers are a mechanism by which PlainBox can enumerate jobs and whitelists. |
2551 | +Providers are a mechanism by which PlainBox can enumerate jobs and test plans. |
2552 | Currently there are only v1 (as in version one) providers that basically have |
2553 | to behave as CheckBox itself (mini CheckBox forks for example) |
2554 | |
2555 | -There is ongoing work and discussion on V2 providers that would have a |
2556 | -lower-level interface and would be able to define new job types, new whitelist |
2557 | -types and generally all the next-gen semantics. |
2558 | - |
2559 | -PlainBox does not come with any real provider by default. PlainBox sometimes |
2560 | -creates special dummy providers that have particular data in them for testing. |
2561 | - |
2562 | - |
2563 | V1 providers |
2564 | ------------ |
2565 | |
2566 | @@ -43,9 +35,6 @@ this is also described by :class:`IProvider1`:: |
2567 | * there is a directory with '.txt' or '.txt.in' files with RFC822-encoded |
2568 | job definitions. The definitions need a particular set of keys to work. |
2569 | |
2570 | - * there is a directory with '.whitelist' files that contain a list (one per |
2571 | - line) of job definitions to execute. |
2572 | - |
2573 | * there is a directory with additional executables (added to PATH) |
2574 | |
2575 | * there is a directory with an additional python3 libraries (added to |
2576 | diff --git a/plainbox/impl/providers/exporters/data/checkbox.html b/plainbox/impl/providers/exporters/data/checkbox.html |
2577 | index 8b6e5c6..9bc8eba 100644 |
2578 | --- a/plainbox/impl/providers/exporters/data/checkbox.html |
2579 | +++ b/plainbox/impl/providers/exporters/data/checkbox.html |
2580 | @@ -225,7 +225,7 @@ |
2581 | </tr> |
2582 | </thead> |
2583 | <tbody> |
2584 | - {%- for job_id, job_state in job_state_map|dictsort if job_state.result.outcome != None and job_state.job.plugin not in ("resource", "local", "attachment") %} |
2585 | + {%- for job_id, job_state in job_state_map|dictsort if job_state.result.outcome != None and job_state.job.plugin not in ("resource", "attachment") %} |
2586 | <tr> |
2587 | <td>{{ job_state.job.tr_summary() }}</td> |
2588 | <td style='font-weight: bold; color: {{ job_state.result.outcome_meta().color_hex }}'>{{ job_state.result.outcome_meta().tr_label }}</td> |
2589 | diff --git a/plainbox/impl/providers/exporters/data/checkbox.json b/plainbox/impl/providers/exporters/data/checkbox.json |
2590 | index 00ed7e1..cd49b05 100644 |
2591 | --- a/plainbox/impl/providers/exporters/data/checkbox.json |
2592 | +++ b/plainbox/impl/providers/exporters/data/checkbox.json |
2593 | @@ -36,7 +36,7 @@ |
2594 | }, |
2595 | {%- endif %} |
2596 | "results": [ |
2597 | - {%- for job_id, job_state in job_state_map|dictsort if job_state.result.outcome != None and job_state.job.plugin not in ("resource", "local", "attachment") %} |
2598 | + {%- for job_id, job_state in job_state_map|dictsort if job_state.result.outcome != None and job_state.job.plugin not in ("resource", "attachment") %} |
2599 | { |
2600 | "id": "{{ job_id|strip_ns }}", |
2601 | "name": "{{ job_state.job.tr_summary() }}", |
2602 | diff --git a/plainbox/impl/providers/exporters/units/exporter.pxu b/plainbox/impl/providers/exporters/units/exporter.pxu |
2603 | index 0d7ec67..65f96eb 100644 |
2604 | --- a/plainbox/impl/providers/exporters/units/exporter.pxu |
2605 | +++ b/plainbox/impl/providers/exporters/units/exporter.pxu |
2606 | @@ -13,12 +13,6 @@ file_extension: json |
2607 | data: {"template": "checkbox.json"} |
2608 | |
2609 | unit: exporter |
2610 | -id: rfc822 |
2611 | -_summary: Generate RCF822 output |
2612 | -entry_point: rfc822 |
2613 | -file_extension: rfc822 |
2614 | - |
2615 | -unit: exporter |
2616 | id: text |
2617 | _summary: Generate plain text output |
2618 | entry_point: text |
2619 | @@ -36,7 +30,7 @@ _summary: Generate an Excel 2007+ XLSX document |
2620 | entry_point: xlsx |
2621 | file_extension: xlsx |
2622 | options: |
2623 | - with-sys-info, with-summary, with-job-description, with-text-attachments, with-unit-categories |
2624 | + with-sys-info, with-summary, with-job-description, with-text-attachments |
2625 | |
2626 | unit: exporter |
2627 | id: global |
2628 | diff --git a/plainbox/impl/providers/stubbox/units/jobs/local.pxu b/plainbox/impl/providers/stubbox/units/jobs/local.pxu |
2629 | deleted file mode 100644 |
2630 | index 0f41e6c..0000000 |
2631 | --- a/plainbox/impl/providers/stubbox/units/jobs/local.pxu |
2632 | +++ /dev/null |
2633 | @@ -1,9 +0,0 @@ |
2634 | -id: stub/local/true |
2635 | -_summary: A job generated by another job |
2636 | -# TRANSLATORS: don't translate 'local' below. |
2637 | -_description: |
2638 | - Check success result from shell test case (generated from a local job) |
2639 | -plugin: shell |
2640 | -flags: preserve-locale |
2641 | -command: true |
2642 | -estimated_duration: 0.1 |
2643 | diff --git a/plainbox/impl/providers/stubbox/units/jobs/multilevel.pxu b/plainbox/impl/providers/stubbox/units/jobs/multilevel.pxu |
2644 | deleted file mode 100644 |
2645 | index f01a716..0000000 |
2646 | --- a/plainbox/impl/providers/stubbox/units/jobs/multilevel.pxu |
2647 | +++ /dev/null |
2648 | @@ -1,25 +0,0 @@ |
2649 | -id: stub/multilevel |
2650 | -_summary: A generated generator job that generates two more jobs |
2651 | -_description: Multilevel tests |
2652 | -plugin: local |
2653 | -flags: preserve-locale |
2654 | -command: |
2655 | - cat <<'EOF' |
2656 | - id: stub/multilevel_1 |
2657 | - _summary: Generated multi-level job 1 |
2658 | - _description: This is just a sample multilevel test. Test 1. |
2659 | - plugin: shell |
2660 | - command: echo 1 |
2661 | - estimated_duration: 0.1 |
2662 | - EOF |
2663 | - echo "" |
2664 | - cat <<'EOF' |
2665 | - id: stub/multilevel_2 |
2666 | - _summary: Generated multi-level job 2 |
2667 | - _description: This is just a sample multilevel test. Test 2. |
2668 | - plugin: shell |
2669 | - command: echo 2 |
2670 | - estimated_duration: 0.1 |
2671 | - EOF |
2672 | - echo "" |
2673 | -estimated_duration: 0.1 |
2674 | diff --git a/plainbox/impl/providers/stubbox/units/jobs/representative.pxu b/plainbox/impl/providers/stubbox/units/jobs/representative.pxu |
2675 | index f7d7f60..8bab00d 100644 |
2676 | --- a/plainbox/impl/providers/stubbox/units/jobs/representative.pxu |
2677 | +++ b/plainbox/impl/providers/stubbox/units/jobs/representative.pxu |
2678 | @@ -29,15 +29,6 @@ command: |
2679 | estimated_duration: 0.1 |
2680 | category_id: plugin-representative |
2681 | |
2682 | -id: representative/plugin/local |
2683 | -_summary: Job with plugin=local |
2684 | -_description: Job with plugin=local |
2685 | -plugin: local |
2686 | -flags: preserve-locale |
2687 | -command: : |
2688 | -estimated_duration: 0.1 |
2689 | -category_id: plugin-representative |
2690 | - |
2691 | id: representative/plugin/attachment |
2692 | _summary: Job with plugin=attachment |
2693 | _description: Job with plugin=attachment |
2694 | diff --git a/plainbox/impl/providers/stubbox/units/jobs/stub.pxu b/plainbox/impl/providers/stubbox/units/jobs/stub.pxu |
2695 | index 7bdfc8d..55c1d62 100644 |
2696 | --- a/plainbox/impl/providers/stubbox/units/jobs/stub.pxu |
2697 | +++ b/plainbox/impl/providers/stubbox/units/jobs/stub.pxu |
2698 | @@ -328,30 +328,6 @@ command: false |
2699 | estimated_duration: 25 |
2700 | category_id: split-field-representative |
2701 | |
2702 | -id: __local__ |
2703 | -_summary: A job generating one more job |
2704 | -_description: |
2705 | - This job generates the stub/local/true job |
2706 | -plugin: local |
2707 | -flags: preserve-locale |
2708 | -command: |
2709 | - shopt -s extglob |
2710 | - cat $PLAINBOX_PROVIDER_UNITS/jobs/local.pxu |
2711 | -estimated_duration: 0.1 |
2712 | -category_id: plugin-representative |
2713 | - |
2714 | -id: __multilevel__ |
2715 | -_summary: A job generating more generator jobs |
2716 | -_description: |
2717 | - This job generates stub/multilevel which in turn can |
2718 | - generate stub/multilevel_1 and stub/multilevel_2 |
2719 | -plugin: local |
2720 | -flags: preserve-locale |
2721 | -command: |
2722 | - shopt -s extglob |
2723 | - cat $PLAINBOX_PROVIDER_UNITS/jobs/multilevel.pxu |
2724 | -estimated_duration: 0.1 |
2725 | - |
2726 | id: stub/root |
2727 | _summary: A job that runs as root |
2728 | _description: |
2729 | diff --git a/plainbox/impl/providers/stubbox/whitelists/stub.whitelist b/plainbox/impl/providers/stubbox/whitelists/stub.whitelist |
2730 | deleted file mode 100644 |
2731 | index 3c06fd3..0000000 |
2732 | --- a/plainbox/impl/providers/stubbox/whitelists/stub.whitelist |
2733 | +++ /dev/null |
2734 | @@ -1,23 +0,0 @@ |
2735 | -# Shell job that always works |
2736 | -stub/true |
2737 | -# Shell job that always fails |
2738 | -stub/false |
2739 | -# User-* Job collection |
2740 | -stub/user-verify |
2741 | -stub/user-interact |
2742 | -stub/user-interact-verify |
2743 | -# A manual job |
2744 | -stub/manual |
2745 | -# A shell job with a dependency that always works |
2746 | -stub/dependency/good |
2747 | -# A shell job with a dependency that always fails |
2748 | -stub/dependency/bad |
2749 | -# A shell job that requires a resource which is available |
2750 | -stub/requirement/good |
2751 | -# A shell job that requires a resource which is not available |
2752 | -stub/requirement/bad |
2753 | -__local__ |
2754 | -stub/local/true |
2755 | -stub/multilevel |
2756 | -stub/multilevel.* |
2757 | -stub/root |
2758 | diff --git a/plainbox/impl/providers/stubbox/whitelists/stub1.whitelist b/plainbox/impl/providers/stubbox/whitelists/stub1.whitelist |
2759 | deleted file mode 100644 |
2760 | index d33c8a4..0000000 |
2761 | --- a/plainbox/impl/providers/stubbox/whitelists/stub1.whitelist |
2762 | +++ /dev/null |
2763 | @@ -1,7 +0,0 @@ |
2764 | -stub/true |
2765 | -stub/dependency/bad |
2766 | -# stub_package |
2767 | -stub/requirement/good |
2768 | -stub/requirement/bad |
2769 | -stub/multilevel |
2770 | -stub/multilevel.* |
2771 | diff --git a/plainbox/impl/providers/stubbox/whitelists/stub2.whitelist b/plainbox/impl/providers/stubbox/whitelists/stub2.whitelist |
2772 | deleted file mode 100644 |
2773 | index 7785d21..0000000 |
2774 | --- a/plainbox/impl/providers/stubbox/whitelists/stub2.whitelist |
2775 | +++ /dev/null |
2776 | @@ -1,7 +0,0 @@ |
2777 | -stub/false |
2778 | -stub/dependency/good |
2779 | -stub/dependency/bad |
2780 | -# stub_package |
2781 | -stub/manual |
2782 | -__local__ |
2783 | -stub/local/true |
2784 | diff --git a/plainbox/impl/result.py b/plainbox/impl/result.py |
2785 | index ec325e6..e31b069 100644 |
2786 | --- a/plainbox/impl/result.py |
2787 | +++ b/plainbox/impl/result.py |
2788 | @@ -461,30 +461,6 @@ class MemoryJobResult(_JobResultBase): |
2789 | " or special the IOLogRecord tuple") |
2790 | |
2791 | |
2792 | -class GzipFile(gzip.GzipFile): |
2793 | - |
2794 | - """ |
2795 | - Subclass of GzipFile that works around missing read1() on python3.2. |
2796 | - |
2797 | - See: http://bugs.python.org/issue10791 |
2798 | - """ |
2799 | - |
2800 | - def _read_gzip_header(self): |
2801 | - """ |
2802 | - Ignore the non-compressed garbage at the end of the file |
2803 | - |
2804 | - See: https://bugs.python.org/issue24301 |
2805 | - """ |
2806 | - |
2807 | - try: |
2808 | - return super()._read_gzip_header() |
2809 | - except OSError: |
2810 | - return False |
2811 | - |
2812 | - def read1(self, n): |
2813 | - return self.read(n) |
2814 | - |
2815 | - |
2816 | class DiskJobResult(_JobResultBase): |
2817 | |
2818 | """ |
2819 | @@ -505,7 +481,7 @@ class DiskJobResult(_JobResultBase): |
2820 | def get_io_log(self): |
2821 | record_path = self.io_log_filename |
2822 | if record_path: |
2823 | - with GzipFile(record_path, mode='rb') as gzip_stream, \ |
2824 | + with gzip.GzipFile(record_path, mode='rb') as gzip_stream, \ |
2825 | io.TextIOWrapper(gzip_stream, encoding='UTF-8') as stream: |
2826 | for record in IOLogRecordReader(stream): |
2827 | yield record |
2828 | diff --git a/plainbox/impl/runner.py b/plainbox/impl/runner.py |
2829 | index 8c22892..e5a6e28 100644 |
2830 | --- a/plainbox/impl/runner.py |
2831 | +++ b/plainbox/impl/runner.py |
2832 | @@ -275,7 +275,7 @@ class JobRunner(IJobRunner): |
2833 | """ |
2834 | |
2835 | # List of plugins that are still executed |
2836 | - _DRY_RUN_PLUGINS = ('local', 'resource', 'attachment') |
2837 | + _DRY_RUN_PLUGINS = ('resource', 'attachment') |
2838 | |
2839 | def __init__(self, session_dir, provider_list, jobs_io_log_dir, |
2840 | command_io_delegate=None, dry_run=False, |
2841 | @@ -520,32 +520,6 @@ class JobRunner(IJobRunner): |
2842 | job, job_state, config).get_result() |
2843 | return result |
2844 | |
2845 | - def run_local_job(self, job, job_state, config): |
2846 | - """ |
2847 | - Method called to run a job with plugin field equal to 'local'. |
2848 | - |
2849 | - The 'local' job implements the following scenario: |
2850 | - |
2851 | - * Maybe display the description to the user |
2852 | - * The API states that :meth:`JobRunner.run_job()` should only be |
2853 | - called at this time. |
2854 | - * Run the command and wait for it to finish |
2855 | - * Decide on the outcome based on the return code |
2856 | - * The method ends here |
2857 | - |
2858 | - .. note:: |
2859 | - Local jobs are similar to resource jobs, in that the output matters |
2860 | - more than the return code. Unlike resource jobs and attachment |
2861 | - jobs, the output is expected to be a job definition in the |
2862 | - canonical RFC822 format. Local jobs are discouraged (due to some |
2863 | - complexities they introduce) but only supported way of generating |
2864 | - additional jobs at runtime. |
2865 | - """ |
2866 | - if job.plugin != "local": |
2867 | - # TRANSLATORS: please keep 'plugin' untranslated |
2868 | - raise ValueError(_("bad job plugin value")) |
2869 | - return self._just_run_command(job, job_state, config).get_result() |
2870 | - |
2871 | def run_manual_job(self, job, job_state, config): |
2872 | """ |
2873 | Method called to run a job with plugin field equal to 'manual'. |
2874 | @@ -741,7 +715,7 @@ class JobRunner(IJobRunner): |
2875 | delegate, io_log_gen = self._prepare_io_handling(job, config) |
2876 | # Create a subprocess.Popen() like object that uses the delegate |
2877 | # system to observe all IO as it occurs in real time. |
2878 | - delegate_cls = self._get_delegate_cls(config) |
2879 | + delegate_cls = extcmd.ExternalCommandWithDelegate |
2880 | extcmd_popen = delegate_cls(delegate) |
2881 | # Stream all IOLogRecord entries to disk |
2882 | record_path = self.get_record_path_for_job(job) |
2883 | @@ -892,7 +866,7 @@ class JobRunner(IJobRunner): |
2884 | delegate, io_log_gen = self._prepare_io_handling(job, config) |
2885 | # Create a subprocess.Popen() like object that uses the delegate |
2886 | # system to observe all IO as it occurs in real time. |
2887 | - delegate_cls = self._get_delegate_cls(config) |
2888 | + delegate_cls = extcmd.ExternalCommandWithDelegate |
2889 | flags = 0 |
2890 | # Use chunked IO for jobs that explicitly request this |
2891 | if 'use-chunked-io' in job.get_flag_set(): |
2892 | @@ -996,14 +970,3 @@ class JobRunner(IJobRunner): |
2893 | logger.warning( |
2894 | _("Please store desired files in $PLAINBOX_SESSION_SHARE and" |
2895 | " use regular temporary files for everything else")) |
2896 | - |
2897 | - def _get_delegate_cls(self, config): |
2898 | - if (sys.version_info[0:2] >= (3, 4) and sys.platform == 'linux' |
2899 | - and config.extcmd == "glibc"): |
2900 | - logger.debug("Using glibc-based command runner") |
2901 | - from plainbox.vendor.extcmd.glibc import ( |
2902 | - GlibcExternalCommandWithDelegate) |
2903 | - return GlibcExternalCommandWithDelegate |
2904 | - else: |
2905 | - logger.debug("Using classic thread-based command runner") |
2906 | - return extcmd.ExternalCommandWithDelegate |
2907 | diff --git a/plainbox/impl/secure/launcher1.py b/plainbox/impl/secure/launcher1.py |
2908 | index eaa1a8e..9515cc1 100644 |
2909 | --- a/plainbox/impl/secure/launcher1.py |
2910 | +++ b/plainbox/impl/secure/launcher1.py |
2911 | @@ -121,11 +121,7 @@ class TrustedLauncher: |
2912 | logging.error( |
2913 | _("Syntax error in record generated from %s: %s"), job, exc) |
2914 | else: |
2915 | - if job.plugin == 'local': |
2916 | - for record in record_list: |
2917 | - job = JobDefinition.from_rfc822_record(record) |
2918 | - job_list.append(job) |
2919 | - elif job.plugin == 'resource': |
2920 | + if job.plugin == 'resource': |
2921 | resource_list = [] |
2922 | for record in record_list: |
2923 | resource = Resource(record.data) |
2924 | @@ -209,9 +205,9 @@ def get_parser_for_sphinx(): |
2925 | group.add_argument( |
2926 | '-g', '--generator', |
2927 | metavar=_('CHECKSUM'), |
2928 | - # TRANSLATORS: don't translate 'local' in the sentence below. It |
2929 | - # denotes a special type of job, not its location. |
2930 | - help=_('also run a job with this checksum (assuming it is a local' |
2931 | + # TRANSLATORS: don't translate 'resource' in the sentence below. It |
2932 | + # denotes a special type of job. |
2933 | + help=_('also run a job with this checksum (assuming it is a resource' |
2934 | ' job)')) |
2935 | group.add_argument( |
2936 | '-G', '--generator-environment', |
2937 | @@ -266,7 +262,7 @@ def main(argv=None): |
2938 | all_providers.load() |
2939 | for plugin in all_providers.get_all_plugins(): |
2940 | launcher.add_job_list(plugin.plugin_object.job_list) |
2941 | - # Run the local job and feed the result back to the launcher |
2942 | + # Run the generator job and feed the result back to the launcher |
2943 | if ns.generator: |
2944 | try: |
2945 | generated_job_list = launcher.run_generator_job( |
2946 | diff --git a/plainbox/impl/secure/providers/__init__.py b/plainbox/impl/secure/providers/__init__.py |
2947 | index 8808022..066d2ee 100644 |
2948 | --- a/plainbox/impl/secure/providers/__init__.py |
2949 | +++ b/plainbox/impl/secure/providers/__init__.py |
2950 | @@ -21,18 +21,10 @@ |
2951 | :mod:`plainbox.impl.secure.providers` -- providers package |
2952 | ========================================================== |
2953 | |
2954 | -Providers are a mechanism by which PlainBox can enumerate jobs and whitelists. |
2955 | +Providers are a mechanism by which PlainBox can enumerate jobs and test plans. |
2956 | Currently there are only v1 (as in version one) providers that basically have |
2957 | to behave as CheckBox itself (mini CheckBox forks for example) |
2958 | |
2959 | -There is ongoing work and discussion on V2 providers that would have a |
2960 | -lower-level interface and would be able to define new job types, new whitelist |
2961 | -types and generally all the next-gen semantics. |
2962 | - |
2963 | -PlainBox does not come with any real provider by default. PlainBox sometimes |
2964 | -creates special dummy providers that have particular data in them for testing. |
2965 | - |
2966 | - |
2967 | V1 providers |
2968 | ------------ |
2969 | |
2970 | @@ -42,9 +34,6 @@ this is also described by :class:`plainbox.abc.IProvider1`:: |
2971 | * there is a directory with '.txt' or '.txt.in' files with RFC822-encoded |
2972 | job definitions. The definitions need a particular set of keys to work. |
2973 | |
2974 | - * there is a directory with '.whitelist' files that contain a list (one per |
2975 | - line) of job definitions to execute. |
2976 | - |
2977 | * there is a directory with additional executables (added to PATH) |
2978 | |
2979 | * there is a directory with an additional python3 libraries (added to |
2980 | diff --git a/plainbox/impl/secure/providers/test_v1.py b/plainbox/impl/secure/providers/test_v1.py |
2981 | index d54a3c2..b0eb001 100644 |
2982 | --- a/plainbox/impl/secure/providers/test_v1.py |
2983 | +++ b/plainbox/impl/secure/providers/test_v1.py |
2984 | @@ -37,8 +37,6 @@ from plainbox.impl.secure.providers.v1 import Provider1Definition |
2985 | from plainbox.impl.secure.providers.v1 import Provider1PlugIn |
2986 | from plainbox.impl.secure.providers.v1 import UnitPlugIn |
2987 | from plainbox.impl.secure.providers.v1 import VersionValidator |
2988 | -from plainbox.impl.secure.providers.v1 import WhiteListPlugIn |
2989 | -from plainbox.impl.secure.qualifiers import WhiteList |
2990 | from plainbox.impl.secure.rfc822 import FileTextSource |
2991 | from plainbox.impl.secure.rfc822 import Origin |
2992 | from plainbox.impl.unit.file import FileUnit |
2993 | @@ -156,7 +154,6 @@ class Provider1DefinitionTests(TestCase): |
2994 | "gettext_domain = domain\n" |
2995 | "units_dir = /some/directory/units\n" |
2996 | "jobs_dir = /some/directory/jobs\n" |
2997 | - "whitelists_dir = /some/directory/whitelists\n" |
2998 | "data_dir = /some/directory/data\n" |
2999 | "bin_dir = /some/directory/bin\n" |
3000 | "locale_dir = /some/directory/locale\n" |
3001 | @@ -168,7 +165,6 @@ class Provider1DefinitionTests(TestCase): |
3002 | self.assertEqual(def_.location, Unset) |
3003 | self.assertEqual(def_.units_dir, "/some/directory/units") |
3004 | self.assertEqual(def_.jobs_dir, "/some/directory/jobs") |
3005 | - self.assertEqual(def_.whitelists_dir, "/some/directory/whitelists") |
3006 | self.assertEqual(def_.data_dir, "/some/directory/data") |
3007 | self.assertEqual(def_.bin_dir, "/some/directory/bin") |
3008 | self.assertEqual(def_.locale_dir, "/some/directory/locale") |
3009 | @@ -211,7 +207,6 @@ class Provider1DefinitionTests(TestCase): |
3010 | self.assertEqual(def_.location, "/some/directory") |
3011 | self.assertEqual(def_.units_dir, Unset) |
3012 | self.assertEqual(def_.jobs_dir, Unset) |
3013 | - self.assertEqual(def_.whitelists_dir, Unset) |
3014 | self.assertEqual(def_.data_dir, Unset) |
3015 | self.assertEqual(def_.bin_dir, Unset) |
3016 | self.assertEqual(def_.locale_dir, Unset) |
3017 | @@ -369,11 +364,11 @@ class Provider1DefinitionTests(TestCase): |
3018 | |
3019 | def test_init_validation__foo_dir_unset(self): |
3020 | """ |
3021 | - verify that Provider1Definition allows 'jobs_dir', 'whitelists_dir', |
3022 | - 'data_dir', 'bin_dir' and 'locale_dir' fields to be unset |
3023 | + verify that Provider1Definition allows 'jobs_dir', 'data_dir', |
3024 | + 'bin_dir' and 'locale_dir' fields to be unset |
3025 | """ |
3026 | - for attr in ('units_dir', 'jobs_dir', 'whitelists_dir', 'data_dir', |
3027 | - 'bin_dir', 'locale_dir'): |
3028 | + for attr in ('units_dir', 'jobs_dir', 'data_dir', 'bin_dir', |
3029 | + 'locale_dir'): |
3030 | def_ = Provider1Definition() |
3031 | setattr(def_, attr, Unset) |
3032 | self.assertEqual(getattr(def_, attr), Unset) |
3033 | @@ -381,11 +376,10 @@ class Provider1DefinitionTests(TestCase): |
3034 | def test_init_validation__foo_dir_is_empty(self): |
3035 | """ |
3036 | verify that Provider1Definition ensures that 'jobs_dir', |
3037 | - 'whitelists_dir', 'data_dir', 'bin_dir' and 'locale_dir' fields are not |
3038 | - empty |
3039 | + 'data_dir', 'bin_dir' and 'locale_dir' fields are not empty |
3040 | """ |
3041 | - for attr in ('units_dir', 'jobs_dir', 'whitelists_dir', 'data_dir', |
3042 | - 'bin_dir', 'locale_dir'): |
3043 | + for attr in ('units_dir', 'jobs_dir', 'data_dir', 'bin_dir', |
3044 | + 'locale_dir'): |
3045 | def_ = Provider1Definition() |
3046 | with self.assertRaises(ValidationError) as boom: |
3047 | setattr(def_, attr, '') |
3048 | @@ -394,11 +388,11 @@ class Provider1DefinitionTests(TestCase): |
3049 | def test_init_validation__foo_dir_relative(self): |
3050 | """ |
3051 | verify that Provider1Definition ensures that 'jobs_dir', |
3052 | - 'whitelists_dir', 'data_dir', 'bin_dir' and 'locale_dir' fields are not |
3053 | - a relative pathname |
3054 | + 'data_dir', 'bin_dir' and 'locale_dir' fields are not a relative |
3055 | + pathname |
3056 | """ |
3057 | - for attr in ('units_dir', 'jobs_dir', 'whitelists_dir', 'data_dir', |
3058 | - 'bin_dir', 'locale_dir'): |
3059 | + for attr in ('units_dir', 'jobs_dir', 'data_dir', 'bin_dir', |
3060 | + 'locale_dir'): |
3061 | def_ = Provider1Definition() |
3062 | with self.assertRaises(ValidationError) as boom: |
3063 | setattr(def_, attr, 'some/place') |
3064 | @@ -407,11 +401,11 @@ class Provider1DefinitionTests(TestCase): |
3065 | def test_init_validation__foo_dir_doesnt_exist(self): |
3066 | """ |
3067 | verify that Provider1Definition ensures that 'jobs_dir', |
3068 | - 'whitelists_dir', 'data_dir', 'bin_dir' and 'locale_dir' fields are not |
3069 | - pointing to a non-existing directory |
3070 | + 'data_dir', 'bin_dir' and 'locale_dir' fields are not pointing to a |
3071 | + non-existing directory |
3072 | """ |
3073 | - for attr in ('units_dir', 'jobs_dir', 'whitelists_dir', 'data_dir', |
3074 | - 'bin_dir', 'locale_dir'): |
3075 | + for attr in ('units_dir', 'jobs_dir', 'data_dir', 'bin_dir', |
3076 | + 'locale_dir'): |
3077 | def_ = Provider1Definition() |
3078 | with self.assertRaises(ValidationError) as boom: |
3079 | with mock.patch('os.path.isdir') as mock_isdir: |
3080 | @@ -437,7 +431,6 @@ class Provider1PlugInTests(TestCase): |
3081 | DEF_TEXT_w_dirs = DEF_TEXT + ( |
3082 | "units_dir = /some/directory/units\n" |
3083 | "jobs_dir = /some/directory/jobs\n" |
3084 | - "whitelists_dir = /some/directory/whitelists\n" |
3085 | "data_dir = /some/directory/data\n" |
3086 | "bin_dir = /some/directory/bin\n" |
3087 | "locale_dir = /some/directory/locale\n" |
3088 | @@ -488,7 +481,6 @@ class Provider1PlugInTests(TestCase): |
3089 | provider = self.plugin.plugin_object |
3090 | self.assertEqual(provider.units_dir, None) |
3091 | self.assertEqual(provider.jobs_dir, None) |
3092 | - self.assertEqual(provider.whitelists_dir, None) |
3093 | self.assertEqual(provider.data_dir, None) |
3094 | self.assertEqual(provider.bin_dir, None) |
3095 | self.assertEqual(provider.build_bin_dir, None) |
3096 | @@ -505,7 +497,6 @@ class Provider1PlugInTests(TestCase): |
3097 | provider = self.plugin_w_location.plugin_object |
3098 | self.assertEqual(provider.units_dir, "/some/directory/units") |
3099 | self.assertEqual(provider.jobs_dir, "/some/directory/jobs") |
3100 | - self.assertEqual(provider.whitelists_dir, "/some/directory/whitelists") |
3101 | self.assertEqual(provider.data_dir, "/some/directory/data") |
3102 | self.assertEqual(provider.bin_dir, "/some/directory/bin") |
3103 | self.assertEqual(provider.build_bin_dir, "/some/directory/build/bin") |
3104 | @@ -523,7 +514,6 @@ class Provider1PlugInTests(TestCase): |
3105 | provider = self.plugin_w_location_w_no_dirs.plugin_object |
3106 | self.assertEqual(provider.units_dir, None) |
3107 | self.assertEqual(provider.jobs_dir, None) |
3108 | - self.assertEqual(provider.whitelists_dir, None) |
3109 | self.assertEqual(provider.data_dir, None) |
3110 | self.assertEqual(provider.bin_dir, None) |
3111 | self.assertEqual(provider.build_bin_dir, "/some/directory/build/bin") |
3112 | @@ -539,7 +529,6 @@ class Provider1PlugInTests(TestCase): |
3113 | provider = self.plugin_w_dirs.plugin_object |
3114 | self.assertEqual(provider.units_dir, "/some/directory/units") |
3115 | self.assertEqual(provider.jobs_dir, "/some/directory/jobs") |
3116 | - self.assertEqual(provider.whitelists_dir, "/some/directory/whitelists") |
3117 | self.assertEqual(provider.data_dir, "/some/directory/data") |
3118 | self.assertEqual(provider.bin_dir, "/some/directory/bin") |
3119 | self.assertEqual(provider.build_bin_dir, None) |
3120 | @@ -548,62 +537,6 @@ class Provider1PlugInTests(TestCase): |
3121 | self.assertEqual(provider.base_dir, None) |
3122 | |
3123 | |
3124 | -class WhiteListPlugInTests(TestCase): |
3125 | - """ |
3126 | - Tests for WhiteListPlugIn |
3127 | - """ |
3128 | - |
3129 | - LOAD_TIME = 42 |
3130 | - |
3131 | - def setUp(self): |
3132 | - self.plugin = WhiteListPlugIn( |
3133 | - "/path/to/some.whitelist", "foo\nbar\n", self.LOAD_TIME) |
3134 | - |
3135 | - def test_plugin_name(self): |
3136 | - """ |
3137 | - verify that the WhiteListPlugIn.plugin_name property returns |
3138 | - WhiteList.name |
3139 | - """ |
3140 | - self.assertEqual(self.plugin.plugin_name, "some") |
3141 | - |
3142 | - def test_plugin_object(self): |
3143 | - """ |
3144 | - verify that the WhiteListPlugIn.plugin_object property returns a |
3145 | - WhiteList |
3146 | - """ |
3147 | - self.assertIsInstance(self.plugin.plugin_object, WhiteList) |
3148 | - |
3149 | - def test_plugin_load_time(self): |
3150 | - self.assertEqual(self.plugin.plugin_load_time, self.LOAD_TIME) |
3151 | - |
3152 | - def test_whitelist_data(self): |
3153 | - """ |
3154 | - verify the contents of the loaded whitelist object |
3155 | - """ |
3156 | - self.assertEqual( |
3157 | - self.plugin.plugin_object.qualifier_list[0].pattern_text, "^foo$") |
3158 | - self.assertEqual( |
3159 | - self.plugin.plugin_object.qualifier_list[1].pattern_text, "^bar$") |
3160 | - self.assertEqual(self.plugin.plugin_object.name, 'some') |
3161 | - self.assertEqual( |
3162 | - self.plugin.plugin_object.origin, |
3163 | - Origin(FileTextSource('/path/to/some.whitelist'), 1, 2)) |
3164 | - |
3165 | - def test_init_failing(self): |
3166 | - """ |
3167 | - verify how WhiteList() initializer works if something is wrong |
3168 | - """ |
3169 | - # The pattern is purposefully invalid |
3170 | - with self.assertRaises(PlugInError) as boom: |
3171 | - WhiteListPlugIn("/path/to/some.whitelist", "*", self.LOAD_TIME) |
3172 | - # NOTE: we should have syntax error for whitelists that keeps track or |
3173 | - # line we're at to help developers figure out where errors such as this |
3174 | - # are coming from. |
3175 | - self.assertEqual( |
3176 | - str(boom.exception), |
3177 | - ("Cannot load '/path/to/some.whitelist': nothing to repeat")) |
3178 | - |
3179 | - |
3180 | class UnitPlugInTests(TestCase): |
3181 | """ |
3182 | Tests for UnitPlugIn |
3183 | @@ -686,7 +619,6 @@ class Provider1Tests(TestCase): |
3184 | GETTEXT_DOMAIN = "domain" |
3185 | UNITS_DIR = "units-dir" |
3186 | JOBS_DIR = "jobs-dir" |
3187 | - WHITELISTS_DIR = "whitelists-dir" |
3188 | DATA_DIR = "data-dir" |
3189 | BIN_DIR = "bin-dir" |
3190 | LOCALE_DIR = "locale-dir" |
3191 | @@ -698,8 +630,7 @@ class Provider1Tests(TestCase): |
3192 | self.provider = Provider1( |
3193 | self.NAME, self.NAMESPACE, self.VERSION, self.DESCRIPTION, |
3194 | self.SECURE, self.GETTEXT_DOMAIN, self.UNITS_DIR, self.JOBS_DIR, |
3195 | - self.WHITELISTS_DIR, self.DATA_DIR, self.BIN_DIR, self.LOCALE_DIR, |
3196 | - self.BASE_DIR, |
3197 | + self.DATA_DIR, self.BIN_DIR, self.LOCALE_DIR, self.BASE_DIR, |
3198 | # We are using dummy job definitions so let's not shout about those |
3199 | # being invalid in each test |
3200 | validate=False) |
3201 | @@ -762,12 +693,6 @@ class Provider1Tests(TestCase): |
3202 | """ |
3203 | self.assertEqual(self.provider.jobs_dir, self.JOBS_DIR) |
3204 | |
3205 | - def test_whitelists_dir(self): |
3206 | - """ |
3207 | - Verify that Provider1.whitelists_dir attribute is set correctly |
3208 | - """ |
3209 | - self.assertEqual(self.provider.whitelists_dir, self.WHITELISTS_DIR) |
3210 | - |
3211 | def test_data_dir(self): |
3212 | """ |
3213 | Verify that Provider1.data_dir attribute is set correctly |
3214 | @@ -901,8 +826,7 @@ class Provider1Tests(TestCase): |
3215 | Provider1( |
3216 | self.NAME, self.NAMESPACE, self.VERSION, self.DESCRIPTION, |
3217 | self.SECURE, self.GETTEXT_DOMAIN, self.UNITS_DIR, self.JOBS_DIR, |
3218 | - self.WHITELISTS_DIR, self.DATA_DIR, self.BIN_DIR, self.LOCALE_DIR, |
3219 | - self.BASE_DIR) |
3220 | + self.DATA_DIR, self.BIN_DIR, self.LOCALE_DIR, self.BASE_DIR) |
3221 | mock_gettext.bindtextdomain.assert_called_once_with( |
3222 | self.GETTEXT_DOMAIN, self.LOCALE_DIR) |
3223 | |
3224 | @@ -915,6 +839,6 @@ class Provider1Tests(TestCase): |
3225 | Provider1( |
3226 | self.NAME, self.NAMESPACE, self.VERSION, self.DESCRIPTION, |
3227 | self.SECURE, self.GETTEXT_DOMAIN, self.UNITS_DIR, self.JOBS_DIR, |
3228 | - self.WHITELISTS_DIR, self.DATA_DIR, self.BIN_DIR, locale_dir=None, |
3229 | + self.DATA_DIR, self.BIN_DIR, locale_dir=None, |
3230 | base_dir=self.BASE_DIR) |
3231 | self.assertEqual(mock_gettext.bindtextdomain.call_args_list, []) |
3232 | diff --git a/plainbox/impl/secure/providers/v1.py b/plainbox/impl/secure/providers/v1.py |
3233 | index f1a6e58..18d3213 100644 |
3234 | --- a/plainbox/impl/secure/providers/v1.py |
3235 | +++ b/plainbox/impl/secure/providers/v1.py |
3236 | @@ -44,7 +44,6 @@ from plainbox.impl.secure.plugins import LazyFsPlugInCollection |
3237 | from plainbox.impl.secure.plugins import PlugIn |
3238 | from plainbox.impl.secure.plugins import PlugInError |
3239 | from plainbox.impl.secure.plugins import now |
3240 | -from plainbox.impl.secure.qualifiers import WhiteList |
3241 | from plainbox.impl.secure.rfc822 import FileTextSource |
3242 | from plainbox.impl.secure.rfc822 import RFC822SyntaxError |
3243 | from plainbox.impl.secure.rfc822 import load_rfc822_records |
3244 | @@ -63,17 +62,14 @@ class ProviderContentPlugIn(PlugIn): |
3245 | """ |
3246 | PlugIn class for loading provider content. |
3247 | |
3248 | - Provider content comes in two shapes and sizes: |
3249 | + Provider content: |
3250 | - units (of any kind) |
3251 | - - whitelists |
3252 | |
3253 | The actual logic on how to load everything is encapsulated in |
3254 | :meth:`wrap()` though its return value is not so useful. |
3255 | |
3256 | :attr unit_list: |
3257 | The list of loaded units |
3258 | - :attr whitelist_list: |
3259 | - The list of loaded whitelists |
3260 | """ |
3261 | |
3262 | def __init__(self, filename, text, load_time, provider, *, |
3263 | @@ -94,12 +90,9 @@ class ProviderContentPlugIn(PlugIn): |
3264 | wrap_time = now() - start_time |
3265 | super().__init__(filename, inspect_result, load_time, wrap_time) |
3266 | self.unit_list = [] |
3267 | - self.whitelist_list = [] |
3268 | # And load all of the content from that file |
3269 | self.unit_list.extend(self.discover_units( |
3270 | inspect_result, filename, text, provider)) |
3271 | - self.whitelist_list.extend(self.discover_whitelists( |
3272 | - inspect_result, filename, text, provider)) |
3273 | |
3274 | def inspect(self, filename: str, text: str, provider: "Provider1", |
3275 | validate: bool, validation_kwargs: "Dict[str, Any]", check: |
3276 | @@ -110,9 +103,8 @@ class ProviderContentPlugIn(PlugIn): |
3277 | :meth:`plugin_object` |
3278 | |
3279 | .. note:: |
3280 | - This method must *not* access neither :attr:`unit_list` nor |
3281 | - :attr:`whitelist_list`. If needed, it can collect its own state in |
3282 | - private instance attributes. |
3283 | + This method must *not* access :attr:`unit_list`. If needed, it can |
3284 | + collect its own state in private instance attributes. |
3285 | """ |
3286 | |
3287 | def discover_units( |
3288 | @@ -132,23 +124,6 @@ class ProviderContentPlugIn(PlugIn): |
3289 | """ |
3290 | yield self.make_file_unit(filename, provider) |
3291 | |
3292 | - def discover_whitelists( |
3293 | - self, inspect_result: "Any", filename: str, text: str, |
3294 | - provider: "Provider1" |
3295 | - ) -> "Iterable[WhiteList]": |
3296 | - """ |
3297 | - Discover all whitelists that were loaded by this plug-in |
3298 | - |
3299 | - :param wrap_result: |
3300 | - whatever was returned on the call to :meth:`wrap()`. |
3301 | - :returns: |
3302 | - an iterable of whitelists. |
3303 | - |
3304 | - .. note:: |
3305 | - this method is always called *after* :meth:`wrap()`. |
3306 | - """ |
3307 | - return () |
3308 | - |
3309 | def make_file_unit(self, filename, provider, role=None, base=None): |
3310 | if role is None or base is None: |
3311 | role, base, plugin_cls = provider.classify(filename) |
3312 | @@ -161,71 +136,6 @@ class ProviderContentPlugIn(PlugIn): |
3313 | virtual=True) |
3314 | |
3315 | |
3316 | -class WhiteListPlugIn(ProviderContentPlugIn): |
3317 | - """ |
3318 | - A specialized :class:`plainbox.impl.secure.plugins.IPlugIn` that loads |
3319 | - :class:`plainbox.impl.secure.qualifiers.WhiteList` instances from a file. |
3320 | - """ |
3321 | - |
3322 | - def inspect(self, filename: str, text: str, provider: "Provider1", |
3323 | - validate: bool, validation_kwargs: "Dict[str, Any]", check: |
3324 | - bool, context: "???") -> "WhiteList": |
3325 | - if provider is not None: |
3326 | - implicit_namespace = provider.namespace |
3327 | - else: |
3328 | - implicit_namespace = None |
3329 | - origin = Origin(FileTextSource(filename), 1, text.count('\n')) |
3330 | - return WhiteList.from_string( |
3331 | - text, filename=filename, origin=origin, |
3332 | - implicit_namespace=implicit_namespace) |
3333 | - |
3334 | - def discover_units( |
3335 | - self, inspect_result: "WhiteList", filename: str, text: str, |
3336 | - provider: "Provider1" |
3337 | - ) -> "Iterable[Unit]": |
3338 | - if provider is not None: |
3339 | - yield self.make_file_unit( |
3340 | - filename, provider, |
3341 | - # NOTE: don't guess what this file is for |
3342 | - role=FileRole.legacy_whitelist, base=provider.whitelists_dir) |
3343 | - yield self.make_test_plan_unit(filename, text, provider) |
3344 | - |
3345 | - def discover_whitelists( |
3346 | - self, inspect_result: "WhiteList", filename: str, text: str, |
3347 | - provider: "Provider1" |
3348 | - ) -> "Iterable[WhiteList]": |
3349 | - yield inspect_result |
3350 | - |
3351 | - def make_test_plan_unit(self, filename, text, provider): |
3352 | - name = os.path.basename(os.path.splitext(filename)[0]) |
3353 | - origin = Origin(FileTextSource(filename), 1, text.count('\n')) |
3354 | - field_offset_map = {'include': 0} |
3355 | - return TestPlanUnit({ |
3356 | - 'unit': TestPlanUnit.Meta.name, |
3357 | - 'id': name, |
3358 | - 'name': name, |
3359 | - 'include': str(text), # delazify content |
3360 | - }, origin=origin, provider=provider, field_offset_map=field_offset_map, |
3361 | - virtual=True) |
3362 | - |
3363 | - # NOTE: This version of __init__() exists solely so that provider can |
3364 | - # default to None. This is still used in some places and must be supported. |
3365 | - def __init__(self, filename, text, load_time, provider=None, *, |
3366 | - validate=False, validation_kwargs=None, |
3367 | - check=True, context=None): |
3368 | - super().__init__( |
3369 | - filename, text, load_time, provider, validate=validate, |
3370 | - validation_kwargs=validation_kwargs, check=check, context=context) |
3371 | - |
3372 | - # NOTE: this version of plugin_name() is just for legacy code support |
3373 | - @property |
3374 | - def plugin_name(self): |
3375 | - """ |
3376 | - plugin name, the name of the WhiteList |
3377 | - """ |
3378 | - return self.plugin_object.name |
3379 | - |
3380 | - |
3381 | class UnitPlugIn(ProviderContentPlugIn): |
3382 | """ |
3383 | A specialized :class:`plainbox.impl.secure.plugins.IPlugIn` that loads a |
3384 | @@ -307,23 +217,6 @@ class UnitPlugIn(ProviderContentPlugIn): |
3385 | yield unit |
3386 | yield self.make_file_unit(filename, provider) |
3387 | |
3388 | - def discover_whitelists( |
3389 | - self, inspect_result: "List[Unit]", filename: str, text: str, |
3390 | - provider: "Provider1" |
3391 | - ) -> "Iterable[WhiteList]": |
3392 | - for unit in (unit for unit in inspect_result |
3393 | - if unit.Meta.name == 'test plan'): |
3394 | - if unit.include is not None: |
3395 | - patterns = [] |
3396 | - for line in unit.include.split('\n'): |
3397 | - if '#' in line: |
3398 | - line = line.split('#')[0] |
3399 | - if line: |
3400 | - patterns.append('${}^'.format(line)) |
3401 | - yield WhiteList( |
3402 | - patterns, name=unit.partial_id, origin=unit.origin, |
3403 | - implicit_namespace=unit.provider.namespace) |
3404 | - |
3405 | # NOTE: this version of plugin_object() is just for legacy code support |
3406 | @property |
3407 | def plugin_object(self): |
3408 | @@ -393,8 +286,6 @@ class ProviderContentEnumerator: |
3409 | dir_list.append(provider.bin_dir) |
3410 | if provider.locale_dir: |
3411 | dir_list.append(provider.locale_dir) |
3412 | - if provider.whitelists_dir: |
3413 | - dir_list.append(provider.whitelists_dir) |
3414 | # Find all the files that belong to a provider |
3415 | self._content_collection = LazyFsPlugInCollection( |
3416 | dir_list, ext=None, recursive=True) |
3417 | @@ -429,8 +320,7 @@ class ProviderContentClassifier: |
3418 | |
3419 | The secondary role is to provide a hint on what PlugIn to use to load such |
3420 | content (as units). In practice the majority of files are loaded with the |
3421 | - :class:`UnitPlugIn` class. Legacy ``.whitelist`` files are loaded with the |
3422 | - :class:`WhiteListPlugIn` class instead. All other files are handled by the |
3423 | + :class:`UnitPlugIn` class. All other files are handled by the |
3424 | :class:`ProviderContentPlugIn`. |
3425 | |
3426 | .. note:: |
3427 | @@ -508,8 +398,6 @@ class ProviderContentClassifier: |
3428 | classify_fn_list.append(self._classify_pxu_jobs) |
3429 | if self.provider.units_dir: |
3430 | classify_fn_list.append(self._classify_pxu_units) |
3431 | - if self.provider.whitelists_dir: |
3432 | - classify_fn_list.append(self._classify_whitelist) |
3433 | if self.provider.data_dir: |
3434 | classify_fn_list.append(self._classify_data) |
3435 | if self.provider.bin_dir: |
3436 | @@ -568,13 +456,6 @@ class ProviderContentClassifier: |
3437 | return (FileRole.unit_source, self.provider.units_dir, |
3438 | UnitPlugIn) |
3439 | |
3440 | - def _classify_whitelist(self, filename: str): |
3441 | - """ classify .whitelist files in whitelist_dir as whitelist """ |
3442 | - if (filename.startswith(self.provider.whitelists_dir) and |
3443 | - filename.endswith(".whitelist")): |
3444 | - return (FileRole.legacy_whitelist, self.provider.whitelists_dir, |
3445 | - WhiteListPlugIn) |
3446 | - |
3447 | def _classify_data(self, filename: str): |
3448 | """ classify files in data_dir as data """ |
3449 | if filename.startswith(self.provider.data_dir): |
3450 | @@ -676,7 +557,7 @@ class ProviderContentLoader: |
3451 | :attr is_loaded: |
3452 | Flag indicating if the content loader has loaded all of the content |
3453 | :attr unit_list: |
3454 | - A list of loaded whitelist objects |
3455 | + A list of loaded units objects |
3456 | :attr problem_list: |
3457 | A list of problems experienced while loading any of the content |
3458 | :attr path_map: |
3459 | @@ -691,7 +572,6 @@ class ProviderContentLoader: |
3460 | self.provider = provider |
3461 | self.is_loaded = False |
3462 | self.unit_list = [] |
3463 | - self.whitelist_list = [] |
3464 | self.problem_list = [] |
3465 | self.path_map = collections.defaultdict(list) # path -> list(unit) |
3466 | self.id_map = collections.defaultdict(list) # id -> list(unit) |
3467 | @@ -720,7 +600,6 @@ class ProviderContentLoader: |
3468 | self.problem_list.append(exc) |
3469 | else: |
3470 | self.unit_list.extend(plugin.unit_list) |
3471 | - self.whitelist_list.extend(plugin.whitelist_list) |
3472 | for unit in plugin.unit_list: |
3473 | if hasattr(unit.Meta.fields, 'id'): |
3474 | self.id_map[unit.id].append(unit) |
3475 | @@ -732,7 +611,7 @@ class Provider1(IProvider1): |
3476 | """ |
3477 | A v1 provider implementation. |
3478 | |
3479 | - A provider is a container of jobs and whitelists. It provides additional |
3480 | + A provider is a container of jobs and test plans. It provides additional |
3481 | meta-data and knows about location of essential directories to both load |
3482 | structured data and provide runtime information for job execution. |
3483 | |
3484 | @@ -741,8 +620,8 @@ class Provider1(IProvider1): |
3485 | """ |
3486 | |
3487 | def __init__(self, name, namespace, version, description, secure, |
3488 | - gettext_domain, units_dir, jobs_dir, whitelists_dir, data_dir, |
3489 | - bin_dir, locale_dir, base_dir, *, validate=False, |
3490 | + gettext_domain, units_dir, jobs_dir, data_dir, bin_dir, |
3491 | + locale_dir, base_dir, *, validate=False, |
3492 | validation_kwargs=None, check=True, context=None): |
3493 | """ |
3494 | Initialize a provider with a set of meta-data and directories. |
3495 | @@ -778,9 +657,6 @@ class Provider1(IProvider1): |
3496 | :param jobs_dir: |
3497 | path of the directory with job definitions |
3498 | |
3499 | - :param whitelists_dir: |
3500 | - path of the directory with whitelists definitions (aka test-plans) |
3501 | - |
3502 | :param data_dir: |
3503 | path of the directory with files used by jobs at runtime |
3504 | |
3505 | @@ -791,8 +667,8 @@ class Provider1(IProvider1): |
3506 | path of the directory with locale database (translation catalogs) |
3507 | |
3508 | :param base_dir: |
3509 | - path of the directory with (perhaps) all of jobs_dir, |
3510 | - whitelists_dir, data_dir, bin_dir, locale_dir. This may be None. |
3511 | + path of the directory with (perhaps) all of jobs_dir, data_dir, |
3512 | + bin_dir, locale_dir. This may be None. |
3513 | This is also the effective value of $CHECKBOX_SHARE |
3514 | |
3515 | :param validate: |
3516 | @@ -820,7 +696,6 @@ class Provider1(IProvider1): |
3517 | # Directories |
3518 | self._units_dir = units_dir |
3519 | self._jobs_dir = jobs_dir |
3520 | - self._whitelists_dir = whitelists_dir |
3521 | self._data_dir = data_dir |
3522 | self._bin_dir = bin_dir |
3523 | self._locale_dir = locale_dir |
3524 | @@ -843,9 +718,6 @@ class Provider1(IProvider1): |
3525 | if not self._loader.is_loaded: |
3526 | self._loader.load(self._load_kwargs) |
3527 | |
3528 | - def _load_whitelists(self): |
3529 | - self._ensure_loaded() |
3530 | - |
3531 | def _load_units(self, validate, validation_kwargs, check, context): |
3532 | self._ensure_loaded() |
3533 | |
3534 | @@ -890,10 +762,10 @@ class Provider1(IProvider1): |
3535 | definition.description, secure, |
3536 | definition.effective_gettext_domain, |
3537 | definition.effective_units_dir, definition.effective_jobs_dir, |
3538 | - definition.effective_whitelists_dir, definition.effective_data_dir, |
3539 | - definition.effective_bin_dir, definition.effective_locale_dir, |
3540 | - definition.location or None, validate=validate, |
3541 | - validation_kwargs=validation_kwargs, check=check, context=context) |
3542 | + definition.effective_data_dir, definition.effective_bin_dir, |
3543 | + definition.effective_locale_dir, definition.location or None, |
3544 | + validate=validate, validation_kwargs=validation_kwargs, |
3545 | + check=check, context=context) |
3546 | |
3547 | def __repr__(self): |
3548 | return "<{} name:{!r}>".format(self.__class__.__name__, self.name) |
3549 | @@ -966,13 +838,6 @@ class Provider1(IProvider1): |
3550 | return self._jobs_dir |
3551 | |
3552 | @property |
3553 | - def whitelists_dir(self): |
3554 | - """ |
3555 | - absolute path of the whitelist directory |
3556 | - """ |
3557 | - return self._whitelists_dir |
3558 | - |
3559 | - @property |
3560 | def data_dir(self): |
3561 | """ |
3562 | absolute path of the data directory |
3563 | @@ -1002,8 +867,8 @@ class Provider1(IProvider1): |
3564 | @property |
3565 | def base_dir(self): |
3566 | """ |
3567 | - path of the directory with (perhaps) all of jobs_dir, whitelists_dir, |
3568 | - data_dir, bin_dir, locale_dir. This may be None |
3569 | + path of the directory with (perhaps) all of jobs_dir, data_dir, |
3570 | + bin_dir, locale_dir. This may be None |
3571 | """ |
3572 | return self._base_dir |
3573 | |
3574 | @@ -1136,20 +1001,6 @@ class Provider1(IProvider1): |
3575 | unit.role in (FileRole.script, FileRole.binary)) |
3576 | |
3577 | @property |
3578 | - def whitelist_list(self): |
3579 | - """ |
3580 | - List of loaded whitelists. |
3581 | - |
3582 | - .. warning:: |
3583 | - :class:`WhiteList` is currently deprecated. You should never need |
3584 | - to access them in any new code. They are entirely replaced by |
3585 | - :class:`TestPlan`. This property is provided for completeness and |
3586 | - it will be **removed** once whitelists classes are no longer used. |
3587 | - """ |
3588 | - self._ensure_loaded() |
3589 | - return self._loader.whitelist_list |
3590 | - |
3591 | - @property |
3592 | def problem_list(self): |
3593 | """ |
3594 | list of problems encountered by the loading process |
3595 | @@ -1506,43 +1357,6 @@ class Provider1Definition(Config): |
3596 | if implicit is not None and os.path.isdir(implicit): |
3597 | return implicit |
3598 | |
3599 | - whitelists_dir = Variable( |
3600 | - section='PlainBox Provider', |
3601 | - help_text=_("Pathname of the directory with whitelists definitions"), |
3602 | - validator_list=[ |
3603 | - # NOTE: it *can* be unset |
3604 | - NotEmptyValidator(), |
3605 | - AbsolutePathValidator(), |
3606 | - ExistingDirectoryValidator(), |
3607 | - ]) |
3608 | - |
3609 | - @property |
3610 | - def implicit_whitelists_dir(self): |
3611 | - """ |
3612 | - implicit value of whitelists_dir (if Unset) |
3613 | - |
3614 | - The implicit value is only defined if location is not Unset. It is the |
3615 | - 'whitelists' subdirectory of the directory that location points to. |
3616 | - """ |
3617 | - if self.location is not Unset: |
3618 | - return os.path.join(self.location, "whitelists") |
3619 | - |
3620 | - @property |
3621 | - def effective_whitelists_dir(self): |
3622 | - """ |
3623 | - effective value of whitelists_dir |
3624 | - |
3625 | - The effective value is :meth:`whitelists_dir` itself, unless it is |
3626 | - Unset. If it is Unset the effective value is the |
3627 | - :meth:`implicit_whitelists_dir`, if that value would be valid. The |
3628 | - effective value may be None. |
3629 | - """ |
3630 | - if self.whitelists_dir is not Unset: |
3631 | - return self.whitelists_dir |
3632 | - implicit = self.implicit_whitelists_dir |
3633 | - if implicit is not None and os.path.isdir(implicit): |
3634 | - return implicit |
3635 | - |
3636 | data_dir = Variable( |
3637 | section='PlainBox Provider', |
3638 | help_text=_("Pathname of the directory with provider data"), |
3639 | @@ -1708,7 +1522,6 @@ class Provider1PlugIn(PlugIn): |
3640 | definition.location = os.path.dirname(filename) |
3641 | definition.units_dir = Unset |
3642 | definition.jobs_dir = Unset |
3643 | - definition.whitelists_dir = Unset |
3644 | definition.data_dir = Unset |
3645 | definition.bin_dir = Unset |
3646 | definition.locale_dir = Unset |
3647 | diff --git a/plainbox/impl/secure/qualifiers.py b/plainbox/impl/secure/qualifiers.py |
3648 | index 773d953..d1ee659 100644 |
3649 | --- a/plainbox/impl/secure/qualifiers.py |
3650 | +++ b/plainbox/impl/secure/qualifiers.py |
3651 | @@ -210,28 +210,6 @@ class JobIdQualifier(SimpleQualifier): |
3652 | self.__class__.__name__, self._id, self._inclusive) |
3653 | |
3654 | |
3655 | -class NonLocalJobQualifier(SimpleQualifier): |
3656 | - """ |
3657 | - A JobQualifier that designates only non local jobs |
3658 | - """ |
3659 | - |
3660 | - def __init__(self, origin, inclusive=True): |
3661 | - super().__init__(origin, inclusive) |
3662 | - |
3663 | - def get_simple_match(self, job): |
3664 | - """ |
3665 | - Check if the given job matches this qualifier. |
3666 | - |
3667 | - This method should not be called directly, it is an implementation |
3668 | - detail of SimpleQualifier class. |
3669 | - """ |
3670 | - return job.plugin != 'local' |
3671 | - |
3672 | - def __repr__(self): |
3673 | - return "{0}(inclusive={1})".format( |
3674 | - self.__class__.__name__, self._inclusive) |
3675 | - |
3676 | - |
3677 | class IMatcher(metaclass=abc.ABCMeta): |
3678 | """ |
3679 | Interface for objects that perform some kind of comparison on a value |
3680 | @@ -441,206 +419,6 @@ class NonPrimitiveQualifierOrigin(Exception): |
3681 | """ |
3682 | |
3683 | |
3684 | -# NOTE: using CompositeQualifier seems strange but it's a tested proven |
3685 | -# component so all we have to ensure is that we read the whitelist files |
3686 | -# correctly. |
3687 | -class WhiteList(CompositeQualifier): |
3688 | - """ |
3689 | - A qualifier that understands checkbox whitelist files. |
3690 | - |
3691 | - A whitelist file is a plain text, line oriented file. Each line represents |
3692 | - a regular expression pattern that can be matched against the id of a job. |
3693 | - |
3694 | - The file can contain simple shell-style comments that begin with the pound |
3695 | - or hash key (#). Those are ignored. Comments can span both a fraction of a |
3696 | - line as well as the whole line. |
3697 | - |
3698 | - For historical reasons each pattern has an implicit '^' and '$' prepended |
3699 | - and appended (respectively) to the actual pattern specified in the file. |
3700 | - """ |
3701 | - |
3702 | - def __init__(self, pattern_list, name=None, origin=None, |
3703 | - implicit_namespace=None): |
3704 | - """ |
3705 | - Initialize a WhiteList object with the specified list of patterns. |
3706 | - |
3707 | - The patterns must be already mangled with '^' and '$'. |
3708 | - """ |
3709 | - self._name = name |
3710 | - self._origin = origin |
3711 | - self._implicit_namespace = implicit_namespace |
3712 | - if implicit_namespace is not None: |
3713 | - # If we have an implicit namespace then transform all the patterns |
3714 | - # without the namespace operator ('::') |
3715 | - namespace_pattern = implicit_namespace.replace('.', '\\.') |
3716 | - |
3717 | - def transform_pattern(maybe_partial_id_pattern): |
3718 | - if "::" not in maybe_partial_id_pattern: |
3719 | - return "^{}::{}$".format( |
3720 | - namespace_pattern, maybe_partial_id_pattern[1:-1]) |
3721 | - else: |
3722 | - return maybe_partial_id_pattern |
3723 | - qualifier_list = [ |
3724 | - RegExpJobQualifier( |
3725 | - transform_pattern(pattern), origin, inclusive=True) |
3726 | - for pattern in pattern_list] |
3727 | - else: |
3728 | - # Otherwise just use the patterns directly |
3729 | - qualifier_list = [ |
3730 | - RegExpJobQualifier(pattern, origin, inclusive=True) |
3731 | - for pattern in pattern_list] |
3732 | - super().__init__(qualifier_list) |
3733 | - |
3734 | - def __repr__(self): |
3735 | - return "<{} name:{!r}>".format(self.__class__.__name__, self.name) |
3736 | - |
3737 | - @property |
3738 | - def name(self): |
3739 | - """ |
3740 | - name of this WhiteList (might be None) |
3741 | - """ |
3742 | - return self._name |
3743 | - |
3744 | - @name.setter |
3745 | - def name(self, value): |
3746 | - """ |
3747 | - set a new name for a WhiteList |
3748 | - """ |
3749 | - self._name = value |
3750 | - |
3751 | - @property |
3752 | - def origin(self): |
3753 | - """ |
3754 | - origin object associated with this WhiteList (might be None) |
3755 | - """ |
3756 | - return self._origin |
3757 | - |
3758 | - @property |
3759 | - def implicit_namespace(self): |
3760 | - """ |
3761 | - namespace used to qualify patterns without explicit namespace |
3762 | - """ |
3763 | - return self._implicit_namespace |
3764 | - |
3765 | - @classmethod |
3766 | - def from_file(cls, pathname, implicit_namespace=None): |
3767 | - """ |
3768 | - Load and initialize the WhiteList object from the specified file. |
3769 | - |
3770 | - :param pathname: |
3771 | - file to load |
3772 | - :param implicit_namespace: |
3773 | - (optional) implicit namespace for jobs that are using partial |
3774 | - identifiers (all jobs) |
3775 | - :returns: |
3776 | - a fresh WhiteList object |
3777 | - """ |
3778 | - pattern_list, max_lineno = cls._load_patterns(pathname) |
3779 | - name = os.path.splitext(os.path.basename(pathname))[0] |
3780 | - origin = Origin(FileTextSource(pathname), 1, max_lineno) |
3781 | - return cls(pattern_list, name, origin, implicit_namespace) |
3782 | - |
3783 | - @classmethod |
3784 | - def from_string(cls, text, *, filename=None, name=None, origin=None, |
3785 | - implicit_namespace=None): |
3786 | - """ |
3787 | - Load and initialize the WhiteList object from the specified string. |
3788 | - |
3789 | - :param text: |
3790 | - full text of the whitelist |
3791 | - :param filename: |
3792 | - (optional, keyword-only) filename from which text was read from. |
3793 | - This simulates a call to :meth:`from_file()` which properly |
3794 | - computes the name and origin of the whitelist. |
3795 | - :param name: |
3796 | - (optional) name of the whitelist, only used if filename is not |
3797 | - specified. |
3798 | - :param origin: |
3799 | - (optional) origin of the whitelist, only used if a filename is not |
3800 | - specified. If omitted a default origin value will be constructed |
3801 | - out of UnknownTextSource instance |
3802 | - :param implicit_namespace: |
3803 | - (optional) implicit namespace for jobs that are using partial |
3804 | - identifiers (all jobs) |
3805 | - :returns: |
3806 | - a fresh WhiteList object |
3807 | - |
3808 | - The optional filename or a pair of name and origin arguments may be |
3809 | - provided in order to have additional meta-data. This is typically |
3810 | - needed when the :meth:`from_file()` method cannot be used as the caller |
3811 | - already has the full text of the intended file available. |
3812 | - """ |
3813 | - _logger.debug("Loaded whitelist from %r", filename) |
3814 | - pattern_list, max_lineno = cls._parse_patterns(text) |
3815 | - # generate name and origin if filename is provided |
3816 | - if filename is not None: |
3817 | - name = WhiteList.name_from_filename(filename) |
3818 | - origin = Origin(FileTextSource(filename), 1, max_lineno) |
3819 | - else: |
3820 | - # otherwise generate origin if it's not specified |
3821 | - if origin is None: |
3822 | - origin = Origin(UnknownTextSource(), 1, max_lineno) |
3823 | - return cls(pattern_list, name, origin, implicit_namespace) |
3824 | - |
3825 | - @classmethod |
3826 | - def name_from_filename(cls, filename): |
3827 | - """ |
3828 | - Compute the name of a whitelist based on the name |
3829 | - of the file it is stored in. |
3830 | - """ |
3831 | - return os.path.splitext(os.path.basename(filename))[0] |
3832 | - |
3833 | - @classmethod |
3834 | - def _parse_patterns(cls, text): |
3835 | - """ |
3836 | - Load whitelist patterns from the specified text |
3837 | - |
3838 | - :param text: |
3839 | - string of text, including newlines, to parse |
3840 | - :returns: |
3841 | - (pattern_list, lineno) where lineno is the final line number |
3842 | - (1-based) and pattern_list is a list of regular expression strings |
3843 | - parsed from the whitelist. |
3844 | - """ |
3845 | - from plainbox.impl.xparsers import Re |
3846 | - from plainbox.impl.xparsers import Visitor |
3847 | - from plainbox.impl.xparsers import WhiteList |
3848 | - |
3849 | - class WhiteListVisitor(Visitor): |
3850 | - |
3851 | - def __init__(self): |
3852 | - self.pattern_list = [] |
3853 | - self.lineno = 0 |
3854 | - |
3855 | - def visit_Re_node(self, node: Re): |
3856 | - self.pattern_list.append(r"^{}$".format(node.text.strip())) |
3857 | - self.lineno = max(node.lineno, self.lineno) |
3858 | - return super().generic_visit(node) |
3859 | - |
3860 | - visit_ReFixed_node = visit_Re_node |
3861 | - visit_RePattern_node = visit_Re_node |
3862 | - visit_ReErr_node = visit_Re_node |
3863 | - |
3864 | - visitor = WhiteListVisitor() |
3865 | - visitor.visit(WhiteList.parse(text)) |
3866 | - return visitor.pattern_list, visitor.lineno |
3867 | - |
3868 | - @classmethod |
3869 | - def _load_patterns(cls, pathname): |
3870 | - """ |
3871 | - Load whitelist patterns from the specified file |
3872 | - |
3873 | - :param pathname: |
3874 | - pathname of the file to load and parse |
3875 | - :returns: |
3876 | - (pattern_list, lineno) where lineno is the final line number |
3877 | - (1-based) and pattern_list is a list of regular expression strings |
3878 | - parsed from the whitelist. |
3879 | - """ |
3880 | - with open(pathname, "rt", encoding="UTF-8") as stream: |
3881 | - return cls._parse_patterns(stream.read()) |
3882 | - |
3883 | - |
3884 | def get_flat_primitive_qualifier_list(qualifier_list): |
3885 | return list(itertools.chain(*[ |
3886 | qual.get_primitive_qualifiers() |
3887 | @@ -659,7 +437,7 @@ def select_jobs(job_list, qualifier_list): |
3888 | A sub-list of JobDefinition objects, selected from job_list. |
3889 | """ |
3890 | # Flatten the qualifier list, so that we can see the fine structure of |
3891 | - # composite objects, such as whitelists. |
3892 | + # composite objects. |
3893 | flat_qualifier_list = get_flat_primitive_qualifier_list(qualifier_list) |
3894 | # Short-circuit if there are no jobs to select. Min is used later and this |
3895 | # will allow us to assume that the matrix is not empty. |
3896 | diff --git a/plainbox/impl/secure/test_launcher1.py b/plainbox/impl/secure/test_launcher1.py |
3897 | index c270be5..247b306 100644 |
3898 | --- a/plainbox/impl/secure/test_launcher1.py |
3899 | +++ b/plainbox/impl/secure/test_launcher1.py |
3900 | @@ -110,33 +110,6 @@ class TrustedLauncherTests(TestCase): |
3901 | # Ensure that the return value of subprocess.call() is returned |
3902 | self.assertEqual(retval, mock_call()) |
3903 | |
3904 | - @mock.patch.dict('os.environ', clear=True) |
3905 | - @mock.patch('plainbox.impl.job.JobDefinition.from_rfc822_record') |
3906 | - @mock.patch('plainbox.impl.secure.launcher1.load_rfc822_records') |
3907 | - @mock.patch('subprocess.check_output') |
3908 | - def test_run_local_job(self, mock_check_output, mock_load_rfc822_records, |
3909 | - mock_from_rfc822_record): |
3910 | - # Create a mock job and add it to the launcher |
3911 | - job = mock.Mock(spec=JobDefinition, name='job', plugin='local') |
3912 | - self.launcher.add_job_list([job]) |
3913 | - # Create two mock rfc822 records |
3914 | - record1 = mock.Mock(spec=RFC822Record, name='record') |
3915 | - record2 = mock.Mock(spec=RFC822Record, name='record') |
3916 | - # Ensure that load_rfc822_records() returns some mocked records |
3917 | - mock_load_rfc822_records.return_value = [record1, record2] |
3918 | - # Run the tested method |
3919 | - job_list = self.launcher.run_generator_job(job.checksum, None) |
3920 | - # Ensure that we run the job command via job.shell |
3921 | - mock_check_output.assert_called_with( |
3922 | - [job.shell, '-c', job.command], env={}, universal_newlines=True) |
3923 | - # Ensure that we parse all of the output |
3924 | - mock_load_rfc822_records.assert_called_with( |
3925 | - mock_check_output(), source=JobOutputTextSource(job)) |
3926 | - # Ensure that we return the jobs back |
3927 | - self.assertEqual(len(job_list), 2) |
3928 | - self.assertEqual(job_list[0], mock_from_rfc822_record(record1)) |
3929 | - self.assertEqual(job_list[1], mock_from_rfc822_record(record2)) |
3930 | - |
3931 | |
3932 | class MainTests(TestCase): |
3933 | """ |
3934 | @@ -189,7 +162,7 @@ class MainTests(TestCase): |
3935 | -g CHECKSUM, --generator CHECKSUM |
3936 | also run a job with this checksum (assuming \ |
3937 | it is a |
3938 | - local job) |
3939 | + resource job) |
3940 | -G NAME=VALUE [NAME=VALUE ...], --generator-environment NAME=VALUE \ |
3941 | [NAME=VALUE ...] |
3942 | environment passed to the generator job |
3943 | @@ -257,9 +230,9 @@ it is a |
3944 | verify what happens when `plainbox-trusted-launcher-1` is called with |
3945 | both --hash and --via that both are okay and designate existing jobs. |
3946 | """ |
3947 | - # Create a mock (local) job, give it a predictable checksum |
3948 | - local_job = mock.Mock( |
3949 | - name='local_job', |
3950 | + # Create a mock generator job, give it a predictable checksum |
3951 | + generator_job = mock.Mock( |
3952 | + name='generator_job', |
3953 | spec=JobDefinition, |
3954 | checksum='5678') |
3955 | # Create a mock (target) job, give it a predictable checksum |
3956 | @@ -267,16 +240,16 @@ it is a |
3957 | name='target_job', |
3958 | spec=JobDefinition, |
3959 | checksum='1234') |
3960 | - # Ensure this local job is enumerated by the provider |
3961 | - self.provider.job_list = [local_job] |
3962 | - # Ensure that the target job is generated by the local job |
3963 | - mock_launcher.run_local_job.return_value = [target_job] |
3964 | + # Ensure this generator job is enumerated by the provider |
3965 | + self.provider.job_list = [generator_job] |
3966 | + # Ensure that the target job is generated by the generator job |
3967 | + mock_launcher.run_generator_job.return_value = [target_job] |
3968 | # Run the program with io intercept |
3969 | with TestIO(combined=True) as io: |
3970 | retval = main(['--target=1234', '--generator=5678']) |
3971 | - # Ensure that the local job command was invoked |
3972 | + # Ensure that the generator job command was invoked |
3973 | mock_launcher().run_generator_job.assert_called_with( |
3974 | - local_job.checksum, None) |
3975 | + generator_job.checksum, None) |
3976 | # Ensure that the target job command was invoked |
3977 | mock_launcher().run_shell_from_job.assert_called_with( |
3978 | target_job.checksum, None) |
3979 | diff --git a/plainbox/impl/secure/test_qualifiers.py b/plainbox/impl/secure/test_qualifiers.py |
3980 | index b02cf2b..e177796 100644 |
3981 | --- a/plainbox/impl/secure/test_qualifiers.py |
3982 | +++ b/plainbox/impl/secure/test_qualifiers.py |
3983 | @@ -39,14 +39,12 @@ from plainbox.impl.secure.qualifiers import CompositeQualifier |
3984 | from plainbox.impl.secure.qualifiers import FieldQualifier |
3985 | from plainbox.impl.secure.qualifiers import IMatcher |
3986 | from plainbox.impl.secure.qualifiers import JobIdQualifier |
3987 | -from plainbox.impl.secure.qualifiers import NonLocalJobQualifier |
3988 | from plainbox.impl.secure.qualifiers import NonPrimitiveQualifierOrigin |
3989 | from plainbox.impl.secure.qualifiers import OperatorMatcher |
3990 | from plainbox.impl.secure.qualifiers import PatternMatcher |
3991 | from plainbox.impl.secure.qualifiers import RegExpJobQualifier |
3992 | from plainbox.impl.secure.qualifiers import select_jobs |
3993 | from plainbox.impl.secure.qualifiers import SimpleQualifier |
3994 | -from plainbox.impl.secure.qualifiers import WhiteList |
3995 | from plainbox.impl.testing_utils import make_job |
3996 | from plainbox.vendor import mock |
3997 | |
3998 | @@ -378,56 +376,6 @@ class JobIdQualifierTests(TestCase): |
3999 | JobIdQualifier('*', self.origin).designates(make_job('name'))) |
4000 | |
4001 | |
4002 | -class NonLocalJobQualifierTests(TestCase): |
4003 | - """ |
4004 | - Test cases for NonLocalJobQualifier class |
4005 | - """ |
4006 | - |
4007 | - def setUp(self): |
4008 | - self.origin = mock.Mock(name='origin', spec_set=Origin) |
4009 | - self.qualifier = NonLocalJobQualifier(self.origin) |
4010 | - |
4011 | - def test_init(self): |
4012 | - """ |
4013 | - verify that init assigns stuff to properties correctly |
4014 | - """ |
4015 | - self.assertEqual(self.qualifier.origin, self.origin) |
4016 | - |
4017 | - def test_is_primitive(self): |
4018 | - """ |
4019 | - verify that LocalJobQualifier.is_primitive is True |
4020 | - """ |
4021 | - self.assertTrue(self.qualifier.is_primitive) |
4022 | - |
4023 | - def test_repr(self): |
4024 | - """ |
4025 | - verify that NonLocalJobQualifier.__repr__() works as expected |
4026 | - """ |
4027 | - self.assertEqual( |
4028 | - repr(self.qualifier), "NonLocalJobQualifier(inclusive=True)") |
4029 | - |
4030 | - def test_get_vote(self): |
4031 | - """ |
4032 | - verify that NonLocalJobQualifier.get_vote() works as expected |
4033 | - """ |
4034 | - self.assertEqual( |
4035 | - NonLocalJobQualifier(self.origin).get_vote( |
4036 | - JobDefinition({'name': 'foo', 'plugin': 'shell'})), |
4037 | - IJobQualifier.VOTE_INCLUDE) |
4038 | - self.assertEqual( |
4039 | - NonLocalJobQualifier(self.origin, inclusive=False).get_vote( |
4040 | - JobDefinition({'name': 'foo', 'plugin': 'shell'})), |
4041 | - IJobQualifier.VOTE_EXCLUDE) |
4042 | - self.assertEqual( |
4043 | - NonLocalJobQualifier(self.origin).get_vote( |
4044 | - JobDefinition({'name': 'bar', 'plugin': 'local'})), |
4045 | - IJobQualifier.VOTE_IGNORE) |
4046 | - self.assertEqual( |
4047 | - NonLocalJobQualifier(self.origin, inclusive=False).get_vote( |
4048 | - JobDefinition({'name': 'bar', 'plugin': 'local'})), |
4049 | - IJobQualifier.VOTE_IGNORE) |
4050 | - |
4051 | - |
4052 | class CompositeQualifierTests(TestCase): |
4053 | """ |
4054 | Test cases for CompositeQualifier class |
4055 | @@ -537,185 +485,6 @@ class CompositeQualifierTests(TestCase): |
4056 | CompositeQualifier([]).origin |
4057 | |
4058 | |
4059 | -class WhiteListTests(TestCase): |
4060 | - """ |
4061 | - Test cases for WhiteList class |
4062 | - """ |
4063 | - |
4064 | - _name = 'whitelist.txt' |
4065 | - |
4066 | - _content = [ |
4067 | - "# this is a comment", |
4068 | - "foo # this is another comment", |
4069 | - "bar", |
4070 | - "" |
4071 | - ] |
4072 | - |
4073 | - @contextmanager |
4074 | - def mocked_file(self, name, content): |
4075 | - m_open = mock.MagicMock(name='open', spec=open) |
4076 | - m_stream = mock.MagicMock(spec=TextIOWrapper) |
4077 | - m_stream.__enter__.return_value = m_stream |
4078 | - # The next two lines are complementary, either will suffice but the |
4079 | - # test may need changes if the code that reads stuff changes. |
4080 | - m_stream.__iter__.side_effect = lambda: iter(content) |
4081 | - m_stream.read.return_value = "\n".join(content) |
4082 | - m_open.return_value = m_stream |
4083 | - with mock.patch('plainbox.impl.secure.qualifiers.open', m_open, |
4084 | - create=True): |
4085 | - yield |
4086 | - m_open.assert_called_once_with(name, "rt", encoding="UTF-8") |
4087 | - |
4088 | - def test_load_patterns(self): |
4089 | - with self.mocked_file(self._name, self._content): |
4090 | - pattern_list, max_lineno = WhiteList._load_patterns(self._name) |
4091 | - self.assertEqual(pattern_list, ['^foo$', '^bar$']) |
4092 | - self.assertEqual(max_lineno, 3) |
4093 | - |
4094 | - def test_designates(self): |
4095 | - """ |
4096 | - verify that WhiteList.designates() works |
4097 | - """ |
4098 | - self.assertTrue( |
4099 | - WhiteList.from_string("foo").designates(make_job('foo'))) |
4100 | - self.assertTrue( |
4101 | - WhiteList.from_string("foo\nbar\n").designates(make_job('foo'))) |
4102 | - self.assertTrue( |
4103 | - WhiteList.from_string("foo\nbar\n").designates(make_job('bar'))) |
4104 | - # Note, it's not matching either! |
4105 | - self.assertFalse( |
4106 | - WhiteList.from_string("foo").designates(make_job('foobar'))) |
4107 | - self.assertFalse( |
4108 | - WhiteList.from_string("bar").designates(make_job('foobar'))) |
4109 | - |
4110 | - def test_from_file(self): |
4111 | - """ |
4112 | - verify that WhiteList.from_file() works |
4113 | - """ |
4114 | - with self.mocked_file(self._name, self._content): |
4115 | - whitelist = WhiteList.from_file(self._name) |
4116 | - # verify that the patterns are okay |
4117 | - self.assertEqual( |
4118 | - repr(whitelist.qualifier_list[0]), |
4119 | - "RegExpJobQualifier('^foo$', inclusive=True)") |
4120 | - # verify that whitelist name got set |
4121 | - self.assertEqual(whitelist.name, "whitelist") |
4122 | - # verify that the origin got set |
4123 | - self.assertEqual( |
4124 | - whitelist.origin, |
4125 | - Origin(FileTextSource("whitelist.txt"), 1, 3)) |
4126 | - |
4127 | - def test_from_string(self): |
4128 | - """ |
4129 | - verify that WhiteList.from_string() works |
4130 | - """ |
4131 | - whitelist = WhiteList.from_string("\n".join(self._content)) |
4132 | - # verify that the patterns are okay |
4133 | - self.assertEqual( |
4134 | - repr(whitelist.qualifier_list[0]), |
4135 | - "RegExpJobQualifier('^foo$', inclusive=True)") |
4136 | - # verify that whitelist name is the empty default |
4137 | - self.assertEqual(whitelist.name, None) |
4138 | - # verify that the origin got set to the default constructed value |
4139 | - self.assertEqual(whitelist.origin, Origin(UnknownTextSource(), 1, 3)) |
4140 | - |
4141 | - def test_from_empty_string(self): |
4142 | - """ |
4143 | - verify that WhiteList.from_string("") works |
4144 | - """ |
4145 | - WhiteList.from_string("") |
4146 | - |
4147 | - def test_from_string__with_name_and_origin(self): |
4148 | - """ |
4149 | - verify that WhiteList.from_string() works when passing name and origin |
4150 | - """ |
4151 | - # construct a whitelist with some dummy data, the names, pathnames and |
4152 | - # line ranges are arbitrary |
4153 | - whitelist = WhiteList.from_string( |
4154 | - "\n".join(self._content), name="somefile", |
4155 | - origin=Origin(FileTextSource("somefile.txt"), 1, 3)) |
4156 | - # verify that the patterns are okay |
4157 | - self.assertEqual( |
4158 | - repr(whitelist.qualifier_list[0]), |
4159 | - "RegExpJobQualifier('^foo$', inclusive=True)") |
4160 | - # verify that whitelist name is copied |
4161 | - self.assertEqual(whitelist.name, "somefile") |
4162 | - # verify that the origin is copied |
4163 | - self.assertEqual( |
4164 | - whitelist.origin, Origin(FileTextSource("somefile.txt"), 1, 3)) |
4165 | - |
4166 | - def test_from_string__with_filename(self): |
4167 | - """ |
4168 | - verify that WhiteList.from_string() works when passing filename |
4169 | - """ |
4170 | - # construct a whitelist with some dummy data, the names, pathnames and |
4171 | - # line ranges are arbitrary |
4172 | - whitelist = WhiteList.from_string( |
4173 | - "\n".join(self._content), filename="somefile.txt") |
4174 | - # verify that the patterns are okay |
4175 | - self.assertEqual( |
4176 | - repr(whitelist.qualifier_list[0]), |
4177 | - "RegExpJobQualifier('^foo$', inclusive=True)") |
4178 | - # verify that whitelist name is derived from the filename |
4179 | - self.assertEqual(whitelist.name, "somefile") |
4180 | - # verify that the origin is properly derived from the filename |
4181 | - self.assertEqual( |
4182 | - whitelist.origin, Origin(FileTextSource("somefile.txt"), 1, 3)) |
4183 | - |
4184 | - def test_repr(self): |
4185 | - """ |
4186 | - verify that custom repr works |
4187 | - """ |
4188 | - whitelist = WhiteList([], name="test") |
4189 | - self.assertEqual(repr(whitelist), "<WhiteList name:'test'>") |
4190 | - |
4191 | - def test_name_getter(self): |
4192 | - """ |
4193 | - verify that WhiteList.name getter works |
4194 | - """ |
4195 | - self.assertEqual(WhiteList([], "foo").name, "foo") |
4196 | - |
4197 | - def test_name_setter(self): |
4198 | - """ |
4199 | - verify that WhiteList.name setter works |
4200 | - """ |
4201 | - whitelist = WhiteList([], "foo") |
4202 | - whitelist.name = "bar" |
4203 | - self.assertEqual(whitelist.name, "bar") |
4204 | - |
4205 | - def test_name_from_filename(self): |
4206 | - """ |
4207 | - verify how name_from_filename() works |
4208 | - """ |
4209 | - self.assertEqual( |
4210 | - WhiteList.name_from_filename("some/path/foo.whitelist"), "foo") |
4211 | - self.assertEqual(WhiteList.name_from_filename("foo.whitelist"), "foo") |
4212 | - self.assertEqual(WhiteList.name_from_filename("foo."), "foo") |
4213 | - self.assertEqual(WhiteList.name_from_filename("foo"), "foo") |
4214 | - self.assertEqual( |
4215 | - WhiteList.name_from_filename("foo.notawhitelist"), "foo") |
4216 | - |
4217 | - def test_namespace_behavior(self): |
4218 | - """ |
4219 | - verify that WhiteList() correctly respects namespace declarations |
4220 | - and uses implict_namespace to fully qualifiy all patterns |
4221 | - """ |
4222 | - whitelist = WhiteList.from_string( |
4223 | - "foo\n" |
4224 | - "example\\.org::bar\n", |
4225 | - implicit_namespace="other.example.org") |
4226 | - # verify that the implicit namespace was recorded |
4227 | - self.assertEqual( |
4228 | - whitelist.implicit_namespace, "other.example.org") |
4229 | - # verify that the patterns are okay |
4230 | - self.assertEqual( |
4231 | - whitelist.qualifier_list[0].pattern_text, |
4232 | - "^other\\.example\\.org::foo$") |
4233 | - self.assertEqual( |
4234 | - whitelist.qualifier_list[1].pattern_text, |
4235 | - "^example\\.org::bar$") |
4236 | - |
4237 | - |
4238 | class FunctionTests(TestCase): |
4239 | |
4240 | def setUp(self): |
4241 | diff --git a/plainbox/impl/session/assistant.py b/plainbox/impl/session/assistant.py |
4242 | index 1ea7ad7..db60157 100644 |
4243 | --- a/plainbox/impl/session/assistant.py |
4244 | +++ b/plainbox/impl/session/assistant.py |
4245 | @@ -44,6 +44,7 @@ from plainbox.impl.developer import UnexpectedMethodCall |
4246 | from plainbox.impl.developer import UsageExpectation |
4247 | from plainbox.impl.jobcache import ResourceJobCache |
4248 | from plainbox.impl.result import JobResultBuilder |
4249 | +from plainbox.impl.providers import get_providers |
4250 | from plainbox.impl.runner import JobRunner |
4251 | from plainbox.impl.runner import JobRunnerUIDelegate |
4252 | from plainbox.impl.secure.origin import Origin |
4253 | @@ -60,7 +61,6 @@ from plainbox.impl.session.restart import detect_restart_strategy |
4254 | from plainbox.impl.session.storage import SessionStorageRepository |
4255 | from plainbox.impl.transport import OAuthTransport |
4256 | from plainbox.impl.transport import TransportError |
4257 | -from plainbox.public import get_providers |
4258 | from plainbox.vendor import morris |
4259 | |
4260 | _logger = logging.getLogger("plainbox.session.assistant") |
4261 | @@ -837,9 +837,8 @@ class SessionAssistant: |
4262 | |
4263 | During the bootstrap phase resource jobs that are associated with job |
4264 | templates may generate new jobs according to the information specified |
4265 | - in the template. In addition, local jobs can generate arbitrary |
4266 | - (unrestricted) units. Both of those mechanism are subject to the |
4267 | - validation system (invalid units are discarded). |
4268 | + in the template. This mechanism is subject to the validation system |
4269 | + (invalid units are discarded). |
4270 | |
4271 | When this method returns (which can take a while) the session is now |
4272 | ready for running any jobs. |
4273 | @@ -850,8 +849,8 @@ class SessionAssistant: |
4274 | """ |
4275 | UsageExpectation.of(self).enforce() |
4276 | # NOTE: there is next-to-none UI here as bootstrap jobs are limited to |
4277 | - # just resource and local jobs (including their dependencies) so there |
4278 | - # should be very little UI required. |
4279 | + # just resource jobs (including their dependencies) so there should be |
4280 | + # very little UI required. |
4281 | desired_job_list = select_jobs( |
4282 | self._context.state.job_list, |
4283 | [plan.get_bootstrap_qualifier() for plan in ( |
4284 | diff --git a/plainbox/impl/session/jobs.py b/plainbox/impl/session/jobs.py |
4285 | index 22a138b..809c72c 100644 |
4286 | --- a/plainbox/impl/session/jobs.py |
4287 | +++ b/plainbox/impl/session/jobs.py |
4288 | @@ -27,6 +27,7 @@ particular session. The :class:`JobState` class holds references to a |
4289 | that prevent the job from being runnable in a particular session. |
4290 | """ |
4291 | |
4292 | +from enum import IntEnum |
4293 | import logging |
4294 | |
4295 | from plainbox.abc import IJobResult |
4296 | @@ -35,7 +36,6 @@ from plainbox.impl import pod |
4297 | from plainbox.impl.resource import ResourceExpression |
4298 | from plainbox.impl.result import MemoryJobResult |
4299 | from plainbox.impl.unit.job import JobDefinition |
4300 | -from plainbox.vendor.enum import IntEnum |
4301 | |
4302 | logger = logging.getLogger("plainbox.session.jobs") |
4303 | |
4304 | diff --git a/plainbox/impl/session/manager.py b/plainbox/impl/session/manager.py |
4305 | index 9a056fa..20616d0 100644 |
4306 | --- a/plainbox/impl/session/manager.py |
4307 | +++ b/plainbox/impl/session/manager.py |
4308 | @@ -39,6 +39,7 @@ import tempfile |
4309 | |
4310 | from plainbox.i18n import gettext as _, ngettext |
4311 | from plainbox.impl import pod |
4312 | +from plainbox.impl.providers import get_providers |
4313 | from plainbox.impl.session.resume import SessionResumeHelper |
4314 | from plainbox.impl.session.state import SessionDeviceContext |
4315 | from plainbox.impl.session.state import SessionState |
4316 | @@ -47,7 +48,6 @@ from plainbox.impl.session.storage import SessionStorage |
4317 | from plainbox.impl.session.storage import SessionStorageRepository |
4318 | from plainbox.impl.session.suspend import SessionSuspendHelper |
4319 | from plainbox.impl.unit.testplan import TestPlanUnit |
4320 | -from plainbox.public import get_providers |
4321 | from plainbox.vendor import morris |
4322 | |
4323 | logger = logging.getLogger("plainbox.session.manager") |
4324 | @@ -187,7 +187,7 @@ class SessionManager(pod.POD): |
4325 | return self.default_device_context.state |
4326 | |
4327 | @classmethod |
4328 | - def create(cls, repo=None, legacy_mode=False, prefix='pbox-'): |
4329 | + def create(cls, repo=None, prefix='pbox-'): |
4330 | """ |
4331 | Create an empty session manager. |
4332 | |
4333 | @@ -205,24 +205,18 @@ class SessionManager(pod.POD): |
4334 | constructed with the default location. |
4335 | :ptype repo: |
4336 | :class:`~plainbox.impl.session.storage.SessionStorageRepository`. |
4337 | - :param legacy_mode: |
4338 | - Propagated to |
4339 | - :meth:`~plainbox.impl.session.storage.SessionStorage.create()` to |
4340 | - ensure that legacy (single session) mode is used. |
4341 | - :ptype legacy_mode: |
4342 | - bool |
4343 | :return: |
4344 | fresh :class:`SessionManager` instance |
4345 | """ |
4346 | logger.debug("SessionManager.create()") |
4347 | if repo is None: |
4348 | repo = SessionStorageRepository() |
4349 | - storage = SessionStorage.create(repo.location, legacy_mode, prefix) |
4350 | + storage = SessionStorage.create(repo.location, prefix) |
4351 | WellKnownDirsHelper(storage).populate() |
4352 | return cls([], storage) |
4353 | |
4354 | @classmethod |
4355 | - def create_with_state(cls, state, repo=None, legacy_mode=False): |
4356 | + def create_with_state(cls, state, repo=None): |
4357 | """ |
4358 | Create a session manager by wrapping existing session state. |
4359 | |
4360 | @@ -237,26 +231,19 @@ class SessionManager(pod.POD): |
4361 | constructed with the default location. |
4362 | :ptype repo: |
4363 | :class:`~plainbox.impl.session.storage.SessionStorageRepository`. |
4364 | - :param legacy_mode: |
4365 | - Propagated to |
4366 | - :meth:`~plainbox.impl.session.storage.SessionStorage.create()` |
4367 | - to ensure that legacy (single session) mode is used. |
4368 | - :ptype legacy_mode: |
4369 | - bool |
4370 | :return: |
4371 | fresh :class:`SessionManager` instance |
4372 | """ |
4373 | logger.debug("SessionManager.create_with_state()") |
4374 | if repo is None: |
4375 | repo = SessionStorageRepository() |
4376 | - storage = SessionStorage.create(repo.location, legacy_mode) |
4377 | + storage = SessionStorage.create(repo.location) |
4378 | WellKnownDirsHelper(storage).populate() |
4379 | context = SessionDeviceContext(state) |
4380 | return cls([context], storage) |
4381 | |
4382 | @classmethod |
4383 | - def create_with_unit_list(cls, unit_list=None, repo=None, |
4384 | - legacy_mode=False): |
4385 | + def create_with_unit_list(cls, unit_list=None, repo=None): |
4386 | """ |
4387 | Create a session manager with a fresh session. |
4388 | |
4389 | @@ -272,12 +259,6 @@ class SessionManager(pod.POD): |
4390 | constructed with the default location. |
4391 | :ptype repo: |
4392 | :class:`~plainbox.impl.session.storage.SessionStorageRepository`. |
4393 | - :param legacy_mode: |
4394 | - Propagated to |
4395 | - :meth:`~plainbox.impl.session.storage.SessionStorage.create()` |
4396 | - to ensure that legacy (single session) mode is used. |
4397 | - :ptype legacy_mode: |
4398 | - bool |
4399 | :return: |
4400 | fresh :class:`SessionManager` instance |
4401 | """ |
4402 | @@ -287,7 +268,7 @@ class SessionManager(pod.POD): |
4403 | state = SessionState(unit_list) |
4404 | if repo is None: |
4405 | repo = SessionStorageRepository() |
4406 | - storage = SessionStorage.create(repo.location, legacy_mode) |
4407 | + storage = SessionStorage.create(repo.location) |
4408 | context = SessionDeviceContext(state) |
4409 | WellKnownDirsHelper(storage).populate() |
4410 | return cls([context], storage) |
4411 | @@ -480,7 +461,6 @@ class SessionManager(pod.POD): |
4412 | 'com.canonical.plainbox::html': 'html', |
4413 | 'com.canonical.plainbox::json': 'json', |
4414 | 'com.canonical.plainbox::junit': 'junit', |
4415 | - 'com.canonical.plainbox::rfc822': 'rfc822', |
4416 | 'com.canonical.plainbox::tar': 'tar', |
4417 | 'com.canonical.plainbox::text': 'text', |
4418 | 'com.canonical.plainbox::xlsx': 'xlsx' |
4419 | @@ -498,8 +478,7 @@ class SessionManager(pod.POD): |
4420 | Identifier of the exporter unit (which must have been loaded |
4421 | into the session device context of the first device). For |
4422 | backwards compatibility this can also be any of the legacy |
4423 | - identifiers ``tar``, ``html``, ``json``, ``rfc822``, ``text`` or |
4424 | - ``xlsx``. |
4425 | + identifiers ``tar``, ``html``, ``json``, ``text`` or ``xlsx``. |
4426 | :param option_list: |
4427 | (optional) A list of options to pass to the exporter. Each option |
4428 | is a string. Some strings may be of form 'key=value' but those are |
4429 | diff --git a/plainbox/impl/session/state.py b/plainbox/impl/session/state.py |
4430 | index 0631de1..c7c6601 100644 |
4431 | --- a/plainbox/impl/session/state.py |
4432 | +++ b/plainbox/impl/session/state.py |
4433 | @@ -645,10 +645,6 @@ class SessionState: |
4434 | Not all the jobs from this list are going to be executed (or selected |
4435 | for execution) by the user. |
4436 | |
4437 | - It may change at runtime because of local jobs. Note that in upcoming |
4438 | - changes this will start out empty and will be changeable dynamically. |
4439 | - It can still change due to local jobs but there is no API yes. |
4440 | - |
4441 | This list cannot have any duplicates, if that is the case a |
4442 | :class:`DependencyDuplicateError` is raised. This has to be handled |
4443 | externally and is a sign that the job database is corrupted or has |
4444 | @@ -660,8 +656,7 @@ class SessionState: |
4445 | This list contains all the known units, including all the know job |
4446 | definitions (and in the future, all test plans). |
4447 | |
4448 | - It may change at runtime because of local jobs and template |
4449 | - instantiations. |
4450 | + It may change at runtime because of template instantiations. |
4451 | |
4452 | :ivar dict job_state_map: mapping that tracks the state of each job |
4453 | |
4454 | @@ -971,7 +966,7 @@ class SessionState: |
4455 | if job.automated and estimate_automated is not None: |
4456 | if job.estimated_duration is not None: |
4457 | estimate_automated += job.estimated_duration |
4458 | - elif job.plugin != 'local': |
4459 | + else: |
4460 | estimate_automated = None |
4461 | elif not job.automated and estimate_manual is not None: |
4462 | # We add a fixed extra amount of seconds to the run time |
4463 | @@ -993,8 +988,8 @@ class SessionState: |
4464 | Results also change the ready map (jobs that can run) because of |
4465 | dependency relations. |
4466 | |
4467 | - Some results have deeper meaning, those are results for local and |
4468 | - resource jobs. They are discussed in detail below: |
4469 | + Some results have deeper meaning, those are results for resource jobs. |
4470 | + They are discussed in detail below: |
4471 | |
4472 | Resource jobs produce resource records which are used as data to run |
4473 | requirement expressions against. Each time a result for a resource job |
4474 | @@ -1002,11 +997,6 @@ class SessionState: |
4475 | records. A new entry is created in the resource map (entirely replacing |
4476 | any old entries), with a list of the resources that were parsed from |
4477 | the IO log. |
4478 | - |
4479 | - Local jobs produce more jobs. Like with resource jobs, their IO log is |
4480 | - parsed and interpreted as additional jobs. Unlike in resource jobs |
4481 | - local jobs don't replace anything. They cannot replace an existing job |
4482 | - with the same id. |
4483 | """ |
4484 | job.controller.observe_result(self, job, result) |
4485 | self._recompute_job_readiness() |
4486 | @@ -1108,8 +1098,7 @@ class SessionState: |
4487 | return new_job |
4488 | else: |
4489 | # If there is a clash report DependencyDuplicateError only when the |
4490 | - # hashes are different. This prevents a common "problem" where |
4491 | - # "__foo__" local jobs just load all jobs from the "foo" category. |
4492 | + # hashes are different. |
4493 | if new_job != existing_job: |
4494 | raise DependencyDuplicateError(existing_job, new_job) |
4495 | self._add_job_siblings_unit(new_job, recompute) |
4496 | diff --git a/plainbox/impl/session/storage.py b/plainbox/impl/session/storage.py |
4497 | index de9620a..c290a24 100644 |
4498 | --- a/plainbox/impl/session/storage.py |
4499 | +++ b/plainbox/impl/session/storage.py |
4500 | @@ -54,8 +54,6 @@ class SessionStorageRepository: |
4501 | :meth:SessionStorage.remove()`) |
4502 | """ |
4503 | |
4504 | - _LAST_SESSION_SYMLINK = "last-session" |
4505 | - |
4506 | def __init__(self, location=None): |
4507 | """ |
4508 | Initialize new repository at the specified location. |
4509 | @@ -75,35 +73,6 @@ class SessionStorageRepository: |
4510 | """ |
4511 | return self._location |
4512 | |
4513 | - def get_last_storage(self): |
4514 | - """ |
4515 | - Find the last session storage object created in this repository. |
4516 | - |
4517 | - :returns: |
4518 | - SessionStorage object associated with the last session created in |
4519 | - this repository using legacy mode. |
4520 | - |
4521 | - .. note:: |
4522 | - This will only return storage objects that were created using |
4523 | - legacy mode. Nonlegacy storage objects will not be returned this |
4524 | - way. |
4525 | - """ |
4526 | - pathname = os.path.join(self.location, self._LAST_SESSION_SYMLINK) |
4527 | - try: |
4528 | - last_storage = os.readlink(pathname) |
4529 | - except OSError: |
4530 | - # The symlink can be gone or not be a real symlink |
4531 | - # in that case just ignore it and return None |
4532 | - return None |
4533 | - else: |
4534 | - # The link may be relative so let's ensure we know the full |
4535 | - # pathname for the subsequent check (which may be performed |
4536 | - # from another directory) |
4537 | - last_storage = os.path.join(self._location, last_storage) |
4538 | - # If the link points to a directory, assume it's okay |
4539 | - if os.path.isdir(last_storage): |
4540 | - return SessionStorage(last_storage) |
4541 | - |
4542 | def get_storage_list(self): |
4543 | """ |
4544 | Enumerate stored sessions in the repository. |
4545 | @@ -251,7 +220,7 @@ class SessionStorage: |
4546 | return os.path.join(self._location, self._SESSION_FILE) |
4547 | |
4548 | @classmethod |
4549 | - def create(cls, base_dir, legacy_mode=False, prefix='pbox-'): |
4550 | + def create(cls, base_dir, prefix='pbox-'): |
4551 | """ |
4552 | Create a new :class:`SessionStorage` in a random subdirectory |
4553 | of the specified base directory. The base directory is also |
4554 | @@ -262,25 +231,9 @@ class SessionStorage: |
4555 | Typically the base directory should be obtained from |
4556 | :meth:`SessionStorageRepository.get_default_location()` |
4557 | |
4558 | - :param legacy_mode: |
4559 | - If False (defaults to True) then the caller is expected to |
4560 | - handle multiple sessions by itself. |
4561 | - |
4562 | :param prefix: |
4563 | String which should prefix all session filenames. The prefix is |
4564 | sluggified before use. |
4565 | - |
4566 | - .. note:: |
4567 | - Legacy mode is where applications using PlainBox API can only |
4568 | - handle one session. Creating another session replaces whatever was |
4569 | - stored before. In non-legacy mode applications can enumerate |
4570 | - sessions, create arbitrary number of sessions at the same time |
4571 | - and remove sessions once they are no longer necessary. |
4572 | - |
4573 | - Legacy mode is implemented with a symbolic link called |
4574 | - 'last-session' that keeps track of the last session created using |
4575 | - ``legacy_mode=True``. When a new legacy-mode session is created |
4576 | - the target of that symlink is read and recursively removed. |
4577 | """ |
4578 | if not os.path.exists(base_dir): |
4579 | os.makedirs(base_dir) |
4580 | @@ -298,57 +251,8 @@ class SessionStorage: |
4581 | os.mkdir(location) |
4582 | logger.debug(_("Created new storage in %r"), location) |
4583 | self = cls(location) |
4584 | - if legacy_mode: |
4585 | - self._replace_legacy_session(base_dir) |
4586 | return self |
4587 | |
4588 | - def _replace_legacy_session(self, base_dir): |
4589 | - """ |
4590 | - Remove the previous legacy session and update the 'last-session' |
4591 | - symlink so that it points to this session storage directory. |
4592 | - """ |
4593 | - symlink_pathname = os.path.join( |
4594 | - base_dir, SessionStorageRepository._LAST_SESSION_SYMLINK) |
4595 | - # Try to read and remove the storage referenced to by last-session |
4596 | - # symlink. This can fail if the link file is gone (which is harmless) |
4597 | - # or when it is not an actual symlink (which means that the |
4598 | - # repository is corrupted). |
4599 | - try: |
4600 | - symlink_target = os.readlink(symlink_pathname) |
4601 | - except OSError as exc: |
4602 | - if exc.errno == errno.ENOENT: |
4603 | - pass |
4604 | - elif exc.errno == errno.EINVAL: |
4605 | - logger.warning( |
4606 | - _("%r is not a symlink, repository %r must be corrupted"), |
4607 | - symlink_pathname, base_dir) |
4608 | - else: |
4609 | - logger.warning( |
4610 | - _("Unable to read symlink target from %r: %r"), |
4611 | - symlink_pathname, exc) |
4612 | - else: |
4613 | - logger.debug( |
4614 | - _("Removing storage associated with last session %r"), |
4615 | - symlink_target) |
4616 | - # Remove the old session, note that the symlink may be broken so |
4617 | - # let's ignore any errors here |
4618 | - shutil.rmtree(symlink_target, ignore_errors=True) |
4619 | - # Remove the last-session symlink itself |
4620 | - logger.debug( |
4621 | - _("Removing symlink associated with last session: %r"), |
4622 | - symlink_pathname) |
4623 | - os.unlink(symlink_pathname) |
4624 | - finally: |
4625 | - # Finally put the last-session synlink that points to this storage |
4626 | - logger.debug( |
4627 | - _("Linking storage %r to last session"), self.location) |
4628 | - try: |
4629 | - os.symlink(self.location, symlink_pathname) |
4630 | - except OSError as exc: |
4631 | - logger.error( |
4632 | - _("Cannot link %r as %r: %r"), |
4633 | - self.location, symlink_pathname, exc) |
4634 | - |
4635 | def remove(self): |
4636 | """ |
4637 | Remove all filesystem entries associated with this instance. |
4638 | @@ -367,164 +271,7 @@ class SessionStorage: |
4639 | |
4640 | :raises IOError, OSError: |
4641 | on various problems related to accessing the filesystem |
4642 | - |
4643 | - :raises NotImplementedError: |
4644 | - when openat(2) is not available on this platform. Should never |
4645 | - happen on Linux or Windows where appropriate checks divert to a |
4646 | - correct implementation that is not using them. |
4647 | - """ |
4648 | - if sys.platform == 'linux' or sys.platform == 'linux2': |
4649 | - if sys.version_info[0:2] >= (3, 3): |
4650 | - return self._load_checkpoint_unix_py33() |
4651 | - else: |
4652 | - return self._load_checkpoint_unix_py32() |
4653 | - elif sys.platform == 'win32': |
4654 | - return self._load_checkpoint_win32_py33() |
4655 | - raise NotImplementedError( |
4656 | - "platform/python combination is not supported: {} + {}".format( |
4657 | - sys.version, sys.platform)) |
4658 | - |
4659 | - def save_checkpoint(self, data): |
4660 | - """ |
4661 | - Save checkpoint data to the filesystem. |
4662 | - |
4663 | - The directory associated with this :class:`SessionStorage` must already |
4664 | - exist. Typically the instance should be obtained by calling |
4665 | - :meth:`SessionStorage.create()` which will ensure that this is already |
4666 | - the case. |
4667 | - |
4668 | - :raises TypeError: |
4669 | - if data is not a bytes object. |
4670 | - |
4671 | - :raises LockedStorageError: |
4672 | - if leftovers from previous save_checkpoint() have been detected. |
4673 | - Normally those should never be here but in certain cases that is |
4674 | - possible. Callers might want to call :meth:`break_lock()` |
4675 | - to resolve the problem and try again. |
4676 | - |
4677 | - :raises IOError, OSError: |
4678 | - on various problems related to accessing the filesystem. |
4679 | - Typically permission errors may be reported here. |
4680 | - |
4681 | - :raises NotImplementedError: |
4682 | - when openat(2), renameat(2), unlinkat(2) are not available on this |
4683 | - platform. Should never happen on Linux or Windows where appropriate |
4684 | - checks divert to a correct implementation that is not using them. |
4685 | """ |
4686 | - if sys.platform == 'linux' or sys.platform == 'linux2': |
4687 | - if sys.version_info[0:2] >= (3, 3): |
4688 | - return self._save_checkpoint_unix_py33(data) |
4689 | - else: |
4690 | - return self._save_checkpoint_unix_py32(data) |
4691 | - elif sys.platform == 'win32': |
4692 | - if sys.version_info[0:2] >= (3, 3): |
4693 | - return self._save_checkpoint_win32_py33(data) |
4694 | - raise NotImplementedError( |
4695 | - "platform/python combination is not supported: {} + {}".format( |
4696 | - sys.version, sys.platform)) |
4697 | - |
4698 | - def break_lock(self): |
4699 | - """ |
4700 | - Forcibly unlock the storage by removing a file created during |
4701 | - atomic filesystem operations of save_checkpoint(). |
4702 | - |
4703 | - This method might be useful if save_checkpoint() |
4704 | - raises LockedStorageError. It removes the "next" file that is used |
4705 | - for atomic rename. |
4706 | - """ |
4707 | - _next_session_pathname = os.path.join( |
4708 | - self._location, self._SESSION_FILE_NEXT) |
4709 | - logger.debug( |
4710 | - # TRANSLATORS: unlinking as in deleting a file |
4711 | - # Please keep the 'next' string untranslated |
4712 | - _("Forcibly unlinking 'next' file %r"), _next_session_pathname) |
4713 | - os.unlink(_next_session_pathname) |
4714 | - |
4715 | - def _load_checkpoint_win32_py33(self): |
4716 | - logger.debug(_("Loading checkpoint (%s)"), "Windows") |
4717 | - _session_pathname = os.path.join(self._location, self._SESSION_FILE) |
4718 | - try: |
4719 | - # Open the current session file in the location directory |
4720 | - session_fd = os.open(_session_pathname, os.O_RDONLY | os.O_BINARY) |
4721 | - logger.debug( |
4722 | - _("Opened session state file %r as descriptor %d"), |
4723 | - _session_pathname, session_fd) |
4724 | - # Stat the file to know how much to read |
4725 | - session_stat = os.fstat(session_fd) |
4726 | - logger.debug( |
4727 | - # TRANSLATORS: stat is a system call name, don't translate it |
4728 | - _("Stat'ed session state file: %s"), session_stat) |
4729 | - try: |
4730 | - # Read session data |
4731 | - logger.debug(ngettext( |
4732 | - "Reading %d byte of session state", |
4733 | - "Reading %d bytes of session state", |
4734 | - session_stat.st_size), session_stat.st_size) |
4735 | - data = os.read(session_fd, session_stat.st_size) |
4736 | - logger.debug(ngettext( |
4737 | - "Read %d byte of session state", |
4738 | - "Read %d bytes of session state", len(data)), len(data)) |
4739 | - if len(data) != session_stat.st_size: |
4740 | - raise IOError(_("partial read?")) |
4741 | - finally: |
4742 | - # Close the session file |
4743 | - logger.debug(_("Closed descriptor %d"), session_fd) |
4744 | - os.close(session_fd) |
4745 | - except IOError as exc: |
4746 | - if exc.errno == errno.ENOENT: |
4747 | - # Treat lack of 'session' file as an empty file |
4748 | - return b'' |
4749 | - raise |
4750 | - else: |
4751 | - return data |
4752 | - |
4753 | - def _load_checkpoint_unix_py32(self): |
4754 | - _session_pathname = os.path.join(self._location, self._SESSION_FILE) |
4755 | - # Open the location directory |
4756 | - location_fd = os.open(self._location, os.O_DIRECTORY) |
4757 | - logger.debug( |
4758 | - _("Opened session directory %r as descriptor %d"), |
4759 | - self._location, location_fd) |
4760 | - try: |
4761 | - # Open the current session file in the location directory |
4762 | - session_fd = os.open(_session_pathname, os.O_RDONLY) |
4763 | - logger.debug( |
4764 | - _("Opened session state file %r as descriptor %d"), |
4765 | - _session_pathname, session_fd) |
4766 | - # Stat the file to know how much to read |
4767 | - session_stat = os.fstat(session_fd) |
4768 | - logger.debug( |
4769 | - # TRANSLATORS: stat is a system call name, don't translate it |
4770 | - _("Stat'ed session state file: %s"), session_stat) |
4771 | - try: |
4772 | - # Read session data |
4773 | - logger.debug(ngettext( |
4774 | - "Reading %d byte of session state", |
4775 | - "Reading %d bytes of session state", |
4776 | - session_stat.st_size), session_stat.st_size) |
4777 | - data = os.read(session_fd, session_stat.st_size) |
4778 | - logger.debug(ngettext( |
4779 | - "Read %d byte of session state", |
4780 | - "Read %d bytes of session state", len(data)), len(data)) |
4781 | - if len(data) != session_stat.st_size: |
4782 | - raise IOError(_("partial read?")) |
4783 | - finally: |
4784 | - # Close the session file |
4785 | - logger.debug(_("Closed descriptor %d"), session_fd) |
4786 | - os.close(session_fd) |
4787 | - except IOError as exc: |
4788 | - if exc.errno == errno.ENOENT: |
4789 | - # Treat lack of 'session' file as an empty file |
4790 | - return b'' |
4791 | - raise |
4792 | - else: |
4793 | - return data |
4794 | - finally: |
4795 | - # Close the location directory |
4796 | - logger.debug(_("Closed descriptor %d"), location_fd) |
4797 | - os.close(location_fd) |
4798 | - |
4799 | - def _load_checkpoint_unix_py33(self): |
4800 | # Open the location directory |
4801 | location_fd = os.open(self._location, os.O_DIRECTORY) |
4802 | try: |
4803 | @@ -552,225 +299,28 @@ class SessionStorage: |
4804 | # Close the location directory |
4805 | os.close(location_fd) |
4806 | |
4807 | - def _save_checkpoint_win32_py33(self, data): |
4808 | - # NOTE: this is like _save_checkpoint_py32 but without location_fd |
4809 | - # wich cannot be opened on windows (no os.O_DIRECTORY) |
4810 | - # |
4811 | - # NOTE: The windows version is relatively new and under-tested |
4812 | - # but then again we don't expect to run tests *on* windows, only |
4813 | - # *from* windows so hard data retention requirements are of lesser |
4814 | - # importance. |
4815 | - if not isinstance(data, bytes): |
4816 | - raise TypeError("data must be bytes") |
4817 | - logger.debug(ngettext( |
4818 | - "Saving %d byte of data (%s)", |
4819 | - "Saving %d bytes of data (%s)", |
4820 | - len(data)), len(data), "Windows") |
4821 | - # Helper pathnames, needed because we don't have *at functions |
4822 | - _next_session_pathname = os.path.join( |
4823 | - self._location, self._SESSION_FILE_NEXT) |
4824 | - _session_pathname = os.path.join(self._location, self._SESSION_FILE) |
4825 | - # Open the "next" file in the location_directory |
4826 | - # |
4827 | - # Use "write" + "create" + "exclusive" flags so that no race |
4828 | - # condition is possible. |
4829 | - # |
4830 | - # This will never return -1, it throws IOError when anything is |
4831 | - # wrong. The caller has to catch this. |
4832 | - # |
4833 | - # As a special exception, this code handles EEXISTS and converts |
4834 | - # that to LockedStorageError that can be especially handled by |
4835 | - # some layer above. |
4836 | - try: |
4837 | - next_session_fd = os.open( |
4838 | - _next_session_pathname, |
4839 | - os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_BINARY, 0o644) |
4840 | - except IOError as exc: |
4841 | - if exc.errno == errno.EEXISTS: |
4842 | - raise LockedStorageError() |
4843 | - else: |
4844 | - raise |
4845 | - logger.debug( |
4846 | - _("Opened next session file %s as descriptor %d"), |
4847 | - _next_session_pathname, next_session_fd) |
4848 | - try: |
4849 | - # Write session data to disk |
4850 | - # |
4851 | - # I cannot find conclusive evidence but it seems that |
4852 | - # os.write() handles partial writes internally. In case we do |
4853 | - # get a partial write _or_ we run out of disk space, raise an |
4854 | - # explicit IOError. |
4855 | - num_written = os.write(next_session_fd, data) |
4856 | - logger.debug(ngettext( |
4857 | - "Wrote %d byte of data to descriptor %d", |
4858 | - "Wrote %d bytes of data to descriptor %d", |
4859 | - num_written), num_written, next_session_fd) |
4860 | - if num_written != len(data): |
4861 | - raise IOError(_("partial write?")) |
4862 | - except Exception as exc: |
4863 | - logger.warning(_("Unable to complete write: %s"), exc) |
4864 | - # If anything goes wrong we should unlink the next file. |
4865 | - # TRANSLATORS: unlinking as in deleting a file |
4866 | - logger.warning(_("Unlinking %r: %r"), _next_session_pathname, exc) |
4867 | - os.unlink(_next_session_pathname) |
4868 | - else: |
4869 | - # If the write was successful we must flush kernel buffers. |
4870 | - # |
4871 | - # We want to be sure this data is really on disk by now as we |
4872 | - # may crash the machine soon after this method exits. |
4873 | - logger.debug( |
4874 | - # TRANSLATORS: please don't translate fsync() |
4875 | - _("Calling fsync() on descriptor %d"), next_session_fd) |
4876 | - try: |
4877 | - os.fsync(next_session_fd) |
4878 | - except OSError as exc: |
4879 | - logger.warning(_("Cannot synchronize file %r: %s"), |
4880 | - _next_session_pathname, exc) |
4881 | - finally: |
4882 | - # Close the new session file |
4883 | - logger.debug(_("Closing descriptor %d"), next_session_fd) |
4884 | - os.close(next_session_fd) |
4885 | - # Rename FILE_NEXT over FILE. |
4886 | - logger.debug(_("Renaming %r to %r"), |
4887 | - _next_session_pathname, _session_pathname) |
4888 | - try: |
4889 | - os.replace(_next_session_pathname, _session_pathname) |
4890 | - except Exception as exc: |
4891 | - # Same as above, if we fail we need to unlink the next file |
4892 | - # otherwise any other attempts will not be able to open() it |
4893 | - # with O_EXCL flag. |
4894 | - logger.warning( |
4895 | - _("Unable to rename/overwrite %r to %r: %r"), |
4896 | - _next_session_pathname, _session_pathname, exc) |
4897 | - # TRANSLATORS: unlinking as in deleting a file |
4898 | - logger.warning(_("Unlinking %r"), _next_session_pathname) |
4899 | - os.unlink(_next_session_pathname) |
4900 | - |
4901 | - def _save_checkpoint_unix_py32(self, data): |
4902 | - # NOTE: this is like _save_checkpoint_py33 but without all the |
4903 | - # *at() functions (openat, renameat) |
4904 | - # |
4905 | - # Since we cannot use those functions there is an implicit race |
4906 | - # condition on all open() calls with another process that renames |
4907 | - # any of the directories that are part of the opened path. |
4908 | - # |
4909 | - # I don't think we can really do anything about this in userspace |
4910 | - # so this, python 3.2 specific version, just does the best effort |
4911 | - # implementation. Some of the comments were redacted but |
4912 | - # but keep in mind that the rename race is always there. |
4913 | - if not isinstance(data, bytes): |
4914 | - raise TypeError("data must be bytes") |
4915 | - logger.debug(ngettext( |
4916 | - "Saving %d byte of data (%s)", |
4917 | - "Saving %d bytes of data (%s)", |
4918 | - len(data)), len(data), "UNIX, python 3.2 or older") |
4919 | - # Helper pathnames, needed because we don't have *at functions |
4920 | - _next_session_pathname = os.path.join( |
4921 | - self._location, self._SESSION_FILE_NEXT) |
4922 | - _session_pathname = os.path.join(self._location, self._SESSION_FILE) |
4923 | - # Open the location directory, we need to fsync that later |
4924 | - # XXX: this may fail, maybe we should keep the fd open all the time? |
4925 | - location_fd = os.open(self._location, os.O_DIRECTORY) |
4926 | - logger.debug( |
4927 | - _("Opened %r as descriptor %d"), self._location, location_fd) |
4928 | - try: |
4929 | - # Open the "next" file in the location_directory |
4930 | - # |
4931 | - # Use "write" + "create" + "exclusive" flags so that no race |
4932 | - # condition is possible. |
4933 | - # |
4934 | - # This will never return -1, it throws IOError when anything is |
4935 | - # wrong. The caller has to catch this. |
4936 | - # |
4937 | - # As a special exception, this code handles EEXISTS and converts |
4938 | - # that to LockedStorageError that can be especially handled by |
4939 | - # some layer above. |
4940 | - try: |
4941 | - next_session_fd = os.open( |
4942 | - _next_session_pathname, |
4943 | - os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o644) |
4944 | - except IOError as exc: |
4945 | - if exc.errno == errno.EEXISTS: |
4946 | - raise LockedStorageError() |
4947 | - else: |
4948 | - raise |
4949 | - logger.debug( |
4950 | - _("Opened next session file %s as descriptor %d"), |
4951 | - _next_session_pathname, next_session_fd) |
4952 | - try: |
4953 | - # Write session data to disk |
4954 | - # |
4955 | - # I cannot find conclusive evidence but it seems that |
4956 | - # os.write() handles partial writes internally. In case we do |
4957 | - # get a partial write _or_ we run out of disk space, raise an |
4958 | - # explicit IOError. |
4959 | - num_written = os.write(next_session_fd, data) |
4960 | - logger.debug(ngettext( |
4961 | - "Wrote %d byte of data to descriptor %d", |
4962 | - "Wrote %d bytes of data to descriptor %d", |
4963 | - num_written), num_written, next_session_fd) |
4964 | - if num_written != len(data): |
4965 | - raise IOError(_("partial write?")) |
4966 | - except Exception as exc: |
4967 | - logger.warning(_("Unable to complete write: %r"), exc) |
4968 | - # If anything goes wrong we should unlink the next file. |
4969 | - # TRANSLATORS: unlinking as in deleting a file |
4970 | - logger.warning(_("Unlinking %r"), _next_session_pathname) |
4971 | - os.unlink(_next_session_pathname) |
4972 | - else: |
4973 | - # If the write was successful we must flush kernel buffers. |
4974 | - # |
4975 | - # We want to be sure this data is really on disk by now as we |
4976 | - # may crash the machine soon after this method exits. |
4977 | - logger.debug( |
4978 | - # TRANSLATORS: please don't translate fsync() |
4979 | - _("Calling fsync() on descriptor %d"), next_session_fd) |
4980 | - try: |
4981 | - os.fsync(next_session_fd) |
4982 | - except OSError as exc: |
4983 | - logger.warning(_("Cannot synchronize file %r: %s"), |
4984 | - _next_session_pathname, exc) |
4985 | - finally: |
4986 | - # Close the new session file |
4987 | - logger.debug(_("Closing descriptor %d"), next_session_fd) |
4988 | - os.close(next_session_fd) |
4989 | - # Rename FILE_NEXT over FILE. |
4990 | - logger.debug(_("Renaming %r to %r"), |
4991 | - _next_session_pathname, _session_pathname) |
4992 | - try: |
4993 | - os.rename(_next_session_pathname, _session_pathname) |
4994 | - except Exception as exc: |
4995 | - # Same as above, if we fail we need to unlink the next file |
4996 | - # otherwise any other attempts will not be able to open() it |
4997 | - # with O_EXCL flag. |
4998 | - logger.warning( |
4999 | - _("Unable to rename/overwrite %r to %r: %r"), |
5000 | - _next_session_pathname, _session_pathname, exc) |
Great branch. Also literally :)
Few observations:
Saving storage now uses the py33 method that uses python API for 'saving- at-location' . This is necessary for corner cases when saving. This is the only thing that makes checkbox not run on OSX at the moment.
Lazy modules were not actually used properly, as many of the modules are imported at checkbox start-up, so their laziness was not actually employed.
Overall, awesome stuff covered in awesomesauce. Great christmas present!
Huge +1!