Merge lp:~allenap/maas/parallel-tests-reliability into lp:~maas-committers/maas/trunk
- parallel-tests-reliability
- Merge into trunk
Status: | Merged |
---|---|
Approved by: | Gavin Panella |
Approved revision: | no longer in the source branch. |
Merged at revision: | 5664 |
Proposed branch: | lp:~allenap/maas/parallel-tests-reliability |
Merge into: | lp:~maas-committers/maas/trunk |
Diff against target: |
694 lines (+157/-94) 21 files modified
Makefile (+2/-7) media/README (+4/-4) required-packages/forbidden (+3/-0) src/maasserver/api/tests/test_discoveries.py (+5/-1) src/maasserver/djangosettings/development.py (+3/-1) src/maasserver/regiondservices/tests/test_ntp.py (+0/-6) src/maasserver/testing/resources.py (+6/-1) src/maasserver/utils/tests/test_dblocks.py (+30/-15) src/maasserver/utils/tests/test_dbtasks.py (+1/-1) src/maasserver/websockets/handlers/tests/test_subnet.py (+11/-7) src/maastesting/fixtures.py (+21/-7) src/maastesting/testcase.py (+8/-1) src/maastesting/tests/test_conflict_markers.py (+20/-17) src/maastesting/tests/test_fixtures.py (+11/-0) src/metadataserver/models/scriptset.py (+4/-0) src/provisioningserver/dhcp/__init__.py (+5/-5) src/provisioningserver/rpc/tests/test_clusterservice.py (+2/-0) src/provisioningserver/tests/test_config.py (+4/-2) src/provisioningserver/utils/fs.py (+5/-4) src/provisioningserver/utils/tests/test_fs.py (+4/-3) src/provisioningserver/utils/tests/test_utils.py (+8/-12) |
To merge this branch: | bzr merge lp:~allenap/maas/parallel-tests-reliability |
Related bugs: |
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Blake Rouse (community) | Approve | ||
Review via email: mp+315553@code.launchpad.net |
Commit message
Make bin/test.parallel reliable.
Description of the change
I have run bin/test.parallel 10 times with only one failure. This turned out to be a broken test; see the fix in https:/
I haven't swapped bin/test.parallel into the `make test` slot because it does not yet produce XUnit/JUnit output which Jenkins consumes. I also would like to get feedback from other team members to see if it is working reliably for them in different environments to my own.
Gavin Panella (allenap) wrote : | # |
Thanks. I fixed a couple of problems you encountered when running bin/test.parallel.
I'm not sure how to fix the "No address found for host localhost" problem you hit (http://
I would like to get test.js working again, locally, but there's no obvious reason why it breaks.
Blake Rouse (blake-rouse) wrote : | # |
Well test.js works locally for me. It also must work on the lander as its a requirement for branches to land. As long as this branch doesn't remove that requirement I am good with landing it, once that test failure I am getting is fixed.
Gavin Panella (allenap) wrote : | # |
Re-enabled test.js, and test failure fixed. Thanks!
Preview Diff
1 | === modified file 'Makefile' |
2 | --- Makefile 2017-01-20 11:30:38 +0000 |
3 | +++ Makefile 2017-01-25 16:51:52 +0000 |
4 | @@ -90,10 +90,6 @@ |
5 | echo '/src/**/*.pyc' >> $@ |
6 | echo '/etc/**/*.pyc' >> $@ |
7 | |
8 | -run/etc/ntp.conf: templates/ntp.conf |
9 | - @mkdir -p $(@D) |
10 | - @cp templates/ntp.conf $@ |
11 | - |
12 | configure-buildout: |
13 | utilities/configure-buildout |
14 | |
15 | @@ -163,8 +159,7 @@ |
16 | @touch --no-create $@ |
17 | |
18 | bin/test.rack: \ |
19 | - bin/buildout buildout.cfg versions.cfg setup.py bin/maas-rack \ |
20 | - run/etc/ntp.conf |
21 | + bin/buildout buildout.cfg versions.cfg setup.py bin/maas-rack |
22 | $(buildout) install rack-test |
23 | @touch --no-create $@ |
24 | |
25 | @@ -397,7 +392,7 @@ |
26 | find . -type d -name '__pycache__' -print0 | xargs -r0 $(RM) -r |
27 | find . -type f -name '*~' -print0 | xargs -r0 $(RM) |
28 | find . -type f -name dropin.cache -print0 | xargs -r0 $(RM) |
29 | - $(RM) -r media/demo/* media/development |
30 | + $(RM) -r media/demo/* media/development media/development.* |
31 | $(RM) $(js_enums) $(js_enums).tmp |
32 | $(RM) src/maasserver/data/templates.py |
33 | $(RM) *.log |
34 | |
35 | === modified file 'media/README' |
36 | --- media/README 2016-12-07 12:46:14 +0000 |
37 | +++ media/README 2017-01-25 16:51:52 +0000 |
38 | @@ -1,5 +1,5 @@ |
39 | This folder contains somewhat ephemeral things: subfolders serve as |
40 | -MEDIA_ROOT for maasserver.djangosettings.demo and .development |
41 | -environments. The media/demo directory should always exist and not be |
42 | -deleted, though its contents can be. The media/development directory |
43 | -should be created and destroyed by tests, as needed. |
44 | +MEDIA_ROOT for `demo` and `development` environments. The media/demo |
45 | +directory should always exist and not be deleted, though its contents |
46 | +can be. The media/development.$$ (where $$ is the PID of the running |
47 | +process) directories are created and destroyed by tests, as needed. |
48 | |
49 | === modified file 'required-packages/forbidden' |
50 | --- required-packages/forbidden 2014-09-24 13:05:22 +0000 |
51 | +++ required-packages/forbidden 2017-01-25 16:51:52 +0000 |
52 | @@ -0,0 +1,3 @@ |
53 | +python3-testresources |
54 | +python3-testscenarios |
55 | +python3-testtools |
56 | |
57 | === renamed file 'templates/ntp.conf' => 'run/etc/ntp.conf' |
58 | === modified file 'src/maasserver/api/tests/test_discoveries.py' |
59 | --- src/maasserver/api/tests/test_discoveries.py 2016-12-12 14:17:23 +0000 |
60 | +++ src/maasserver/api/tests/test_discoveries.py 2017-01-25 16:51:52 +0000 |
61 | @@ -9,6 +9,7 @@ |
62 | import http.client |
63 | import json |
64 | import random |
65 | +from unittest.mock import ANY |
66 | |
67 | from django.conf import settings |
68 | from django.core.urlresolvers import reverse |
69 | @@ -519,8 +520,11 @@ |
70 | scan_all_rack_networks(cidrs=cidrs) |
71 | self.assertThat( |
72 | self.call_racks_sync_mock, MockCalledOnceWith( |
73 | - cluster.ScanNetworks, controllers=self.started, |
74 | + cluster.ScanNetworks, controllers=ANY, |
75 | kwargs={'cidrs': cidrs})) |
76 | + # Check `controllers` separately because its order may vary. |
77 | + controllers = self.call_racks_sync_mock.call_args[1]["controllers"] |
78 | + self.assertItemsEqual(self.started, controllers) |
79 | |
80 | def test__calls_racks_synchronously_with_force_ping(self): |
81 | scan_all_rack_networks(ping=True) |
82 | |
83 | === modified file 'src/maasserver/djangosettings/development.py' |
84 | --- src/maasserver/djangosettings/development.py 2017-01-20 08:22:34 +0000 |
85 | +++ src/maasserver/djangosettings/development.py 2017-01-25 16:51:52 +0000 |
86 | @@ -63,7 +63,9 @@ |
87 | |
88 | # Absolute filesystem path to the directory that will hold user-uploaded files. |
89 | # Example: "/home/media/media.lawrence.com/media/" |
90 | -MEDIA_ROOT = abspath("media/development") |
91 | +# To allow tests to run in parallel the current process's PID is part of this |
92 | +# path. See MediaRootFixture for details. |
93 | +MEDIA_ROOT = abspath("media/development.%d" % os.getpid()) |
94 | |
95 | INTERNAL_IPS = ('127.0.0.1', '::1') |
96 | |
97 | |
98 | === modified file 'src/maasserver/regiondservices/tests/test_ntp.py' |
99 | --- src/maasserver/regiondservices/tests/test_ntp.py 2016-12-12 14:17:23 +0000 |
100 | +++ src/maasserver/regiondservices/tests/test_ntp.py 2017-01-25 16:51:52 +0000 |
101 | @@ -5,8 +5,6 @@ |
102 | |
103 | __all__ = [] |
104 | |
105 | -from os.path import join |
106 | - |
107 | from crochet import wait_for |
108 | from maasserver.models.config import Config |
109 | from maasserver.regiondservices import ntp |
110 | @@ -123,10 +121,6 @@ |
111 | # Ensure that we never actually execute against systemd. |
112 | self.patch_autospec(service_monitor, "restartService") |
113 | |
114 | - maasroot = self.useFixture(MAASRootFixture()).path |
115 | - with open(join(maasroot, "etc", "ntp.conf"), "w") as ntp_conf: |
116 | - ntp_conf.write("# Placeholder NTP configuration file.\n") |
117 | - |
118 | with TwistedLoggerFixture() as logger: |
119 | yield service._tryUpdate() |
120 | |
121 | |
122 | === modified file 'src/maasserver/testing/resources.py' |
123 | --- src/maasserver/testing/resources.py 2017-01-20 08:21:22 +0000 |
124 | +++ src/maasserver/testing/resources.py 2017-01-25 16:51:52 +0000 |
125 | @@ -217,6 +217,7 @@ |
126 | |
127 | def make(self, dependencies): |
128 | databases = dependencies["templates"] |
129 | + clusterlock = databases.cluster.lock |
130 | with databases.cluster.connect() as conn: |
131 | with conn.cursor() as cursor: |
132 | for database in databases: |
133 | @@ -226,7 +227,11 @@ |
134 | stmt = ( |
135 | "CREATE DATABASE %s WITH TEMPLATE %s" |
136 | % (dbname, template)) |
137 | - cursor.execute(stmt) |
138 | + # Create the database with a shared lock to the cluster to |
139 | + # avoid racing a DjangoPristineDatabaseManager.make in a |
140 | + # concurrently running test process. |
141 | + with clusterlock.shared: |
142 | + cursor.execute(stmt) |
143 | debug( |
144 | "Created {dbname}; statement: {stmt}", |
145 | dbname=dbname, stmt=stmt) |
146 | |
147 | === modified file 'src/maasserver/utils/tests/test_dblocks.py' |
148 | --- src/maasserver/utils/tests/test_dblocks.py 2015-12-01 18:12:59 +0000 |
149 | +++ src/maasserver/utils/tests/test_dblocks.py 2017-01-25 16:51:52 +0000 |
150 | @@ -9,7 +9,7 @@ |
151 | closing, |
152 | contextmanager, |
153 | ) |
154 | -from random import randint |
155 | +from random import randrange |
156 | import sys |
157 | |
158 | from django.db import ( |
159 | @@ -25,18 +25,24 @@ |
160 | from maasserver.utils import dblocks |
161 | from testtools.matchers import Equals |
162 | |
163 | +# Use "high" objid numbers to avoid conflicts with predeclared locks. |
164 | +objid_min = 2 << 10 |
165 | +objid_max = 2 << 16 |
166 | + |
167 | |
168 | def get_locks(): |
169 | - """Return the set of locks held.""" |
170 | - stmt = "SELECT objid FROM pg_locks WHERE classid = %s" |
171 | + """Return the set of locks held between `objid_min` and `objid_max`.""" |
172 | with closing(connection.cursor()) as cursor: |
173 | - cursor.execute(stmt, [dblocks.classid]) |
174 | + cursor.execute( |
175 | + "SELECT objid FROM pg_locks " |
176 | + "WHERE classid = %s AND objid >= %s AND objid < %s", |
177 | + [dblocks.classid, objid_min, objid_max]) |
178 | return {result[0] for result in cursor.fetchall()} |
179 | |
180 | |
181 | def random_objid(): |
182 | """Return a 'high' objid that's won't coincide with predeclared locks.""" |
183 | - return randint(2 << 10, 2 << 16) |
184 | + return randrange(objid_min, objid_max) |
185 | |
186 | |
187 | @transaction.atomic |
188 | @@ -99,10 +105,15 @@ |
189 | objid = random_objid() |
190 | lock = self.make_lock(objid) |
191 | |
192 | - locks_held_before = get_locks() |
193 | - with lock: |
194 | - locks_held = get_locks() |
195 | - locks_held_after = get_locks() |
196 | + # Take an exclusive lock on the database cluster to prevent this |
197 | + # section from running concurrently with any other thusly delimited |
198 | + # section. Note that `self.databases` is the test resource for the |
199 | + # database(s), which then has a reference to the cluster resource. |
200 | + with self.databases.cluster.lock.exclusive: |
201 | + locks_held_before = get_locks() |
202 | + with lock: |
203 | + locks_held = get_locks() |
204 | + locks_held_after = get_locks() |
205 | |
206 | locks_obtained = locks_held - locks_held_before |
207 | self.assertEqual({objid}, locks_obtained) |
208 | @@ -266,12 +277,16 @@ |
209 | objid = random_objid() |
210 | lock = self.make_lock(objid) |
211 | |
212 | - with transaction.atomic(): |
213 | - locks_held_before = get_locks() |
214 | - with lock: |
215 | - locks_held = get_locks() |
216 | - locks_held_after = get_locks() |
217 | - locks_held_after_txn = get_locks() |
218 | + # Take an exclusive lock on the database cluster to prevent this |
219 | + # section from running concurrently with any other thusly delimited |
220 | + # section. |
221 | + with self.databases.cluster.lock.exclusive: |
222 | + with transaction.atomic(): |
223 | + locks_held_before = get_locks() |
224 | + with lock: |
225 | + locks_held = get_locks() |
226 | + locks_held_after = get_locks() |
227 | + locks_held_after_txn = get_locks() |
228 | |
229 | locks_obtained = locks_held - locks_held_before |
230 | self.assertEqual({objid}, locks_obtained) |
231 | |
232 | === modified file 'src/maasserver/utils/tests/test_dbtasks.py' |
233 | --- src/maasserver/utils/tests/test_dbtasks.py 2016-06-22 17:03:02 +0000 |
234 | +++ src/maasserver/utils/tests/test_dbtasks.py 2017-01-25 16:51:52 +0000 |
235 | @@ -291,7 +291,7 @@ |
236 | |
237 | self.assertDocTestMatches( |
238 | """\ |
239 | - Unhandled failure in database task. |
240 | + ...Unhandled failure in database task. |
241 | Traceback (most recent call last): |
242 | ... |
243 | builtins.ZeroDivisionError: ... |
244 | |
245 | === modified file 'src/maasserver/websockets/handlers/tests/test_subnet.py' |
246 | --- src/maasserver/websockets/handlers/tests/test_subnet.py 2016-12-20 06:52:13 +0000 |
247 | +++ src/maasserver/websockets/handlers/tests/test_subnet.py 2017-01-25 16:51:52 +0000 |
248 | @@ -5,6 +5,7 @@ |
249 | |
250 | __all__ = [] |
251 | |
252 | +import re |
253 | from unittest.mock import sentinel |
254 | |
255 | from fixtures import FakeLogger |
256 | @@ -15,14 +16,14 @@ |
257 | from maasserver.utils.orm import reload_object |
258 | from maasserver.websockets.base import dehydrate_datetime |
259 | from maasserver.websockets.handlers.subnet import SubnetHandler |
260 | -from maastesting.matchers import ( |
261 | - DocTestMatches, |
262 | - MockCalledOnceWith, |
263 | -) |
264 | +from maastesting.matchers import MockCalledOnceWith |
265 | from netaddr import IPNetwork |
266 | from provisioningserver.utils.network import IPRangeStatistics |
267 | from testtools import ExpectedException |
268 | -from testtools.matchers import Equals |
269 | +from testtools.matchers import ( |
270 | + Equals, |
271 | + MatchesRegex, |
272 | +) |
273 | |
274 | |
275 | class TestSubnetHandler(MAASServerTestCase): |
276 | @@ -160,8 +161,11 @@ |
277 | handler.scan({ |
278 | "id": subnet.id, |
279 | }) |
280 | - self.assertThat(logger.output, DocTestMatches( |
281 | - "User...%s...scan...%s" % (user.username, cidr))) |
282 | + # Use MatchesRegex here rather than DocTestMatches because usernames |
283 | + # can contain characters that confuse DocTestMatches (e.g. periods). |
284 | + self.assertThat(logger.output, MatchesRegex( |
285 | + "User '%s' initiated a neighbour discovery scan against subnet: %s" |
286 | + % (re.escape(user.username), re.escape(str(cidr))))) |
287 | |
288 | def test__scan_ipv6_fails(self): |
289 | user = factory.make_admin() |
290 | |
291 | === modified file 'src/maastesting/fixtures.py' |
292 | --- src/maastesting/fixtures.py 2016-08-09 00:14:06 +0000 |
293 | +++ src/maastesting/fixtures.py 2017-01-25 16:51:52 +0000 |
294 | @@ -22,10 +22,9 @@ |
295 | ) |
296 | import logging |
297 | import os |
298 | +from pathlib import Path |
299 | from subprocess import ( |
300 | CalledProcessError, |
301 | - check_output, |
302 | - DEVNULL, |
303 | PIPE, |
304 | Popen, |
305 | ) |
306 | @@ -404,7 +403,7 @@ |
307 | |
308 | |
309 | class MAASRootFixture(fixtures.Fixture): |
310 | - """Copy an existing `MAAS_ROOT` to a new temporary location. |
311 | + """Create a new, pristine, `MAAS_ROOT` in a temporary location. |
312 | |
313 | Also updates `MAAS_ROOT` in the environment to point to this new location. |
314 | """ |
315 | @@ -416,10 +415,25 @@ |
316 | raise NotADirectoryError("MAAS_ROOT is not defined.") |
317 | elif os.path.isdir(maasroot): |
318 | self.path = self.useFixture(TempDirectory()).join("run") |
319 | - # Neither shutil.copytree nor /bin/cp deal with relative symlinks, |
320 | - # but /bin/cp is much faster at copying dereferenced files. |
321 | - command = "/bin/cp", "-r", "--dereference", maasroot, self.path |
322 | - check_output(command, stdin=DEVNULL, stderr=PIPE) |
323 | + # Work only in $MAAS_ROOT/run; reference the old $MAAS_ROOT. |
324 | + etc = Path(self.path).joinpath("etc") |
325 | + src = Path(maasroot) |
326 | + # Create and populate $MAAS_ROOT/run/etc/{ntp,ntp.conf}. The |
327 | + # `.keep` file is not strictly necessary, but it's created for |
328 | + # consistency with the source tree's `run` directory. |
329 | + ntp = etc.joinpath("ntp") |
330 | + ntp.mkdir(parents=True) |
331 | + ntp.joinpath(".keep").touch() |
332 | + ntp_conf = etc.joinpath("ntp.conf") |
333 | + ntp_conf.write_bytes(src.joinpath("etc", "ntp.conf").read_bytes()) |
334 | + # Create and populate $MAAS_ROOT/run/etc/maas. |
335 | + maas = etc.joinpath("maas") |
336 | + maas.mkdir(parents=True) |
337 | + maas.joinpath("drivers.yaml").symlink_to( |
338 | + src.joinpath("etc", "maas", "drivers.yaml").resolve()) |
339 | + maas.joinpath("templates").symlink_to( |
340 | + src.joinpath("etc", "maas", "templates").resolve()) |
341 | + # Update the environment. |
342 | self.useFixture(EnvironmentVariable("MAAS_ROOT", self.path)) |
343 | else: |
344 | raise NotADirectoryError( |
345 | |
346 | === modified file 'src/maastesting/testcase.py' |
347 | --- src/maastesting/testcase.py 2017-01-11 09:19:28 +0000 |
348 | +++ src/maastesting/testcase.py 2017-01-25 16:51:52 +0000 |
349 | @@ -24,7 +24,10 @@ |
350 | import crochet |
351 | from maastesting.crochet import EventualResultCatchingMixin |
352 | from maastesting.factory import factory |
353 | -from maastesting.fixtures import TempDirectory |
354 | +from maastesting.fixtures import ( |
355 | + MAASRootFixture, |
356 | + TempDirectory, |
357 | +) |
358 | from maastesting.matchers import DocTestMatches |
359 | from maastesting.runtest import ( |
360 | MAASCrochetRunTest, |
361 | @@ -154,6 +157,10 @@ |
362 | return MAASRunTest |
363 | |
364 | def setUp(self): |
365 | + # Every test gets a pristine MAAS_ROOT, when it is defined. |
366 | + if "MAAS_ROOT" in os.environ: |
367 | + self.useFixture(MAASRootFixture()) |
368 | + |
369 | # Capture Twisted logs and add them as a test detail. |
370 | twistedLog = self.useFixture(TwistedLoggerFixture()) |
371 | self.addDetail("Twisted logs", twistedLog.getContent()) |
372 | |
373 | === modified file 'src/maastesting/tests/test_conflict_markers.py' |
374 | --- src/maastesting/tests/test_conflict_markers.py 2015-12-01 18:12:59 +0000 |
375 | +++ src/maastesting/tests/test_conflict_markers.py 2017-01-25 16:51:52 +0000 |
376 | @@ -5,12 +5,16 @@ |
377 | |
378 | __all__ = [] |
379 | |
380 | -from pipes import quote |
381 | +from os.path import ( |
382 | + isdir, |
383 | + join, |
384 | +) |
385 | from subprocess import ( |
386 | PIPE, |
387 | Popen, |
388 | STDOUT, |
389 | ) |
390 | +from unittest import skipIf |
391 | |
392 | from maastesting import root |
393 | from maastesting.testcase import MAASTestCase |
394 | @@ -19,25 +23,24 @@ |
395 | UTF8_TEXT, |
396 | ) |
397 | |
398 | -# Do not use '=======' as a conflict marker since it's |
399 | -# used in docstrings. |
400 | -# Express the conflict markers so that this very file won't contain |
401 | -# them. |
402 | -CONFLICT_MARKERS = "<" * 7, ">" * 7 |
403 | - |
404 | |
405 | class TestConflictMarkers(MAASTestCase): |
406 | + """Search for merge conflict markers if this is a branch.""" |
407 | |
408 | - def execute(self, *command): |
409 | - process = Popen(command, stdout=PIPE, stderr=STDOUT, cwd=root) |
410 | + @skipIf(not isdir(join(root, ".bzr")), "Not a branch.") |
411 | + def test_no_conflict_markers(self): |
412 | + # Do not search for '=======' as a conflict marker since it's used in |
413 | + # docstrings, search for angle brackets instead. Express the conflict |
414 | + # markers as a regular expression so that this very file won't match. |
415 | + command = ( |
416 | + "bzr ls --kind=file --recursive --versioned --null | " |
417 | + "xargs -r0 egrep -I '[<]{7}|[>]{7}' -C 3") |
418 | + process = Popen( |
419 | + command, shell=True, stdout=PIPE, stderr=STDOUT, cwd=root) |
420 | output, _ = process.communicate() |
421 | if len(output) != 0: |
422 | - name = "stdout/err from `%s`" % " ".join(map(quote, command)) |
423 | + name = "stdout/err from `%s`" % command |
424 | self.addDetail(name, Content(UTF8_TEXT, lambda: [output])) |
425 | - self.assertEqual('', output, "Conflict markers present!") |
426 | - self.assertEqual(1, process.wait(), "(return code is not one)") |
427 | - |
428 | - def test_no_conflict_markers(self): |
429 | - command = ["egrep", "-rI", "--exclude=*~", "--exclude-dir=include"] |
430 | - command.append("|".join(CONFLICT_MARKERS)) |
431 | - self.execute(*command) |
432 | + self.fail("Conflict markers present!") |
433 | + # Don't check the process's exit code because xargs muddles things. |
434 | + # Checking the output should suffice. |
435 | |
436 | === modified file 'src/maastesting/tests/test_fixtures.py' |
437 | --- src/maastesting/tests/test_fixtures.py 2016-08-09 00:14:06 +0000 |
438 | +++ src/maastesting/tests/test_fixtures.py 2017-01-25 16:51:52 +0000 |
439 | @@ -266,6 +266,17 @@ |
440 | self.assertThat(fixture.path, Not(SamePath(self.maasroot))) |
441 | files_expected = set(listdirs(self.maasroot)) |
442 | files_observed = set(listdirs(fixture.path)) |
443 | + |
444 | + # Some files, like regiond.conf and rackd.conf, may be created by |
445 | + # concurrently running test suites. In particular, regiond.conf is |
446 | + # created as a side-effect of loading Django's settings which |
447 | + # takes place before tests are running. We ignore these files. |
448 | + files_missing = files_expected - files_observed |
449 | + if "etc/maas/regiond.conf" in files_missing: |
450 | + files_expected.remove("etc/maas/regiond.conf") |
451 | + if "etc/maas/rackd.conf" in files_missing: |
452 | + files_expected.remove("etc/maas/rackd.conf") |
453 | + |
454 | self.assertThat(files_observed, Equals(files_expected)) |
455 | self.assertThat(fixture.path, Not(PathExists())) |
456 | |
457 | |
458 | === modified file 'src/metadataserver/models/scriptset.py' |
459 | --- src/metadataserver/models/scriptset.py 2017-01-21 00:49:34 +0000 |
460 | +++ src/metadataserver/models/scriptset.py 2017-01-25 16:51:52 +0000 |
461 | @@ -16,6 +16,7 @@ |
462 | ) |
463 | from maasserver.models.cleansave import CleanSave |
464 | from maasserver.preseed import CURTIN_INSTALL_LOG |
465 | +from metadataserver import DefaultMeta |
466 | from metadataserver.enum import ( |
467 | RESULT_TYPE, |
468 | RESULT_TYPE_CHOICES, |
469 | @@ -91,6 +92,9 @@ |
470 | |
471 | class ScriptSet(CleanSave, Model): |
472 | |
473 | + class Meta(DefaultMeta): |
474 | + """Needed for South to recognize this model.""" |
475 | + |
476 | objects = ScriptSetManager() |
477 | |
478 | last_ping = DateTimeField(blank=True, null=True) |
479 | |
480 | === modified file 'src/provisioningserver/dhcp/__init__.py' |
481 | --- src/provisioningserver/dhcp/__init__.py 2016-04-12 22:22:23 +0000 |
482 | +++ src/provisioningserver/dhcp/__init__.py 2017-01-25 16:51:52 +0000 |
483 | @@ -13,7 +13,7 @@ |
484 | abstractproperty, |
485 | ) |
486 | |
487 | -from provisioningserver.path import get_path |
488 | +from provisioningserver.path import get_tentative_path |
489 | |
490 | # Location of the DHCPv4 configuration file. |
491 | DHCPv4_CONFIG_FILE = '/var/lib/maas/dhcpd.conf' |
492 | @@ -65,8 +65,8 @@ |
493 | |
494 | descriptive_name = "DHCPv4" |
495 | template_basename = 'dhcpd.conf.template' |
496 | - interfaces_filename = get_path(DHCPv4_INTERFACES_FILE) |
497 | - config_filename = get_path(DHCPv4_CONFIG_FILE) |
498 | + interfaces_filename = get_tentative_path(DHCPv4_INTERFACES_FILE) |
499 | + config_filename = get_tentative_path(DHCPv4_CONFIG_FILE) |
500 | dhcp_service = "dhcpd" |
501 | ipv6 = False |
502 | |
503 | @@ -79,7 +79,7 @@ |
504 | |
505 | descriptive_name = "DHCPv6" |
506 | template_basename = 'dhcpd6.conf.template' |
507 | - interfaces_filename = get_path(DHCPv6_INTERFACES_FILE) |
508 | - config_filename = get_path(DHCPv6_CONFIG_FILE) |
509 | + interfaces_filename = get_tentative_path(DHCPv6_INTERFACES_FILE) |
510 | + config_filename = get_tentative_path(DHCPv6_CONFIG_FILE) |
511 | dhcp_service = "dhcpd6" |
512 | ipv6 = True |
513 | |
514 | === modified file 'src/provisioningserver/rpc/tests/test_clusterservice.py' |
515 | --- src/provisioningserver/rpc/tests/test_clusterservice.py 2017-01-24 09:24:36 +0000 |
516 | +++ src/provisioningserver/rpc/tests/test_clusterservice.py 2017-01-25 16:51:52 +0000 |
517 | @@ -1229,6 +1229,7 @@ |
518 | # self.assertTrue(client.isSecure()) |
519 | |
520 | def test_authenticateRegion_accepts_matching_digests(self): |
521 | + set_shared_secret_on_filesystem(factory.make_bytes()) |
522 | client = self.make_running_client() |
523 | |
524 | def calculate_digest(_, message): |
525 | @@ -1243,6 +1244,7 @@ |
526 | self.assertTrue(extract_result(d)) |
527 | |
528 | def test_authenticateRegion_rejects_non_matching_digests(self): |
529 | + set_shared_secret_on_filesystem(factory.make_bytes()) |
530 | client = self.make_running_client() |
531 | |
532 | def calculate_digest(_, message): |
533 | |
534 | === modified file 'src/provisioningserver/tests/test_config.py' |
535 | --- src/provisioningserver/tests/test_config.py 2016-08-11 01:50:21 +0000 |
536 | +++ src/provisioningserver/tests/test_config.py 2017-01-25 16:51:52 +0000 |
537 | @@ -19,6 +19,7 @@ |
538 | from fixtures import EnvironmentVariableFixture |
539 | import formencode |
540 | import formencode.validators |
541 | +import maastesting |
542 | from maastesting.factory import factory |
543 | from maastesting.fixtures import ImportErrorFixture |
544 | from maastesting.matchers import ( |
545 | @@ -638,8 +639,9 @@ |
546 | self.assertEqual({"tftp_port": example_port}, config.store) |
547 | |
548 | def test_default_tftp_root(self): |
549 | - maas_root = os.getenv("MAAS_ROOT") |
550 | - self.assertIsNotNone(maas_root) |
551 | + # The default tftp_root is calculated relative to MAAS_ROOT at module |
552 | + # import time, so we need to recreate that value. |
553 | + maas_root = os.path.join(maastesting.root, "run") |
554 | config = ClusterConfiguration({}) |
555 | self.assertEqual( |
556 | os.path.join(maas_root, "var/lib/maas/boot-resources/current"), |
557 | |
558 | === modified file 'src/provisioningserver/utils/fs.py' |
559 | --- src/provisioningserver/utils/fs.py 2016-12-14 08:43:09 +0000 |
560 | +++ src/provisioningserver/utils/fs.py 2017-01-25 16:51:52 +0000 |
561 | @@ -45,6 +45,7 @@ |
562 | import threading |
563 | from time import sleep |
564 | |
565 | +from provisioningserver.path import get_path |
566 | from provisioningserver.utils import sudo |
567 | from provisioningserver.utils.shell import ExternalProcessError |
568 | from provisioningserver.utils.twisted import retries |
569 | @@ -485,7 +486,7 @@ |
570 | |
571 | |
572 | class RunLock(SystemLock): |
573 | - """Always create a lock file at ``/run/lock/maas@${path,modified}``. |
574 | + """Lock file at ``${MAAS_ROOT}/run/lock/maas@${path,modified}``. |
575 | |
576 | This implements an advisory file lock, by proxy, on the given file-system |
577 | path. This is especially useful if you do not have permissions to the |
578 | @@ -498,12 +499,12 @@ |
579 | def __init__(self, path): |
580 | abspath = FilePath(path).asTextMode().path.lstrip("/") |
581 | discriminator = abspath.replace(":", "::").replace("/", ":") |
582 | - lockpath = "/run/lock/maas@%s" % discriminator |
583 | + lockpath = get_path("run", "lock", "maas@%s" % discriminator) |
584 | super(RunLock, self).__init__(lockpath) |
585 | |
586 | |
587 | class NamedLock(SystemLock): |
588 | - """Always create a lock file at ``/run/lock/maas.${name}``. |
589 | + """Lock file at ``${MAAS_ROOT}/run/lock/maas.${name}``. |
590 | |
591 | This implements an advisory lock, by proxy, for an abstract name. The name |
592 | must be a string and can contain only numerical digits, hyphens, and ASCII |
593 | @@ -524,5 +525,5 @@ |
594 | "Lock name contains illegal characters: %s" |
595 | % "".join(sorted(illegal))) |
596 | else: |
597 | - lockpath = "/run/lock/maas:%s" % name |
598 | + lockpath = get_path("run", "lock", "maas:%s" % name) |
599 | super(NamedLock, self).__init__(lockpath) |
600 | |
601 | === modified file 'src/provisioningserver/utils/tests/test_fs.py' |
602 | --- src/provisioningserver/utils/tests/test_fs.py 2017-01-06 09:50:17 +0000 |
603 | +++ src/provisioningserver/utils/tests/test_fs.py 2017-01-25 16:51:52 +0000 |
604 | @@ -33,6 +33,7 @@ |
605 | from maastesting.testcase import MAASTestCase |
606 | from maastesting.utils import age_file |
607 | import provisioningserver.config |
608 | +from provisioningserver.path import get_tentative_path |
609 | from provisioningserver.utils.fs import ( |
610 | atomic_copy, |
611 | atomic_delete, |
612 | @@ -773,13 +774,13 @@ |
613 | |
614 | def test__string_path(self): |
615 | filename = '/foo/bar/123:456.txt' |
616 | - expected = '/run/lock/maas@foo:bar:123::456.txt' |
617 | + expected = get_tentative_path('/run/lock/maas@foo:bar:123::456.txt') |
618 | observed = RunLock(filename).path |
619 | self.assertEqual(expected, observed) |
620 | |
621 | def test__byte_path(self): |
622 | filename = b'/foo/bar/123:456.txt' |
623 | - expected = '/run/lock/maas@foo:bar:123::456.txt' |
624 | + expected = get_tentative_path('/run/lock/maas@foo:bar:123::456.txt') |
625 | observed = RunLock(filename).path |
626 | self.assertEqual(expected, observed) |
627 | |
628 | @@ -789,7 +790,7 @@ |
629 | |
630 | def test__string_name(self): |
631 | name = factory.make_name("lock") |
632 | - expected = '/run/lock/maas:' + name |
633 | + expected = get_tentative_path('/run/lock/maas:' + name) |
634 | observed = NamedLock(name).path |
635 | self.assertEqual(expected, observed) |
636 | |
637 | |
638 | === modified file 'src/provisioningserver/utils/tests/test_utils.py' |
639 | --- src/provisioningserver/utils/tests/test_utils.py 2016-09-22 02:53:33 +0000 |
640 | +++ src/provisioningserver/utils/tests/test_utils.py 2017-01-25 16:51:52 +0000 |
641 | @@ -13,7 +13,6 @@ |
642 | from unittest.mock import sentinel |
643 | |
644 | from fixtures import EnvironmentVariableFixture |
645 | -from maastesting import root |
646 | from maastesting.factory import factory |
647 | from maastesting.testcase import MAASTestCase |
648 | import provisioningserver |
649 | @@ -40,20 +39,17 @@ |
650 | ) |
651 | |
652 | |
653 | -def get_branch_dir(*path): |
654 | - """Locate a file or directory relative to ``MAAS_ROOT``. |
655 | - |
656 | - This function assumes that ``MAAS_ROOT`` has been set to a ``run`` |
657 | - subdirectory of this working-tree's root. |
658 | - """ |
659 | - return os.path.abspath(os.path.join(root, "run", *path)) |
660 | +def get_run_path(*path): |
661 | + """Locate a file or directory relative to ``MAAS_ROOT``.""" |
662 | + maas_root = os.environ["MAAS_ROOT"] |
663 | + return os.path.abspath(os.path.join(maas_root, *path)) |
664 | |
665 | |
666 | class TestLocateConfig(MAASTestCase): |
667 | """Tests for `locate_config`.""" |
668 | |
669 | def test_returns_branch_etc_maas(self): |
670 | - self.assertEqual(get_branch_dir('etc/maas'), locate_config()) |
671 | + self.assertEqual(get_run_path('etc/maas'), locate_config()) |
672 | self.assertThat(locate_config(), DirExists()) |
673 | |
674 | def test_defaults_to_global_etc_maas_if_variable_is_unset(self): |
675 | @@ -71,18 +67,18 @@ |
676 | def test_locates_config_file(self): |
677 | filename = factory.make_string() |
678 | self.assertEqual( |
679 | - get_branch_dir('etc/maas/', filename), |
680 | + get_run_path('etc/maas/', filename), |
681 | locate_config(filename)) |
682 | |
683 | def test_locates_full_path(self): |
684 | path = [factory.make_string() for counter in range(3)] |
685 | self.assertEqual( |
686 | - get_branch_dir('etc/maas/', *path), |
687 | + get_run_path('etc/maas/', *path), |
688 | locate_config(*path)) |
689 | |
690 | def test_normalizes_path(self): |
691 | self.assertEqual( |
692 | - get_branch_dir('etc/maas/bar/szot'), |
693 | + get_run_path('etc/maas/bar/szot'), |
694 | locate_config('foo/.././bar///szot')) |
695 | |
696 |
Looks good. Just a couple of questions.