Merge lp:~ltrager/maas/lp1569084 into lp:~maas-committers/maas/trunk

Proposed by Lee Trager
Status: Merged
Approved by: Lee Trager
Approved revision: no longer in the source branch.
Merged at revision: 4923
Proposed branch: lp:~ltrager/maas/lp1569084
Merge into: lp:~maas-committers/maas/trunk
Diff against target: 355 lines (+169/-66)
5 files modified
src/maasserver/api/tests/test_rackcontroller.py (+2/-1)
src/maasserver/clusterrpc/boot_images.py (+0/-17)
src/maasserver/clusterrpc/tests/test_boot_images.py (+1/-25)
src/maasserver/models/node.py (+31/-4)
src/maasserver/models/tests/test_node.py (+135/-19)
To merge this branch: bzr merge lp:~ltrager/maas/lp1569084
Reviewer Review Type Date Requested Status
Blake Rouse (community) Approve
Review via email: mp+291600@code.launchpad.net

Commit message

Returns whether the boot images are in sync

Description of the change

Returns whether the boot images are in sync, currently being synced, or out of sync. I figure this information out by looking at the timestamp from /var/lib/maas/boot-resources/snapshot-DATE. I then check if any new boot resource files have been created since the given timestamp.

I also moved is_import_boot_images_running_for onto the rack controller model like refresh and addchassis.

To post a comment you must log in.
Revision history for this message
Blake Rouse (blake-rouse) wrote :

This will not work. The way you are determining if the images have been synced is incorrect. Using the time from the snapshot is not correct. The updated time for the boot images is updated everytime the region checked for new images. If no new images are needed then nothing is done but the time is updated. That updated time will be after rack controller synced and the rack controller will not sync any images because it already has what it needs.

The correct way of checking is comparing the images result you get back from the rack controller with the data in boot resources. This is how it was done in <2.0 with cluster controllers and should be the same here.

See here on how to do it, most of this code is still around should be rather easy to connect to a rack controllers:

http://bazaar.launchpad.net/~maas-committers/maas/1.9/view/head:/src/maasserver/models/nodegroup.py#L310

review: Needs Fixing
Revision history for this message
Lee Trager (ltrager) wrote :

Thanks for the review and pointing me in the right direction. I forgot to look at what we did previously and just started coding. I've updated the MP to use BootResource.objects.boot_images_are_in_sync which is what was done in 1.9.

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Thanks for fixing this. Looks much better.

review: Approve
Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (32.8 KiB)

The attempt to merge lp:~ltrager/maas/lp1569084 into lp:maas failed. Below is the output from the failed tests.

Get:1 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Hit:2 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,761 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,543 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,362 kB]
Fetched 19.9 MB in 3s (5,378 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu2).
archdetect-deb is already the newest version (1.117ubuntu1).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
firefox is already the newest version (45.0.1+build1-0ubuntu1).
freeipmi-tools is already the newest version (1.4.11-1ubuntu1).
git is already the newest version (1:2.7.4-0...

Revision history for this message
Blake Rouse (blake-rouse) wrote :

Looks like you got a lint error.

Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (1.0 MiB)

The attempt to merge lp:~ltrager/maas/lp1569084 into lp:maas failed. Below is the output from the failed tests.

Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:2 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Hit:3 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:4 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial-backports InRelease
Get:5 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main Sources [865 kB]
Get:6 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe Sources [7,760 kB]
Get:7 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,183 kB]
Get:8 http://prodstack-zone-2.clouds.archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,543 kB]
Fetched 17.6 MB in 3s (5,305 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
    --no-install-recommends install apache2 archdetect-deb authbind bash bind9 bind9utils build-essential bzr-builddeb chromium-browser chromium-chromedriver curl daemontools debhelper dh-apport dh-systemd distro-info dnsutils firefox freeipmi-tools git gjs ipython isc-dhcp-common libjs-angularjs libjs-jquery libjs-jquery-hotkeys libjs-yui3-full libjs-yui3-min libpq-dev make nodejs-legacy npm postgresql pxelinux python3-all python3-apt python3-bson python3-convoy python3-coverage python3-crochet python3-cssselect python3-curtin python3-dev python3-distro-info python3-django python3-django-nose python3-django-piston3 python3-dnspython python3-docutils python3-formencode python3-hivex python3-httplib2 python3-jinja2 python3-jsonschema python3-lxml python3-mock python3-netaddr python3-netifaces python3-oauth python3-oauthlib python3-openssl python3-paramiko python3-petname python3-pexpect python3-psycopg2 python3-pyinotify python3-pyparsing python3-pyvmomi python3-requests python3-seamicroclient python3-setuptools python3-simplestreams python3-sphinx python3-tempita python3-twisted python3-txtftp python3-tz python3-yaml python3-zope.interface python-bson python-crochet python-django python-django-piston python-djorm-ext-pgarray python-formencode python-lxml python-netaddr python-netifaces python-pocket-lint python-psycopg2 python-tempita python-twisted python-yaml socat syslinux-common tgt ubuntu-cloudimage-keyring wget xvfb
Reading package lists...
Building dependency tree...
Reading state information...
apache2 is already the newest version (2.4.18-2ubuntu2).
archdetect-deb is already the newest version (1.117ubuntu1).
authbind is already the newest version (2.1.1+nmu1).
bash is already the newest version (4.3-14ubuntu1).
bind9 is already the newest version (1:9.10.3.dfsg.P4-8).
bind9utils is already the newest version (1:9.10.3.dfsg.P4-8).
build-essential is already the newest version (12.1ubuntu2).
curl is already the newest version (7.47.0-1ubuntu2).
debhelper is already the newest version (9.20160115ubuntu3).
distro-info is already the newest version (0.14build1).
dnsutils is already the newest version (1:9.10.3.dfsg.P4-8).
firefox is already the newest version (45.0.1+build1-0ubuntu1).
freeipmi-tools is al...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/maasserver/api/tests/test_rackcontroller.py'
2--- src/maasserver/api/tests/test_rackcontroller.py 2016-04-08 19:23:26 +0000
3+++ src/maasserver/api/tests/test_rackcontroller.py 2016-04-14 17:36:28 +0000
4@@ -78,7 +78,8 @@
5 http.client.OK, response.status_code,
6 explain_unexpected_response(http.client.OK, response))
7 self.assertItemsEqual(
8- ['connected', 'images'], json_load_bytes(response.content))
9+ ['connected', 'images', 'status'],
10+ json_load_bytes(response.content))
11
12
13 class TestRackControllersAPI(APITestCase):
14
15=== modified file 'src/maasserver/clusterrpc/boot_images.py'
16--- src/maasserver/clusterrpc/boot_images.py 2016-03-28 13:54:47 +0000
17+++ src/maasserver/clusterrpc/boot_images.py 2016-04-14 17:36:28 +0000
18@@ -10,7 +10,6 @@
19 "get_boot_images_for",
20 "get_common_available_boot_images",
21 "is_import_boot_images_running",
22- "is_import_boot_images_running_for",
23 ]
24
25 from collections import Sequence
26@@ -81,22 +80,6 @@
27
28
29 @synchronous
30-def is_import_boot_images_running_for(rack_controller):
31- """Return True if the rack_controller is currently import boot images.
32-
33- :param rack_controller: The RackController.
34-
35- :raises NoConnectionsAvailable: When no connections to the rack controller
36- are available for use.
37- :raises crochet.TimeoutError: If a response has not been received within
38- 30 seconds.
39- """
40- client = getClientFor(rack_controller.system_id, timeout=1)
41- call = client(IsImportBootImagesRunning)
42- return call.wait(30).get("running")
43-
44-
45-@synchronous
46 def get_boot_images(rack_controller):
47 """Obtain the avaliable boot images of this rack controller.
48
49
50=== modified file 'src/maasserver/clusterrpc/tests/test_boot_images.py'
51--- src/maasserver/clusterrpc/tests/test_boot_images.py 2016-03-28 13:54:47 +0000
52+++ src/maasserver/clusterrpc/tests/test_boot_images.py 2016-04-14 17:36:28 +0000
53@@ -17,7 +17,6 @@
54 get_boot_images_for,
55 get_common_available_boot_images,
56 is_import_boot_images_running,
57- is_import_boot_images_running_for,
58 )
59 from maasserver.clusterrpc.testing.boot_images import make_rpc_boot_image
60 from maasserver.enum import BOOT_RESOURCE_TYPE
61@@ -52,10 +51,7 @@
62 compose_image_path,
63 locate_tftp_path,
64 )
65-from provisioningserver.rpc import (
66- boot_images,
67- clusterservice,
68-)
69+from provisioningserver.rpc import boot_images
70 from provisioningserver.rpc.cluster import (
71 ImportBootImages,
72 ListBootImages,
73@@ -151,26 +147,6 @@
74 self.assertTrue(is_import_boot_images_running())
75
76
77-class TestIsImportBootImagesRunningFor(MAASServerTestCase):
78- """Tests for `is_import_boot_images_running_for`."""
79-
80- def test_returns_True(self):
81- mock_is_running = self.patch(
82- clusterservice, "is_import_boot_images_running")
83- mock_is_running.return_value = True
84- rack_controller = factory.make_RackController()
85- self.useFixture(RunningClusterRPCFixture())
86- self.assertTrue(is_import_boot_images_running_for(rack_controller))
87-
88- def test_returns_False(self):
89- mock_is_running = self.patch(
90- clusterservice, "is_import_boot_images_running")
91- mock_is_running.return_value = False
92- rack_controller = factory.make_RackController()
93- self.useFixture(RunningClusterRPCFixture())
94- self.assertFalse(is_import_boot_images_running_for(rack_controller))
95-
96-
97 def prepare_tftp_root(test):
98 """Create a `current` directory and configure its use."""
99 test.tftp_root = os.path.join(test.make_dir(), 'current')
100
101=== modified file 'src/maasserver/models/node.py'
102--- src/maasserver/models/node.py 2016-04-14 00:38:16 +0000
103+++ src/maasserver/models/node.py 2016-04-14 17:36:28 +0000
104@@ -24,6 +24,7 @@
105 import socket
106 from urllib.parse import urlparse
107
108+from crochet import TimeoutError
109 from django.contrib.auth.models import User
110 from django.core.exceptions import (
111 PermissionDenied,
112@@ -87,6 +88,7 @@
113 MAC,
114 )
115 from maasserver.models.bmc import BMC
116+from maasserver.models.bootresource import BootResource
117 from maasserver.models.cleansave import CleanSave
118 from maasserver.models.config import Config
119 from maasserver.models.domain import Domain
120@@ -159,6 +161,7 @@
121 from provisioningserver.power import QUERY_POWER_TYPES
122 from provisioningserver.rpc.cluster import (
123 AddChassis,
124+ IsImportBootImagesRunning,
125 RefreshRackControllerInfo,
126 )
127 from provisioningserver.rpc.exceptions import (
128@@ -3620,7 +3623,17 @@
129 try:
130 # Combine all boot images one per name and arch
131 downloaded_boot_images = defaultdict(set)
132- for image in get_boot_images(self):
133+ boot_images = get_boot_images(self)
134+ # Determine the status of the boot images
135+ if not BootResource.objects.boot_images_are_in_sync(boot_images):
136+ if self.is_import_boot_images_running():
137+ status = "Syncing"
138+ else:
139+ status = "Out of sync"
140+ else:
141+ status = "Synced"
142+
143+ for image in boot_images:
144 if image['osystem'] == 'custom':
145 name = image['release']
146 else:
147@@ -3637,9 +3650,23 @@
148 'architecture': arch,
149 'subarches': sorted(subarches),
150 } for (name, arch), subarches in downloaded_boot_images.items()]
151- return {'images': images, 'connected': True}
152- except NoConnectionsAvailable:
153- return {'images': [], 'connected': False}
154+
155+ return {'images': images, 'connected': True, 'status': status}
156+ except (NoConnectionsAvailable, TimeoutError):
157+ return {'images': [], 'connected': False, 'status': 'Unknown'}
158+
159+ def is_import_boot_images_running(self):
160+ """Return whether the boot images are running
161+
162+ :raises NoConnectionsAvailable: When no connections to the rack
163+ controller are available for use.
164+ :raises crochet.TimeoutError: If a response has not been received
165+ within 30 seconds.
166+ """
167+ client = getClientFor(self.system_id, timeout=1)
168+ call = client(IsImportBootImagesRunning)
169+ response = call.wait(30)
170+ return response['running']
171
172
173 class RegionController(Node):
174
175=== modified file 'src/maasserver/models/tests/test_node.py'
176--- src/maasserver/models/tests/test_node.py 2016-04-14 00:38:16 +0000
177+++ src/maasserver/models/tests/test_node.py 2016-04-14 17:36:28 +0000
178@@ -40,6 +40,7 @@
179 from maasserver.models import (
180 bmc as bmc_module,
181 BondInterface,
182+ BootResource,
183 BridgeInterface,
184 Config,
185 Device,
186@@ -135,6 +136,7 @@
187 from provisioningserver.power.schema import JSON_POWER_TYPE_PARAMETERS
188 from provisioningserver.rpc.cluster import (
189 AddChassis,
190+ IsImportBootImagesRunning,
191 RefreshRackControllerInfo,
192 )
193 from provisioningserver.rpc.exceptions import (
194@@ -6899,28 +6901,142 @@
195 'subarchitecture': 'hwe-x',
196 },
197 ]
198- self.assertItemsEqual(
199- {
200- 'connected': True,
201- 'images': [
202- {
203- 'name': 'ubuntu/trusty',
204- 'architecture': 'amd64',
205- 'subarches': ['generic', 'hwe-t', 'hwe-x'],
206- },
207- {
208- 'name': 'custom_os',
209- 'architecture': 'amd64',
210- 'subarches': ['generic'],
211- }
212- ]
213- }, rack_controller.list_boot_images())
214+ self.patch(
215+ BootResource.objects,
216+ 'boot_images_are_in_sync').return_value = True
217+ images = rack_controller.list_boot_images()
218+ self.assertTrue(images['connected'])
219+ self.assertItemsEqual([
220+ {
221+ 'name': 'ubuntu/trusty',
222+ 'architecture': 'amd64',
223+ 'subarches': ['generic', 'hwe-t', 'hwe-x'],
224+ },
225+ {
226+ 'name': 'custom_os',
227+ 'architecture': 'amd64',
228+ 'subarches': ['generic'],
229+ }], images['images'])
230+ self.assertEquals('Synced', images['status'])
231
232 def test_list_boot_images_when_disconnected(self):
233 rack_controller = factory.make_RackController()
234- self.assertItemsEqual(
235- {'connected': False, 'images': []},
236- rack_controller.list_boot_images())
237+ images = rack_controller.list_boot_images()
238+ self.assertEquals(False, images['connected'])
239+ self.assertItemsEqual([], images['images'])
240+ self.assertEquals('Unknown', images['status'])
241+
242+ def test_list_boot_images_syncing(self):
243+ rack_controller = factory.make_RackController()
244+ self.patch(boot_images, 'get_boot_images').return_value = [
245+ {
246+ 'release': 'custom_os',
247+ 'osystem': 'custom',
248+ 'architecture': 'amd64',
249+ 'subarchitecture': 'generic',
250+ },
251+ {
252+ 'release': 'trusty',
253+ 'osystem': 'ubuntu',
254+ 'architecture': 'amd64',
255+ 'subarchitecture': 'generic',
256+ },
257+ {
258+ 'release': 'trusty',
259+ 'osystem': 'ubuntu',
260+ 'architecture': 'amd64',
261+ 'subarchitecture': 'hwe-t',
262+ },
263+ {
264+ 'release': 'trusty',
265+ 'osystem': 'ubuntu',
266+ 'architecture': 'amd64',
267+ 'subarchitecture': 'hwe-x',
268+ },
269+ ]
270+ self.patch(
271+ BootResource.objects,
272+ 'boot_images_are_in_sync').return_value = False
273+ self.patch(
274+ rack_controller,
275+ 'is_import_boot_images_running').return_value = True
276+ images = rack_controller.list_boot_images()
277+ self.assertTrue(images['connected'])
278+ self.assertItemsEqual([
279+ {
280+ 'name': 'ubuntu/trusty',
281+ 'architecture': 'amd64',
282+ 'subarches': ['generic', 'hwe-t', 'hwe-x'],
283+ },
284+ {
285+ 'name': 'custom_os',
286+ 'architecture': 'amd64',
287+ 'subarches': ['generic'],
288+ }], images['images'])
289+ self.assertEquals('Syncing', images['status'])
290+
291+ def test_list_boot_images_out_of_sync(self):
292+ rack_controller = factory.make_RackController()
293+ self.patch(boot_images, 'get_boot_images').return_value = [
294+ {
295+ 'release': 'custom_os',
296+ 'osystem': 'custom',
297+ 'architecture': 'amd64',
298+ 'subarchitecture': 'generic',
299+ },
300+ {
301+ 'release': 'trusty',
302+ 'osystem': 'ubuntu',
303+ 'architecture': 'amd64',
304+ 'subarchitecture': 'generic',
305+ },
306+ {
307+ 'release': 'trusty',
308+ 'osystem': 'ubuntu',
309+ 'architecture': 'amd64',
310+ 'subarchitecture': 'hwe-t',
311+ },
312+ {
313+ 'release': 'trusty',
314+ 'osystem': 'ubuntu',
315+ 'architecture': 'amd64',
316+ 'subarchitecture': 'hwe-x',
317+ },
318+ ]
319+ self.patch(
320+ BootResource.objects,
321+ 'boot_images_are_in_sync').return_value = False
322+ self.patch(
323+ rack_controller,
324+ 'is_import_boot_images_running').return_value = False
325+ images = rack_controller.list_boot_images()
326+ self.assertTrue(images['connected'])
327+ self.assertItemsEqual([
328+ {
329+ 'name': 'ubuntu/trusty',
330+ 'architecture': 'amd64',
331+ 'subarches': ['generic', 'hwe-t', 'hwe-x'],
332+ },
333+ {
334+ 'name': 'custom_os',
335+ 'architecture': 'amd64',
336+ 'subarches': ['generic'],
337+ }], images['images'])
338+ self.assertEquals('Out of sync', images['status'])
339+
340+ def test_is_import_images_running(self):
341+ running = factory.pick_bool()
342+ rackcontroller = factory.make_RackController()
343+ self.useFixture(RegionEventLoopFixture("rpc"))
344+ self.useFixture(RunningEventLoopFixture())
345+ fixture = self.useFixture(MockLiveRegionToClusterRPCFixture())
346+ protocol = fixture.makeCluster(
347+ rackcontroller, IsImportBootImagesRunning)
348+ protocol.IsImportBootImagesRunning.return_value = defer.succeed({
349+ 'running': running,
350+ })
351+ self.assertEquals(
352+ running, rackcontroller.is_import_boot_images_running())
353
354
355 class TestRegionController(MAASServerTestCase):