MAAS

Merge lp:~gmb/maas/celery-pass-blob-to-mipf into lp:~maas-committers/maas/trunk

  1. celery-pass-blob-to-mipf
  2. Merge into trunk
Proposed by Graham Binns
Status: Merged
Approved by: Graham Binns
Approved revision: no longer in the source branch.
Merged at revision: 2358
Proposed branch: lp:~gmb/maas/celery-pass-blob-to-mipf
Merge into: lp:~maas-committers/maas/trunk
Diff against target: 245 lines (+52/-16)
11 files modified
etc/maas/bootresources.yaml (+1/-1)
src/maasserver/models/nodegroup.py (+3/-0)
src/maasserver/models/tests/test_nodegroup.py (+12/-0)
src/provisioningserver/config.py (+1/-1)
src/provisioningserver/import_images/download_descriptions.py (+1/-1)
src/provisioningserver/import_images/download_resources.py (+1/-1)
src/provisioningserver/import_images/tests/test_boot_resources.py (+3/-3)
src/provisioningserver/import_images/tests/test_download_resources.py (+4/-4)
src/provisioningserver/tasks.py (+5/-3)
src/provisioningserver/tests/test_config.py (+2/-2)
src/provisioningserver/tests/test_tasks.py (+19/-0)
To merge this branch: bzr merge lp:~gmb/maas/celery-pass-blob-to-mipf
Reviewer Review Type Date Requested Status
Jeroen T. Vermeulen (community) Approve
Review via email: mp+220650@code.launchpad.net

Commit message

Update the import_boot_images celery job so that it fetches configuration from the DB and not from bootresources.yaml, which is going away.

sources['path'] because sources['url'] in the import_images script to reflect the way that BootSources are represented in the database.

To post a comment you must log in.
Revision history for this message
Jeroen T. Vermeulen (jtv) wrote :

Should that "because" in the second paragraph of the commit message be "becomes"?

review: Approve
Revision history for this message
Graham Binns (gmb) wrote :

On 22 May 2014 15:01, Jeroen T. Vermeulen <email address hidden> wrote:
> Review: Approve
>
> Should that "because" in the second paragraph of the commit message be "becomes"?

Yes :)

> Diff comments:
>
>> === modified file 'etc/maas/bootresources.yaml'
>> --- etc/maas/bootresources.yaml 2014-04-25 18:03:38 +0000
>> +++ etc/maas/bootresources.yaml 2014-05-22 13:50:08 +0000
>> @@ -52,7 +52,7 @@
>> storage: "/var/lib/maas/boot-resources/"
>>
>> sources:
>> - - path: "http://maas.ubuntu.com/images/ephemeral-v2/releases/"
>> + - url: "http://maas.ubuntu.com/images/ephemeral-v2/releases/"
>> keyring: "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg"
>> selections:
>> - release: "trusty"
>>
>> === modified file 'src/maasserver/models/nodegroup.py'
>> --- src/maasserver/models/nodegroup.py 2014-05-21 17:03:00 +0000
>> +++ src/maasserver/models/nodegroup.py 2014-05-22 13:50:08 +0000
>> @@ -283,9 +283,12 @@
>> """
>> # Avoid circular imports.
>> from maasserver.models import Config
>> + config = [
>> + source.to_dict() for source in self.bootsource_set.all()]
>> task_kwargs = {
>> 'callback': report_boot_images.subtask(
>> options={'queue': self.uuid}),
>> + 'config': config,
>
> Wasn't this called "sources" now?

Er, yes.

Revision history for this message
MAAS Lander (maas-lander) wrote :
Download full text (27.4 KiB)

The attempt to merge lp:~gmb/maas/celery-pass-blob-to-mipf into lp:maas failed. Below is the output from the failed tests.

Ign http://security.ubuntu.com trusty-security InRelease
Get:1 http://security.ubuntu.com trusty-security Release.gpg [933 B]
Ign http://nova.clouds.archive.ubuntu.com trusty InRelease
Get:2 http://security.ubuntu.com trusty-security Release [58.5 kB]
Ign http://nova.clouds.archive.ubuntu.com trusty-updates InRelease
Hit http://nova.clouds.archive.ubuntu.com trusty Release.gpg
Get:3 http://nova.clouds.archive.ubuntu.com trusty-updates Release.gpg [933 B]
Hit http://nova.clouds.archive.ubuntu.com trusty Release
Get:4 http://nova.clouds.archive.ubuntu.com trusty-updates Release [58.5 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty/main Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Sources
Hit http://nova.clouds.archive.ubuntu.com trusty/main amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/universe amd64 Packages
Hit http://nova.clouds.archive.ubuntu.com trusty/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en
Get:5 http://security.ubuntu.com trusty-security/main Sources [15.6 kB]
Get:6 http://security.ubuntu.com trusty-security/universe Sources [4,212 B]
Get:7 http://nova.clouds.archive.ubuntu.com trusty-updates/main Sources [40.2 kB]
Get:8 http://security.ubuntu.com trusty-security/main amd64 Packages [49.4 kB]
Get:9 http://nova.clouds.archive.ubuntu.com trusty-updates/universe Sources [24.2 kB]
Get:10 http://security.ubuntu.com trusty-security/universe amd64 Packages [17.7 kB]
Hit http://security.ubuntu.com trusty-security/main Translation-en
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://security.ubuntu.com trusty-security/main Translation-en_US
Ign http://security.ubuntu.com trusty-security/universe Translation-en_US
Get:11 http://nova.clouds.archive.ubuntu.com trusty-updates/main amd64 Packages [94.1 kB]
Get:12 http://nova.clouds.archive.ubuntu.com trusty-updates/universe amd64 Packages [65.2 kB]
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en
Hit http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en
Ign http://nova.clouds.archive.ubuntu.com trusty/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty/universe Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/main Translation-en_US
Ign http://nova.clouds.archive.ubuntu.com trusty-updates/universe Translation-en_US
Fetched 429 kB in 0s (2,065 kB/s)
Reading package lists...
sudo DEBIAN_FRONTEND=noninteractive apt-get -y \
     --no-install-recommends install apache2 bind9 bind9utils build-essential bzr-builddeb curl daemontools debhelper dh-apport distro-info dnsutils firefox freeipmi-tools ipython isc-dhcp-common libjs-raphael libjs-yui3-full libjs-yui3-min libpq-dev make postgresql python-amqplib python-bzrlib python-celery python-convoy python-crochet python-cssselect python-curtin python-dev python-distro-info python-django python-django-piston python-django-south python-djorm-ext-pgarray python-docutils python-formencode python-httplib2 python-jinja2 pyth...

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'etc/maas/bootresources.yaml'
2--- etc/maas/bootresources.yaml 2014-04-25 18:03:38 +0000
3+++ etc/maas/bootresources.yaml 2014-05-22 15:12:53 +0000
4@@ -52,7 +52,7 @@
5 storage: "/var/lib/maas/boot-resources/"
6
7 sources:
8- - path: "http://maas.ubuntu.com/images/ephemeral-v2/releases/"
9+ - url: "http://maas.ubuntu.com/images/ephemeral-v2/releases/"
10 keyring: "/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg"
11 selections:
12 - release: "trusty"
13
14=== modified file 'src/maasserver/models/nodegroup.py'
15--- src/maasserver/models/nodegroup.py 2014-05-21 17:03:00 +0000
16+++ src/maasserver/models/nodegroup.py 2014-05-22 15:12:53 +0000
17@@ -283,9 +283,12 @@
18 """
19 # Avoid circular imports.
20 from maasserver.models import Config
21+ sources = [
22+ source.to_dict() for source in self.bootsource_set.all()]
23 task_kwargs = {
24 'callback': report_boot_images.subtask(
25 options={'queue': self.uuid}),
26+ 'sources': sources,
27 }
28 http_proxy = Config.objects.get_config('http_proxy')
29 if http_proxy is not None:
30
31=== modified file 'src/maasserver/models/tests/test_nodegroup.py'
32--- src/maasserver/models/tests/test_nodegroup.py 2014-05-19 16:07:59 +0000
33+++ src/maasserver/models/tests/test_nodegroup.py 2014-05-22 15:12:53 +0000
34@@ -466,6 +466,18 @@
35 recorder.apply_async.call_args[1]['kwargs']['callback'],
36 )
37
38+ def test_import_boot_images_passes_sources_list(self):
39+ recorder = self.patch(nodegroup_module, 'import_boot_images')
40+ self.patch(nodegroup_module, 'report_boot_images', Mock())
41+ nodegroup = factory.make_node_group()
42+ nodegroup.ensure_boot_source_definition()
43+ sources = [
44+ source.to_dict() for source in nodegroup.bootsource_set.all()]
45+ nodegroup.import_boot_images()
46+ [call] = recorder.mock_calls
47+ _, _, kwargs = call
48+ self.assertEqual(sources, kwargs['kwargs'].get('sources'))
49+
50 def test_ensure_boot_source_definition_creates_default_source(self):
51 cluster = factory.make_node_group()
52 cluster.ensure_boot_source_definition()
53
54=== modified file 'src/provisioningserver/config.py'
55--- src/provisioningserver/config.py 2014-05-22 10:27:13 +0000
56+++ src/provisioningserver/config.py 2014-05-22 15:12:53 +0000
57@@ -181,7 +181,7 @@
58
59 if_key_missing = None
60
61- path = String(
62+ url = String(
63 if_missing="http://maas.ubuntu.com/images/ephemeral-v2/releases/")
64 keyring = String(
65 if_missing="/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg")
66
67=== modified file 'src/provisioningserver/import_images/download_descriptions.py'
68--- src/provisioningserver/import_images/download_descriptions.py 2014-05-22 11:14:09 +0000
69+++ src/provisioningserver/import_images/download_descriptions.py 2014-05-22 15:12:53 +0000
70@@ -188,6 +188,6 @@
71 boot = BootImageMapping()
72 for source in sources:
73 repo_boot = download_image_descriptions(
74- source['path'], keyring=source['keyring'])
75+ source['url'], keyring=source['keyring'])
76 boot_merge(boot, repo_boot, source['selections'])
77 return boot
78
79=== modified file 'src/provisioningserver/import_images/download_resources.py'
80--- src/provisioningserver/import_images/download_resources.py 2014-05-22 10:13:52 +0000
81+++ src/provisioningserver/import_images/download_resources.py 2014-05-22 15:12:53 +0000
82@@ -293,7 +293,7 @@
83
84 for source in sources:
85 download_boot_resources(
86- source['path'], store, ubuntu_path, product_mapping,
87+ source['url'], store, ubuntu_path, product_mapping,
88 keyring_file=source.get('keyring'),
89 keyring_data=source.get('keyring_data'))
90
91
92=== modified file 'src/provisioningserver/import_images/tests/test_boot_resources.py'
93--- src/provisioningserver/import_images/tests/test_boot_resources.py 2014-05-22 10:27:13 +0000
94+++ src/provisioningserver/import_images/tests/test_boot_resources.py 2014-05-22 15:12:53 +0000
95@@ -251,7 +251,7 @@
96 'boot': {
97 'sources': [
98 {
99- 'path': self.repo,
100+ 'url': self.repo,
101 'selections': [
102 {
103 'release': self.release,
104@@ -352,7 +352,7 @@
105 'boot': {
106 'sources': [
107 {
108- 'path': self.make_dir(),
109+ 'url': self.make_dir(),
110 'keyring': factory.make_name('keyring'),
111 'selections': [
112 {'release': factory.make_name('release')},
113@@ -419,7 +419,7 @@
114 {
115 'keyring': factory.make_name("keyring"),
116 'keyring_data': '',
117- 'path': factory.make_name("something"),
118+ 'url': factory.make_name("something"),
119 'selections': [
120 {
121 'release': factory.make_name("release"),
122
123=== modified file 'src/provisioningserver/import_images/tests/test_download_resources.py'
124--- src/provisioningserver/import_images/tests/test_download_resources.py 2014-05-22 13:40:09 +0000
125+++ src/provisioningserver/import_images/tests/test_download_resources.py 2014-05-22 15:12:53 +0000
126@@ -42,7 +42,7 @@
127 def test_accepts_keyring_data_in_sources(self):
128 keyring_data = b64encode("A keyring!")
129 source = {
130- 'path': 'http://example.com',
131+ 'url': 'http://example.com',
132 'keyring_data': keyring_data,
133 'selections': [{
134 'release': 'trusty',
135@@ -67,7 +67,7 @@
136 self.assertThat(
137 fake,
138 MockCalledWith(
139- source['path'], file_store, ubuntu_path, None,
140+ source['url'], file_store, ubuntu_path, None,
141 keyring_file=None, keyring_data=keyring_data))
142
143 def test_returns_snapshot_path(self):
144@@ -91,7 +91,7 @@
145 cache_path = os.path.join(storage_path, 'cache')
146 file_store = FileStore(cache_path)
147 source = {
148- 'path': 'http://example.com',
149+ 'url': 'http://example.com',
150 'keyring': self.make_file("keyring"),
151 }
152 product_mapping = ProductMapping()
153@@ -102,7 +102,7 @@
154 self.assertThat(
155 fake,
156 MockCalledWith(
157- source['path'], file_store, ubuntu_path, product_mapping,
158+ source['url'], file_store, ubuntu_path, product_mapping,
159 keyring_file=source['keyring'], keyring_data=None))
160
161
162
163=== modified file 'src/provisioningserver/tasks.py'
164--- src/provisioningserver/tasks.py 2014-05-22 11:14:09 +0000
165+++ src/provisioningserver/tasks.py 2014-05-22 15:12:53 +0000
166@@ -450,8 +450,10 @@
167
168 @task
169 @log_exception_text
170-def import_boot_images(http_proxy=None, callback=None):
171- config = boot_resources.read_config()
172+def import_boot_images(http_proxy=None, callback=None, sources=None):
173+ if sources is None:
174+ config = boot_resources.read_config()
175+ sources = config['boot']['sources']
176 variables = {
177 'GNUPGHOME': MAAS_USER_GPGHOME,
178 }
179@@ -459,7 +461,7 @@
180 variables['http_proxy'] = http_proxy
181 variables['https_proxy'] = http_proxy
182 with environment_variables(variables):
183- boot_resources.import_images(config['boot']['sources'])
184+ boot_resources.import_images(sources)
185 if callback is not None:
186 callback.delay()
187
188
189=== modified file 'src/provisioningserver/tests/test_config.py'
190--- src/provisioningserver/tests/test_config.py 2014-05-16 11:55:28 +0000
191+++ src/provisioningserver/tests/test_config.py 2014-05-22 15:12:53 +0000
192@@ -431,7 +431,7 @@
193 'boot': {
194 'sources': [
195 {
196- 'path': (
197+ 'url': (
198 'http://maas.ubuntu.com/images/ephemeral-v2/releases/'
199 ),
200 'keyring': (
201@@ -456,7 +456,7 @@
202 'boot': {
203 'sources': [
204 {
205- 'path': (
206+ 'url': (
207 'http://maas.ubuntu.com/'
208 'images/ephemeral-v2/releases/'),
209 'keyring': (
210
211=== modified file 'src/provisioningserver/tests/test_tasks.py'
212--- src/provisioningserver/tests/test_tasks.py 2014-05-05 18:51:41 +0000
213+++ src/provisioningserver/tests/test_tasks.py 2014-05-22 15:12:53 +0000
214@@ -634,6 +634,7 @@
215 """Fake function; records a copy of the environment."""
216
217 def __call__(self, *args, **kwargs):
218+ self.args = args
219 self.env = os.environ.copy()
220
221 return self.patch(boot_resources, 'import_images', CaptureEnv())
222@@ -678,6 +679,24 @@
223 import_boot_images(callback=mock_callback)
224 self.assertThat(mock_callback.delay, MockCalledOnceWith())
225
226+ def test_import_boot_images_accepts_sources_parameter(self):
227+ fake = self.patch(boot_resources, 'import_images')
228+ sources = [
229+ {
230+ 'path': "http://example.com",
231+ 'selections': [
232+ {
233+ 'release': "trusty",
234+ 'arches': ["amd64"],
235+ 'subarches': ["generic"],
236+ 'labels': ["release"]
237+ },
238+ ],
239+ },
240+ ],
241+ import_boot_images(sources=sources)
242+ self.assertThat(fake, MockCalledOnceWith(sources))
243+
244
245 class TestAddUCSM(PservTestCase):
246