Merge lp:~ltrager/maas/lp1656425_2.1 into lp:maas/2.1
- lp1656425_2.1
- Merge into 2.1
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Lee Trager | ||||
Approved revision: | no longer in the source branch. | ||||
Merged at revision: | 5580 | ||||
Proposed branch: | lp:~ltrager/maas/lp1656425_2.1 | ||||
Merge into: | lp:maas/2.1 | ||||
Diff against target: |
303 lines (+145/-33) 3 files modified
docs/changelog.rst (+1/-0) src/provisioningserver/import_images/download_resources.py (+25/-3) src/provisioningserver/import_images/tests/test_download_resources.py (+119/-30) |
||||
To merge this branch: | bzr merge lp:~ltrager/maas/lp1656425_2.1 | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Lee Trager (community) | Approve | ||
Review via email: mp+315156@code.launchpad.net |
Commit message
Backport r5642 from trunk: Only create hard links for the product subarch, not its subarches
Every Ubuntu product in a SimpleStream contains a list of subarches which contain what subarches are a subset of the current product. The rack controller was creating hard links for each subarch in this list. Kernel flavors contain the same subarches list as the generic kernel as the only difference between them is their config. For example both ga-16.04 and ga-16.04-lowlatency have a subarches list of "hwe-{p,
Description of the change
MAAS Lander (maas-lander) wrote : | # |
The attempt to merge lp:~ltrager/maas/lp1656425_2.1 into lp:maas/2.1 failed. Below is the output from the failed tests.
Get:1 http://
Hit:2 http://
Get:3 http://
Hit:4 http://
Fetched 204 kB in 0s (462 kB/s)
Reading package lists...
sudo DEBIAN_
--no-
Reading package lists...
Building dependency tree...
Reading state information...
authbind is already the newest version (2.1.1+nmu1).
avahi-utils is already the newest version (0.6.32~
build-essential is already the newest version (12.1ubuntu2).
debhelper is already the newest version (9.20160115ubun
distro-info is already the newest version (0.14build1).
git is already the newest version (1:2.7.4-0ubuntu1).
libjs-angularjs is already the newest version (1.2.28-1ubuntu2).
libjs-jquery is already the newest version (1.11.3+dfsg-4).
libjs-yui3-full is already the newest version (3.5.1-1ubuntu3).
libjs-yui3-min is already the newest version (3.5.1-1ubuntu3).
make is already the newest version (4.1-6).
postgresql is already the newest version (9.5+173).
pxelinux is already the newest version (3:6.03+
python-formencode is already the newest version (1.3.0-0ubuntu5).
python-lxml is already the newest version (3.5.0-1build1).
python-netaddr is already the newest version (0.7.18-1).
python-netifaces is already the newest version (0.10.4-0.1build2).
python-psycopg2 is already the newes...
Preview Diff
1 | === modified file 'docs/changelog.rst' |
2 | --- docs/changelog.rst 2017-01-12 14:14:52 +0000 |
3 | +++ docs/changelog.rst 2017-01-19 17:54:18 +0000 |
4 | @@ -14,6 +14,7 @@ |
5 | |
6 | LP: #1654412 Unable to set edge kernel as the min_hwe_kernel and deploy with the default kernel |
7 | |
8 | +LP: #1656425 Ephemeral environments using the wrong kernel |
9 | |
10 | 2.1.3 |
11 | ===== |
12 | |
13 | === modified file 'src/provisioningserver/import_images/download_resources.py' |
14 | --- src/provisioningserver/import_images/download_resources.py 2016-10-28 15:58:32 +0000 |
15 | +++ src/provisioningserver/import_images/download_resources.py 2017-01-19 17:54:18 +0000 |
16 | @@ -1,4 +1,4 @@ |
17 | -# Copyright 2014-2016 Canonical Ltd. This software is licensed under the |
18 | +# Copyright 2014-2017 Canonical Ltd. This software is licensed under the |
19 | # GNU Affero General Public License version 3 (see the file LICENSE). |
20 | |
21 | """Simplestreams code to download boot resources.""" |
22 | @@ -287,11 +287,33 @@ |
23 | self.store, filename, tag, checksums, size, contentsource) |
24 | |
25 | osystem = get_os_from_product(item) |
26 | - subarches = self.product_mapping.get(item) |
27 | + |
28 | + # link_resources creates a hardlink for every subarch. Every Ubuntu |
29 | + # product in a SimpleStream contains a list of subarches which list |
30 | + # what subarches are a subset of that subarch. For example Xenial |
31 | + # ga-16.04 has the subarches list hwe-{p,q,r,s,t,u,v,w},ga-16.04. |
32 | + # Kernel flavors are the same arch, the only difference is the kernel |
33 | + # config. So ga-16.04-lowlatency has the same subarch list as ga-16.04. |
34 | + # If we create hard links for all subarches a kernel flavor may |
35 | + # overwrite the generic kernel hard link. This happens if a kernel |
36 | + # flavor is processed after the generic kernel. Since MAAS doesn't use |
37 | + # the other hard links only create hard links for the subarch of the |
38 | + # product we have and a rolling link if it's a rolling kernel. |
39 | + if 'subarch' in item: |
40 | + # MAAS uses the 'generic' subarch when it doesn't know which |
41 | + # subarch to use. This happens during enlistment and commissioning. |
42 | + # Allow the 'generic' kflavor to own the 'generic' hardlink. |
43 | + if item.get('kflavor') == 'generic': |
44 | + subarches = {item['subarch'], 'generic'} |
45 | + else: |
46 | + subarches = {item['subarch']} |
47 | + else: |
48 | + subarches = {'generic'} |
49 | + |
50 | if item.get('rolling', False): |
51 | subarch_parts = item['subarch'].split('-') |
52 | subarch_parts[1] = 'rolling' |
53 | - subarches.append('-'.join(subarch_parts)) |
54 | + subarches.add('-'.join(subarch_parts)) |
55 | link_resources( |
56 | snapshot_path=self.root_path, links=links, |
57 | osystem=osystem, arch=item['arch'], release=item['release'], |
58 | |
59 | === modified file 'src/provisioningserver/import_images/tests/test_download_resources.py' |
60 | --- src/provisioningserver/import_images/tests/test_download_resources.py 2016-10-12 15:26:17 +0000 |
61 | +++ src/provisioningserver/import_images/tests/test_download_resources.py 2017-01-19 17:54:18 +0000 |
62 | @@ -182,124 +182,213 @@ |
63 | class TestRepoWriter(MAASTestCase): |
64 | """Tests for `RepoWriter`.""" |
65 | |
66 | - def make_product(self, ftype=None): |
67 | - if ftype is None: |
68 | - ftype = factory.make_name('ftype') |
69 | + def make_product(self, **kwargs): |
70 | return { |
71 | 'content_id': 'maas:v2:download', |
72 | 'product_name': factory.make_string(), |
73 | 'version_name': datetime.utcnow().strftime('%Y%m%d'), |
74 | 'sha256': factory.make_name('sha256'), |
75 | 'size': random.randint(2, 2**16), |
76 | - 'ftype': ftype, |
77 | + 'ftype': factory.make_name('ftype'), |
78 | 'path': '/path/to/%s' % factory.make_name('filename'), |
79 | 'os': factory.make_name('os'), |
80 | 'release': factory.make_name('release'), |
81 | 'arch': factory.make_name('arch'), |
82 | 'label': factory.make_name('label'), |
83 | + **kwargs, |
84 | } |
85 | |
86 | def test_inserts_archive(self): |
87 | - product = self.make_product('archive.tar.xz') |
88 | product_mapping = ProductMapping() |
89 | subarch = factory.make_name('subarch') |
90 | + product = self.make_product(ftype='archive.tar.xz', subarch=subarch) |
91 | product_mapping.add(product, subarch) |
92 | repo_writer = download_resources.RepoWriter( |
93 | None, None, product_mapping) |
94 | self.patch( |
95 | download_resources, 'products_exdata').return_value = product |
96 | + # Prevent MAAS from trying to actually write the file. |
97 | mock_extract_archive_tar = self.patch( |
98 | download_resources, 'extract_archive_tar') |
99 | mock_link_resources = self.patch(download_resources, 'link_resources') |
100 | + # We only need to provide the product as the other fields are only used |
101 | + # when writing the actual files to disk. |
102 | repo_writer.insert_item(product, None, None, None, None) |
103 | + # None is used for the store and the content source as we're not |
104 | + # writing anything to disk. |
105 | self.assertThat( |
106 | mock_extract_archive_tar, |
107 | MockCalledOnceWith( |
108 | - mock.ANY, os.path.basename(product['path']), product['sha256'], |
109 | + None, os.path.basename(product['path']), product['sha256'], |
110 | {'sha256': product['sha256']}, product['size'], None)) |
111 | + # links are mocked out by the mock_insert_file above. |
112 | self.assertThat( |
113 | mock_link_resources, |
114 | MockCalledOnceWith( |
115 | snapshot_path=None, links=mock.ANY, osystem=product['os'], |
116 | arch=product['arch'], release=product['release'], |
117 | - label=product['label'], subarches=[subarch], |
118 | + label=product['label'], subarches={subarch}, |
119 | bootloader_type=None)) |
120 | |
121 | def test_inserts_root_image(self): |
122 | - product = self.make_product('root-image.gz') |
123 | product_mapping = ProductMapping() |
124 | subarch = factory.make_name('subarch') |
125 | + product = self.make_product(ftype='root-image.gz', subarch=subarch) |
126 | product_mapping.add(product, subarch) |
127 | repo_writer = download_resources.RepoWriter( |
128 | None, None, product_mapping) |
129 | self.patch( |
130 | download_resources, 'products_exdata').return_value = product |
131 | + # Prevent MAAS from trying to actually write the file. |
132 | mock_insert_root_image = self.patch( |
133 | download_resources, 'insert_root_image') |
134 | mock_link_resources = self.patch(download_resources, 'link_resources') |
135 | + # We only need to provide the product as the other fields are only used |
136 | + # when writing the actual files to disk. |
137 | repo_writer.insert_item(product, None, None, None, None) |
138 | + # None is used for the store and the content source as we're not |
139 | + # writing anything to disk. |
140 | self.assertThat( |
141 | mock_insert_root_image, |
142 | MockCalledOnceWith( |
143 | - mock.ANY, product['sha256'], {'sha256': product['sha256']}, |
144 | + None, product['sha256'], {'sha256': product['sha256']}, |
145 | product['size'], None)) |
146 | + # links are mocked out by the mock_insert_file above. |
147 | self.assertThat( |
148 | mock_link_resources, |
149 | MockCalledOnceWith( |
150 | snapshot_path=None, links=mock.ANY, osystem=product['os'], |
151 | arch=product['arch'], release=product['release'], |
152 | - label=product['label'], subarches=[subarch], |
153 | + label=product['label'], subarches={subarch}, |
154 | bootloader_type=None)) |
155 | |
156 | def test_inserts_file(self): |
157 | - product = self.make_product() |
158 | product_mapping = ProductMapping() |
159 | subarch = factory.make_name('subarch') |
160 | + product = self.make_product(subarch=subarch) |
161 | product_mapping.add(product, subarch) |
162 | repo_writer = download_resources.RepoWriter( |
163 | None, None, product_mapping) |
164 | self.patch( |
165 | download_resources, 'products_exdata').return_value = product |
166 | + # Prevent MAAS from trying to actually write the file. |
167 | mock_insert_file = self.patch(download_resources, 'insert_file') |
168 | mock_link_resources = self.patch(download_resources, 'link_resources') |
169 | + # We only need to provide the product as the other fields are only used |
170 | + # when writing the actual files to disk. |
171 | repo_writer.insert_item(product, None, None, None, None) |
172 | + # None is used for the store and the content source as we're not |
173 | + # writing anything to disk. |
174 | self.assertThat( |
175 | mock_insert_file, |
176 | MockCalledOnceWith( |
177 | - mock.ANY, os.path.basename(product['path']), product['sha256'], |
178 | + None, os.path.basename(product['path']), product['sha256'], |
179 | {'sha256': product['sha256']}, product['size'], None)) |
180 | + # links are mocked out by the mock_insert_file above. |
181 | self.assertThat( |
182 | mock_link_resources, |
183 | MockCalledOnceWith( |
184 | snapshot_path=None, links=mock.ANY, osystem=product['os'], |
185 | arch=product['arch'], release=product['release'], |
186 | - label=product['label'], subarches=[subarch], |
187 | + label=product['label'], subarches={subarch}, |
188 | bootloader_type=None)) |
189 | |
190 | def test_inserts_rolling_links(self): |
191 | - product = self.make_product() |
192 | - product['subarch'] = 'hwe-16.04' |
193 | - product['rolling'] = True |
194 | product_mapping = ProductMapping() |
195 | + product = self.make_product(subarch='hwe-16.04', rolling=True) |
196 | product_mapping.add(product, 'hwe-16.04') |
197 | repo_writer = download_resources.RepoWriter( |
198 | None, None, product_mapping) |
199 | self.patch( |
200 | download_resources, 'products_exdata').return_value = product |
201 | - mock_insert_file = self.patch(download_resources, 'insert_file') |
202 | - mock_link_resources = self.patch(download_resources, 'link_resources') |
203 | - repo_writer.insert_item(product, None, None, None, None) |
204 | - self.assertThat( |
205 | - mock_insert_file, |
206 | - MockCalledOnceWith( |
207 | - mock.ANY, os.path.basename(product['path']), product['sha256'], |
208 | - {'sha256': product['sha256']}, product['size'], None)) |
209 | - self.assertThat( |
210 | - mock_link_resources, |
211 | - MockCalledOnceWith( |
212 | - snapshot_path=None, links=mock.ANY, osystem=product['os'], |
213 | - arch=product['arch'], release=product['release'], |
214 | - label=product['label'], subarches=['hwe-16.04', 'hwe-rolling'], |
215 | + # Prevent MAAS from trying to actually write the file. |
216 | + mock_insert_file = self.patch(download_resources, 'insert_file') |
217 | + mock_link_resources = self.patch(download_resources, 'link_resources') |
218 | + # We only need to provide the product as the other fields are only used |
219 | + # when writing the actual files to disk. |
220 | + repo_writer.insert_item(product, None, None, None, None) |
221 | + # None is used for the store and the content source as we're not |
222 | + # writing anything to disk. |
223 | + self.assertThat( |
224 | + mock_insert_file, |
225 | + MockCalledOnceWith( |
226 | + None, os.path.basename(product['path']), product['sha256'], |
227 | + {'sha256': product['sha256']}, product['size'], None)) |
228 | + # links are mocked out by the mock_insert_file above. |
229 | + self.assertThat( |
230 | + mock_link_resources, |
231 | + MockCalledOnceWith( |
232 | + snapshot_path=None, links=mock.ANY, osystem=product['os'], |
233 | + arch=product['arch'], release=product['release'], |
234 | + label=product['label'], subarches={'hwe-16.04', 'hwe-rolling'}, |
235 | + bootloader_type=None)) |
236 | + |
237 | + def test_only_creates_links_for_its_own_subarch(self): |
238 | + # Regression test for LP:1656425 |
239 | + product_name = factory.make_name('product_name') |
240 | + version_name = factory.make_name('version_name') |
241 | + product_mapping = ProductMapping() |
242 | + for subarch in [ |
243 | + 'hwe-p', 'hwe-q', 'hwe-r', 'hwe-s', 'hwe-t', 'hwe-u', 'hwe-v', |
244 | + 'hwe-w', 'ga-16.04']: |
245 | + product = self.make_product( |
246 | + product_name=product_name, version_name=version_name, |
247 | + subarch=subarch) |
248 | + product_mapping.add(product, subarch) |
249 | + repo_writer = download_resources.RepoWriter( |
250 | + None, None, product_mapping) |
251 | + self.patch( |
252 | + download_resources, 'products_exdata').return_value = product |
253 | + # Prevent MAAS from trying to actually write the file. |
254 | + mock_insert_file = self.patch(download_resources, 'insert_file') |
255 | + mock_link_resources = self.patch(download_resources, 'link_resources') |
256 | + # We only need to provide the product as the other fields are only used |
257 | + # when writing the actual files to disk. |
258 | + repo_writer.insert_item(product, None, None, None, None) |
259 | + # None is used for the store and the content source as we're not |
260 | + # writing anything to disk. |
261 | + self.assertThat( |
262 | + mock_insert_file, |
263 | + MockCalledOnceWith( |
264 | + None, os.path.basename(product['path']), product['sha256'], |
265 | + {'sha256': product['sha256']}, product['size'], None)) |
266 | + # links are mocked out by the mock_insert_file above. |
267 | + self.assertThat( |
268 | + mock_link_resources, |
269 | + MockCalledOnceWith( |
270 | + snapshot_path=None, links=mock.ANY, osystem=product['os'], |
271 | + arch=product['arch'], release=product['release'], |
272 | + label=product['label'], subarches={'ga-16.04'}, |
273 | + bootloader_type=None)) |
274 | + |
275 | + def test_inserts_generic_link_for_generic_kflavor(self): |
276 | + product_mapping = ProductMapping() |
277 | + product = self.make_product(subarch='ga-16.04', kflavor='generic') |
278 | + product_mapping.add(product, 'ga-16.04') |
279 | + repo_writer = download_resources.RepoWriter( |
280 | + None, None, product_mapping) |
281 | + self.patch( |
282 | + download_resources, 'products_exdata').return_value = product |
283 | + # Prevent MAAS from trying to actually write the file. |
284 | + mock_insert_file = self.patch(download_resources, 'insert_file') |
285 | + mock_link_resources = self.patch(download_resources, 'link_resources') |
286 | + # We only need to provide the product as the other fields are only used |
287 | + # when writing the actual files to disk. |
288 | + repo_writer.insert_item(product, None, None, None, None) |
289 | + # None is used for the store and the content source as we're not |
290 | + # writing anything to disk. |
291 | + self.assertThat( |
292 | + mock_insert_file, |
293 | + MockCalledOnceWith( |
294 | + None, os.path.basename(product['path']), product['sha256'], |
295 | + {'sha256': product['sha256']}, product['size'], None)) |
296 | + # links are mocked out by the mock_insert_file above. |
297 | + self.assertThat( |
298 | + mock_link_resources, |
299 | + MockCalledOnceWith( |
300 | + snapshot_path=None, links=mock.ANY, osystem=product['os'], |
301 | + arch=product['arch'], release=product['release'], |
302 | + label=product['label'], subarches={'ga-16.04', 'generic'}, |
303 | bootloader_type=None)) |
304 | |
305 |
Reviewed in https:/ /code.launchpad .net/~ltrager/ maas/lp1656425/ +merge/ 314761