Merge ~raharper/curtin:feature/support-uc20-images into curtin:master

Proposed by Ryan Harper
Status: Merged
Approved by: Ryan Harper
Approved revision: c96a736d2cf5c02a8149dd6cd0e83bb5a431fef5
Merge reported by: Server Team CI bot
Merged at revision: not available
Proposed branch: ~raharper/curtin:feature/support-uc20-images
Merge into: curtin:master
Diff against target: 439 lines (+209/-22)
9 files modified
curtin/commands/block_meta.py (+8/-1)
curtin/commands/curthooks.py (+8/-3)
curtin/distro.py (+18/-1)
tests/unittests/test_commands_block_meta.py (+2/-2)
tests/unittests/test_curthooks.py (+74/-5)
tests/unittests/test_distro.py (+51/-2)
tests/vmtests/__init__.py (+21/-6)
tests/vmtests/releases.py (+18/-2)
tests/vmtests/test_ubuntu_core.py (+9/-0)
Reviewer Review Type Date Requested Status
Server Team CI bot continuous-integration Approve
Chad Smith Approve
Review via email: mp+380552@code.launchpad.net

Commit message

Add support for installing Ubuntu Core 20 images

Update detection methods for Ubuntu Core 20. Use new agreed upon location
for writing out cloud.cfg.d directory with contents provided from MAAS.
Add vmtest coverage for UC20 image as well.

To post a comment you must log in.
Revision history for this message
Server Team CI bot (server-team-bot) wrote :

FAILED: Continuous integration, rev:5e3715d5ddaddd02fe85cab0b594aa3d1ac13ce6
https://jenkins.ubuntu.com/server/job/curtin-ci/12/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-vote/2091/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/12//rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
087aaf8... by Ryan Harper

uc20: update location to write cloud-config

https://github.com/snapcore/snapd/pull/8299

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks for this Ryan.

To refresh context on integration testing uc20 and debugging, can you specify the one-liner vmtest run you can perform to beat that into my head.

Revision history for this message
Ryan Harper (raharper) wrote :

Since the images are not published under images.maas.io we have hack we employ on diglett where I've manually downloaded a UC20 image, unpacked it and created the directory paths that published images would, and in this MR, the path is hard-coded.

rm -rf output; IMAGE_DIR=/srv/tmp/rharper/images CURTIN_VMTEST_KEEP_DATA_FAIL=all ./tools/jenkins-runner tests/vmtests/test_ubuntu_core.py:UbuntuCore20TestUbuntuCore

Thanks for the review. I'll update with some changes.

5f2cd54... by Ryan Harper

block-meta: add documentation on paths used with block.get_root_device()

bf7ac11... by Ryan Harper

curthooks: fix typo in ubuntu_core_curthooks comment.

36cbfad... by Ryan Harper

Update core16/core18 docstrings to mention their specific version

27fb624... by Ryan Harper

Rename mock variable to indicate its use

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) wrote :

Thanks for the comments and clarifications here. LGTM

review: Approve
c96a736... by Ryan Harper

Don't enable the vmtest by default, images are not published

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/curtin/commands/block_meta.py b/curtin/commands/block_meta.py
2index b0dcb81..6c6bfc7 100644
3--- a/curtin/commands/block_meta.py
4+++ b/curtin/commands/block_meta.py
5@@ -135,7 +135,14 @@ def write_image_to_disk(source, dev):
6 '--', source['uri'], devnode])
7 util.subp(['partprobe', devnode])
8 udevadm_settle()
9- paths = ["curtin", "system-data/var/lib/snapd"]
10+ # Images from MAAS have well-known/required paths present
11+ # on the rootfs partition. Use these values to select the
12+ # root (target) partition to complete installation.
13+ #
14+ # /curtin -> Most Ubuntu Images
15+ # /system-data/var/lib/snapd -> UbuntuCore 16 or 18
16+ # /snaps -> UbuntuCore20
17+ paths = ["curtin", "system-data/var/lib/snapd", "snaps"]
18 return block.get_root_device([devname], paths=paths)
19
20
21diff --git a/curtin/commands/curthooks.py b/curtin/commands/curthooks.py
22index 3c5d38f..ad90fa1 100644
23--- a/curtin/commands/curthooks.py
24+++ b/curtin/commands/curthooks.py
25@@ -1264,13 +1264,18 @@ def handle_cloudconfig(cfg, base_dir=None):
26
27
28 def ubuntu_core_curthooks(cfg, target=None):
29- """ Ubuntu-Core 16 images cannot execute standard curthooks
30- Instead we copy in any cloud-init configuration to
31- the 'LABEL=writable' partition mounted at target.
32+ """ Ubuntu-Core images cannot execute standard curthooks.
33+ Instead, for core16/18 we copy in any cloud-init configuration to
34+ the 'LABEL=writable' partition mounted at target. For core20, we
35+ write a cloud-config.d directory in the 'ubuntu-seed' location.
36 """
37
38 ubuntu_core_target = os.path.join(target, "system-data")
39 cc_target = os.path.join(ubuntu_core_target, 'etc/cloud/cloud.cfg.d')
40+ if not os.path.exists(ubuntu_core_target): # uc20
41+ ubuntu_core_target = target
42+ cc_target = os.path.join(ubuntu_core_target, 'data', 'etc',
43+ 'cloud', 'cloud.cfg.d')
44
45 cloudconfig = cfg.get('cloudconfig', None)
46 if cloudconfig:
47diff --git a/curtin/distro.py b/curtin/distro.py
48index ed178bd..1f62e7a 100644
49--- a/curtin/distro.py
50+++ b/curtin/distro.py
51@@ -131,10 +131,27 @@ def get_osfamily(target=None):
52
53
54 def is_ubuntu_core(target=None):
55- """Check if Ubuntu-Core specific directory is present at target"""
56+ """Check if any Ubuntu-Core specific directory is present at target"""
57+ return any([is_ubuntu_core_16(target),
58+ is_ubuntu_core_18(target),
59+ is_ubuntu_core_20(target)])
60+
61+
62+def is_ubuntu_core_16(target=None):
63+ """Check if Ubuntu-Core 16 specific directory is present at target"""
64 return os.path.exists(target_path(target, 'system-data/var/lib/snapd'))
65
66
67+def is_ubuntu_core_18(target=None):
68+ """Check if Ubuntu-Core 18 specific directory is present at target"""
69+ return is_ubuntu_core_16(target)
70+
71+
72+def is_ubuntu_core_20(target=None):
73+ """Check if Ubuntu-Core 20 specific directory is present at target"""
74+ return os.path.exists(target_path(target, 'snaps'))
75+
76+
77 def is_centos(target=None):
78 """Check if CentOS specific file is present at target"""
79 return os.path.exists(target_path(target, 'etc/centos-release'))
80diff --git a/tests/unittests/test_commands_block_meta.py b/tests/unittests/test_commands_block_meta.py
81index d7715c0..231b2d1 100644
82--- a/tests/unittests/test_commands_block_meta.py
83+++ b/tests/unittests/test_commands_block_meta.py
84@@ -148,7 +148,7 @@ class TestBlockMetaSimple(CiTestCase):
85 self.mock_subp.assert_has_calls([call(args=wget),
86 call(['partprobe', devnode]),
87 call(['udevadm', 'settle'])])
88- paths = ["curtin", "system-data/var/lib/snapd"]
89+ paths = ["curtin", "system-data/var/lib/snapd", "snaps"]
90 self.mock_block_get_root_device.assert_called_with([devname],
91 paths=paths)
92
93@@ -171,7 +171,7 @@ class TestBlockMetaSimple(CiTestCase):
94 self.mock_subp.assert_has_calls([call(args=wget),
95 call(['partprobe', devnode]),
96 call(['udevadm', 'settle'])])
97- paths = ["curtin", "system-data/var/lib/snapd"]
98+ paths = ["curtin", "system-data/var/lib/snapd", "snaps"]
99 self.mock_block_get_root_device.assert_called_with([devname],
100 paths=paths)
101
102diff --git a/tests/unittests/test_curthooks.py b/tests/unittests/test_curthooks.py
103index 72bb24e..5bc4bc0 100644
104--- a/tests/unittests/test_curthooks.py
105+++ b/tests/unittests/test_curthooks.py
106@@ -879,14 +879,31 @@ class TestUefiRemoveDuplicateEntries(CiTestCase):
107
108
109 class TestUbuntuCoreHooks(CiTestCase):
110+
111+ def _make_uc16(self, target):
112+ ucpath = os.path.join(target, 'system-data', 'var/lib/snapd')
113+ util.ensure_dir(ucpath)
114+ return ucpath
115+
116+ def _make_uc20(self, target):
117+ ucpath = os.path.join(target, 'snaps')
118+ util.ensure_dir(ucpath)
119+ return ucpath
120+
121 def setUp(self):
122 super(TestUbuntuCoreHooks, self).setUp()
123 self.target = None
124
125- def test_target_is_ubuntu_core(self):
126+ def test_target_is_ubuntu_core_16(self):
127+ self.target = self.tmp_dir()
128+ ubuntu_core_path = self._make_uc16(self.target)
129+ self.assertTrue(os.path.isdir(ubuntu_core_path))
130+ is_core = distro.is_ubuntu_core(self.target)
131+ self.assertTrue(is_core)
132+
133+ def test_target_is_ubuntu_core_20(self):
134 self.target = self.tmp_dir()
135- ubuntu_core_path = os.path.join(self.target, 'system-data',
136- 'var/lib/snapd')
137+ ubuntu_core_path = self._make_uc20(self.target)
138 util.ensure_dir(ubuntu_core_path)
139 self.assertTrue(os.path.isdir(ubuntu_core_path))
140 is_core = distro.is_ubuntu_core(self.target)
141@@ -952,6 +969,8 @@ class TestUbuntuCoreHooks(CiTestCase):
142 }
143 }
144 }
145+ uc_cloud = os.path.join(self.target, 'system-data')
146+ util.ensure_dir(uc_cloud)
147 curthooks.ubuntu_core_curthooks(cfg, target=self.target)
148
149 self.assertEqual(len(mock_del_file.call_args_list), 0)
150@@ -964,9 +983,32 @@ class TestUbuntuCoreHooks(CiTestCase):
151 @patch('curtin.util.write_file')
152 @patch('curtin.util.del_file')
153 @patch('curtin.commands.curthooks.handle_cloudconfig')
154+ def test_curthooks_uc20_cloud_config(self, mock_handle_cc, mock_del_file,
155+ mock_write_file):
156+ self.target = self.tmp_dir()
157+ self._make_uc20(self.target)
158+ cfg = {
159+ 'cloudconfig': {
160+ 'file1': {
161+ 'content': "Hello World!\n",
162+ }
163+ }
164+ }
165+ curthooks.ubuntu_core_curthooks(cfg, target=self.target)
166+ self.assertEqual(len(mock_del_file.call_args_list), 0)
167+ cc_path = os.path.join(self.target,
168+ 'data', 'etc', 'cloud', 'cloud.cfg.d')
169+ mock_handle_cc.assert_called_with(cfg.get('cloudconfig'),
170+ base_dir=cc_path)
171+ self.assertEqual(len(mock_write_file.call_args_list), 0)
172+
173+ @patch('curtin.util.write_file')
174+ @patch('curtin.util.del_file')
175+ @patch('curtin.commands.curthooks.handle_cloudconfig')
176 def test_curthooks_net_config(self, mock_handle_cc, mock_del_file,
177 mock_write_file):
178 self.target = self.tmp_dir()
179+ self._make_uc16(self.target)
180 cfg = {
181 'network': {
182 'version': '1',
183@@ -974,12 +1016,12 @@ class TestUbuntuCoreHooks(CiTestCase):
184 'name': 'eth0', 'subnets': [{'type': 'dhcp4'}]}]
185 }
186 }
187+ uc_cloud = os.path.join(self.target, 'system-data')
188 curthooks.ubuntu_core_curthooks(cfg, target=self.target)
189
190 self.assertEqual(len(mock_del_file.call_args_list), 0)
191 self.assertEqual(len(mock_handle_cc.call_args_list), 0)
192- netcfg_path = os.path.join(self.target,
193- 'system-data',
194+ netcfg_path = os.path.join(uc_cloud,
195 'etc/cloud/cloud.cfg.d',
196 '50-curtin-networking.cfg')
197 netcfg = config.dump_config({'network': cfg.get('network')})
198@@ -987,6 +1029,33 @@ class TestUbuntuCoreHooks(CiTestCase):
199 content=netcfg)
200 self.assertEqual(len(mock_del_file.call_args_list), 0)
201
202+ @patch('curtin.util.write_file')
203+ @patch('curtin.util.del_file')
204+ @patch('curtin.commands.curthooks.handle_cloudconfig')
205+ def test_curthooks_uc20_net_config(self, mock_handle_cc, mock_del_file,
206+ mock_write_file):
207+ self.target = self.tmp_dir()
208+ self._make_uc20(self.target)
209+ cfg = {
210+ 'network': {
211+ 'version': '1',
212+ 'config': [{'type': 'physical',
213+ 'name': 'eth0', 'subnets': [{'type': 'dhcp4'}]}]
214+ }
215+ }
216+ uc_cloud = os.path.join(self.target,
217+ 'data', 'etc', 'cloud', 'cloud.cfg.d')
218+ curthooks.ubuntu_core_curthooks(cfg, target=self.target)
219+
220+ self.assertEqual(len(mock_del_file.call_args_list), 0)
221+ self.assertEqual(len(mock_handle_cc.call_args_list), 0)
222+ netcfg_path = os.path.join(uc_cloud,
223+ '50-curtin-networking.cfg')
224+ netcfg = config.dump_config({'network': cfg.get('network')})
225+ mock_write_file.assert_called_with(netcfg_path,
226+ content=netcfg)
227+ self.assertEqual(len(mock_del_file.call_args_list), 0)
228+
229 @patch('curtin.commands.curthooks.futil.write_files')
230 def test_handle_cloudconfig(self, mock_write_files):
231 cc_target = "tmpXXXX/systemd-data/etc/cloud/cloud.cfg.d"
232diff --git a/tests/unittests/test_distro.py b/tests/unittests/test_distro.py
233index dc1038c..c994963 100644
234--- a/tests/unittests/test_distro.py
235+++ b/tests/unittests/test_distro.py
236@@ -193,16 +193,65 @@ class TestDistroInfo(CiTestCase):
237
238 class TestDistroIdentity(CiTestCase):
239
240+ ubuntu_core_os_path_side_effects = [
241+ [True, True, True],
242+ [True, True, False],
243+ [True, False, True],
244+ [True, False, False],
245+ [False, True, True],
246+ [False, True, False],
247+ [False, False, True],
248+ ]
249+
250 def setUp(self):
251 super(TestDistroIdentity, self).setUp()
252 self.add_patch('curtin.distro.os.path.exists', 'mock_os_path')
253
254- def test_is_ubuntu_core(self):
255+ def test_is_ubuntu_core_16(self):
256 for exists in [True, False]:
257 self.mock_os_path.return_value = exists
258- self.assertEqual(exists, distro.is_ubuntu_core())
259+ self.assertEqual(exists, distro.is_ubuntu_core_16())
260 self.mock_os_path.assert_called_with('/system-data/var/lib/snapd')
261
262+ def test_is_ubuntu_core_18(self):
263+ for exists in [True, False]:
264+ self.mock_os_path.return_value = exists
265+ self.assertEqual(exists, distro.is_ubuntu_core_18())
266+ self.mock_os_path.assert_called_with('/system-data/var/lib/snapd')
267+
268+ def test_is_ubuntu_core_is_core20(self):
269+ for exists in [True, False]:
270+ self.mock_os_path.return_value = exists
271+ self.assertEqual(exists, distro.is_ubuntu_core_20())
272+ self.mock_os_path.assert_called_with('/snaps')
273+
274+ def test_is_ubuntu_core_true(self):
275+ side_effects = self.ubuntu_core_os_path_side_effects
276+ for true_effect in side_effects:
277+ self.mock_os_path.side_effect = iter(true_effect)
278+ self.assertTrue(distro.is_ubuntu_core())
279+
280+ expected_calls = [
281+ mock.call('/system-data/var/lib/snapd'),
282+ mock.call('/system-data/var/lib/snapd'),
283+ mock.call('/snaps')]
284+ expected_nr_calls = len(side_effects) * len(expected_calls)
285+ self.assertEqual(expected_nr_calls, self.mock_os_path.call_count)
286+ self.mock_os_path.assert_has_calls(
287+ expected_calls * len(side_effects))
288+
289+ def test_is_ubuntu_core_false(self):
290+ self.mock_os_path.return_value = False
291+ self.assertFalse(distro.is_ubuntu_core())
292+
293+ expected_calls = [
294+ mock.call('/system-data/var/lib/snapd'),
295+ mock.call('/system-data/var/lib/snapd'),
296+ mock.call('/snaps')]
297+ expected_nr_calls = 3
298+ self.assertEqual(expected_nr_calls, self.mock_os_path.call_count)
299+ self.mock_os_path.assert_has_calls(expected_calls)
300+
301 def test_is_centos(self):
302 for exists in [True, False]:
303 self.mock_os_path.return_value = exists
304diff --git a/tests/vmtests/__init__.py b/tests/vmtests/__init__.py
305index b87784b..06b1b48 100644
306--- a/tests/vmtests/__init__.py
307+++ b/tests/vmtests/__init__.py
308@@ -66,6 +66,8 @@ _TOPDIR = None
309
310 UC16_IMAGE = os.path.join(IMAGE_DIR,
311 'ubuntu-core-16/amd64/20170217/root-image.xz')
312+UC20_IMAGE = os.path.join(IMAGE_DIR, ('ubuntu-core-20/amd64/20200304/'
313+ 'ubuntu-core-20-amd64.img.xz'))
314
315
316 def remove_empty_dir(dirpath):
317@@ -670,9 +672,16 @@ class VMBaseClass(TestCase):
318
319 tftype = cls.target_ftype
320 if tftype in ["root-image.xz"]:
321- logger.info('get-testfiles UC16 hack!')
322- target_ftypes = {'root-image.xz': UC16_IMAGE}
323- target_img_verstr = "UbuntuCore 16"
324+ logger.info('get-testfiles UC hack!')
325+ if cls.target_release == 'ubuntu-core-16':
326+ target_ftypes = {'root-image.xz': UC16_IMAGE}
327+ target_img_verstr = "UbuntuCore 16"
328+ elif cls.target_release == 'ubuntu-core-20':
329+ target_ftypes = {'root-image.xz': UC20_IMAGE}
330+ target_img_verstr = "UbuntuCore 20"
331+ else:
332+ raise ValueError(
333+ "Unknown target_release=%s" % cls.target_release)
334 elif cls.target_release == cls.release:
335 target_ftypes = ftypes.copy()
336 target_img_verstr = eph_img_verstr
337@@ -891,7 +900,7 @@ class VMBaseClass(TestCase):
338 if not cls.collect_scripts:
339 cls.collect_scripts = (
340 DEFAULT_COLLECT_SCRIPTS['common'] +
341- DEFAULT_COLLECT_SCRIPTS[cls.target_distro])
342+ DEFAULT_COLLECT_SCRIPTS.get(cls.target_distro, []))
343 else:
344 raise RuntimeError('cls collect scripts not empty: %s' %
345 cls.collect_scripts)
346@@ -1077,7 +1086,10 @@ class VMBaseClass(TestCase):
347 disks.extend(cls.build_iscsi_disks())
348
349 # class config file and vmtest defaults
350- configs = [cls.conf_file, 'examples/tests/vmtest_defaults.yaml']
351+ configs = [cls.conf_file]
352+ if cls.target_distro not in ['ubuntu-core']:
353+ configs.append('examples/tests/vmtest_defaults.yaml')
354+
355 # proxy config
356 cls.proxy = get_apt_proxy()
357 if cls.proxy is not None and not cls.td.restored:
358@@ -1927,7 +1939,10 @@ class VMBaseClass(TestCase):
359
360 def has_storage_config(self):
361 '''check if test used storage config'''
362- return len(self.get_storage_config()) > 0
363+ try:
364+ return len(self.get_storage_config()) > 0
365+ except FileNotFoundError:
366+ return False
367
368 @skip_if_flag('expected_failure')
369 def test_swaps_used(self):
370diff --git a/tests/vmtests/releases.py b/tests/vmtests/releases.py
371index bf8be2f..629b96e 100644
372--- a/tests/vmtests/releases.py
373+++ b/tests/vmtests/releases.py
374@@ -26,7 +26,7 @@ class _CentosFromUbuntuBase(_UbuntuBase):
375
376 class _UbuntuCoreUbuntuBase(_UbuntuBase):
377 # base for installing UbuntuCore root-image.xz from ubuntu base
378- target_distro = "ubuntu-core-16"
379+ target_distro = "ubuntu-core"
380 target_ftype = "root-image.xz"
381 kflavor = None
382
383@@ -57,7 +57,21 @@ class _UbuntuCore16FromXenialBase(_UbuntuCoreUbuntuBase):
384 release = "xenial"
385 # release for target
386 target_release = "ubuntu-core-16"
387- target_distro = "ubuntu-core"
388+
389+
390+class _UbuntuCore18FromBionicBase(_UbuntuCoreUbuntuBase):
391+ # release for boot
392+ release = "bionic"
393+ # release for target
394+ target_release = "ubuntu-core-18"
395+
396+
397+class _UbuntuCore20FromFocalBase(_UbuntuCoreUbuntuBase):
398+ # release for boot
399+ release = "focal"
400+ # release for target
401+ target_release = "ubuntu-core-20"
402+ mem = "2048"
403
404
405 class _Centos66FromXenialBase(_CentosFromUbuntuBase):
406@@ -201,6 +215,8 @@ class _CentosReleases(object):
407
408 class _UbuntuCoreReleases(object):
409 uc16fromxenial = _UbuntuCore16FromXenialBase
410+ uc18frombionic = _UbuntuCore18FromBionicBase
411+ uc20fromfocal = _UbuntuCore20FromFocalBase
412
413
414 base_vm_classes = _Releases
415diff --git a/tests/vmtests/test_ubuntu_core.py b/tests/vmtests/test_ubuntu_core.py
416index a282940..bd62175 100644
417--- a/tests/vmtests/test_ubuntu_core.py
418+++ b/tests/vmtests/test_ubuntu_core.py
419@@ -18,6 +18,7 @@ class TestUbuntuCoreAbs(VMBaseClass):
420 cp -a /etc/cloud ./etc_cloud |:
421 cp -a /home . |:
422 cp -a /var/lib/extrausers . |:
423+ find /boot > ./boot.files
424
425 exit 0
426 """)]
427@@ -44,4 +45,12 @@ class TestUbuntuCoreAbs(VMBaseClass):
428 class UbuntuCore16TestUbuntuCore(relbase.uc16fromxenial, TestUbuntuCoreAbs):
429 __test__ = False
430
431+
432+class UbuntuCore20TestUbuntuCore(relbase.uc20fromfocal, TestUbuntuCoreAbs):
433+ uefi = True
434+ __test__ = False
435+ mem = 2048
436+ nr_cpus = 2
437+
438+
439 # vi: ts=4 expandtab syntax=python

Subscribers

People subscribed via source and target branches