Merge ~smoser/curtin:feature/bug-1746348-fsimage-support into curtin:master

Proposed by Scott Moser
Status: Merged
Merge reported by: Ryan Harper
Merged at revision: f0f1bcf227b1022c361f53b1b27fb35c0b802b93
Proposed branch: ~smoser/curtin:feature/bug-1746348-fsimage-support
Merge into: curtin:master
Diff against target: 583 lines (+256/-81)
8 files modified
curtin/commands/extract.py (+59/-31)
curtin/url_helper.py (+94/-4)
curtin/util.py (+8/-1)
doc/topics/config.rst (+3/-0)
tests/vmtests/__init__.py (+49/-35)
tests/vmtests/releases.py (+13/-0)
tests/vmtests/test_uefi_basic.py (+14/-0)
tools/launch (+16/-10)
Reviewer Review Type Date Requested Status
Lee Trager (community) Approve
Server Team CI bot continuous-integration Approve
Ryan Harper Approve
Review via email: mp+336872@code.launchpad.net

Commit message

Add support for installing sources that are a filesystem image.

This adds support for sources that are a 'filesystem image' via local
file path or via http or https. It will work for any filesystem
image type that is mountable by the running kernel.

The end result is that this works:
  sudo ./bin/curtin install --config=my.yaml \
    http://cloud-images.ubuntu.com/..../some.squashfs

To accomplish this:
 a.) if the source is a http/https url, then download it to temporary file.
 b.) mount the file loopback (mount -o loop,ro).
 c.) copy the contents to the target
 d.) unmount
 e.) if downloaded remove the file.

In order to do this we needed some mechanism for downloading a url
in pieces rather than all into memory as 'geturl' uses. So I've
added UrlReader and 'download'.

Also here, a fix for the default headers to include the actual curtin
version per version.version_string() rather than '.1'.

MAAS 2.3 and 2.4 plan to use this functionality to install Ubuntu 12.04
while booted in 16.04. There are two tests added that excercise that
path that will be enabled later. To support testing hwe here,
also have added explicit setting of target_kernel_package in the class.

LP: #1746348

Description of the change

I was able to test with the proposed precise squashfs stream:
 https://code.launchpad.net/~smoser/curtin/+git/curtin/+merge/337080

$ export IMAGE_DIR=/tmp/my-image-dir
$ python3 tests/vmtests/image_sync.py mirror --max=1 --verbose --arches=amd64 --source=http://people.canonical.com/~rcj/mpmirror/streams/v1/index.sjson --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg "$IMAGE_DIR" "release~(precise|xenial)"
$ ./tools/jenkins-runner tests/vmtests/test_uefi_basic.py

To post a comment you must log in.
Revision history for this message
Scott Moser (smoser) wrote :

no tests yet, but tested installation using both url and file from xenial squashfs image on http://cloud-images.ubuntu.com as described in
https://gist.github.com/smoser/375123ef1ef098be23cc856a5772c5c8

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
fb9128f... by Scott Moser

tools/launch: support --publish to a subdir

This allows:
  --publish=/my/path/to/image:somedir/somefile

Before you could only publish to the top level PUBDIR.
That issue would be exposed if you did:
  --publish=/path/to/boot/squashfs
  --publish=/path/to/target/squashfs

the second would silently override the first.
It still does, but now you can put into subdirs.

88bc44c... by Scott Moser

vmtest: Enablement of precise installations using xenial boot.

This works a bit on vmtest base class to more cleanly support
installations of one OS while booted into another.

Also enables some one test of Precise to verify.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

The goal of this merge is to enable installation of a Precise squashfs
while booting either trusty or xenial. Since we do not currently have
a precise squashfs I instead worked on installing a trusty squashfs image
while booted in xenial.

The test I re-enabled is called PreciseUefiTestBasic.

It currently installs fine but fails boot.
The issue can be seen in the boot log at
  http://paste.ubuntu.com/26498529/ .
| [ 3.055946] SGI XFS with ACLs, security attributes, realtime, large block/inode numbers, no debug enabled
| [ 3.062047] XFS (vda3): Version 5 superblock detected. This kernel has EXPERIMENTAL support enabled!
| [ 3.062047] Use of these features in this kernel is at your own risk!
| [ 3.063697] XFS (vda3): Superblock has unknown read-only compatible features (0x1) enabled.
| [ 3.064568] XFS (vda3): Attempted to mount read-only compatible filesystem read-write.
| [ 3.064568] Filesystem can only be safely mounted read only.
| [ 3.068021] XFS (vda3): SB validate failed with error 22.

The issue here I believe is that Xenial's mkfs.xfs enabled some features
by default that were not supported in the Trusty (3.13) kernel. So the
install works fine, but when the target tries to boot it fails. We are
shielded from such mismatch issues currently as we booting and installing
the same release, so the default mkfs options match.

I can't be sure yet whether or not this would be a problem when booting
trusty and installing precise, but it is a general issue that could
definitely show itself.

To validate, the following "fixes" the boot problem:

--- a/examples/tests/uefi_basic.yaml
+++ b/examples/tests/uefi_basic.yaml
@@ -49,7 +49,7 @@ storage:
   - id: id_home_format
     label: home
     type: format
- fstype: xfs
+ fstype: ext4
     volume: id_disk0_part3

Revision history for this message
Scott Moser (smoser) wrote :

another option...
 http://paste.ubuntu.com/26498603/

basically just allowing the storage_config to pass an array of options taht will go into the mkfs command.

that also produced actually booting system.

a85f8b7... by Scott Moser

move downloading to a url_helper.download method that adds speed info.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

OK, I'm happy with the current state of this branch for landing.
It doesn't handle the issue discussed above, but that is really
not related to support for installing a filesystem image.

The 'download' currently gives no status, but at the end does have a nice
message like:

[ 51.934046] cloud-init[1254]: Downloaded 287375360 bytes from http://10.245.1
68.20:35549/target/squashfs to /tmp/tmpgqud66pl.img in 1.57s (174.35Mbps)

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Chad Smith (chad.smith) :
Revision history for this message
Ryan Harper (raharper) wrote :

The extract logic currently is like:

dd-
  continue

cp://
  copy_to_target

isfile:
  extract_root_tgz_file

file://
  extract_root_tgz_file

http[s]://
  extract_root_tgz_url

Can we just remove _tgz_ from the method name,
and add the squashfs handling in there? Something like

  extract_root_from_file
  extract_root_from_url

Then we could push some of the logic for source and format hanlding into those two functions.
Something like this:

    if source['type'].startswith('dd-'):
        continue
    if source['uri'].startswith("cp://"):
        copy_to_target(source['uri'], target)
    elif os.path.isfile(source['uri']) or source['uri'].startswith("file://"):
        extract_root_from_file(source['uri'], fmt=source['type'],
                                target=target)
    elif (source['uri'].startswith("http://") or
          source['uri'].startswith("https://")):
        extract_root_from_url(source['uri'], fmt=source['type'],
                              target=target)
    else:
        raise TypeError("do not know how to extract '%s'" % source['uri'])

def extract_root_from_file(source, fmt=None, target=None):
    if source.startswith('file://'):
        source = source[len('file://'):]
    if fmt == "fsimage":
        return extract_root_fsimage(source, target)
    else:
        curtin.util.subp(args=['tar', '-C', target] +
                         tar_xattr_opts() +
                         ['-Sxpzf', source, '--numeric-owner'])

def extract_root_from_url(source, fmt=None, target=None):
    if fmt == "fsimage":
        return extract_root_fsimage_url(source, target)
    else:
        curtin.util.subp(args=['sh', '-cf',
                           ('wget "$1" --progress=dot:mega -O - |'
                            'smtar -C "$2" ' + ' '.join(tar_xattr_opts()) +
                            ' ' + '-Sxpf - --numeric-owner'),
                           '--', source, target])

Revision history for this message
Scott Moser (smoser) :
Revision history for this message
Scott Moser (smoser) :
3541263... by Scott Moser

address some feedback from chad.

20e2684... by Scott Moser

explicitly set some things in PreciseBase

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
8af54f1... by Scott Moser

add returns to extract_root_tgz calls

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Needs Fixing (continuous-integration)
dd224f0... by Scott Moser

support specifying kernel package.

The get_kernel_config didn't work for precise kernel names.
  linux-generic .. linux-generic-lts-trusty
So just specify them manually.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
1613ee0... by Scott Moser

address feedback / simplify

e900c89... by Scott Moser

add urls that are just path, not file://<path> .

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Scott Moser (smoser) wrote :

I've addressed the feedback i think. i haven't tested, but plealse re-review.

f0525d8... by Scott Moser

disable PreciseUefi tests. Later enable when they work.

Revision history for this message
Ryan Harper (raharper) wrote :

This looks great, just a few questions/comments in line.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Lee Trager (ltrager) wrote :

The Precise image currently in the stream is a gzip compressed ext4 image. It seems this supports SquashFS files by mounting them and coping the files. Why not support mounting the Precise ext4 image so the streams don't have to be modified to add a SquashFS image?

If Curtin is supporting SquashFS why not use unsquashfs and skip mounting it?

review: Needs Information
Revision history for this message
Scott Moser (smoser) wrote :

> The Precise image currently in the stream is a gzip compressed ext4 image. It
> seems this supports SquashFS files by mounting them and coping the files. Why
> not support mounting the Precise ext4 image so the streams don't have to be
> modified to add a SquashFS image?
>

We could support that, but I'd rather not. We'd have to download the
image (possibly decompressing as we read) and then mount it. That image
is 1.4G so we'd then be using 1.4G of tmpfs space. It seems
generally superior to make the squashfs image availalbe in the
stream.. Then we have
 - consistency for all releases in v3 stream
 - less memory required

> If Curtin is supporting SquashFS why not use unsquashfs and skip
> mounting it?

a.) That would certainly be one path, however it then requires a package
to do what the kernel can do just as well. We are guaranteed root to
run curtin so this seems more straight forward. We also then are able
to re-use the 'cp://' handler as we're mounting and copying from
an existing directory.

b.) the non-unsquashfs path supports *any* filesystem image that the
kernel booted can mount.

What is wrong with mounting an image?

If squashfs was "streamable" then you'd have a much better argument.
But to my knowledge it is not. Ie, you have to download the whole
image before you can start 'unsquashfs'. If we could
 wget <url> -O - | unsquashfs --cd /target/dir

then that'd be nice. I dont think the squashfs filesystem format
would really allow it though.

Revision history for this message
Scott Moser (smoser) wrote :

I put up
 https://code.launchpad.net/~smoser/curtin/+git/curtin/+merge/337192
which removes xfs from the uefi basic test.
that will solve/avoid the problems i've discussed above.

the goal is to later enable a 'FileSystemStress' sort of test that
would verify support for each filesystem type and could be run
with precise installs from xenial.

fe3fe29... by Scott Moser

disable PreciseUefi tests. Later enable when they work.

Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
1e53905... by Scott Moser

change installation of PreciseBase to use precise.

e7d6656... by Scott Moser

fix class name

7bb6fa1... by Scott Moser

sanitize_source: do not ever call 'squashfs' an image type

Revision history for this message
Ryan Harper (raharper) wrote :

I think the commit message should clarify fsimage means squasfs-based image; especially regarding the comment about other image types.

If we want to keep fsimage as generic (it *could* be used to install the ext4 root-image) then maybe we should add a test (even if that's not what we'd use in practice). Thoughts?

Some more questions and items inline.

Revision history for this message
Scott Moser (smoser) wrote :

fsimage does not mean 'squashfs-based image'.
it means filesystem image.

I'd like to keep it 'fsimage' as that is what is implemented.

Revision history for this message
Chad Smith (chad.smith) :
e685e5a... by Scott Moser

mention other file system types than squash

51b13c9... by Scott Moser

avoid valueerror if precise was not in the list

31c1664... by Scott Moser

update download per feedback.

e4cf526... by Scott Moser

raise OSerror rather than runtimeerror.

f0f1bcf... by Scott Moser

address suggested change for _path_from_file usage.

Revision history for this message
Ryan Harper (raharper) wrote :

One more question on the curtin.util source sanitize w.r.t auto converting squash to fsimage types.

Revision history for this message
Scott Moser (smoser) :
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Ryan Harper (raharper) wrote :

I'm ok with leaving it in.

review: Approve
Revision history for this message
Server Team CI bot (server-team-bot) wrote :
review: Approve (continuous-integration)
Revision history for this message
Lee Trager (ltrager) wrote :

Thanks for the explanation!

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/curtin/commands/extract.py b/curtin/commands/extract.py
2index b9f1baf..69a9d18 100644
3--- a/curtin/commands/extract.py
4+++ b/curtin/commands/extract.py
5@@ -2,12 +2,14 @@
6
7 import os
8 import sys
9+import tempfile
10
11 import curtin.config
12 from curtin.log import LOG
13-import curtin.util
14+from curtin import util
15 from curtin.futil import write_files
16 from curtin.reporter import events
17+from curtin import url_helper
18
19 from . import populate_one_subcmd
20
21@@ -31,27 +33,56 @@ def tar_xattr_opts(cmd=None):
22 if isinstance(cmd, str):
23 cmd = [cmd]
24
25- (out, _err) = curtin.util.subp(cmd + ['--help'], capture=True)
26+ (out, _err) = util.subp(cmd + ['--help'], capture=True)
27
28 if "xattr" in out:
29 return ['--xattrs', '--xattrs-include=*']
30 return []
31
32
33-def extract_root_tgz_url(source, target):
34+def extract_root_tgz_url(url, target):
35 # extract a -root.tar.gz url in the 'target' directory
36- #
37- # Uses smtar to avoid specifying the compression type
38- curtin.util.subp(args=['sh', '-cf',
39- ('wget "$1" --progress=dot:mega -O - |'
40- 'smtar -C "$2" ' + ' '.join(tar_xattr_opts()) +
41- ' ' + '-Sxpf - --numeric-owner'),
42- '--', source, target])
43-
44+ path = _path_from_file_url(url)
45+ if path != url or os.path.isfile(path):
46+ util.subp(args=['tar', '-C', target] + tar_xattr_opts() +
47+ ['-Sxpzf', path, '--numeric-owner'])
48+ return
49
50-def extract_root_tgz_file(source, target):
51- curtin.util.subp(args=['tar', '-C', target] +
52- tar_xattr_opts() + ['-Sxpzf', source, '--numeric-owner'])
53+ # Uses smtar to avoid specifying the compression type
54+ util.subp(args=['sh', '-cf',
55+ ('wget "$1" --progress=dot:mega -O - |'
56+ 'smtar -C "$2" ' + ' '.join(tar_xattr_opts()) +
57+ ' ' + '-Sxpf - --numeric-owner'),
58+ '--', url, target])
59+
60+
61+def extract_root_fsimage_url(url, target):
62+ path = _path_from_file_url(url)
63+ if path != url or os.path.isfile(path):
64+ return _extract_root_fsimage(path(url), target)
65+
66+ wfp = tempfile.NamedTemporaryFile(suffix=".img", delete=False)
67+ wfp.close()
68+ try:
69+ url_helper.download(url, wfp.name)
70+ return _extract_root_fsimage(wfp.name, target)
71+ finally:
72+ os.unlink(wfp.name)
73+
74+
75+def _extract_root_fsimage(path, target):
76+ mp = tempfile.mkdtemp()
77+ try:
78+ util.subp(['mount', '-o', 'loop,ro', path, mp], capture=True)
79+ except util.ProcessExecutionError as e:
80+ LOG.error("Failed to mount '%s' for extraction: %s", path, e)
81+ os.rmdir(mp)
82+ raise e
83+ try:
84+ return copy_to_target(mp, target)
85+ finally:
86+ util.subp(['umount', mp])
87+ os.rmdir(mp)
88
89
90 def copy_to_target(source, target):
91@@ -59,17 +90,21 @@ def copy_to_target(source, target):
92 source = source[5:]
93 source = os.path.abspath(source)
94
95- curtin.util.subp(args=['sh', '-c',
96- ('mkdir -p "$2" && cd "$2" && '
97- 'rsync -aXHAS --one-file-system "$1/" .'),
98- '--', source, target])
99+ util.subp(args=['sh', '-c',
100+ ('mkdir -p "$2" && cd "$2" && '
101+ 'rsync -aXHAS --one-file-system "$1/" .'),
102+ '--', source, target])
103+
104+
105+def _path_from_file_url(url):
106+ return url[7:] if url.startswith("file://") else url
107
108
109 def extract(args):
110 if not args.target:
111 raise ValueError("Target must be defined or set in environment")
112
113- state = curtin.util.load_command_environment()
114+ state = util.load_command_environment()
115 cfg = curtin.config.load_command_config(args, state)
116
117 sources = args.sources
118@@ -82,6 +117,8 @@ def extract(args):
119 if isinstance(sources, dict):
120 sources = [sources[k] for k in sorted(sources.keys())]
121
122+ sources = [util.sanitize_source(s) for s in sources]
123+
124 LOG.debug("Installing sources: %s to target at %s" % (sources, target))
125 stack_prefix = state.get('report_stack_prefix', '')
126
127@@ -94,19 +131,10 @@ def extract(args):
128 continue
129 if source['uri'].startswith("cp://"):
130 copy_to_target(source['uri'], target)
131- elif os.path.isfile(source['uri']):
132- extract_root_tgz_file(source['uri'], target)
133- elif source['uri'].startswith("file://"):
134- extract_root_tgz_file(
135- source['uri'][len("file://"):],
136- target)
137- elif (source['uri'].startswith("http://") or
138- source['uri'].startswith("https://")):
139- extract_root_tgz_url(source['uri'], target)
140+ elif source['type'] == "fsimage":
141+ extract_root_fsimage_url(source['uri'], target=target)
142 else:
143- raise TypeError(
144- "do not know how to extract '%s'" %
145- source['uri'])
146+ extract_root_tgz_url(source['uri'], target=target)
147
148 if cfg.get('write_files'):
149 LOG.info("Applying write_files from config.")
150diff --git a/curtin/url_helper.py b/curtin/url_helper.py
151index 75ea633..0221ad5 100644
152--- a/curtin/url_helper.py
153+++ b/curtin/url_helper.py
154@@ -9,6 +9,8 @@ import time
155 import uuid
156 from functools import partial
157
158+from curtin import version
159+
160 try:
161 from urllib import request as _u_re # pylint: disable=no-name-in-module
162 from urllib import error as _u_e # pylint: disable=no-name-in-module
163@@ -25,6 +27,8 @@ from .log import LOG
164
165 error = urllib_error
166
167+DEFAULT_HEADERS = {'User-Agent': 'Curtin/' + version.version_string()}
168+
169
170 class _ReRaisedException(Exception):
171 exc = None
172@@ -34,14 +38,100 @@ class _ReRaisedException(Exception):
173 self.exc = exc
174
175
176-def _geturl(url, headers=None, headers_cb=None, exception_cb=None, data=None):
177- def_headers = {'User-Agent': 'Curtin/0.1'}
178+class UrlReader(object):
179+ fp = None
180+
181+ def __init__(self, url, headers=None, data=None):
182+ headers = _get_headers(headers)
183+ self.url = url
184+ try:
185+ req = urllib_request.Request(url=url, data=data, headers=headers)
186+ self.fp = urllib_request.urlopen(req)
187+ except urllib_error.HTTPError as exc:
188+ raise UrlError(exc, code=exc.code, headers=exc.headers, url=url,
189+ reason=exc.reason)
190+ except Exception as exc:
191+ raise UrlError(exc, code=None, headers=None, url=url,
192+ reason="unknown")
193+
194+ self.info = self.fp.info()
195+ self.size = self.info.get('content-length', -1)
196+
197+ def read(self, buflen):
198+ try:
199+ return self.fp.read(buflen)
200+ except urllib_error.HTTPError as exc:
201+ raise UrlError(exc, code=exc.code, headers=exc.headers,
202+ url=self.url, reason=exc.reason)
203+ except Exception as exc:
204+ raise UrlError(exc, code=None, headers=None, url=self.url,
205+ reason="unknown")
206+
207+ def close(self):
208+ if not self.fp:
209+ return
210+ try:
211+ self.fp.close()
212+ finally:
213+ self.fp = None
214
215+ def __enter__(self):
216+ return self
217+
218+ def __exit__(self, etype, value, trace):
219+ self.close()
220+
221+
222+def download(url, path, reporthook=None, data=None):
223+ """Download url to path.
224+
225+ reporthook is compatible with py3 urllib.request.urlretrieve.
226+ urlretrieve does not exist in py2."""
227+
228+ buflen = 8192
229+ wfp = open(path, "wb")
230+
231+ try:
232+ buf = None
233+ blocknum = 0
234+ fsize = 0
235+ start = time.time()
236+ with UrlReader(url) as rfp:
237+ if reporthook:
238+ reporthook(blocknum, buflen, rfp.size)
239+
240+ while True:
241+ buf = rfp.read(buflen)
242+ if not buf:
243+ break
244+ blocknum += 1
245+ if reporthook:
246+ reporthook(blocknum, buflen, rfp.size)
247+ rlen = len(buf)
248+ wlen = wfp.write(buf)
249+ if rlen != wlen:
250+ raise OSError(
251+ "Short write to %s. Tried write of %d bytes "
252+ "but wrote only %d" % (path, rlen, wlen))
253+ fsize += rlen
254+ timedelta = time.time() - start
255+ LOG.debug("Downloaded %d bytes from %s to %s in %.2fs (%.2fMbps)",
256+ fsize, url, path, timedelta, fsize / timedelta / 1024 / 1024)
257+ return path, rfp.info
258+ finally:
259+ wfp.close()
260+
261+
262+def _get_headers(headers=None):
263+ allheaders = DEFAULT_HEADERS.copy()
264 if headers is not None:
265- def_headers.update(headers)
266+ allheaders.update(headers)
267+ return allheaders
268
269- headers = def_headers
270
271+def _geturl(url, headers=None, headers_cb=None, exception_cb=None, data=None):
272+
273+ headers = _get_headers(headers)
274 if headers_cb:
275 headers.update(headers_cb(url))
276
277diff --git a/curtin/util.py b/curtin/util.py
278index d343339..12a5446 100644
279--- a/curtin/util.py
280+++ b/curtin/util.py
281@@ -1084,13 +1084,20 @@ def sanitize_source(source):
282 # already sanitized?
283 return source
284 supported = ['tgz', 'dd-tgz', 'dd-tbz', 'dd-txz', 'dd-tar', 'dd-bz2',
285- 'dd-gz', 'dd-xz', 'dd-raw']
286+ 'dd-gz', 'dd-xz', 'dd-raw', 'fsimage']
287 deftype = 'tgz'
288 for i in supported:
289 prefix = i + ":"
290 if source.startswith(prefix):
291 return {'type': i, 'uri': source[len(prefix):]}
292
293+ # translate squashfs: to fsimage type.
294+ if source.startswith("squashfs:"):
295+ return {'type': 'fsimage', 'uri': source[len("squashfs:")]}
296+
297+ if source.endswith("squashfs") or source.endswith("squash"):
298+ return {'type': 'fsimage', 'uri': source}
299+
300 LOG.debug("unknown type for url '%s', assuming type '%s'", source, deftype)
301 # default to tgz for unknown types
302 return {'type': deftype, 'uri': source}
303diff --git a/doc/topics/config.rst b/doc/topics/config.rst
304index 0f11b53..fdc524f 100644
305--- a/doc/topics/config.rst
306+++ b/doc/topics/config.rst
307@@ -435,6 +435,9 @@ configures the method used to copy the image to the target system.
308 - **cp://**: Use ``rsync`` command to copy source directory to target.
309 - **file://**: Use ``tar`` command to extract source to target.
310 - **http[s]://**: Use ``wget | tar`` commands to extract source to target.
311+- **fsimage://**: mount filesystem image and copy contents to target.
312+ Local file or url are supported. Filesystem can be any filesystem type
313+ mountable by the running kernel.
314
315 **Example Cloud-image**::
316
317diff --git a/tests/vmtests/__init__.py b/tests/vmtests/__init__.py
318index f6515af..60882a2 100644
319--- a/tests/vmtests/__init__.py
320+++ b/tests/vmtests/__init__.py
321@@ -392,6 +392,7 @@ class VMBaseClass(TestCase):
322 target_release = None
323 target_krel = None
324 target_ftype = "squashfs"
325+ target_kernel_package = None
326
327 _debian_packages = None
328
329@@ -414,31 +415,32 @@ class VMBaseClass(TestCase):
330 subarch=cls.subarch if cls.subarch else None,
331 sync=CURTIN_VMTEST_IMAGE_SYNC,
332 ftypes=('boot-initrd', 'boot-kernel', cls.ephemeral_ftype))
333- logger.debug("Install Image %s\n, ftypes: %s\n",
334- eph_img_verstr, ftypes)
335+
336 if not cls.target_krel and cls.krel:
337 cls.target_krel = cls.krel
338
339- # get local absolute filesystem paths for the OS tarball to be
340- # installed
341- if cls.target_ftype in ["vmtest.root-tgz", "squashfs"]:
342- target_img_verstr, found = get_images(
343+ tftype = cls.target_ftype
344+ if tftype in ["root-image.xz"]:
345+ logger.info('get-testfiles UC16 hack!')
346+ target_ftypes = {'root-image.xz': UC16_IMAGE}
347+ target_img_verstr = "UbuntuCore 16"
348+ elif cls.target_release == cls.release:
349+ target_ftypes = ftypes.copy()
350+ target_img_verstr = eph_img_verstr
351+ else:
352+ target_img_verstr, target_ftypes = get_images(
353 IMAGE_SRC_URL, IMAGE_DIR,
354 cls.target_distro if cls.target_distro else cls.distro,
355 cls.target_release if cls.target_release else cls.release,
356 cls.arch, subarch=cls.subarch if cls.subarch else None,
357 kflavor=cls.kflavor if cls.kflavor else None,
358 krel=cls.target_krel, sync=CURTIN_VMTEST_IMAGE_SYNC,
359- ftypes=(cls.target_ftype,))
360- logger.debug("Target Tarball %s\n, ftypes: %s\n",
361- target_img_verstr, found)
362- elif cls.target_ftype in ["root-image.xz"]:
363- logger.info('get-testfiles UC16 hack!')
364- found = {'root-image.xz': UC16_IMAGE}
365- target_img_verstr = "UbuntuCore 16"
366- logger.info("Ephemeral Image Version:[%s] Target Image Version:[%s]",
367- eph_img_verstr, target_img_verstr)
368- ftypes.update(found)
369+ ftypes=(tftype,))
370+
371+ ftypes["target/%s" % tftype] = target_ftypes[tftype]
372+ logger.debug(
373+ "Install Image Version = %s\n, Target Image Version = %s\n"
374+ "ftypes: %s", eph_img_verstr, target_img_verstr, ftypes)
375 return ftypes
376
377 @classmethod
378@@ -621,6 +623,9 @@ class VMBaseClass(TestCase):
379 @classmethod
380 def get_kernel_package(cls):
381 """ Return the kernel package name for this class """
382+ if cls.target_kernel_package:
383+ return cls.target_kernel_package
384+
385 package = 'linux-image-' + cls.kflavor
386 if cls.subarch is None or cls.subarch.startswith('ga'):
387 return package
388@@ -634,6 +639,9 @@ class VMBaseClass(TestCase):
389
390 Returns: config dictionary only for hwe, or edge subclass
391 """
392+ if cls.target_kernel_package:
393+ return {'kernel': {'package': cls.target_kernel_package}}
394+
395 if cls.subarch is None or cls.subarch.startswith('ga'):
396 return None
397
398@@ -689,12 +697,12 @@ class VMBaseClass(TestCase):
399 cmd.extend(["--append=" + cls.extra_kern_args])
400
401 ftypes = cls.get_test_files()
402+ root_pubpath = "root/" + cls.ephemeral_ftype
403 # trusty can't yet use root=URL due to LP:#1735046
404 if cls.release in ['trusty']:
405 root_url = "/dev/disk/by-id/virtio-boot-disk"
406 else:
407- root_url = "squash:PUBURL/%s" % (
408- os.path.basename(ftypes[cls.ephemeral_ftype]))
409+ root_url = "squash:PUBURL/" + root_pubpath
410 # configure ephemeral boot environment
411 cmd.extend([
412 "--root-arg=root=%s" % root_url,
413@@ -710,16 +718,17 @@ class VMBaseClass(TestCase):
414 cmd.extend(["--append=iscsi_auto"])
415
416 # publish the ephemeral image (used in root=URL)
417- cmd.append("--publish=%s" % ftypes[cls.ephemeral_ftype])
418+ cmd.append("--publish=%s:%s" % (ftypes[cls.ephemeral_ftype],
419+ root_pubpath))
420 logger.info("Publishing ephemeral image as %s", cmd[-1])
421- # publish the target image
422- cmd.append("--publish=%s" % ftypes[cls.target_ftype])
423- logger.info("Publishing target image as %s", cmd[-1])
424
425 # set curtin install source
426- install_src = cls.get_install_source(ftypes)
427-
428+ install_src, publishes = cls.get_install_source(ftypes)
429 logger.info("Curtin install source URI: %s", install_src)
430+ if len(publishes):
431+ cmd.extend(['--publish=%s' % p for p in publishes])
432+ logger.info("Publishing install sources: %s",
433+ cmd[-len(publishes):])
434
435 # check for network configuration
436 cls.network_state = curtin_net.parse_net_config(cls.conf_file)
437@@ -1082,19 +1091,21 @@ class VMBaseClass(TestCase):
438
439 @classmethod
440 def get_install_source(cls, ftypes):
441- if cls.target_ftype == 'squashfs':
442- # If we're installing from squashfs source then direct
443- # curtin to install from the read-only undermount
444+ """Return install uri and a list of files needed to be published."""
445+ # if release (install environment) is the same as target
446+ # target (thing to install) then install via cp://
447+ if cls.release == cls.target_release:
448 install_src = "cp:///media/root-ro"
449- else:
450- if cls.target_ftype == 'root-image.xz':
451- stype = "dd-xz"
452- else:
453- stype = "tgz"
454- src = os.path.basename(ftypes[cls.target_ftype])
455- install_src = "%s:PUBURL/%s" % (stype, src)
456+ return install_src, []
457
458- return install_src
459+ # publish the file to target/<ftype>
460+ # and set the install source to <type>:PUBURL/target/<ftype>
461+ ttype2stype = {'root-image.xz': 'dd-xz', 'squashfs': 'fsimage'}
462+ ftype = cls.target_ftype
463+ pubpath = "target/" + ftype
464+ src = "%s:PUBURL/%s" % (ttype2stype.get(ftype, 'tgz'), pubpath)
465+ publishes = [ftypes["target/" + ftype] + ":" + pubpath]
466+ return src, publishes
467
468 @classmethod
469 def tearDownClass(cls):
470@@ -1711,6 +1722,9 @@ def is_unsupported_ubuntu(release):
471 elif util.which(udi):
472 _UNSUPPORTED_UBUNTU = util.subp(
473 [udi, '--unsupported'], capture=True)[0].splitlines()
474+ # precise ESM support.
475+ if 'precise' in _UNSUPPORTED_UBUNTU:
476+ _UNSUPPORTED_UBUNTU.remove('precise')
477 else:
478 # no way to tell.
479 return None
480diff --git a/tests/vmtests/releases.py b/tests/vmtests/releases.py
481index f365824..7c133db 100644
482--- a/tests/vmtests/releases.py
483+++ b/tests/vmtests/releases.py
484@@ -46,6 +46,17 @@ class _Centos66FromXenialBase(_CentosFromUbuntuBase):
485 target_release = "centos66"
486
487
488+class _PreciseBase(_UbuntuBase):
489+ release = "xenial"
490+ target_release = "precise"
491+ target_distro = "ubuntu"
492+ target_ftype = "squashfs"
493+
494+
495+class _PreciseHWET(_PreciseBase):
496+ target_kernel_package = 'linux-generic-lts-trusty'
497+
498+
499 class _TrustyBase(_UbuntuBase):
500 release = "trusty"
501
502@@ -105,6 +116,8 @@ class _BionicBase(_UbuntuBase):
503
504 class _Releases(object):
505 trusty = _TrustyBase
506+ precise = _PreciseBase
507+ precise_hwe_t = _PreciseHWET
508 trusty_hwe_u = _TrustyHWEU
509 trusty_hwe_v = _TrustyHWEV
510 trusty_hwe_w = _TrustyHWEW
511diff --git a/tests/vmtests/test_uefi_basic.py b/tests/vmtests/test_uefi_basic.py
512index 712075c..d6a58eb 100644
513--- a/tests/vmtests/test_uefi_basic.py
514+++ b/tests/vmtests/test_uefi_basic.py
515@@ -78,6 +78,20 @@ class TestBasicAbs(VMBaseClass):
516 self.assertEqual(self.disk_block_size, size)
517
518
519+class PreciseUefiTestBasic(relbase.precise, TestBasicAbs):
520+ __test__ = False
521+
522+ def test_ptable(self):
523+ print("test_ptable does not work for Precise")
524+
525+ def test_dname(self):
526+ print("test_dname does not work for Precise")
527+
528+
529+class PreciseHWETUefiTestBasic(relbase.precise_hwe_t, PreciseUefiTestBasic):
530+ __test__ = False
531+
532+
533 class TrustyUefiTestBasic(relbase.trusty, TestBasicAbs):
534 __test__ = True
535
536diff --git a/tools/launch b/tools/launch
537index 179dd82..b273de4 100755
538--- a/tools/launch
539+++ b/tools/launch
540@@ -624,7 +624,12 @@ main() {
541 debug 1 "added $ip to existing no_proxy (${no_proxy})";;
542 esac
543
544- local tok tok_split src pub fpath
545+ start_http "${TEMP_D}" "$ip" "$http_port" "${HTTP_TRIES}" </dev/null ||
546+ { error "failed to start http service"; return 1; }
547+ http_port=$_RET
548+ burl="http://$ip:${http_port}"
549+
550+ local tok tok_split src pub fpath rdir
551 # tok in pubs looks like file[:pubname]
552 # link them into the temp dir for publishing
553 for tok in "${pubs[@]}"; do
554@@ -633,19 +638,20 @@ main() {
555 pub=${tok_split[1]}
556 fpath=$(readlink -f "$src") ||
557 { error "'$src': failed to get path"; return 1; }
558- if [ -z "$pub" ]; then
559+ if [ -n "$pub" ]; then
560+ rdir=$(dirname "$pub")
561+ [ -d "${TEMP_D}/$rdir" ] || mkdir -p "${TEMP_D}/$rdir" || {
562+ error "Failed to make <pubdir>/$rdir for publish of $pub";
563+ return 1;
564+ }
565+ else
566 pub="${src##*/}"
567 fi
568- ln -sf "$fpath" "${TEMP_D}/${pub}"
569- debug 1 "publishing: $fpath to ${TEMP_D}/${pub}"
570+ ln -sf "$fpath" "${TEMP_D}/${pub}" ||
571+ { error "failed to link $fpath to <pubdir>/$pub"; return 1; }
572+ debug 1 "publishing: $fpath to $burl/${pub}"
573 done
574
575- start_http "${TEMP_D}" "$ip" "$http_port" "${HTTP_TRIES}" </dev/null ||
576- { error "failed to start http service"; return 1; }
577- http_port=$_RET
578- burl="http://$ip:${http_port}"
579-
580-
581 local addargs="" f=""
582 addargs=( )
583 for f in "${addfiles[@]}"; do

Subscribers

People subscribed via source and target branches