Merge ~danudey/curtin:squashfs_prefix_fix into curtin:master

[Daniel Fox]

Fix parsing of squashfs: uri prefix for installation sources

util.santitize_source was truncating uris which used the squashfs:
prefix by taking the index of a character rather than a slice starting
at that character. This changes fixes this issue and provides the
intended behaviour and adds a deprecation warning to point to
the supported URI: "squashfs://".

LP: #1851271

[Ryan Harper (raharper)]

Hi Daniel,

Thank you for contributing to curtin.

To contribute, you must sign the Canonical Contributor License Agreement (CLA) [1].

If you have already signed it as an individual, your Launchpad user will be listed in the contributor-agreement-canonical launchpad group [2]. Unfortunately there is no easy way to check if an organization or company you are doing work for has signed. If you are unsure or have questions, email <email address hidden> or ping powersj in #curtin channel via freenode.

For information on how to sign, please see the HACKING document [3].

Thanks again, and please feel free to reach out with any questions.

–
[1] http://www.canonical.com/contributors
[2] https://launchpad.net/~contributor-agreement-canonical/+members
[3] http://curtin.readthedocs.io/en/latest/topics/hacking.html

[Ryan Harper (raharper)]

Thanks. This patch looks good. Let's also get a documentation update like so included:

http://paste.ubuntu.com/p/St2pDdBqSH/

review: Needs Fixing

[Daniel Fox (danudey)]

Agreement signed.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:66a7bcea0b7c8fa337d95e33338b3bb15a38b612
https://jenkins.ubuntu.com/server/job/curtin-ci/3779/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3779/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3779/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3779/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3779/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3779//rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Hi Daniel,

Could you pick up the doc patch and apply it to your branch and push that here?

http://paste.ubuntu.com/p/St2pDdBqSH/

review: Needs Information

[Daniel Fox (danudey)]

> Hi Daniel,
>
> Could you pick up the doc patch and apply it to your branch and push that
> here?
>
> http://paste.ubuntu.com/p/St2pDdBqSH/

All done, sorry for the delay!

[Ryan Harper (raharper)]

Thanks!

review: Approve

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
Unapproved changes made after approval.
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/1402/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/682/

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
Unapproved changes made after approval.
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/1403/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/684/

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
Unapproved changes made after approval.
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/1404/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/685/

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

review: Approve (continuous-integration)

[Server Team CI bot (server-team-bot)]

Autolanding: FAILED
Unapproved changes made after approval.
https://jenkins.ubuntu.com/server/job/curtin-autoland-test/1405/
Executed test runs:
    None: https://jenkins.ubuntu.com/server/job/admin-lp-git-autoland/687/

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

I see, you updated this branch to switch to squashfs://

I've an inline comment; we'll need to support both but we
can document the newer, and log a warning with the older.

review: Needs Fixing

688c352... by Daniel Fox <email address hidden> on 2019-11-05

Add back support for `squashfs:` prefix with deprecation warning.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:688c352f5d9fb240ffbbd7f2efc2b328fadfe95e
https://jenkins.ubuntu.com/server/job/curtin-ci/3782/
Executed test runs:
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3782/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3782/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3782/
    FAILURE: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3782/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3782//rebuild

review: Needs Fixing (continuous-integration)

[Ryan Harper (raharper)]

======================================================================
FAIL: Verify squashfs: prefix returns type=fsimage and correct uri.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/jenkins/servers/server/workspace/curtin-ci/nodes/metal-arm64/curtin-3782/tests/unittests/test_util.py", line 1121, in test_supported_squashfs_source_type
    self.assertEqual(expected, result)
AssertionError: {'uri': '/media/filesystem.squashfs', 'type': 'fsimage'} != {'uri': '///media/filesystem.squashfs', 'type': 'fsimage'}
- {'type': 'fsimage', 'uri': '/media/filesystem.squashfs'}
+ {'type': 'fsimage', 'uri': '///media/filesystem.squashfs'}
? ++

You can run this locally with tox

77a9a05... by Daniel Fox <email address hidden> on 2019-11-07

Invert order for deprecated/preferred squashfs prefix

curtin/utils
  - Since squashfs: is a prefix of squashfs:// we need
    to check for the latter before the former to ensure
    we're parsing the right things correctly.

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:77a9a057350eb60aa70fa79bd5491bb47ce32bc0
https://jenkins.ubuntu.com/server/job/curtin-ci/3785/
Executed test runs:
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-arm64/3785/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-ppc64el/3785/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=metal-s390x/3785/
    SUCCESS: https://jenkins.ubuntu.com/server/job/curtin-ci/nodes=torkoal/3785/

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/curtin-ci/3785//rebuild

review: Approve (continuous-integration)

[Ryan Harper (raharper)]

Thanks!

review: Approve

diff --git a/curtin/util.py b/curtin/util.py
index e25402a..eb2228f 100644
--- a/curtin/util.py
+++ b/curtin/util.py
@@ -916,8 +916,14 @@ def sanitize_source(source):
             return {'type': i, 'uri': source[len(prefix):]}
 
     # translate squashfs: to fsimage type.
-    if source.startswith("squashfs:"):
-        return {'type': 'fsimage', 'uri': source[len("squashfs:")]}
+    if source.startswith("squashfs://"):
+        return {'type': 'fsimage', 'uri': source[len("squashfs://"):]}
+
+    elif source.startswith("squashfs:"):
+        LOG.warning("The squashfs: prefix is deprecated and"
+                    "will be removed in a future release."
+                    "Please use squashfs:// instead.")
+        return {'type': 'fsimage', 'uri': source[len("squashfs:"):]}
 
     if source.endswith("squashfs") or source.endswith("squash"):
         return {'type': 'fsimage', 'uri': source}
diff --git a/doc/topics/config.rst b/doc/topics/config.rst
index b7cca43..7aea848 100644
--- a/doc/topics/config.rst
+++ b/doc/topics/config.rst
@@ -555,6 +555,7 @@ configures the method used to copy the image to the target system.
 - **dd-**:  Use ``dd`` command to write image to target.
 - **cp://**: Use ``rsync`` command to copy source directory to target.
 - **file://**: Use ``tar`` command to extract source to target.
+- **squashfs://**: Mount squashfs image and copy contents to target.
 - **http[s]://**: Use ``wget | tar`` commands to extract source to target.
 - **fsimage://** mount filesystem image and copy contents to target.
   Local file or url are supported. Filesystem can be any filesystem type
@@ -649,6 +650,9 @@ This results in Curtin downloading the following URLs::
   sources: 
     - cp:///
 
+**Example squashfs from NFS mount**::
+  sources:
+    - squashfs:///media/filesystem.squashfs
 
 **Example Copy from local tarball**::
 
diff --git a/tests/unittests/test_util.py b/tests/unittests/test_util.py
index 7f05ced..80bb85f 100644
--- a/tests/unittests/test_util.py
+++ b/tests/unittests/test_util.py
@@ -1102,6 +1102,7 @@ class TestSanitizeSource(CiTestCase):
                  'dd-bz2', 'dd-gz', 'dd-xz', 'dd-raw', 'fsimage',
                  'fsimage-layered']
     source_url = 'http://curtin.io/root-fs.foo'
+    squashfs_source_path = "/media/filesystem.squashfs"
 
     def test_supported_sources(self):
         """ Verify supported sources types return expected dictionary. """
@@ -1112,9 +1113,11 @@ class TestSanitizeSource(CiTestCase):
 
     def test_supported_squashfs_source_type(self):
         """ Verify squashfs: prefix returns type=fsimage and correct uri."""
-        stype = 'fsimage'
-        expected = {'type': stype, 'uri': self.source_url}
-        result = util.sanitize_source("%s:%s" % (stype, self.source_url))
+        stype = 'squashfs'
+        target_stype = "fsimage"
+        expected = {'type': target_stype, 'uri': self.squashfs_source_path}
+        result = util.sanitize_source("%s://%s" %
+                                      (stype, self.squashfs_source_path))
         self.assertEqual(expected, result)
 
     def test_supported_squashfs_suffix(self):