Merge lp:~andreserl/maas/use_squashfs_filesystem_2 into lp:maas/trunk

[Andres Rodriguez]

Make use of squashfs when install quantal or up

[Andres Rodriguez (andreserl)]

So Fixed the issue raised by allenap in the code.

The issues about the preseeds evaluating arch... it was discussed with him that symlinking preseeds on arch bases or inheriting would be too much complexity. For now leaving as is.

[Gavin Panella (allenap)]

Looks good :)

As discussed on IRC, there aren't any tests for the templates, but
then there aren't any for the other templates, only general "does
templating work" tests. Because of the confusion with Tempita's
inheritance mechanism, I think we need to revisit this area after
12.10 anyway.

[1]

+    return BootImage.objects.have_image(
+        arch, subarch, node.get_distro_series(), "filesystem")

My one suggestion would be to change "filesystem" to "squashfs" or
something more descriptive.

review: Approve

[Andres Rodriguez (andreserl)]

Hi Gavin,

The squashfs image is the root filesystem image, hence the name. I think it
is appropriate to categorize it that way (as it is now - filesystem).

Cheers
On Oct 2, 2012 3:19 PM, "Andres Rodriguez" <email address hidden> wrote:

> Andres Rodriguez has proposed merging
> lp:~andreserl/maas/use_squashfs_filesystem_2 into lp:maas.
>
> Requested reviews:
> Launchpad code reviewers (launchpad-reviewers)
>
> For more details, see:
>
> https://code.launchpad.net/~andreserl/maas/use_squashfs_filesystem_2/+merge/127577
> --
>
> https://code.launchpad.net/~andreserl/maas/use_squashfs_filesystem_2/+merge/127577
> You are the owner of lp:~andreserl/maas/use_squashfs_filesystem_2.
>
> === modified file 'contrib/maas-http.conf'
> --- contrib/maas-http.conf 2012-09-25 09:48:17 +0000
> +++ contrib/maas-http.conf 2012-10-02 19:13:28 +0000
> @@ -45,6 +45,12 @@
> SetHandler None
> </Directory>
>
> +# Serve squashfs images
> +Alias /MAAS/static/images/ /var/lib/maas/tftp/
> +<Directory /var/lib/maas/tftp/>
> + SetHandler None
> +</Directory>
> +
> # Serve files from staticfiles.
> Alias /MAAS/static/ /usr/share/maas/web/static/
> <Directory /usr/share/maas/web/static/>
>
> === modified file 'contrib/preseeds_v2/generic'
> --- contrib/preseeds_v2/generic 2012-09-21 16:42:19 +0000
> +++ contrib/preseeds_v2/generic 2012-10-02 19:13:28 +0000
> @@ -1,7 +1,17 @@
> {{inherit "preseed_master"}}
> +
> +{{def squashfs_image}}
> +{{if node.distro_series in {'quantal'} and node.architecture in
> {'i386/generic', 'amd64/generic'} and use_squashfs is True }}
> +d-i preseed/early_command string anna-install live-installer
> +d-i live-installer/enable boolean true
> +d-i live-installer/mode select normal
> +d-i live-installer/net-image string http://
> {{server_host}}/MAAS/static/images/{{node.architecture}}/{{node.distro_series}}/filesystem/filesystem.squashfs
> +{{endif}}
> +{{enddef}}
> +
> {{def proxy}}
> d-i mirror/country string manual
> -{{if node.architecture in {'i386', 'amd64'} }}
> +{{if node.architecture in {'i386/generic', 'amd64/generic'} }}
> d-i mirror/http/hostname string archive.ubuntu.com
> d-i mirror/http/directory string /ubuntu
> {{else}}
>
> === modified file 'contrib/preseeds_v2/preseed_master'
> --- contrib/preseeds_v2/preseed_master 2012-08-03 16:33:05 +0000
> +++ contrib/preseeds_v2/preseed_master 2012-10-02 19:13:28 +0000
> @@ -3,6 +3,9 @@
> # * Cloud-init for bare-metal
> # * Cloud-init preseed data
>
> +# Squashfs Installation
> +{{self.squashfs_image}}
> +
> # Locale
> d-i debian-installer/locale string en_US.UTF-8
>
>
> === modified file 'scripts/maas-import-pxe-files'
> --- scripts/maas-import-pxe-files 2012-09-29 03:12:23 +0000
> +++ scripts/maas-import-pxe-files 2012-10-02 19:13:28 +0000
> @@ -45,6 +45,10 @@
> # Default is yes.
> IMPORT_SQUASHFS=${IMPORT_SQUASHFS:-1}
>
> +# Whether to download squashfs images as well: "1" for yes, "0" for no.
> +# Default is yes.
> +IMPORT_SQUASHFS=${IMPORT_SQUASHFS:-1}
> +
> # Whether to download ephemeral images as well: "1" for yes, "0" for no.
> # Default is yes.
> IMPORT_EPHEMERALS=${IMPORT_EPHEMERALS:-1}
> @@ -1...

[Gavin Panella (allenap)]

On 2 October 2012 23:26, Andres Rodriguez <email address hidden> wrote:
> The squashfs image is the root filesystem image, hence the name. I think it
> is appropriate to categorize it that way (as it is now - filesystem).

The problem is that the function is called is_squashfs_image_present,
but it looks for "filesystem" via BootImage. The boot image scanner
makes assumptions about the files within a leaf directory, and doesn't
record them in the database.

The squashfs code is piggybacking on top of the tftp code, so, unless
you change that boot image code, it ought to work with the scheme as
it is. Hence the idea for putting "squashfs" somewhere in the path
where the boot image scanner will find it.

It's not very important though, I just wanted to make you aware of it.

[Andres Rodriguez (andreserl)]

Would it make more sense to change it to is_filesystem_image_present or is is_root_filesystem_image_present? Because the layout of how the directories are stored under the tftproot are:

 - commissioning - > Is a hint to the purpose
 - install -> Is hint to the purpose

So, that's why I added the "filesystem", to give a hint of what it is, or the purpose of it. Ideally, however, I would have loved to have this under install/, but that would result in overwriting the directory

 - filesystem -> hint of the purpose

Other than that, I don't really mind having it change to "squashfs". If you prefer i can just change it to squashfs (which will require to change maas-import-squashfs)

Cheers

[MAAS Lander (maas-lander)]

No commit message specified.

=== modified file 'contrib/maas-http.conf'
--- contrib/maas-http.conf	2012-09-25 09:48:17 +0000
+++ contrib/maas-http.conf	2012-10-02 21:50:23 +0000
@@ -45,6 +45,12 @@
     SetHandler None
 </Directory>
 
+# Serve squashfs images
+Alias /MAAS/static/images/ /var/lib/maas/tftp/
+<Directory /var/lib/maas/tftp/>
+    SetHandler None
+</Directory>
+
 # Serve files from staticfiles.
 Alias /MAAS/static/ /usr/share/maas/web/static/
 <Directory /usr/share/maas/web/static/>

=== modified file 'contrib/preseeds_v2/generic'
--- contrib/preseeds_v2/generic	2012-09-21 16:42:19 +0000
+++ contrib/preseeds_v2/generic	2012-10-02 21:50:23 +0000
@@ -1,7 +1,17 @@
 {{inherit "preseed_master"}}
+
+{{def squashfs_image}}
+{{if node.distro_series in {'quantal'} and node.architecture in {'i386/generic', 'amd64/generic'} and use_squashfs is True }}
+d-i    preseed/early_command string anna-install live-installer
+d-i    live-installer/enable boolean true
+d-i    live-installer/mode select normal
+d-i    live-installer/net-image string http://{{server_host}}/MAAS/static/images/{{node.architecture}}/{{node.distro_series}}/filesystem/filesystem.squashfs
+{{endif}}
+{{enddef}}
+
 {{def proxy}}
 d-i     mirror/country string manual
-{{if node.architecture in {'i386', 'amd64'} }}
+{{if node.architecture in {'i386/generic', 'amd64/generic'} }}
 d-i     mirror/http/hostname string archive.ubuntu.com
 d-i     mirror/http/directory string /ubuntu
 {{else}}

=== modified file 'contrib/preseeds_v2/preseed_master'
--- contrib/preseeds_v2/preseed_master	2012-08-03 16:33:05 +0000
+++ contrib/preseeds_v2/preseed_master	2012-10-02 21:50:23 +0000
@@ -3,6 +3,9 @@
 # * Cloud-init for bare-metal
 # * Cloud-init preseed data
 
+# Squashfs Installation
+{{self.squashfs_image}}
+
 # Locale
 d-i     debian-installer/locale string en_US.UTF-8
 

=== modified file 'scripts/maas-import-pxe-files'
--- scripts/maas-import-pxe-files	2012-10-02 17:00:19 +0000
+++ scripts/maas-import-pxe-files	2012-10-02 21:50:23 +0000
@@ -45,6 +45,10 @@
 # Default is yes.
 IMPORT_SQUASHFS=${IMPORT_SQUASHFS:-1}
 
+# Whether to download squashfs images as well: "1" for yes, "0" for no.
+# Default is yes.
+IMPORT_SQUASHFS=${IMPORT_SQUASHFS:-1}
+
 # Whether to download ephemeral images as well: "1" for yes, "0" for no.
 # Default is yes.
 IMPORT_EPHEMERALS=${IMPORT_EPHEMERALS:-1}
@@ -196,6 +200,15 @@
 }
 
 
+# Download and install the squashfs root images.
+import_squashfs_images() {
+    if test "$IMPORT_SQUASHFS" != "0"
+    then
+        maas-import-squashfs
+    fi
+}
+
+
 # Download and install the ephemeral images.
 import_ephemeral_images() {
     if test "$IMPORT_EPHEMERALS" != "0"

=== modified file 'scripts/maas-import-squashfs'
--- scripts/maas-import-squashfs	2012-10-01 23:16:29 +0000
+++ scripts/maas-import-squashfs	2012-10-02 21:50:23 +0000
@@ -1,4 +1,4 @@
-#!/bin/bash -x
+#!/bin/bash
 #
 # maas-import-squashfs - sync and import squashfs images
 #
@@ -33,7 +33,7 @@
 # Download locations for Ubuntu releases.
 #http://cdimage.ubuntu.com/ubuntu-server/daily/current/
 SQUASHFS_ARCHIVE=${ARCHIVE:-http://cdimage.ubuntu.com/}
-RELEASES="quantal"
+
 # Ubuntu releases that are to be downloaded.
 SUPPORTED_RELEASES=""
 for supported in $(distro-info --supported)

=== modified file 'src/maasserver/preseed.py'
--- src/maasserver/preseed.py	2012-09-27 15:39:38 +0000
+++ src/maasserver/preseed.py	2012-10-02 21:50:23 +0000
@@ -28,7 +28,10 @@
     NODE_STATUS,
     PRESEED_TYPE,
     )
-from maasserver.models import Config
+from maasserver.models import (
+    BootImage,
+    Config,
+    )
 from maasserver.server_address import get_maas_facing_server_host
 from maasserver.utils import absolute_reverse
 import tempita
@@ -228,12 +231,20 @@
             'preseed_data': compose_preseed(node),
             'node_disable_pxe_url': node_disable_pxe_url,
             'node_disable_pxe_data': node_disable_pxe_data,
+            'use_squashfs': is_squashfs_image_present(node),
         }
         context.update(node_context)
 
     return context
 
 
+def is_squashfs_image_present(node):
+    """Whether or not the SquashFS image can be used during installation."""
+    arch, subarch = node.architecture.split("/")
+    return BootImage.objects.have_image(
+        arch, subarch, node.get_distro_series(), "filesystem")
+
+
 def render_preseed(node, prefix, release=''):
     """Find and load a `PreseedTemplate` for the given node.
 

=== modified file 'src/maasserver/tests/test_preseed.py'
--- src/maasserver/tests/test_preseed.py	2012-09-27 15:39:39 +0000
+++ src/maasserver/tests/test_preseed.py	2012-10-02 21:50:23 +0000
@@ -21,6 +21,7 @@
     NODE_STATUS,
     PRESEED_TYPE,
     )
+from maasserver.models import BootImage
 from maasserver.preseed import (
     compose_enlistment_preseed_url,
     compose_preseed_url,
@@ -30,6 +31,7 @@
     get_preseed_context,
     get_preseed_filenames,
     get_preseed_template,
+    is_squashfs_image_present,
     load_preseed_template,
     PreseedTemplate,
     render_preseed,
@@ -295,7 +297,8 @@
         self.assertItemsEqual(
             ['node', 'release', 'metadata_enlist_url',
              'server_host', 'server_url', 'preseed_data',
-             'node_disable_pxe_url', 'node_disable_pxe_data'],
+             'node_disable_pxe_url', 'node_disable_pxe_data',
+             'use_squashfs'],
             context)
 
     def test_get_preseed_context_if_node_None(self):
@@ -309,6 +312,31 @@
             context)
 
 
+class TestSquashFSAvailable(TestCase):
+    """Tests for `is_squashfs_image_present`."""
+
+    # Scenario defaults.
+    arch = "i386"
+    subarch = "generic"
+    series = "quantal"
+    purpose = "filesystem"
+
+    scenarios = (
+        ("mismatch-arch", dict(arch="amd64", present=False)),
+        ("mismatch-subarch", dict(subarch="special", present=False)),
+        ("mismatch-series", dict(series="precise", present=False)),
+        ("mismatch-purpose", dict(purpose="moonraking", present=False)),
+        ("match", dict(present=True)),
+        )
+
+    def test_squashfs_available(self):
+        BootImage.objects.register_image(
+            self.arch, self.subarch, self.series, self.purpose)
+        node = factory.make_node(
+            architecture="i386/generic", distro_series="quantal")
+        self.assertEqual(self.present, is_squashfs_image_present(node))
+
+
 class TestPreseedTemplate(TestCase):
     """Tests for class:`PreseedTemplate`."""