Merge lp:~ubuntu-desktop/ubuntu-cdimage/ubuntu-next-system-image into lp:ubuntu-cdimage

[Didier Roche]

Experimenting transforming ubuntu-desktop-next to a system-image base

[Colin Watson (cjwatson)]

As well as the inline comments below, could you please look into adding unit tests for at least some of this?

review: Needs Fixing

[Didier Roche (didrocks)]

In addition to my inline comments, I did push some fixes which should address all your comments. Working with robert on the tests now.

The release team has been poked and the isotracker change will be soon there (thanks Laney). Please see the reasoning why I'm not renaming the image removing the -next suffix.

Thanks for the initial review (and explanations when there is no need to add the system-image subproject differentiation).

[lp:~ubuntu-desktop/ubuntu-cdimage/ubuntu-next-system-image]

1509. By Didier Roche on 2015-04-22

Ensure we are backward compatible to build older desktop-next images, various typo fixes and factorize some work in functions

[Colin Watson (cjwatson)]

Your call on the image naming. There are some Boolean logic problems here that need to be fixed, but aside from that I'm happy.

review: Approve

[lp:~ubuntu-desktop/ubuntu-cdimage/ubuntu-next-system-image]

1510. By Didier Roche on 2015-04-22

Correct logic for ubuntu-desktop-next and non system-image case

Fix as well some pep8 issues and missing is_live_fs_only() parameter
in call.

1511. By Didier Roche on 2015-04-22

Fix existing ubuntu-desktop-next test

1512. By Didier Roche on 2015-04-22

Merge Robert's ubuntu-desktop-next-test

1513. By Didier Roche on 2015-04-22

Add a test for desktop-next system image

[Didier Roche (didrocks)]

We added a test for ubuntu-desktop-next system image, based on the existing ubuntu-desktop-next one.

Just a note, we are unsure if the artefacts that we counted (as they are the same than the other tests) makes sense.

=== modified file 'etc/crontab'
--- etc/crontab	2015-04-16 20:24:54 +0000
+++ etc/crontab	2015-04-22 15:04:54 +0000
@@ -20,7 +20,7 @@
 32 14 * * *	DIST=trusty for-project ubuntu-gnome cron.daily-live --live
 # regular vivid daily builds
 31 7 * * *	for-project ubuntu cron.daily-live --live
-22 3 * * *	for-project ubuntu-desktop-next cron.daily-live --live
+22 3 * * *	SUBPROJECT=system-image for-project ubuntu-desktop-next cron.daily-preinstalled --live
 14 5 * * *	for-project kubuntu cron.daily-live --live
 24 0 * * *	for-project edubuntu cron.dvd --live
 50 9 * * *	for-project xubuntu cron.daily-live --live

=== modified file 'etc/default-arches'
--- etc/default-arches	2015-03-20 20:01:29 +0000
+++ etc/default-arches	2015-04-22 15:04:54 +0000
@@ -55,7 +55,7 @@
 ubuntu			daily-live		raring-trusty		amd64 amd64+mac i386
 ubuntu			daily-live		utopic-			amd64 i386
 
-ubuntu-desktop-next	*			*			amd64 i386
+ubuntu-desktop-next	*			*			amd64 i386 armhf
 
 ubuntu-gnome		*			*			amd64 i386
 

=== modified file 'etc/qa-products'
--- etc/qa-products	2015-02-25 11:39:07 +0000
+++ etc/qa-products	2015-04-22 15:04:54 +0000
@@ -50,6 +50,7 @@
 Ubuntu Desktop Preinstalled armhf+nexus7	ubuntu			daily-preinstalled	preinstalled-desktop	armhf+nexus7	iso
 
 # Ubuntu Desktop Next
+Ubuntu Desktop (Unity 8) armhf					ubuntu-desktop-next	daily-live		desktop					armhf			iso
 Ubuntu Desktop (Unity 8) amd64					ubuntu-desktop-next	daily-live		desktop					amd64			iso
 Ubuntu Desktop (Unity 8) i386					ubuntu-desktop-next	daily-live		desktop					i386			iso
 

=== modified file 'lib/cdimage/build.py'
--- lib/cdimage/build.py	2015-01-29 04:56:17 +0000
+++ lib/cdimage/build.py	2015-04-22 15:04:54 +0000
@@ -458,7 +458,9 @@
     log_marker("Downloading live filesystem images")
     download_live_filesystems(config)
 
-    if config.project in ("ubuntu-core", "ubuntu-touch"):
+    if (config.project in ("ubuntu-core", "ubuntu-touch") or
+        (config.project == "ubuntu-desktop-next" and
+         config.subproject == "system-image")):
         log_marker("Copying images to debian-cd output directory")
         scratch_dir = os.path.join(
             config.root, "scratch", config.project, config.full_series,
@@ -482,6 +484,16 @@
                     output_prefix = os.path.join(
                         output_dir,
                         "%s-preinstalled-touch-%s" % (config.series, arch))
+                elif config.project == "ubuntu-desktop-next":
+                    if config.image_type == "daily-preinstalled":
+                        output_prefix = os.path.join(
+                            output_dir,
+                            "%s-preinstalled-desktop-next-%s" %
+                            (config.series, arch))
+                    else:
+                        output_prefix = os.path.join(
+                            output_dir, "%s-desktop-next-%s" %
+                            (config.series, arch))
                 shutil.copy2(rootfs, "%s.raw" % output_prefix)
                 with open("%s.type" % output_prefix, "w") as f:
                     print("tar archive", file=f)
@@ -495,7 +507,7 @@
                     if os.path.exists(custom):
                         shutil.copy2(
                             custom, "%s.custom.tar.gz" % output_prefix)
-                if config.project == "ubuntu-core":
+                if config.project in ("ubuntu-core", "ubuntu-desktop-next"):
                     device = "%s.device.tar.gz" % live_prefix
                     if os.path.exists(device):
                         shutil.copy2(
@@ -639,15 +651,23 @@
             body.close()
 
 
+def is_live_fs_only(config):
+    live_fs_only = False
+    if config.project in ("livecd-base", "ubuntu-core", "ubuntu-touch"):
+        live_fs_only = True
+    elif (config.project == "ubuntu-desktop-next" and
+          config.subproject == "system-image"):
+        live_fs_only = True
+    elif config.subproject == "wubi":
+        live_fs_only = True
+    return live_fs_only
+
+
 def build_image_set_locked(config, options, semaphore_state):
     image_type = config.image_type
     config["CDIMAGE_DATE"] = date = next_build_id(config, image_type)
     log_path = None
 
-    live_fs_only = (
-        config.project in ("livecd-base", "ubuntu-core", "ubuntu-touch") or
-        config.subproject == "wubi")
-
     try:
         configure_for_project(config)
         log_path = open_log(config)
@@ -659,7 +679,7 @@
         else:
             tracker_set_rebuild_status(config, [0, 1], 2)
 
-        if not live_fs_only:
+        if not is_live_fs_only(config):
             sync_local_mirror(config, semaphore_state)
 
             if config["LOCAL"]:
@@ -673,7 +693,7 @@
 
         if config["UBUNTU_DEFAULTS_LOCALE"]:
             build_ubuntu_defaults_locale(config)
-        elif live_fs_only:
+        elif is_live_fs_only(config):
             build_livecd_base(config)
         else:
             if not config["CDIMAGE_PREINSTALLED"]:

=== modified file 'lib/cdimage/livefs.py'
--- lib/cdimage/livefs.py	2014-11-13 07:16:42 +0000
+++ lib/cdimage/livefs.py	2015-04-22 15:04:54 +0000
@@ -121,7 +121,9 @@
         elif subarch in ("ac100", "nexus7"):
             options.extend(["-f", "plain"])
 
-    if config.project in ("ubuntu-core", "ubuntu-touch"):
+    if (config.project in ("ubuntu-core", "ubuntu-touch") or
+       (config.project == "ubuntu-desktop-next" and
+       config.subproject == "system-image")):
         options.extend(["-f", "plain"])
 
     if config.subproject == "wubi":
@@ -882,6 +884,8 @@
 
             if (project not in ("livecd-base", "ubuntu-core",
                                 "kubuntu-active") and
+                    (project != "ubuntu-desktop-next" or
+                     config.subproject == "system-image") and
                     (project != "edubuntu" or series >= "precise") and
                     (project != "ubuntukylin" or series < "utopic")):
                 if series <= "feisty":
@@ -912,7 +916,8 @@
                             config, arch, "wubi.exe",
                             "Install %s" % autorun_project)
 
-            if project not in ("livecd-base", "ubuntu-core", "edubuntu"):
+            if project not in ("livecd-base", "ubuntu-core",
+                               "ubuntu-desktop-next", "edubuntu"):
                 if (project in ("kubuntu-active", "ubuntu-netbook",
                                 "ubuntu-moblin-remix") or
                         config["CDIMAGE_DVD"] or
@@ -947,7 +952,7 @@
                 )
             download_live_items(config, arch, "custom.tar.gz")
 
-    if config.project == "ubuntu-core":
+    if config.project in ("ubuntu-core", "ubuntu-desktop-next"):
         for arch in config.arches:
             download_live_items(config, arch, "device.tar.gz")
 

=== modified file 'lib/cdimage/tests/test_build.py'
--- lib/cdimage/tests/test_build.py	2015-01-29 04:56:17 +0000
+++ lib/cdimage/tests/test_build.py	2015-04-22 15:04:54 +0000
@@ -343,6 +343,67 @@
             self.assertTrue(os.path.exists(
                 os.path.join(output_dir, recovery_img)))
 
+    @mock.patch("cdimage.osextras.fetch")
+    def test_ubuntu_desktop_next(self, mock_fetch):
+        def fetch_side_effect(config, source, target):
+            if (target.endswith(".manifest") or
+                    target.endswith(".rootfs.tar.gz")):
+                touch(target)
+            else:
+                raise osextras.FetchError
+
+        mock_fetch.side_effect = fetch_side_effect
+        self.config["PROJECT"] = "ubuntu-desktop-next"
+        self.config["DIST"] = "utopic"
+        self.config["IMAGE_TYPE"] = "daily"
+        self.config["ARCHES"] = "i386"
+        self.capture_logging()
+        build_livecd_base(self.config)
+        self.assertLogEqual([
+            "===== Downloading live filesystem images =====",
+            self.epoch_date,
+        ])
+        output_dir = os.path.join(
+            self.temp_dir, "scratch", "ubuntu-desktop-next", "utopic", "daily",
+            "live")
+        self.assertTrue(os.path.isdir(output_dir))
+        self.assertCountEqual([
+            "i386.manifest",
+            "i386.rootfs.tar.gz",
+        ], os.listdir(output_dir))
+
+    @mock.patch("cdimage.osextras.fetch")
+    def test_ubuntu_desktop_next_system_image(self, mock_fetch):
+        def fetch_side_effect(config, source, target):
+            if (target.endswith(".manifest") or
+                    target.endswith(".rootfs.tar.gz")):
+                touch(target)
+            else:
+                raise osextras.FetchError
+
+        mock_fetch.side_effect = fetch_side_effect
+        self.config["PROJECT"] = "ubuntu-desktop-next"
+        self.config["SUBPROJECT"] = "system-image"
+        self.config["DIST"] = "utopic"
+        self.config["IMAGE_TYPE"] = "daily"
+        self.config["ARCHES"] = "i386"
+        self.capture_logging()
+        build_livecd_base(self.config)
+        self.assertLogEqual([
+            "===== Downloading live filesystem images =====",
+            self.epoch_date,
+            "===== Copying images to debian-cd output directory =====",
+            self.epoch_date,
+        ])
+        output_dir = os.path.join(
+            self.temp_dir, "scratch", "ubuntu-desktop-next", "utopic", "daily",
+            "live")
+        self.assertTrue(os.path.isdir(output_dir))
+        self.assertCountEqual([
+            "i386.manifest",
+            "i386.rootfs.tar.gz",
+        ], os.listdir(output_dir))
+
 
 class TestExtractDebootstrap(TestCase):
     def setUp(self):

=== modified file 'lib/cdimage/tree.py'
--- lib/cdimage/tree.py	2015-02-26 14:29:10 +0000
+++ lib/cdimage/tree.py	2015-04-22 15:04:54 +0000
@@ -380,6 +380,8 @@
                 return "preinstalled-touch"
             elif self.project == "ubuntu-core":
                 return "preinstalled-core"
+            elif self.project == "ubuntu-desktop-next":
+                return "preinstalled-desktop-next"
             else:
                 return "preinstalled-desktop"
         elif self.image_type.endswith("-live"):
@@ -529,6 +531,8 @@
             return "preinstalled touch image"
         elif publish_type == "preinstalled-core":
             return "preinstalled core image"
+        elif publish_type == "preinstalled-desktop-next":
+            return "preinstalled desktop next image"
         elif publish_type == "wubi":
             return "Wubi %s" % cd
         else:
@@ -1107,6 +1111,7 @@
             "preinstalled-mobile", "preinstalled-active",
             "preinstalled-headless", "preinstalled-server",
             "preinstalled-touch", "preinstalled-core", "wubi",
+            "preinstalled-desktop-next",
         )
 
         all_arches = (