Merge lp:~lool/linaro-image-tools/mkfs-uuid into lp:linaro-image-tools/11.11

[Loïc Minier]

Tentative branch; only passed testsuite, didn't create btrfs image yet

Suggestions on testing get_uuid() welcome!

[James Westby (james-w)]

Looks good so far.

For a get_uuid() test I would use the subprocess mocking again to ensure that
it calls the correct thing. It would be testing that at multiple layers, but
I think that's just the price we have to pay here.

Thanks,

James

[lp:~lool/linaro-image-tools/mkfs-uuid]

262. By Loïc Minier on 2011-01-26

Only lookup rootfs UUID when needed.

263. By Loïc Minier on 2011-01-26

Test that get_uuid() runs blkid correctly.

[Loïc Minier (lool)]

But ignoring the return value then -- ok; I guess otherwise I would have to introduce a new Popen mock which supports returning some pre-defined output in communicate().

I've pushed an additional test now, and a cleanup; I'll test the branch now with real images now.

[James Westby (james-w)]

On Wed, 26 Jan 2011 12:29:09 -0000, Loïc Minier <email address hidden> wrote:
> But ignoring the return value then -- ok; I guess otherwise I would
> have to introduce a new Popen mock which supports returning some
> pre-defined output in communicate().

I didn't realise that was missing. I assumed that you would be doing
that.

I don't think it's super critical, but would be nice to have.

Thanks,

James

[lp:~lool/linaro-image-tools/mkfs-uuid]

264. By Loïc Minier on 2011-01-26

Use blkid -p to force probing and ignore the cache in this case since it's
useless; call blkid within sudo as it needs direct device access.

[Loïc Minier (lool)]

With the latest change to use blkid -p (and sudo and -c /dev/null), and with a workaround for the UDisk race issue (time.sleep(5) after creating the rootfs), things are working reliably and I can create an ext3 image and boot it in QEMU.

I tried creating a btrfs image, but I'm using an older hwpack has the latest one doesn't work in QEMU; it fails to insmod btrfs and then falls over. I did however confirm that the proper UUID is set in boot.scr, in etc/fstab, and matches the rootfs' UUID (I manually loop-mounted it to check).

[James Westby (james-w)]

Looks good, apart from the simple conflict.

Should blkid be added to required_commands?

Thanks,

James

review: Approve

[Loïc Minier (lool)]

I am not sure what we keep in required_commands these days; in the case of Debian/Ubuntu it's guaranteed to be there

=== modified file 'debian/control'
--- debian/control	2011-01-19 18:53:01 +0000
+++ debian/control	2011-01-26 15:52:07 +0000
@@ -22,7 +22,6 @@
 Architecture: all
 Depends: ${misc:Depends},
          parted,
-         uuid-runtime,
          dosfstools,
          u-boot-tools | uboot-mkimage,
          python,

=== modified file 'linaro-media-create'
--- linaro-media-create	2011-01-26 12:10:14 +0000
+++ linaro-media-create	2011-01-26 15:52:07 +0000
@@ -8,7 +8,6 @@
 
 from linaro_media_create.boards import (
     board_configs,
-    ROOTFS_UUID,
     )
 from linaro_media_create.check_device import (
     confirm_device_selection_and_ensure_it_is_ready)
@@ -16,6 +15,7 @@
 from linaro_media_create.partitions import (
     Media,
     setup_partitions,
+    get_uuid,
     )
 from linaro_media_create.populate_boot import populate_boot
 from linaro_media_create.rootfs import populate_rootfs
@@ -59,7 +59,11 @@
 def ensure_required_commands(args):
     """Ensure we have the commands that we know are going to be used."""
     required_commands = [
+<<<<<<< TREE
         'mkfs.vfat', 'sfdisk', 'mkimage', 'uuidgen', 'parted']
+=======
+        'mkfs.vfat', 'sfdisk', 'fdisk', 'mkimage', 'parted']
+>>>>>>> MERGE-SOURCE
     if not is_arm_host():
         required_commands.append('qemu-arm-static')
         required_commands.append('qemu-img')
@@ -105,20 +109,22 @@
 
     boot_partition, root_partition = setup_partitions(
         board_config, media, args.image_size, args.boot_label, args.rfs_label,
-        args.rootfs, ROOTFS_UUID, args.should_create_partitions,
-        args.should_format_bootfs, args.should_format_rootfs)
+        args.rootfs, args.should_create_partitions, args.should_format_bootfs,
+        args.should_format_rootfs)
+
+    rootfs_uuid = get_uuid(root_partition)
 
     if args.should_format_bootfs:
         populate_boot(
-            board_config, ROOTFS_DIR, boot_partition, BOOT_DISK, args.device,
-            args.is_live, args.is_lowmem, args.consoles)
+            board_config, ROOTFS_DIR, rootfs_uuid, boot_partition, BOOT_DISK,
+            args.device, args.is_live, args.is_lowmem, args.consoles)
 
     if args.should_format_rootfs:
         create_swap = False
         if args.swap_file is not None:
             create_swap = True
         populate_rootfs(ROOTFS_DIR, ROOT_DISK, root_partition, args.rootfs,
-            ROOTFS_UUID, create_swap, str(args.swap_file),
+            rootfs_uuid, create_swap, str(args.swap_file),
             board_config.mmc_part_offset)
 
     print "Done creating Linaro image on %s" % args.device

=== modified file 'linaro_media_create/__init__.py'
--- linaro_media_create/__init__.py	2011-01-12 21:54:55 +0000
+++ linaro_media_create/__init__.py	2011-01-26 15:52:07 +0000
@@ -52,8 +52,7 @@
     group.add_argument(
         '--live', dest='is_live', action='store_true',
         help=('Create boot command for casper/live images; if this is not '
-              'provided a UUID for the rootfs is generated and used as the '
-              'root= option'))
+              'provided the UUID for the rootfs is used as the root= option'))
     group.add_argument(
         '--live-256m', dest='is_lowmem', action=Live256MegsAction,
         help=('Create boot command for casper/live images; adds '

=== modified file 'linaro_media_create/boards.py'
--- linaro_media_create/boards.py	2011-01-25 20:04:45 +0000
+++ linaro_media_create/boards.py	2011-01-26 15:52:07 +0000
@@ -9,12 +9,9 @@
 import glob
 import os
 import tempfile
-import uuid
 
 from linaro_media_create import cmd_runner
 
-ROOTFS_UUID = str(uuid.uuid4())
-
 
 class BoardConfig(object):
     """The configuration used when building an image for a board."""
@@ -48,7 +45,7 @@
         return ',9,%s,*\n,,,-' % partition_type
 
     @classmethod
-    def _get_boot_cmd(cls, is_live, is_lowmem, consoles):
+    def _get_boot_cmd(cls, is_live, is_lowmem, consoles, rootfs_uuid):
         """Get the boot command for this board.
 
         In general subclasses should not have to override this.
@@ -69,7 +66,7 @@
         serial_opts += ' %s' % cls.extra_serial_opts
 
         lowmem_opt = ''
-        boot_snippet = 'root=UUID=%s' % ROOTFS_UUID
+        boot_snippet = 'root=UUID=%s' % rootfs_uuid
         if is_live:
             serial_opts += ' %s' % cls.live_serial_opts
             boot_snippet = 'boot=casper'
@@ -91,8 +88,9 @@
 
     @classmethod
     def make_boot_files(cls, uboot_parts_dir, is_live, is_lowmem, consoles,
-                        root_dir, boot_dir, boot_script, boot_device_or_file):
-        boot_cmd = cls._get_boot_cmd(is_live, is_lowmem, consoles)
+                        root_dir, rootfs_uuid, boot_dir, boot_script,
+                        boot_device_or_file):
+        boot_cmd = cls._get_boot_cmd(is_live, is_lowmem, consoles, rootfs_uuid)
         cls._make_boot_files(
             uboot_parts_dir, boot_cmd, root_dir, boot_dir, boot_script,
             boot_device_or_file)

=== modified file 'linaro_media_create/partitions.py'
--- linaro_media_create/partitions.py	2011-01-25 20:04:45 +0000
+++ linaro_media_create/partitions.py	2011-01-26 15:52:07 +0000
@@ -1,5 +1,6 @@
 import atexit
 import glob
+import re
 import subprocess
 import time
 
@@ -25,9 +26,8 @@
 # small enough that there's not much benefit in doing that, but if it grows we
 # might want to do it.
 def setup_partitions(board_config, media, image_size, bootfs_label,
-                     rootfs_label, rootfs_type, rootfs_uuid,
-                     should_create_partitions, should_format_bootfs,
-                     should_format_rootfs):
+                     rootfs_label, rootfs_type, should_create_partitions,
+                     should_format_bootfs, should_format_rootfs):
     """Make sure the given device is partitioned to boot the given board.
 
     :param board_config: A BoardConfig class.
@@ -73,13 +73,31 @@
         mkfs = 'mkfs.%s' % rootfs_type
         ensure_partition_is_not_mounted(rootfs)
         proc = cmd_runner.run(
-            [mkfs, '-U', rootfs_uuid, rootfs, '-L', rootfs_label],
+            [mkfs, rootfs, '-L', rootfs_label],
             as_root=True)
         proc.wait()
 
     return bootfs, rootfs
 
 
+def get_uuid(partition):
+    """Find UUID of the given partition."""
+    proc = cmd_runner.run(
+        ['blkid', '-o', 'udev', '-p', '-c', '/dev/null', partition],
+        as_root=True,
+        stdout=subprocess.PIPE)
+    blkid_output, _ = proc.communicate()
+    return _parse_blkid_output(blkid_output)
+
+
+def _parse_blkid_output(output):
+    for line in output.splitlines():
+        uuid_match = re.match("ID_FS_UUID=(.*)", line)
+        if uuid_match:
+            return uuid_match.group(1)
+    return None
+
+
 def ensure_partition_is_not_mounted(partition):
     """Ensure the given partition is not mounted, umounting if necessary."""
     if is_partition_mounted(partition):

=== modified file 'linaro_media_create/populate_boot.py'
--- linaro_media_create/populate_boot.py	2011-01-13 00:00:49 +0000
+++ linaro_media_create/populate_boot.py	2011-01-26 15:52:07 +0000
@@ -4,8 +4,9 @@
 from linaro_media_create import cmd_runner
 
 
-def populate_boot(board_config, chroot_dir, boot_partition, boot_disk,
-                  boot_device_or_file, is_live, is_lowmem, consoles):
+def populate_boot(board_config, chroot_dir, rootfs_uuid, boot_partition,
+                  boot_disk, boot_device_or_file, is_live, is_lowmem,
+                  consoles):
 
     parts_dir = 'boot'
     if is_live:
@@ -33,8 +34,8 @@
              boot_script_name=board_config.boot_script))
 
     board_config.make_boot_files(
-        uboot_parts_dir, is_live, is_lowmem, consoles, chroot_dir, boot_disk,
-        boot_script, boot_device_or_file)
+        uboot_parts_dir, is_live, is_lowmem, consoles, chroot_dir, rootfs_uuid,
+        boot_disk, boot_script, boot_device_or_file)
 
     cmd_runner.run(['sync']).wait()
     try:

=== modified file 'linaro_media_create/tests/test_media_create.py'
--- linaro_media_create/tests/test_media_create.py	2011-01-25 20:04:45 +0000
+++ linaro_media_create/tests/test_media_create.py	2011-01-26 15:52:07 +0000
@@ -27,7 +27,6 @@
     make_boot_script,
     make_uImage,
     make_uInitrd,
-    ROOTFS_UUID,
     _get_file_matching,
     _get_mlo_file,
     _run_mkimage,
@@ -50,6 +49,8 @@
     Media,
     run_sfdisk_commands,
     setup_partitions,
+    get_uuid,
+    _parse_blkid_output,
     )
 from linaro_media_create.rootfs import (
     create_flash_kernel_config,
@@ -172,7 +173,7 @@
                     linaro_media_create.boards, name, mock_func_creator(name)))
 
     def make_boot_files(self, config):
-        config.make_boot_files('', False, False, [], '', '', '', '')
+        config.make_boot_files('', False, False, [], '', '', '', '', '')
 
     def test_vexpress_steps(self):
         config = linaro_media_create.boards.VexpressConfig
@@ -235,71 +236,76 @@
 
     def test_vexpress(self):
         boot_cmd = board_configs['vexpress']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 0:1 0x60008000 uImage; fatload mmc "
             "0:1 0x81000000 uInitrd; bootm 0x60008000 0x81000000'\nsetenv "
             "bootargs ' console=tty0 console=ttyAMA0,38400n8  "
-            "root=UUID=%s rootwait ro'\nboot" % ROOTFS_UUID)
+            "root=UUID=deadbeef rootwait ro'\nboot")
         self.assertEqual(expected, boot_cmd)
 
     def test_mx51evk(self):
         boot_cmd = board_configs['mx51evk']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 0:2 0x90000000 uImage; fatload mmc "
             "0:2 0x90800000 uInitrd; bootm 0x90000000 0x90800000'\nsetenv "
             "bootargs ' console=tty0 console=ttymxc0,115200n8  "
-            "root=UUID=%s rootwait ro'\nboot" % ROOTFS_UUID)
+            "root=UUID=deadbeef rootwait ro'\nboot")
         self.assertEqual(expected, boot_cmd)
 
     def test_ux500(self):
         boot_cmd = board_configs['ux500']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 1:1 0x00100000 uImage; fatload mmc "
             "1:1 0x08000000 uInitrd; bootm 0x00100000 0x08000000'\nsetenv "
             "bootargs ' console=tty0 console=ttyAMA2,115200n8  "
-            "root=UUID=%s rootwait ro earlyprintk rootdelay=1 fixrtc "
+            "root=UUID=deadbeef rootwait ro earlyprintk rootdelay=1 fixrtc "
             "nocompcache mem=96M@0 mem_modem=32M@96M mem=44M@128M "
             "pmem=22M@172M mem=30M@194M mem_mali=32M@224M "
-            "pmem_hwb=54M@256M hwmem=48M@302M mem=152M@360M'\nboot"
-            % ROOTFS_UUID)
+            "pmem_hwb=54M@256M hwmem=48M@302M mem=152M@360M'\nboot")
         self.assertEqual(expected, boot_cmd)
 
     def test_panda(self):
         boot_cmd = board_configs['panda']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 0:1 0x80200000 uImage; fatload mmc "
             "0:1 0x81600000 uInitrd; bootm 0x80200000 0x81600000'\nsetenv "
             "bootargs ' console=tty0 console=ttyO2,115200n8  "
-            "root=UUID=%s rootwait ro earlyprintk fixrtc nocompcache "
+            "root=UUID=deadbeef rootwait ro earlyprintk fixrtc nocompcache "
             "vram=32M omapfb.vram=0:8M mem=463M "
-            "ip=none'\nboot" % ROOTFS_UUID)
+            "ip=none'\nboot")
         self.assertEqual(expected, boot_cmd)
 
     def test_beagle(self):
         boot_cmd = board_configs['beagle']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 0:1 0x80000000 uImage; "
             "fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80000000 "
             "0x81600000'\nsetenv bootargs ' console=tty0 "
-            "console=ttyO2,115200n8  root=UUID=%s rootwait ro earlyprintk "
-            "fixrtc nocompcache vram=12M "
-            "omapfb.mode=dvi:1280x720MR-16@60'\nboot" % ROOTFS_UUID)
+            "console=ttyO2,115200n8  root=UUID=deadbeef rootwait ro "
+            "earlyprintk fixrtc nocompcache vram=12M "
+            "omapfb.mode=dvi:1280x720MR-16@60'\nboot")
         self.assertEqual(expected, boot_cmd)
 
     def test_overo(self):
         boot_cmd = board_configs['overo']._get_boot_cmd(
-            is_live=False, is_lowmem=False, consoles=None)
+            is_live=False, is_lowmem=False, consoles=None,
+            rootfs_uuid="deadbeef")
         expected = (
             "setenv bootcmd 'fatload mmc 0:1 0x80000000 uImage; "
             "fatload mmc 0:1 0x81600000 uInitrd; bootm 0x80000000 "
             "0x81600000'\nsetenv bootargs ' console=tty0 "
-            "console=ttyO2,115200n8  root=UUID=%s rootwait ro earlyprintk'"
-            "\nboot" % ROOTFS_UUID)
+            "console=ttyO2,115200n8  root=UUID=deadbeef rootwait ro "
+            "earlyprintk'\nboot")
         self.assertEqual(expected, boot_cmd)
 
 class TestUnpackBinaryTarball(TestCaseWithFixtures):
@@ -321,6 +327,30 @@
         self.assertEqual(rc, 0)
 
 
+class TestGetUuid(TestCaseWithFixtures):
+
+    def setUp(self):
+        super(TestGetUuid, self).setUp()
+
+    def test_get_uuid(self):
+        fixture = MockCmdRunnerPopenFixture()
+        self.useFixture(fixture)
+        get_uuid("/dev/rootfs")
+        self.assertEquals(
+            [[
+                "sudo", "blkid", "-o", "udev", "-p", "-c", "/dev/null",
+                "/dev/rootfs"]],
+            fixture.mock.calls)
+
+    def test_parse_blkid_output(self):
+        output = (
+            "ID_FS_UUID=67d641db-ea7d-4acf-9f46-5f1f8275dce2\n"
+            "ID_FS_UUID_ENC=67d641db-ea7d-4acf-9f46-5f1f8275dce2\n"
+            "ID_FS_TYPE=ext4\n")
+        uuid = _parse_blkid_output(output)
+        self.assertEquals("67d641db-ea7d-4acf-9f46-5f1f8275dce2", uuid)
+
+
 class TestCmdRunner(TestCaseWithFixtures):
 
     def test_run(self):
@@ -662,10 +692,9 @@
         self.useFixture(MockSomethingFixture(
             partitions, 'get_boot_and_root_loopback_devices',
             lambda image: ('/dev/loop99', '/dev/loop98')))
-        uuid = '2e82008e-1af3-4699-8521-3bf5bac1e67a'
         bootfs_dev, rootfs_dev = setup_partitions(
             board_configs['beagle'], Media(tempfile), '2G', 'boot',
-            'root', 'ext3', uuid, True, True, True)
+            'root', 'ext3', True, True, True)
         self.assertEqual(
              # This is the call that would create the image file.
             [['qemu-img', 'create', '-f', 'raw', tempfile, '2G'],
@@ -675,7 +704,7 @@
              # Make sure changes are written to disk.
              ['sync'],
              ['sudo', 'mkfs.vfat', '-F', '32', bootfs_dev, '-n', 'boot'],
-             ['sudo', 'mkfs.ext3', '-U', uuid, rootfs_dev, '-L', 'root']],
+             ['sudo', 'mkfs.ext3', rootfs_dev, '-L', 'root']],
             popen_fixture.mock.calls)
 
     def test_setup_partitions_for_block_device(self):
@@ -694,10 +723,9 @@
         # Pretend our tempfile is a block device.
         media.is_block_device = True
         popen_fixture = self.useFixture(MockCmdRunnerPopenFixture())
-        uuid = '2e82008e-1af3-4699-8521-3bf5bac1e67a'
         bootfs_dev, rootfs_dev = setup_partitions(
             board_configs['beagle'], media, '2G', 'boot', 'root', 'ext3',
-            uuid, True, True, True)
+            True, True, True)
         self.assertEqual(
             [['sudo', 'parted', '-s', tempfile, 'mklabel', 'msdos'],
              ['sudo', 'sfdisk', '-D', '-H', '255', '-S', '63', tempfile],
@@ -707,7 +735,7 @@
              ['sudo', 'umount', bootfs_dev],
              ['sudo', 'mkfs.vfat', '-F', '32', bootfs_dev, '-n', 'boot'],
              ['sudo', 'umount', rootfs_dev],
-             ['sudo', 'mkfs.ext3', '-U', uuid, rootfs_dev, '-L', 'root']],
+             ['sudo', 'mkfs.ext3', rootfs_dev, '-L', 'root']],
             popen_fixture.mock.calls)