Merge lp:~dpigott/lava-master-image-scripts/move-partitioning-to-first-boot into lp:lava-master-image-scripts

[Dave Pigott]

Get rid of the contrib, second stage code, and make it all happen on first boot. Re-write and tidy the partitioning code. Add a new option to lava-create-master-image to take in testrootfs size as a command line parameter.

Also put the image size back to 1GB. The 3GB was unnecessary, since we deploy onto testrootfs, which is usually 4G and in the LAVA farm is always 8GB.

[Zygmunt Krynicki (zyga)]

450 -LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate"
451 +LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate parted"

Do we need parted? We use fdisk, we also need partprobe and mkfs.ext4 (they may be already contained in other packages)

Otherwise it looks fine on first read +2

[Michael Hudson-Doyle (mwhudson)]

I tested this branch and it seems to work. Nice :)

review: Approve

[Dave Pigott (dpigott)]

> 450 -LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate"
> 451 +LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate parted"
>
> Do we need parted? We use fdisk, we also need partprobe and mkfs.ext4 (they
> may be already contained in other packages)
>
> Otherwise it looks fine on first read +2

It was purely because of partprobe. Wasn't aware of another package that contains it.

[lp:~dpigott/lava-master-image-scripts/move-partitioning-to-first-boot]

61. By Dave Pigott on 2012-05-15

Add a little more commentary

=== modified file 'NEWS'
--- NEWS	2012-05-04 11:46:04 +0000
+++ NEWS	2012-05-15 08:44:18 +0000
@@ -1,11 +1,12 @@
 2012.05:
     
     * Unreleased
-    * Increase the size of the default image to 3G
+    * Increase the size of the default image to 3G - now back at 1G
     * Add support for partitioning SD cards via both native mmc interface and
       USB adapters
     * Rename run-beagle-master-in-qemu.sh to boot-snaphot-in-qemu.sh and allow
       the user to point it at any image pathname
+    * Add partitioning on boot up rather than as a seperate phase
 
 2u12.04:
 

=== modified file 'README'
--- README	2012-05-03 09:53:27 +0000
+++ README	2012-05-15 08:44:18 +0000
@@ -32,25 +32,8 @@
 from the Linaro website. Once prepared the caches "vanilla" image can be reused
 over and over. The tool creates a single file <board>-master.img.
 
-This image is not complete: it lacks the special partitioning layout required by
-LAVA dispatcher. To finish the process copy the image to an SD card as instructed
-by the tool and run:
-
-    $ ./contrib/lava-partition-disk <board type> <device path> <boot mount>
-
-Replace <board type> with one of: beagle, imx53, origen, panda, snowball
-or vexpress.
-
-Replace <device path> with the name of the system device that represents your
-SD card, typically this is /dev/sdX or /dev/mmcblkX (with X being a number or
-letter representing subsequent devices in your system).
-
-Replace <boot mount> with the mount point for the boot partition, typically on
-Ubuntu this will be /media/boot.
-
-This step will be integrated into the tool in the future. At this time the card
-is now ready and can be cloned (like a template). The first boot of a device
-with this card is special. It will initialize the master image and will likely
-take a few extra minutes to finish. The interactive serial console will be
-activated after this process has finished. Once this process is done it is *not
-recommended* to clone the SD card anymore.
+The first boot of a device with this card is special. It will initialize the
+master image, including partitioning and labeling, and will likely take a few
+extra minutes to finish. The interactive serial console will be activated after
+this process has finished. Once this process is done it is *not recommended* to
+clone the SD card anymore.

=== removed directory 'contrib'
=== removed file 'contrib/lava-partition-disk'
--- contrib/lava-partition-disk	2012-05-14 08:38:47 +0000
+++ contrib/lava-partition-disk	1970-01-01 00:00:00 +0000
@@ -1,118 +0,0 @@
-#!/bin/bash
-
-show_usage() {
-    echo "Usage: $0 <board type> <device path> <boot mount> {testrootfs partition size}"
-    echo "Where <board typei> is one of: beagle, imx53, origen, panda,"
-    echo "  snowball or vexpress"
-    echo
-    echo "<device path> is the path to your sd card (e.g. /dev/sdb)"
-    echo "<boot mount> is where the boot partition of the sd card is (e.g. /media/boot"
-    echo
-    echo "The testrootfs partition size is optional and defauts to 4G."
-    echo "Make sure you specify the \"G\" in uppercase"
-}
-
-if [ "$3" = "" ]
-then
-    show_usage
-    exit 1
-fi
-
-PRE_ENV=
-
-case $1 in
-    beagle)
-        PRE_ENV=Y
-        SCHEME=2part;;
-
-    imx53)
-        SCHEME=3part;;
-
-    origen)
-        SCHEME=3part;;
-
-    panda)
-        PRE_ENV=Y
-        SCHEME=2part;;
-
-    snowball)
-        SCHEME=2part;;
-
-    vexpress)
-        SCHEME=2part;;
-
-    *)
-        echo "Board type \"$1\" not recognised"
-        exit 1;;
-esac
-
-DEVICE=$2
-
-BOOT_MOUNT=$3
-
-if [ "$4" = "" ]
-then
-    # SIZE is the size of the testrootfs partition. On an 8G sd you can fit
-    # 4G - put it up to 8G on a 16GB. The rest is allocated to sdcard for
-    # android
-    SIZE="4G"
-else
-    SIZE=$4
-fi
-
-if [ "$PRE_ENV" = "Y" ]
-then
-    cp ./contrib/preEnv.txt $BOOT_MOUNT
-    if [ "$?" != "0" ]
-    then
-        echo "Failed to copy preEnv.txt. Is the disk read only?"
-        exit 1
-    fi
-fi
-
-# origen and imx53 have a different original layout with 3 partitions
-# all others are the same (so far)
-
-if [ $SCHEME = "2part" ]
-then
-    TESTBOOT_PART=3
-    TESTROOTFS_PART=5
-    SDCARD_PART=6
-    LAST_BLOCK=2
-else
-    TESTBOOT_PART=5
-    TESTROOTFS_PART=6
-    SDCARD_PART=7
-    LAST_BLOCK=3
-fi
-
-if [[ $DEVICE =~ 'mmc' ]];
-then
-    PARTITION_PREFIX="p"
-else
-    PARTITION_PREFIX=""
-fi
-
-FIRST_BLOCK_START=`sudo fdisk -S 63 -H 255 -lc $DEVICE | grep "${DEVICE}${PARTITION_PREFIX}${LAST_BLOCK}" | awk '{print $3 + 1}'`
-
-if [ "$FIRST_BLOCK_START" = "" ]
-then
-    echo "Device specification error: $DEVICE"
-    exit 1
-fi
-
-if [ $SCHEME = "2part" ]
-then
-    EXTENDED_BLOCK_START=`echo $FIRST_BLOCK_START | awk '{print $1 + 131072}'`
-    FDISK_SCRIPT="n\np\n3\n$FIRST_BLOCK_START\n+64M\nt\n3\nc\nn\ne\n$EXTENDED_BLOCK_START\n\nn\n\n+$SIZE\nn\n\n\nt\n6\nc\np\nw\n"
-else
-    FDISK_SCRIPT="n\ne\n$FIRST_BLOCK_START\n\nn\n\n+64M\nt\n5\nc\nn\n\n+$SIZE\nn\n\n\nt\n7\nc\np\nw\n"
-fi
-
-echo -e $FDISK_SCRIPT | sudo fdisk -S 63 -H 255 -c $DEVICE
-
-sudo partprobe $DEVICE
-sudo mkfs.vfat -n testboot "${DEVICE}${PARTITION_PREFIX}${TESTBOOT_PART}"
-sudo mkfs.vfat -n sdcard "${DEVICE}${PARTITION_PREFIX}${SDCARD_PART}"
-sudo mkfs.ext3 -L testrootfs "${DEVICE}${PARTITION_PREFIX}${TESTROOTFS_PART}"
-sync

=== removed file 'contrib/preEnv.txt'
--- contrib/preEnv.txt	2012-05-03 09:52:40 +0000
+++ contrib/preEnv.txt	1970-01-01 00:00:00 +0000
@@ -1,1 +0,0 @@
-bootdelay=10

=== modified file 'lava-create-master'
--- lava-create-master	2012-05-14 08:38:47 +0000
+++ lava-create-master	2012-05-15 08:44:18 +0000
@@ -16,6 +16,10 @@
 # When non-empty the tool will stop just before jumping into the rootfs
 export DEBUG=
 
+# Used during partitioning on first boot - size of the testrootsfs partition
+# defaults to 4G (4 GigaBytes)
+export ROOTFS_SIZE="4G"
+
 # TODO: change to BOARD
 dev=
 
@@ -58,7 +62,7 @@
 
 show_usage() {
     # Display the usage line
-    echo "Usage: lava-create-master [--rebuild-vanilla] [--verbose] (--help | --version | DEVICE) [ --output|-o IMAGE]"
+    echo "Usage: lava-create-master [--rebuild-vanilla] [--verbose] (--help | --version | DEVICE) [ --output|-o IMAGE] [--rootfs-size|-r SIZE]"
 }
 
 
@@ -73,6 +77,7 @@
     printf -- " %-9s - %s\n" "--verbose" "Display additional messages"
     printf -- " %-9s - %s\n" "--output" "Save the output as IMAGE"
     printf -- " %-9s - %s\n" "--rebuild-vanilla" "Always rebuild vanilla image"
+    printf -- " %-9s - %s\n" "--rootfs-size" "Size of the testrootfs partition (default 4G)"
     echo
     echo "supported devices:"
     echo
@@ -119,6 +124,10 @@
                 OUTPUT_IMAGE="$2"
                 shift 2
                 ;;
+            --rootfs-size|-r)
+                ROOTFS_SIZE="$2"
+                shift 2
+                ;;
             *)
                 if [ -n "$dev" ]; then
                     show_usage
@@ -335,7 +344,7 @@
             --dev $LMC_DEV \
             --rootfs ext4 \
             --image-file $CACHE_DIR/pristine-images/$dev-vanilla.img \
-            --image-size 3G \
+            --image-size 1G \
             --align-boot-part \
             --binary $CACHE_DIR/image-pieces/$ROOTFS \
             --hwpack $CACHE_DIR/image-pieces/$HWPACK \
@@ -408,6 +417,7 @@
     sed \
         -e "s!@LAVA_VERSION@!$LAVA_VERSION!" \
         -e "s!@LAVA_URL@!$LAVA_URL!" \
+        -e "s!@LAVA_ROOTFS_SIZE@!$ROOTFS_SIZE!" \
         -i \
         $MASTER_MNT/lib/lava/common
 

=== added file 'master-rootfs-extras/bin/lava-partition-disk'
--- master-rootfs-extras/bin/lava-partition-disk	1970-01-01 00:00:00 +0000
+++ master-rootfs-extras/bin/lava-partition-disk	2012-05-15 08:44:18 +0000
@@ -0,0 +1,184 @@
+#!/bin/bash
+
+# By default we build a 4GByte partition. Specifiy on the command line for
+# anything different
+ROOTFS_SIZE="4G"
+
+show_usage() {
+    echo "Usage: $0 [--rootfs-size|-r SIZE]"
+}
+
+show_help() {
+    show_usage
+    echo
+    echo "available options:"
+    echo
+    printf -- " %-9s - %s\n" "--help" "Show this message"
+    printf -- " %-9s - %s\n" "--rootfs-size" "Size of the testrootfs partition (default 4G)"
+    echo
+    echo "See: $LAVA_URL for more information"
+}
+
+parse_argv() {
+    while test -n "$1"; do
+        case "$1" in
+            --help)
+                show_help
+                exit 0
+                ;;
+            --rootfs-size|-r)
+                ROOTFS_SIZE=$2;
+                shift 2
+                ;;
+            *)
+                show_usage
+                echo "Too many arguments, see --help for details"
+                exit 1
+                ;;
+        esac
+    done
+}
+
+# Basically, there are just enough differences between boards that you need
+# to specify which schemes and actions need to take place.
+#
+# PRE_ENV: means that we can put a preEnv.txt in the boot partition and it
+# will be read on boot and used, so we get a default bootdelay of 10s
+#
+# SCHEME: Basically describes the topology of the disk at the start. Some
+# boards have 2 partitions, others 3: TODO Make it sense which one, and even
+# detect that the card doesn't conform (e.g. already partitioned)
+#
+# DEVICE: Fairly self explanatory - the device that will be partitioned
+set_device() {
+    DEVICE_TYPE=`lava-device-info --device-type`
+
+    case $DEVICE_TYPE in
+        lava:beagle)
+            PRE_ENV=Y
+            SCHEME=2part
+            DEVICE="/dev/mmcblk0"
+            ;;
+        lava:imx53)
+            PRE_ENV=N
+            SCHEME=3part
+            DEVICE="/dev/mmcblk0"
+            ;;
+        lava:origen)
+            PRE_ENV=N
+            SCHEME=3part
+            DEVICE="/dev/mmcblk0"
+            ;;
+        lava:panda)
+            PRE_ENV=Y
+            SCHEME=2part
+            DEVICE="/dev/mmcblk0"
+            ;;
+        lava:snowball)
+            PRE_ENV=N
+            SCHEME=2part
+            DEVICE="/dev/mmcblk1"
+            ;;
+        lava:vexpress)
+            PRE_ENV=N
+            SCHEME=2part
+            DEVICE="/dev/mmcblk0"
+            ;;
+        *)
+            echo "Board type \"$1\" not supported"
+            exit 1
+            ;;
+    esac
+
+    # origen and imx53 have a different original layout with 3 partitions
+    # all others are the same (so far). All we're doing is defining the
+    # partition numbers for the partitions we're going to add
+
+    if [ $SCHEME = "2part" ]
+    then
+        TESTBOOT_PART=3
+        TESTROOTFS_PART=5
+        SDCARD_PART=6
+        LAST_BLOCK=2
+    else
+        TESTBOOT_PART=5
+        TESTROOTFS_PART=6
+        SDCARD_PART=7
+        LAST_BLOCK=3
+    fi
+
+    if [[ $DEVICE =~ 'mmc' ]];
+    then
+        PARTITION_PREFIX="p"
+    else
+        PARTITION_PREFIX=""
+    fi
+
+}
+
+
+write_preenv() {
+    mkdir -p /mnt/boot
+    mount /dev/disk/by-label/boot /mnt/boot
+    echo "bootdelay=10" >> /mnt/boot/preEnv.txt
+    if [ "$?" != "0" ]; then
+        echo "Failed to copy preEnv.txt. Is the disk read only?"
+        exit 1
+    fi
+    umount /mnt/boot
+}
+
+# Basically, we need the following scheme:
+# /dev/disk/by-label/boot - the master boot image
+# /dev/disk/by-label/rootfs - the master root filing system
+# /dev/disk/by-label/testboot - the deployed test boot image will go here
+# /dev/disk/by-label/testrootfs - the deployed test rootfs will go here
+# /dev/disk/by-label/sdcard - the Android sdcard partition goes here
+#
+# so first we partition the disk, if it's 2part, then 3 will be the testboot
+# and 4 will be an extended partition containing 5 as the testrootfs and 6
+# as sdcard.
+# For 3part then 4 will be an extended partition containing 5/testboot,
+# 6/testrootfs and 7/sdcard
+partition_disk() {
+    FIRST_BLOCK_START=`fdisk -S 63 -H 255 -lc $DEVICE | grep "${DEVICE}${PARTITION_PREFIX}${LAST_BLOCK}" | awk '{print $3 + 1}'`
+
+    if [ "$FIRST_BLOCK_START" = "" ]
+    then
+        echo "Device specification error: $DEVICE"
+        exit 1
+    fi
+
+    if [ $SCHEME = "2part" ]
+    then
+        # 131072 = 65536 * 2 - fdisk works in 512K blocks, and we want a 64M
+        # partition for testboot.
+        EXTENDED_BLOCK_START=`echo $FIRST_BLOCK_START | awk '{print $1 + 131072}'`
+        FDISK_SCRIPT="n\np\n3\n$FIRST_BLOCK_START\n+64M\nt\n3\nc\nn\ne\n$EXTENDED_BLOCK_START\n\nn\n\n+$ROOTFS_SIZE\nn\n\n\nt\n6\nc\np\nw\n"
+    else
+        FDISK_SCRIPT="n\ne\n$FIRST_BLOCK_START\n\nn\n\n+64M\nt\n5\nc\nn\n\n+$ROOTFS_SIZE\nn\n\n\nt\n7\nc\np\nw\n"
+    fi
+
+    echo -e $FDISK_SCRIPT | fdisk -S 63 -H 255 -c $DEVICE
+}
+
+# Do the partition labelling
+make_fs() {
+    partprobe $DEVICE
+    mkfs.vfat -n testboot "${DEVICE}${PARTITION_PREFIX}${TESTBOOT_PART}"
+    mkfs.vfat -n sdcard "${DEVICE}${PARTITION_PREFIX}${SDCARD_PART}"
+    mkfs.ext3 -L testrootfs "${DEVICE}${PARTITION_PREFIX}${TESTROOTFS_PART}"
+    sync
+}
+
+main() {
+    parse_argv "$@"
+    set_device
+    if [ "$PRE_ENV" = "Y" ]; then
+        write_preenv
+    fi
+    partition_disk
+    make_fs
+}
+
+main "$@"

=== modified file 'master-rootfs-extras/lib/lava/common'
--- master-rootfs-extras/lib/lava/common	2012-03-22 17:46:56 +0000
+++ master-rootfs-extras/lib/lava/common	2012-05-15 08:44:18 +0000
@@ -1,6 +1,7 @@
 # Replaced by lava-create-master
 LAVA_VERSION=@LAVA_VERSION@
 LAVA_URL=@LAVA_URL@
+LAVA_ROOTFS_SIZE=@LAVA_ROOTFS_SIZE@
 
 # Persistent device configuration file
 LAVA_DEVICE_CONF=/var/lib/lava/device.conf

=== modified file 'master-rootfs-extras/lib/lava/lava-master-init'
--- master-rootfs-extras/lib/lava/lava-master-init	2012-05-14 07:05:18 +0000
+++ master-rootfs-extras/lib/lava/lava-master-init	2012-05-15 08:44:18 +0000
@@ -14,7 +14,7 @@
 LAVA_FIRST_BOOT_LOG=/var/log/lava/first-boot.log
 
 # Set of packages to install on first boot
-LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate"
+LAVA_PKGS="wget dosfstools u-boot-tools bzip2 python ntpdate parted"
 
 DEVICE_TYPE=`lava-device-info --device-type`
 
@@ -54,6 +54,12 @@
         echo "Unable to synchronize time" >&2
         return 1
     fi
+    # Do disk partitioning
+    echo "Partitioning disk to LAVA format" >> "$LAVA_FIRST_BOOT_LOG"
+    if ! lava-partition-disk -r $LAVA_ROOTFS_SIZE >> "$LAVA_FIRST_BOOT_LOG" 2>&1; then
+        echo "Unable to partition disk" >&2
+        return 1
+    fi
     if [ "$DEVICE_TYPE" = "lava:panda" ]; then
         echo "Applying sysctl fix for streaming errors on Panda" >> "$LAVA_FIRST_BOOT_LOG"
         echo vm.min_free_kbytes = 32768 >> /etc/sysctl.conf