Merge lp:~liuyq0307/lava-master-image-scripts/add-sdcard into lp:lava-master-image-scripts

[Yongqin Liu]

Add support to creating an sdcard partition into the sdcard disk.
But because the mmcblk0p8 is not supported,
so for Origen and iMX, the sdcard partition keep not used.

The addition of sdcard partition into sdcard disk in this branch will not affect the deployment for Origen and iMX boards.

[Spring Zhang (qzhang)]

imx53 bug is on https://bugs.launchpad.net/lava-master-image-scripts/+bug/1014498, but I think it's related with master kernel.

[Zygmunt Krynicki (zyga)]

<liuyq> could you help to try this branch https://code.launchpad.net/~liuyq0307/lava-master-image-scripts/add-sdcard with a imx53 or origen board?
<liuyq> davepigott, I have tested with panda, it works
<zyga> liuyq, would you mind opening a merge request
 even if it is still work in progress
 (mark it as such if needed)
<davepigott> liuyq: Will test
<liuyq> davepigott, thanks, and I submitted a MP for it. https://code.launchpad.net/~liuyq0307/lava-master-image-scripts/add-sdcard/+merge/110770
<liuyq> davepigott, springz has used it with his imx, but there is a problem about the last 8th partition of sdcard.
 davepigott, I don't know if the problem has relation to the size of his sdcard, the size is 4G in total.
<davepigott> liuyq: Our recommendation is always to use at least 8GB, but I'll check
<liuyq> davepigott, I am a little afraid that we can't create the 8th partition on imx or origen.
<davepigott> liuyq: I'll find out
<zyga> liuyq, why did you move to kpartx?
<liuyq> zyga, partx does not work on my precise, also not work on springz's natty
<zyga> liuyq, I see
<liuyq> zyga, but kparx works work for both
<zyga> liuyq, I'd prefer that to be a separate patch, and drop the # commented out version of partx
* zyga is not really in a position to tell this as I'm not a lava or validation person anymore
<zyga> but that's what I think
<davepigott> zyga: liuyq I completely agree about the commented out code, and we did say that you had to be on precise to make this work properly
<liuyq> zyga, thanks, I will delete the old partx comment when I merge. Here just for the convenience that if the kpartx does not work for all we can easily rollback
<zyga> davepigott, yes but apparently it's hard for some of the assignees to transition to new os release
 sure
 liuyq, I'd rather see a version that works on just kpartx than both comments, that'
 that's why we have bzr for
<davepigott> liuyq: The default rootfs size we set is 4G so it won't work on a 4G card
<liuyq> davepigott, yes, I told springz to specify the size with the command parameters
<liuyq> davepigott, zyga, BTW, we also have a lava-partition-disk under lava-dispatcher/doc/scripts/lava-partition-disk
<zyga> liuyq, oh, I'm happy to kill it if davepigott doesn't mind
<liuyq> davepigott, zyga, should we delete it? it seems almost the same as the one in lava-master-image-scripts, but a little out of date
<zyga> yes
<davepigott> +1

[Zygmunt Krynicki (zyga)]

* split the kpartx from sdcard partition improvements
* remove the old commented out code

review: Needs Fixing

[Dave Pigott (dpigott)]

Will not work with imx53 and Origen. We hit the limit of 8 partitions. Needs rethinking in light of this.

[Zygmunt Krynicki (zyga)]

Interesting:

What are the partitions on imx53:

master:

 boot
 rootfs

test:

 boot
 rootfs / system *
 userdata *
 scdard *

Android partitions marked with *

I assume I'm missing some special boot partitions specific to imx53

[Dave Pigott (dpigott)]

imx53 and origen images have a small first partition as well as boot and rootfs - infrastructure or landing teams team would probably be able to tell us what they are for.

[Spring Zhang (qzhang)]

I upgrade my linux-util to precise version, and found the same error like Yongqin's when using partx.

But I say it doesn't matter, it will finish the image, I can dd it to a SD card and make it boot up, the only missing is that partition making process doesn't happen automatically when first boot up. The error I met:

$ ./lava-create-master imx53 -o ../masteimx53.img --verbose
LAVA master images are easy to make, just three steps:

1) Preparing vanilla Linaro image
 * Inhibiting udisks...
 * Re-using existing vanilla image: /home/spring/.cache/lava/master-image-scripts/pristine-images/imx53-vanilla.img
2) Converting vanilla image to LAVA master image
 * Creating a copy of vanilla image...
 * Attaching master image to loopback device...
 * Adding partitions to the loopback device...
partx: /dev/loop0: error adding partitions 1-3
 * Removing loopback device partitions...
partx: defined range <1:0> does not make sense

$ partx --show ../master_imx53.img
NR START END SECTORS SIZE NAME UUID
 1 1 8191 8191 4M
 2 8192 114687 106496 52M
 3 114688 4194303 4079616 2G

[Yongqin Liu (liuyq0307)]

./lava-create-master will mount the image file locally, and add some file
into it to make it do the partition job when first boot.
(the files are in master-rootfs-extras directory)
but the partx error make it not able to mount the image file locally.

On 20 June 2012 09:28, Spring Zhang <email address hidden> wrote:

> I upgrade my linux-util to precise version, and found the same error like
> Yongqin's when using partx.
>
> But I say it doesn't matter, it will finish the image, I can dd it to a SD
> card and make it boot up, the only missing is that partition making process
> doesn't happen automatically when first boot up. The error I met:
>
> $ ./lava-create-master imx53 -o ../masteimx53.img --verbose
> LAVA master images are easy to make, just three steps:
>
> 1) Preparing vanilla Linaro image
> * Inhibiting udisks...
> * Re-using existing vanilla image:
> /home/spring/.cache/lava/master-image-scripts/pristine-images/imx53-vanilla.img
> 2) Converting vanilla image to LAVA master image
> * Creating a copy of vanilla image...
> * Attaching master image to loopback device...
> * Adding partitions to the loopback device...
> partx: /dev/loop0: error adding partitions 1-3
> * Removing loopback device partitions...
> partx: defined range <1:0> does not make sense
>
> $ partx --show ../master_imx53.img
> NR START END SECTORS SIZE NAME UUID
> 1 1 8191 8191 4M
> 2 8192 114687 106496 52M
> 3 114688 4194303 4079616 2G
>
> --
>
> https://code.launchpad.net/~liuyq0307/lava-master-image-scripts/add-sdcard/+merge/110770
> You are the owner of lp:~liuyq0307/lava-master-image-scripts/add-sdcard.
>

[Yongqin Liu (liuyq0307)]

> * split the kpartx from sdcard partition improvements
> * remove the old commented out code
Split the change to kpartx to this MP
https://code.launchpad.net/~liuyq0307/lava-master-image-scripts/use-kpartx/+merge/111133

please review it

[Dave Pigott (dpigott)]

Given the partitioning limits for origen and imx53, I'd rather see us not do the userdata partitioning on those boards, rather than attempting and failing.

[lp:~liuyq0307/lava-master-image-scripts/add-sdcard]

68. By Yongqin Liu on 2012-06-28

keep not changed for iMX and origen

69. By Yongqin Liu on 2012-06-28

add comment for userdata partition

70. By Yongqin Liu on 2012-06-28

make some lines the same as before

71. By Yongqin Liu on 2012-06-28

add double quote

[Yongqin Liu (liuyq0307)]

> Given the partitioning limits for origen and imx53, I'd rather see us not do
> the userdata partitioning on those boards, rather than attempting and failing.

Yeah, I agree with you.
Now I have updated it to only add the userdata partition for 2part board.
Please review it again.

BTW, I have not test this updated source yet, could you help to test it with the Origen and iMX first?
I will test the 2part with panda tomorrow after the jobs on it now completed.

Thanks,
Yongqin Liu

[Dave Pigott (dpigott)]

review: Approve

[Michael Hudson-Doyle (mwhudson)]

Looks good to me now too.

review: Approve

[lp:~liuyq0307/lava-master-image-scripts/add-sdcard]

72. By Yongqin Liu on 2012-06-29

fix the bug for format userdata partition

73. By Yongqin Liu on 2012-06-29

fix the bug for format userdata partition

74. By Yongqin Liu on 2012-06-29

revert back the fix for trap on EXIT

75. By Yongqin Liu on 2012-06-29

add the for the if condition

=== modified file 'lava-create-master'
--- lava-create-master	2012-06-20 01:57:28 +0000
+++ lava-create-master	2012-06-29 06:58:19 +0000
@@ -20,6 +20,10 @@
 # defaults to 4G (4 GigaBytes)
 export ROOTFS_SIZE="4G"
 
+# Used during partitioning on first boot - size of the userdata partition
+# defaults to 2G (2 GigaBytes)
+export USERDATA_SIZE="2G"
+
 # TODO: change to BOARD
 dev=
 
@@ -81,6 +85,7 @@
     printf -- " %-9s - %s\n" "--output" "Save the output as IMAGE"
     printf -- " %-9s - %s\n" "--rebuild-vanilla" "Always rebuild vanilla image"
     printf -- " %-9s - %s\n" "--rootfs-size" "Size of the testrootfs partition (default 4G)"
+    printf -- " %-9s - %s\n" "--userdata-size" "Size of the userdata partition (default 2G), no meaning for iMX and Origen"
     echo
     echo "supported devices:"
     echo
@@ -131,6 +136,10 @@
                 ROOTFS_SIZE="$2"
                 shift 2
                 ;;
+            --userdata-size|-r)
+                USERDATA_SIZE="$2"
+                shift 2
+                ;;
             *)
                 if [ -n "$dev" ]; then
                     show_usage
@@ -429,6 +438,7 @@
         -e "s!@LAVA_VERSION@!$LAVA_VERSION!" \
         -e "s!@LAVA_URL@!$LAVA_URL!" \
         -e "s!@LAVA_ROOTFS_SIZE@!$ROOTFS_SIZE!" \
+        -e "s!@LAVA_USERDATA_SIZE@!$USERDATA_SIZE!" \
         -i \
         $MASTER_MNT/lib/lava/common
 

=== modified file 'master-rootfs-extras/bin/lava-partition-disk'
--- master-rootfs-extras/bin/lava-partition-disk	2012-05-15 08:31:12 +0000
+++ master-rootfs-extras/bin/lava-partition-disk	2012-06-29 06:58:19 +0000
@@ -4,8 +4,12 @@
 # anything different
 ROOTFS_SIZE="4G"
 
+# By default we build a 2GByte partition. Specifiy on the command line for
+# anything different
+USERDATA_SIZE="2G"
+
 show_usage() {
-    echo "Usage: $0 [--rootfs-size|-r SIZE]"
+    echo "Usage: $0 [--rootfs-size|-r SIZE] [--userdata-size|-u SIZE]"
 }
 
 show_help() {
@@ -15,6 +19,7 @@
     echo
     printf -- " %-9s - %s\n" "--help" "Show this message"
     printf -- " %-9s - %s\n" "--rootfs-size" "Size of the testrootfs partition (default 4G)"
+    printf -- " %-9s - %s\n" "--userdata-size" "Size of the userdata partition (default 2G), no meaning for iMX and Origen"
     echo
     echo "See: $LAVA_URL for more information"
 }
@@ -30,6 +35,10 @@
                 ROOTFS_SIZE=$2;
                 shift 2
                 ;;
+            --userdata-size|-u)
+                USERDATA_SIZE=$2;
+                shift 2
+                ;;
             *)
                 show_usage
                 echo "Too many arguments, see --help for details"
@@ -98,7 +107,8 @@
     then
         TESTBOOT_PART=3
         TESTROOTFS_PART=5
-        SDCARD_PART=6
+        USERDATA_PART=6
+        SDCARD_PART=7
         LAST_BLOCK=2
     else
         TESTBOOT_PART=5
@@ -133,11 +143,12 @@
 # /dev/disk/by-label/rootfs - the master root filing system
 # /dev/disk/by-label/testboot - the deployed test boot image will go here
 # /dev/disk/by-label/testrootfs - the deployed test rootfs will go here
+# /dev/disk/by-label/userdata - the Android userdata partition goes here, but will not exist for iMX and Origen board
 # /dev/disk/by-label/sdcard - the Android sdcard partition goes here
 #
 # so first we partition the disk, if it's 2part, then 3 will be the testboot
 # and 4 will be an extended partition containing 5 as the testrootfs and 6
-# as sdcard.
+# as userdata and 7 as sdcard.
 # For 3part then 4 will be an extended partition containing 5/testboot,
 # 6/testrootfs and 7/sdcard
 partition_disk() {
@@ -154,7 +165,7 @@
         # 131072 = 65536 * 2 - fdisk works in 512K blocks, and we want a 64M
         # partition for testboot.
         EXTENDED_BLOCK_START=`echo $FIRST_BLOCK_START | awk '{print $1 + 131072}'`
-        FDISK_SCRIPT="n\np\n3\n$FIRST_BLOCK_START\n+64M\nt\n3\nc\nn\ne\n$EXTENDED_BLOCK_START\n\nn\n\n+$ROOTFS_SIZE\nn\n\n\nt\n6\nc\np\nw\n"
+        FDISK_SCRIPT="n\np\n3\n$FIRST_BLOCK_START\n+64M\nt\n3\nc\nn\ne\n$EXTENDED_BLOCK_START\n\nn\n\n+$ROOTFS_SIZE\nn\n\n+$USERDATA_SIZE\nn\n\n\nt\n${SDCARD_PART}\nc\np\nw\n"
     else
         FDISK_SCRIPT="n\ne\n$FIRST_BLOCK_START\n\nn\n\n+64M\nt\n5\nc\nn\n\n+$ROOTFS_SIZE\nn\n\n\nt\n7\nc\np\nw\n"
     fi
@@ -168,6 +179,9 @@
     mkfs.vfat -n testboot "${DEVICE}${PARTITION_PREFIX}${TESTBOOT_PART}"
     mkfs.vfat -n sdcard "${DEVICE}${PARTITION_PREFIX}${SDCARD_PART}"
     mkfs.ext3 -L testrootfs "${DEVICE}${PARTITION_PREFIX}${TESTROOTFS_PART}"
+    if [ $SCHEME = "2part" ]; then
+        mkfs.ext4 -L userdata "${DEVICE}${PARTITION_PREFIX}${USERDATA_PART}"
+    fi
     sync
 }
 

=== modified file 'master-rootfs-extras/lib/lava/common'
--- master-rootfs-extras/lib/lava/common	2012-05-14 12:58:45 +0000
+++ master-rootfs-extras/lib/lava/common	2012-06-29 06:58:19 +0000
@@ -2,6 +2,7 @@
 LAVA_VERSION=@LAVA_VERSION@
 LAVA_URL=@LAVA_URL@
 LAVA_ROOTFS_SIZE=@LAVA_ROOTFS_SIZE@
+LAVA_USERDATA_SIZE=@LAVA_USERDATA_SIZE@
 
 # Persistent device configuration file
 LAVA_DEVICE_CONF=/var/lib/lava/device.conf

=== modified file 'master-rootfs-extras/lib/lava/lava-master-init'
--- master-rootfs-extras/lib/lava/lava-master-init	2012-05-14 12:58:45 +0000
+++ master-rootfs-extras/lib/lava/lava-master-init	2012-06-29 06:58:19 +0000
@@ -56,7 +56,7 @@
     fi
     # Do disk partitioning
     echo "Partitioning disk to LAVA format" >> "$LAVA_FIRST_BOOT_LOG"
-    if ! lava-partition-disk -r $LAVA_ROOTFS_SIZE >> "$LAVA_FIRST_BOOT_LOG" 2>&1; then
+    if ! lava-partition-disk -r $LAVA_ROOTFS_SIZE -u $LAVA_USERDATA_SIZE>> "$LAVA_FIRST_BOOT_LOG" 2>&1; then
         echo "Unable to partition disk" >&2
         return 1
     fi