Merge lp:~elopio/snappy/resize_test into lp:~snappy-dev/snappy/snappy-moved-to-github

[Leo Arias]

Added tests for the resize partition forcing a resize and a no-op.

This is not the test I wanted to make because it's far from the initramfs-tools-ubuntu-core source code. Also it's not as fast as it could be if we just test the init script without fully booting snappy.
But this is the easiest approach of all the ones I tried. We can build the initramfs on the same testbed, and use it with a simple reboot. I made notes to keep the research about building the init and making a simple boot. Now it's time to move on.

This is a kind of a complementary test, we add two conditions that are highly unlikely to be true at the same time.

[Oliver Grawert (ogra)]

note that our initrd allows to be compressed by different tools, there is no guarantee we always use xz. so before copying the initrd.img file to initrd.xz there should be a check that picks the right suffix and the right decompression tool in the install-test-initramfs script.

see my desktop initrd vs a current snappy rpi2 initrd:

ogra@anubis:~/Devel/packages$ file /boot/initrd.img-3.13.0-63-generic
/boot/initrd.img-3.13.0-63-generic: gzip compressed data, from Unix, last modified: Sun Sep 6 10:58:31 2015
ogra@anubis:~/Devel/packages$ file /datengrab/rpi/kernel/device/assets/initrd.img
/datengrab/rpi/kernel/device/assets/initrd.img: LZMA compressed data, streamed

I have a script snippet I use in such cases, see http://paste.ubuntu.com/12334035/ (indeed you can trim that to only match xz, gzip and lzma, I don't think we use any of the others for initrd's)

also note that once Bug 1490608 is fixed, the "do_gpt()" function will be dropped and this test needs to be adjusted accordingly.

review: Needs Fixing

[Leo Arias (elopio)]

TODO, rename CurrentPartition to NextBootPartition. Add a proper CurrentPartition to make the test pass in bbb. Check that rpi also uses xz for the initrd.img

[Leo Arias (elopio)]

once the prerequisite lands, this is ready to land too.

[Federico Gimenez (fgimenez)]

LGTM, and has been heavily tested during the past few days :)

I'm getting this in bbb 15.04/edge r155

** Resuming initRAMFSSuite.TestFreeSpaceWithoutResize after reboot
PASS: <autogenerated>:27: initRAMFSSuite.SetUpTest 0.008s

/home/fgimenez/workspace/snappy/1504_validation/_integration-tests/tests/initramfs_test.go:71:
...open /home/fgimenez/workspace/snappy/1504_validation/_integration-tests/tests/initramfs_test.go: no such file or directory
... obtained float64 = 0.12999999523162842
... expected float64 = 5
... The writable partition was resized

Is this expected?

[Leo Arias (elopio)]

No, there's a problem in here.
When we try to resize on mbr, it prints a warning and asks for confirmation.
I don't understand how it worked before. On the morning I ran in bbb with rolling edge and it worked. Now it doesn't work anymore.

review: Needs Fixing

[lp:~elopio/snappy/resize_test]

675. By Leo Arias on 2015-09-22

First remove the original initrd.img.

676. By Leo Arias on 2015-09-23

For mbr, resize the partition with sfdisk and using sectors.

677. By Leo Arias on 2015-09-23

Added comments.

678. By Leo Arias on 2015-09-23

Added setup and teardown.

[Federico Gimenez (fgimenez)]

It's failing because of the initramfs.img filename in SetUpTest and TearDownTest. Fixing that I'm still getting errors in TestFreeSpaceWithoutResize for amd64 rolling edge #191 [1] and bbb rolling edge #187 [2]

[1] http://paste.ubuntu.com/12637246/
[2] http://paste.ubuntu.com/12637539/

review: Needs Fixing

[lp:~elopio/snappy/resize_test]

679. By Leo Arias on 2015-10-07

Merged with trunk.

680. By Leo Arias on 2015-10-07

Fixed the name of the initrd.img

681. By Leo Arias on 2015-10-07

Fixed the call to ExecCommand.

682. By Leo Arias on 2015-10-07

Fixed set up and tear down.

[Leo Arias (elopio)]

Of course, I try to be extra smart at the end and break it all. The problem was that the setup and tear down were reverting the initrd changes.

Please, one more review. And sorry about that.

[Federico Gimenez (fgimenez)]

It works fine on kvm, on bbb I'm still getting one error at the end [1]. This is in rolling/edge #193, does it work for you?

Thanks,

[1] http://paste.ubuntu.com/12713787/

[Leo Arias (elopio)]

@Federico, I've been running this on friday and today and can't reproduce. Latest run was rolling/edge #203.

command1 PASS

Please, if you can still reproduce it, connect through the serial port and record the serial log with ctrl+a H.

[Federico Gimenez (fgimenez)]

Thanks Leo, it's passing on rolling/edge #203

review: Approve

[Leo Arias (elopio)]

I'll merge it before you break it :p

Thanks a lot Federico.

review: Approve

=== modified file '_integration-tests/data/tpl/control'
--- _integration-tests/data/tpl/control	2015-07-22 16:36:14 +0000
+++ _integration-tests/data/tpl/control	2015-10-07 22:21:11 +0000
@@ -1,4 +1,4 @@
 {{ $filter := .Filter }}
 {{ $test := .Test }}
-Test-Command: ./_integration-tests/reboot-wrapper {{ $test }} {{ if $filter }}-gocheck.f {{ $filter }}{{ end }}
+Test-Command: ./_integration-tests/scripts/reboot-wrapper {{ $test }} {{ if $filter }}-gocheck.f {{ $filter }}{{ end }}
 Restrictions: allow-stderr
 
=== added directory '_integration-tests/scripts'
=== renamed file '_integration-tests/tests/get_unpartitioned_space' => '_integration-tests/scripts/get_unpartitioned_space'
=== added file '_integration-tests/scripts/install-test-initramfs'
--- _integration-tests/scripts/install-test-initramfs	1970-01-01 00:00:00 +0000
+++ _integration-tests/scripts/install-test-initramfs	2015-10-07 22:21:11 +0000
@@ -0,0 +1,25 @@
+#! /bin/sh -e
+# Prepare the initramfs image for the tests.
+
+current_boot_dir=$1
+writable_percent=$2
+tree_dir=$(pwd)
+initrd_unpack_dir=$ADT_ARTIFACTS/initrd
+
+cp ${current_boot_dir}/initrd.img /tmp/initrd.xz
+cd /tmp/
+unxz /tmp/initrd.xz
+mkdir $initrd_unpack_dir || true
+cd $initrd_unpack_dir
+cpio -id < /tmp/initrd
+# We need sfdisk to shrink the partition in mbr.
+cp /sbin/sfdisk sbin
+# Libs required by sfdisk.
+cp /lib/*/libfdisk.so.1 lib
+cp /lib/*/libsmartcols.so.1 lib
+cp $tree_dir/_integration-tests/scripts/resize-writable-test scripts/local-premount/
+sed -i "s/writable_percent=.*/writable_percent=${writable_percent}/" scripts/local-premount/resize-writable-test
+sed -i '1i /scripts/local-premount/resize-writable-test "$@"\n[ -e /conf/param.conf ] && . /conf/param.conf' scripts/local-premount/ORDER
+find .|cpio -o -H newc|xz -c -7 --check=crc32 > initrd.img
+sudo rm $current_boot_dir/initrd.img
+sudo mv initrd.img $current_boot_dir
 
=== renamed file '_integration-tests/reboot-wrapper' => '_integration-tests/scripts/reboot-wrapper'
=== added file '_integration-tests/scripts/resize-writable-test'
--- _integration-tests/scripts/resize-writable-test	1970-01-01 00:00:00 +0000
+++ _integration-tests/scripts/resize-writable-test	2015-10-07 22:21:11 +0000
@@ -0,0 +1,123 @@
+#! /bin/sh -ex
+# initramfs local-premount script to resize writable for testing
+
+PREREQ=""
+
+# Output pre-requisites
+prereqs()
+{
+        echo "$PREREQ"
+}
+
+case "$1" in
+    prereqs)
+        prereqs
+        exit 0
+        ;;
+esac
+
+TMPFILE="/run/initramfs/old-table-test.txt"
+LOGFILE="/run/initramfs/resize-writable-test.log"
+
+writable_percent=100
+
+wait-for-root "LABEL=writable" "${ROOTDELAY:-180}" >/dev/null || true
+
+writable_part="$(findfs LABEL=writable)"
+
+syspath="$(dirname $(realpath /sys/class/block/$(basename $writable_part)))"
+device="$(realpath /dev/block/$(cat $syspath/dev))"
+partition=$(cat $syspath/$(basename $writable_part)/partition)
+
+device_size="$(($(cat $syspath/size)/2))"
+
+get_end()
+{
+    NUM=$1
+    lastpart="$(grep -cE ^'[0-9]{1,}': $TMPFILE)"
+    if [ "$lastpart" = "$NUM" ]; then
+        endsize="$(parted -ms $DEV unit B print| grep ^/ | cut -d: -f2|sed 's/B$//')"
+    else
+        # we are not at the end ! get the start of the next partition
+        # (minus 1 byte) instead of using the absolute end of the disk
+        endsize=$(($(parted -ms $DEV unit B print|grep ^$(($num+1)):|\
+            cut -d: -f2|sed 's/B$//')-1))
+    fi
+    echo "endsize: ${endsize}" >/dev/kmsg || true
+    endsize_percent=$(($endsize*$writable_percent/100))
+    echo "endsize percent: ${endsize_percent}" >/dev/kmsg || true
+    echo "$endsize_percent"
+}
+
+do_gpt()
+{
+    DEV=$1
+    # create new empty GPT
+    parted -s $DEV mklabel gpt
+    oIFS=$IFS
+    IFS=:
+    # re-create all partitions from backup table
+    grep -E ^'[0-9]{1,}': $TMPFILE|while read -r num start end size type name flags; do
+        if [ "$name" = "writable" ]; then
+            endsize=$(get_end $num)B
+            parted -s $DEV mkpart \"$name\" \"$type\" $start $endsize
+        else
+            flags="$(echo $flags|sed -e 's/[,;]//g' -e 's/ /:/')"
+            parted -s $DEV mkpart \"$name\" \"$type\" $start $end
+            if [ -n "$flags" ]; then
+                for flag in ${flags}; do
+                    parted -s $DEV set $num $flag on
+                done
+            fi
+        fi
+    done
+    IFS=$oIFS
+    parted -ms $DEV unit B print >/run/initramfs/new-gpt-table.txt 2>/dev/null
+}
+
+do_mbr()
+{
+    DEV=$1
+    PART=$2
+    endsize=$(get_end $PART)
+    # We can't just resize with parted because it prints a warning when you try to shrink the
+    # partition.
+    part_start="$(parted -ms $DEV unit B print | grep ^$PART: | cut -d: -f2 | sed 's/B$//')"
+    part_end=$(($endsize-$part_start))
+    part_end_sectors=$(($part_end/512))
+    echo ", ${part_end_sectors}" | sfdisk -f -N$PART $DEV
+}
+
+
+echo "initrd: resize ${writable_part} to ${writable_percent}% of the disk" >/dev/kmsg || true
+echo "initrd: see ${LOGFILE} for details" >/dev/kmsg || true
+
+# check the filesystem before attempting re-size
+e2fsck -fy $writable_part >>$LOGFILE 2>&1
+# shrink the filesystem to the minimum
+resize2fs -M $writable_part >>$LOGFILE 2>&1
+
+# back up the original partition table for later use or debugging
+parted -ms $device unit B print >$TMPFILE 2>/dev/null
+table="$(parted -ms $device print| grep ^/| cut -d: -f6)"
+case $table in
+    gpt)
+        # do_gpt needs the device name
+        do_gpt $device >>$LOGFILE 2>&1
+        ;;
+    mbr|msdos)
+        # do_mbr needs the device node and partition number
+        do_mbr $device $partition >>$LOGFILE 2>&1
+        resizeopts="-f"
+            ;;
+    *)
+        echo "unknown partition table type, not resizing" >>$LOGFILE
+        exit 0
+        ;;
+esac
+# make sure we re read the partition table in any case
+blockdev --rereadpt $device >>$LOGFILE 2>&1
+# check the filesystem before attempting re-size
+e2fsck -fy $writable_part >>$LOGFILE 2>&1
+# resize the filesystem to full size of the partition
+resize2fs $resizeopts $writable_part >>$LOGFILE 2>&1
 
=== modified file '_integration-tests/tests/initramfs_test.go'
--- _integration-tests/tests/initramfs_test.go	2015-09-08 18:25:45 +0000
+++ _integration-tests/tests/initramfs_test.go	2015-10-07 22:21:11 +0000
@@ -20,11 +20,15 @@
 package tests
 
 import (
+	"os"
 	"os/exec"
+	"path"
 	"strconv"
 	"strings"
 
+	"launchpad.net/snappy/_integration-tests/testutils/cli"
 	"launchpad.net/snappy/_integration-tests/testutils/common"
+	"launchpad.net/snappy/_integration-tests/testutils/partition"
 
 	"gopkg.in/check.v1"
 )
@@ -35,14 +39,69 @@
 	common.SnappySuite
 }
 
-func (s *initRAMFSSuite) TestFreeSpace(c *check.C) {
-	cmd := exec.Command("sh", "_integration-tests/tests/get_unpartitioned_space")
+func getFreeSpacePercent(c *check.C) float64 {
+	cmd := exec.Command("sh", "_integration-tests/scripts/get_unpartitioned_space")
 	free, err := cmd.Output()
 	c.Assert(err, check.IsNil, check.Commentf("Error running the script to get the free space: %s", err))
 	freePercent := strings.TrimRight(strings.TrimSpace(string(free)), "%")
 	freePercentFloat, err := strconv.ParseFloat(freePercent, 32)
 	c.Assert(err, check.IsNil,
 		check.Commentf("Error converting the free space percentage to float: %s", err))
-	c.Assert(freePercentFloat < 10, check.Equals, true,
-		check.Commentf("The free space at the end of the disk is greater than 10%"))
+	return freePercentFloat
+}
+
+func getCurrentBootDir(c *check.C) string {
+	system, err := partition.BootSystem()
+	c.Assert(err, check.IsNil, check.Commentf("Error getting the boot system: %s", err))
+	bootDir := partition.BootDir(system)
+	current, err := partition.CurrentPartition()
+	c.Assert(err, check.IsNil, check.Commentf("Error getting the current partition: %s", err))
+	return path.Join(bootDir, current)
+}
+
+func (s *initRAMFSSuite) SetUpTest(c *check.C) {
+	s.SnappySuite.SetUpTest(c)
+	if common.BeforeReboot() {
+		bootDir := getCurrentBootDir(c)
+		cli.ExecCommand(c, "cp", path.Join(bootDir, "initrd.img"), os.Getenv("ADT_ARTIFACTS"))
+	}
+}
+
+func (s *initRAMFSSuite) TearDownTest(c *check.C) {
+	s.SnappySuite.TearDownTest(c)
+	if !common.IsInRebootProcess() {
+		bootDir := getCurrentBootDir(c)
+		cli.ExecCommand(
+			c, "sudo", "mv", path.Join(os.Getenv("ADT_ARTIFACTS"), "initrd.img"), bootDir)
+	}
+}
+
+func (s *initRAMFSSuite) TestFreeSpaceWithoutResize(c *check.C) {
+	writablePercent := "95"
+	if common.BeforeReboot() {
+		bootDir := getCurrentBootDir(c)
+		cli.ExecCommand(
+			c, "sh", "-x", "_integration-tests/scripts/install-test-initramfs", bootDir, writablePercent)
+		common.Reboot(c)
+	} else if common.AfterReboot(c) {
+		common.RemoveRebootMark(c)
+		freeSpace := getFreeSpacePercent(c)
+		c.Assert(freeSpace, check.Equals, float64(5),
+			check.Commentf("The writable partition was resized"))
+	}
+}
+
+func (s *initRAMFSSuite) TestFreeSpaceWithResize(c *check.C) {
+	if common.BeforeReboot() {
+		bootDir := getCurrentBootDir(c)
+		writablePercent := "85"
+		cli.ExecCommand(
+			c, "sh", "-x", "_integration-tests/scripts/install-test-initramfs", bootDir, writablePercent)
+		common.Reboot(c)
+	} else if common.AfterReboot(c) {
+		common.RemoveRebootMark(c)
+		freeSpace := getFreeSpacePercent(c)
+		c.Assert(freeSpace < 10, check.Equals, true,
+			check.Commentf("The writable partition was not resized"))
+	}
 }
 
=== modified file '_integration-tests/testutils/common/common.go'
--- _integration-tests/testutils/common/common.go	2015-10-01 09:37:04 +0000
+++ _integration-tests/testutils/common/common.go	2015-10-07 22:21:11 +0000
@@ -62,7 +62,7 @@
 		"_integration-tests/data/output/testconfig.json")
 	c.Assert(err, check.IsNil, check.Commentf("Error reading config: %v", err))
 
-	if !isInRebootProcess() {
+	if !IsInRebootProcess() {
 		if Cfg.Update || Cfg.Rollback {
 			switchSystemImageConf(c, Cfg.TargetRelease, Cfg.TargetChannel, "0")
 			// Always use the installed snappy because we are updating from an old
@@ -275,7 +275,8 @@
 	return os.Getenv("ADT_REBOOT_MARK") == mark
 }
 
-func isInRebootProcess() bool {
+// IsInRebootProcess returns True if the suite needs to execute a reboot or has just rebooted.
+func IsInRebootProcess() bool {
 	return !CheckRebootMark("") || NeedsReboot()
 }