Merge lp:~morphis/goget-ubuntu-touch/add-turbo-bootstrap-support into lp:goget-ubuntu-touch

[Simon Fels]

Add bootstrap support for turbo/frieza/cooler devices

The bootloader on turbo/frieza/cooler does not have support for fastboot format <partition> or fastboot boot <image>. Therefor we have to workaround this by using a OEM specific bootloader command to reboot into recovery and adjusting the cleanup of the /cache partition before we push any files.

Also this adds an alias for turbo which is named "smdk" from the bootloader which can't be changed anymore on our production devices.

Corresponding MP for turbo to add the new "format cache" command can be found at https://code.launchpad.net/~morphis/zhongshan/+git/platform_bootable_recovery/+merge/291582

Opened this for a first discussion if we want to add device specific quirks this way.

[Alex Tu (alextu)]

verified the prebuilt one [1] and it works fine on latest engineer mode recovery which enabled adb[2] by command:
$ ubuntu-device-flash touch --channel ubuntu-touch/rc-proposed/meizu.en --recovery-image=${the-recovery[2]}.img --bootstrap --device turbo

[1] https://private-fileshare.canonical.com/~morphis/ubuntu-device-flash
[2] https://git.launchpad.net/~device-release/turbo/+git/recovery/plain/recovery.img

[Alex Tu (alextu)]

LGTM.

review: Approve

[Sergio Schvezov (sergiusens)]

Thanks for your contribution, interesting to see what happened when bootloaders are not taken care of early ;-)

The only thing that worries me here is the fact that we only really format cache because there could be files from a failed "flash" there, by wiping we make sure there is enough space. Maybe a message about this when using turbo would be good and make for an easier troubleshoot.

[Alex Tu (alextu)]

review: Needs Fixing

[Alex Tu (alextu)]

refer the conversation on irc, we might need to do some more operations to simulate fastboot -w then do update, so the proposed way might be 1. reboot recovery to erase data and cache partition 2. reboot recovery again to do update.

<ondra> alextu as for MP for recovery, so I still believe format cache is needed, but it should do proper cache format, or at least delete all from cache, not skip files under /cache/recovery
<ondra> alextu to fix cases when cache is running out of space and there are some old hanging files
<alextu> ondra, hmm ... agree, it seems no reason to keep updates which in /cache/recovery .
* alextu going to upload another mp for that.
<ondra> alextu problem is that then it's not gonna solve problem morphis|off is addressing with that change, when he wants to emulate fastboot -w
<ondra> alextu which should format userdata and cache
<ondra> alextu but then we wipe tarballs which we need to do install update, so it's kinda broken
<ondra> alextu only thing I can think of is to a) move format chache to the end, so it's formatted at the end of update, so it's safe to wipe it all, or we will need two runs, something like "format cache and userdata and reboot back to recovery" so u-d-f can start pushing files
<alextu> ondra, hmm ... sounds option b) is more like what we want to simulate
<ondra> alextu yep
<alextu> ondra, option a) will also remove cache/device-build which is the version of device tarball which updated by u-d-f
<ondra> alextu true

[Bin Li (binli)]

After talked with abeato and morphis, we prepare to add 'rm -rf /cache/*' in u-d-f.

[Bin Li (binli)]

And I read the code, I found the u-d-f will download the tarballs first, then fastboot.format /cache, and reboot into recovery, then call bitPusher to push these tarballs, and before push it will rm previous tarball like below. After that u-d-f push the tarballs into recovery.

adb.Shell("rm -rf /cache/recovery/*.xz /cache/recovery/*.xz.asc")

So for simple I think we just change above line. Is it ok? Thanks!

adb.Shell("rm -rf /cache/*")

[Bin Li (binli)]

Here is my change, and I will test it after building.

https://code.launchpad.net/~binli/goget-ubuntu-touch/refine-turbo-bootstrap-support/+merge/296282

[Sergio Schvezov (sergiusens)]

review: Approve

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Sam behaviour has been observed in frieza and cooler. Can we simply try the "rm -rf /cache/*" if format cache has failed?

review: Needs Fixing

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Tested on cooler now, works fine.

review: Approve

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

Tested on frieza too. Worked as expected.

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~morphis/goget-ubuntu-touch/add-turbo-bootstrap-support into lp:goget-ubuntu-touch failed. Below is the output from the failed tests.

Checking formatting
Formatting wrong in following files
ubuntu-device-flash/touch.go

# we always run in a fresh dir in tarmac
export GOPATH=$(mktemp -d)
trap 'rm -rf "$GOPATH"' EXIT

# this is a hack, but not sure tarmac is golang friendly
mkdir -p $GOPATH/src/launchpad.net/goget-ubuntu-touch
cp -a . $GOPATH/src/launchpad.net/goget-ubuntu-touch
cd $GOPATH/src/launchpad.net/goget-ubuntu-touch

./run-checks

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~morphis/goget-ubuntu-touch/add-turbo-bootstrap-support into lp:goget-ubuntu-touch failed. Below is the output from the failed tests.

Checking formatting
Formatting wrong in following files
ubuntu-device-flash/touch.go

# we always run in a fresh dir in tarmac
export GOPATH=$(mktemp -d)
trap 'rm -rf "$GOPATH"' EXIT

# this is a hack, but not sure tarmac is golang friendly
mkdir -p $GOPATH/src/launchpad.net/goget-ubuntu-touch
cp -a . $GOPATH/src/launchpad.net/goget-ubuntu-touch
cd $GOPATH/src/launchpad.net/goget-ubuntu-touch

./run-checks

[lp:~morphis/goget-ubuntu-touch/add-turbo-bootstrap-support]

239. By Simon Fels on 2016-07-21

Correct formatting

[Robert Bruce Park (robru)]

This MP was released through bileto, but then tarmac merged it prematurely, confusing bileto. If you intend to continue using goget-ubuntu-touch in bileto you need to disable tarmac.

=== modified file 'devices/fastboot.go'
--- devices/fastboot.go	2015-04-16 22:59:18 +0000
+++ devices/fastboot.go	2016-07-21 15:57:39 +0000
@@ -85,3 +85,10 @@
 	}
 	return device, err
 }
+
+// Send OEM specific commands to bootloader
+func (fastboot Fastboot) SendOemCommand(command string) (err error) {
+	cmd := append(fastboot.params, []string{"oem", command}...)
+	err = exec.Command(fastbootCommand, cmd...).Run()
+	return err
+}
 
=== modified file 'ubuntu-device-flash/touch.go'
--- ubuntu-device-flash/touch.go	2016-06-13 14:19:39 +0000
+++ ubuntu-device-flash/touch.go	2016-07-21 15:57:39 +0000
@@ -2,7 +2,7 @@
 // ubuntu-device-flash - Tool to download and flash devices with an Ubuntu Image
 //                       based system
 //
-// Copyright (c) 2013-2014 Canonical Ltd.
+// Copyright (c) 2013-2016 Canonical Ltd.
 //
 // Written by Sergio Schvezov <[e-mail]>
 //
@@ -35,6 +35,7 @@
 	Bootstrap     bool   `long:"bootstrap" description:"bootstrap the system, do this from the bootloader"`
 	Wipe          bool   `long:"wipe" description:"Clear all data after flashing"`
 	Serial        string `long:"serial" description:"Serial of the device to operate"`
+	AdbSerial     string `long:"adb-serial" description:"Serial for ADB to specify the device to operate"`
 	DeveloperMode bool   `long:"developer-mode" description:"Enables developer mode after the factory reset, this is meant for automation and makes the device insecure by default (requires --password)"`
 	AdbKeys       string `long:"adb-keys" description:"Specify a local adb keys files, instead of using default ~/.android/adbkey.pub (requires --developer-mode)"`
 	DeviceTarball string `long:"device-tarball" description:"Specify a local device tarball to override the one from the server (using official Ubuntu images with different device tarballs)"`
@@ -234,17 +235,60 @@
 		if err := touchCmd.fastboot.Flash("recovery", recovery); err != nil {
 			return errors.New("can't flash recovery image")
 		}
-		if err := touchCmd.fastboot.Format("cache"); err != nil {
-			log.Print("Cache formatting was not successful, flashing may fail, " +
-				"check your partitions on device")
-		}
-
-		if err := touchCmd.fastboot.BootImage(recovery); err != nil {
-			return errors.New("Can't boot recovery image")
-		}
+
+		var deviceSupportsCacheFormat bool = true
+		var deviceSupportsBootImage bool = true
+		var deviceSupportsOemRebootRecovery bool = false
+
+		if touchCmd.Device == "turbo" {
+			deviceSupportsCacheFormat = false
+			deviceSupportsBootImage = false
+			deviceSupportsOemRebootRecovery = true
+		}
+		if touchCmd.Device == "frieza" || touchCmd.Device == "cooler" {
+			deviceSupportsCacheFormat = false
+		}
+
+		if deviceSupportsCacheFormat {
+			if err := touchCmd.fastboot.Format("cache"); err != nil {
+				log.Print("Cache formatting was not successful, flashing may fail, " +
+					"check your partitions on device")
+			}
+		}
+
+		if deviceSupportsBootImage {
+			if err := touchCmd.fastboot.BootImage(recovery); err != nil {
+				return errors.New("Can't boot recovery image")
+			}
+		} else if deviceSupportsOemRebootRecovery {
+			// Some bootloaders does not support the BootImage command so we have
+			// to flash the recovery first and then reboot through a OEM specific
+			// command the bootloader offers.
+			if err := touchCmd.fastboot.SendOemCommand("reboot recovery"); err != nil {
+				return errors.New("Can't reboot device")
+			}
+		} else {
+			log.Print("We can't reboot your device automatically. Please reboot " +
+				"your device manually to recovery mode.")
+		}
+
+		log.Print("Waiting for device to enter recovery mode ...")
 		if err := touchCmd.adb.WaitForRecovery(); err != nil {
 			return err
 		}
+
+		if !deviceSupportsCacheFormat {
+			// On some devices we can't run `fastboot format cache` as the bootloader
+			// does not support this command. We're erasing all bits from the
+			// cache partition manually once we've booted into the recovery.
+			if _, err := touchCmd.adb.Shell("rm -rf /cache/*"); err != nil {
+				log.Fatal("Cannot cleanup /cache/ to ensure clean deployment", err)
+			}
+
+			if _, err := touchCmd.adb.Shell("mkdir /cache/recovery"); err != nil {
+				log.Fatal("Failed to recreate filesystem structure on cache partition", err)
+			}
+		}
 	}
 	go bitPusher(touchCmd.adb, files, done)
 	for i := 0; i < totalFiles; i++ {
@@ -309,10 +353,24 @@
 		touchCmd.fastboot.SetSerial(touchCmd.Serial)
 	}
 
+	if touchCmd.AdbSerial != "" {
+		touchCmd.adb.SetSerial(touchCmd.AdbSerial)
+	}
+
 	if touchCmd.Device == "" {
 		if touchCmd.Bootstrap {
 			log.Print("Expecting the device to be in the bootloader... waiting")
 			touchCmd.Device, err = touchCmd.fastboot.GetDevice()
+
+			// For turbo it was missed to put the proper name into the bootloader
+			// and this can't be changed anymore after production already started.
+			// Reflashing the bootloader through an OTA update to fix this is also
+			// very unlikely to happen so we have to work around here and make sure
+			// this never happens again.
+			if touchCmd.Device == "smdk" {
+				touchCmd.Device = "turbo"
+			}
+
 			return err
 		} else {
 			log.Print("Expecting the device to expose an adb interface...")