Merge lp:~sergiusens/goget-ubuntu-touch/installYaml into lp:goget-ubuntu-touch

[Sergio Schvezov]

Stamp a yaml with install/setup information

* Record sideloads to ensure "snappy update" doesn't try to upgrade a system that uses custom
  assets.

Creates a stamp file on the writable partition when --device-part= is specified. This is clearly trivial to defeat, but will allow snappy to refuse to update a system with such a stamp file.

There is a potential gotcha with this approach (albeit unlikely for anyone to hit it): if u-d-f is used to first download the set of s-i tar files and then u-d-f is called again with --device-part= pointing to the valid, official device part just downloaded, the stamp will still be created.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:160
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~jamesodhunt/goget-ubuntu-touch/record-sideload/+merge/256190/+edit-commit-message

http://jenkins.qa.ubuntu.com/job/goget-ubuntu-touch-ci/171/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/goget-ubuntu-touch-vivid-amd64-ci/84/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/goget-ubuntu-touch-vivid-armhf-ci/84/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/goget-ubuntu-touch-vivid-i386-ci/84/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/goget-ubuntu-touch-ci/171/rebuild

review: Needs Fixing (continuous-integration)

[James Hunt (jamesodhunt)]

The snappy part of this work is on MP: https://code.launchpad.net/~jamesodhunt/snappy/refuse-to-update-if-sideloaded/+merge/256296

[Sergio Schvezov (sergiusens)]

As I said on irc, this looks great; I would change:
- writeLocation to a factory reset safe location
- maybe instead of storing sideLoaded and do if sideLoaded you can just if coreCmd.DeviceTarball != "" (which is essentially the same).
- remove the changelog change, I will draft a proper one when this is released into the archive.

[James Hunt (jamesodhunt)]

Thanks for the comments. Branch updated.

[Sergio Schvezov (sergiusens)]

This looks really good, I liked the yaml idea you proposed on irc; and discussed it a bit with mvo and he was in favor of it too.

Just make it /boot/provisioning.yaml or /boot/setup.yaml or /boot/install.yaml (no need to hide it).

Regarding the layout, I'd just keep these:
meta:
    timestamp: 1429185020
    initial-revision: r345
    system-image-server: http://system-image.ubuntu.com

provisioning-tool:
    name: ubuntu-device-flash
    path: /usr/bin/ubuntu-device-flash
    version: 0.20snappy3-0ubuntu1

installation-options:
    size: 3
    size-unit: GB
    output: /tmp/bbb.img
    channel: ubuntu-core/devel-proposed
    device-part: /some/path/file.tgz
    developer-mode: true

Since platform and architecture are already in the oem snap, so it would be duplicate information (oem snaps will become mandatory).

Can't wait to see this make it in!

Oh, there's one minor inline comment.

[James Hunt (jamesodhunt)]

This branch + lp:~jamesodhunt/snappy/install.yaml are what I have so far.

Any thoughts on how we set Version sensibly in writeInstallYaml()? We have the same problem in snappy.

[James Hunt (jamesodhunt)]

The other slight wart is the "GB" in writeInstallYaml().

[James Hunt (jamesodhunt)]

lp:~jamesodhunt/snappy/install.yaml is now essentially complete.

[Sergio Schvezov (sergiusens)]

looks good, we just need to adjust some knobs here and there.

review: Needs Fixing

[James Hunt (jamesodhunt)]

Thanks for reviewing. Branch updated.

[James Hunt (jamesodhunt)]

It tested fine, but it looks like a snappy change earlier today has broken 'udf ... --oem file.snap':
________________________________________
$ sudo ubuntu-device-flash --verbose core --size 3 --channel devel-proposed --developer-mode --output=amd64.img --device-part device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz --oem ./generic-amd64.canonical_1.0_all.snap
Determining oem configuration
2015-04-21 16:46:57 ERROR snappy logger.go:199 ./generic-amd64.canonical_1.0_all.snap failed to install: oem package installation not allowed
./generic-amd64.canonical_1.0_all.snap failed to install: oem package installation not allowed

________________________________________

[James Hunt (jamesodhunt)]

This branch could be the offending one?

https://code.launchpad.net/~sergiusens/snappy/oemBlock/+merge/256919

[James Hunt (jamesodhunt)]

Or is it that udf just needs this branch merging?

https://code.launchpad.net/~sergiusens/goget-ubuntu-touch/oemFlag/+merge/256922

[Sergio Schvezov (sergiusens)]

> Or is it that udf just needs this branch merging?
>
> https://code.launchpad.net/~sergiusens/goget-ubuntu-
> touch/oemFlag/+merge/256922

yes, that is blocked in the pipeline from an MP by steve

[Sergio Schvezov (sergiusens)]

> > Or is it that udf just needs this branch merging?
> >
> > https://code.launchpad.net/~sergiusens/goget-ubuntu-
> > touch/oemFlag/+merge/256922
>
> yes, that is blocked in the pipeline from an MP by steve

Rebased and landed

[James Hunt (jamesodhunt)]

Re-tested:

= amd64 =

$ sudo ubuntu-device-flash --verbose core --size 3 --channel edge --output=amd64.img --device-part device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz --oem ./generic-amd64.canonical_1.
0_all.snap 15.04
Determining oem configuration
Fetching information from server...
Downloading and setting up...
Using a custom device tarball
Searching for hardware.yaml in device part
Download finished for /home/james/device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz
Partitioning...
Download finished for /home/james/.cache/ubuntuimages/ubuntu-core/15.04/edge/generic_amd64/version-26.tar.xz
Download finished for /home/james/.cache/ubuntuimages/pool/ubuntu-4fbe24300a5d5e443a0a455fc3428f780d273bd03673896cfcf51ba0eba4000d.tar.xz
Mapping...
Formatting...
Mapping...
Mounting...
Provisioning...
Installing ./generic-amd64.canonical_1.0_all.snap
Unmounting...
Unmapping...
New image complete
Summary:
 Output: amd64.img
 Architecture: amd64
 Channel: edge
 Version: 26
Launch by running: kvm -m 768 amd64.img
$ sudo kpartx -avs amd64.img
$ sudo mount /dev/mapper/loop0p2 /mnt
$ cat /mnt/install.yaml
meta:
  timestamp: 2015-04-22T09:08:41.393114048+01:00
  initial-version: "26"
  system-image-server: https://system-image.ubuntu.com
tool:
  name: ubuntu-device-flash
  path: /usr/bin/ubuntu-device-flash
  version: ""
options:
  size: 3
  size-unit: GB
  output: amd64.img
  channel: edge
  device-part: device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz
  oem: ./generic-amd64.canonical_1.0_all.snap

= armhf =

$ sudo ubuntu-device-flash --verbose core --size 3 --channel edge --output=armhf.img --device-part device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz --device generic_armhf 15.04 --oem ./beagleblack.canonical_1.7_all.snap
Determining oem configuration
Fetching information from server...
WARNING: this option should only be used to build azure images
Downloading and setting up...
Using a custom device tarball
Searching for hardware.yaml in device part
Download finished for /home/james/device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa0c431.tar.xz
Download finished for /home/james/.cache/ubuntuimages/ubuntu-core/15.04/edge/generic_armhf/version-22.tar.xz
Download finished for /home/james/.cache/ubuntuimages/pool/ubuntu-3599d2c285a4bd0f54d0b13a290c4f9c8b10ae10354afee55237994af7a28f1c.tar.xz
Partitioning...
Mapping...
Formatting...
Mapping...
Mounting...
Provisioning...
Installing ./beagleblack.canonical_1.7_all.snap
Replicating system-a into system-b
Unmounting...
Unmapping...
New image complete
Summary:
 Output: armhf.img
 Architecture: armhf
 Channel: edge
 Version: 22
$ sudo kpartx -avs armhf.img
$ sudo mount /dev/mapper/loop0p2 /mnt
$ cat /mnt/install.yaml
meta:
  timestamp: 2015-04-22T09:22:29.418832806+01:00
  initial-version: "22"
  system-image-server: https://system-image.ubuntu.com
tool:
  name: ubuntu-device-flash
  path: /usr/bin/ubuntu-device-flash
  version: ""
options:
  size: 3
  size-unit: GB
  output: armhf.img
  channel: edge
  device-part: device-e4e9d6632c5266aac95bde68729bcce796509f6abeac8c7ebca25be34aa...

[James Hunt (jamesodhunt)]

Grr, of course although this "works" for grub, it won't work on a running system where it will have to live in /boot/grub/.

[Sergio Schvezov (sergiusens)]

> Grr, of course although this "works" for grub, it won't work on a running
> system where it will have to live in /boot/grub/.

Right, but actually wrong, this is what I mentioned wrt can't find it on a running system, it's actually under /boot/efi/

[James Hunt (jamesodhunt)]

Branch updated.

[Sergio Schvezov (sergiusens)]

For this to land an updated lp:snappy needs to make it to wily

[Sergio Schvezov (sergiusens)]

review: Abstain

[Michael Vogt (mvo)]

Looks good.

review: Approve

[John Lenton (chipaca)]

go go go

review: Approve

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~sergiusens/goget-ubuntu-touch/installYaml into lp:goget-ubuntu-touch failed. Below is the output from the failed tests.

Checking formatting
Formatting wrong in following files
ubuntu-device-flash/core.go

# we always run in a fresh dir in tarmac
export GOPATH=$(mktemp -d)
trap 'rm -rf "$GOPATH"' EXIT

# this is a hack, but not sure tarmac is golang friendly
mkdir -p $GOPATH/src/launchpad.net/goget-ubuntu-touch
cp -a . $GOPATH/src/launchpad.net/goget-ubuntu-touch
cd $GOPATH/src/launchpad.net/goget-ubuntu-touch

./run-checks

[Sergio Schvezov (sergiusens)]

review: Approve

[Snappy Tarmac (snappydevtarmac)]

The attempt to merge lp:~sergiusens/goget-ubuntu-touch/installYaml into lp:goget-ubuntu-touch failed. Below is the output from the failed tests.

Checking formatting
Installing godeps
Install golint
Obtaining dependencies
update github.com/blakesmith/ar failed; trying to fetch newer version
update github.com/cheggaaa/pb failed; trying to fetch newer version
github.com/blakesmith/ar now at c9a977dd0cc1392b023382c7bfa5a22af8d3b730
update github.com/jessevdk/go-flags failed; trying to fetch newer version
github.com/cheggaaa/pb now at e8c7cc515bfde3e267957a3b110080ceed51354e
update github.com/mvo5/goconfigparser failed; trying to fetch newer version
github.com/jessevdk/go-flags now at 15347ef417a300349807983f15af9e65cd2e1b3a
update gopkg.in/yaml.v2 failed; trying to fetch newer version
github.com/mvo5/goconfigparser now at 26426272dda20cc76aa1fa44286dc743d2972fe8
update launchpad.net/gocheck failed; trying to fetch newer version
gopkg.in/yaml.v2 now at 49c95bdc21843256fb6c4e0d370a05f24a0bf213
update launchpad.net/snappy failed; trying to fetch newer version
launchpad.net/gocheck now at <email address hidden>
launchpad.net/snappy now at snappy_tarmac-20150609131158-tl9ndj3aor8m3kna
Building

# we always run in a fresh dir in tarmac
export GOPATH=$(mktemp -d)
trap 'rm -rf "$GOPATH"' EXIT

# this is a hack, but not sure tarmac is golang friendly
mkdir -p $GOPATH/src/launchpad.net/goget-ubuntu-touch
cp -a . $GOPATH/src/launchpad.net/goget-ubuntu-touch
cd $GOPATH/src/launchpad.net/goget-ubuntu-touch

./run-checks
ubuntu-device-flash/core.go:28:2: cannot find package "launchpad.net/snappy/provisioning" in any of:
 /usr/lib/go/src/pkg/launchpad.net/snappy/provisioning (from $GOROOT)
 /home/tarmac/tmp/tmp.9mZ8cTA8TJ/src/launchpad.net/snappy/provisioning (from $GOPATH)

[lp:~sergiusens/goget-ubuntu-touch/installYaml]

191. By Sergio Schvezov on 2015-06-17

godeps godeps godeps...

[Sergio Schvezov (sergiusens)]

review: Approve

=== modified file 'dependencies.tsv'
--- dependencies.tsv	2015-06-09 13:57:34 +0000
+++ dependencies.tsv	2015-06-17 12:21:42 +0000
@@ -4,4 +4,4 @@
 github.com/mvo5/goconfigparser	git	26426272dda20cc76aa1fa44286dc743d2972fe8	2015-02-12T09:37:50Z
 gopkg.in/yaml.v2	git	49c95bdc21843256fb6c4e0d370a05f24a0bf213	2015-02-24T22:57:58Z
 launchpad.net/gocheck	bzr	[e-mail]-20140225173054-xu9zlkf9kxhvow02	87
-launchpad.net/snappy	bzr	snappy_tarmac-20150609131158-tl9ndj3aor8m3kna	493
+launchpad.net/snappy	bzr	snappy_tarmac-20150617101203-6hsfxqrckt6koobw	512

=== modified file 'diskimage/common.go'
--- diskimage/common.go	2015-06-15 15:32:25 +0000
+++ diskimage/common.go	2015-06-17 12:21:42 +0000
@@ -54,6 +54,7 @@
 }
 
 type SystemImage interface {
+	Boot() string
 	System() string
 	Writable() string
 }

=== modified file 'ubuntu-device-flash/core.go'
--- ubuntu-device-flash/core.go	2015-06-12 04:39:13 +0000
+++ ubuntu-device-flash/core.go	2015-06-17 12:21:42 +0000
@@ -25,6 +25,7 @@
 	"gopkg.in/yaml.v2"
 	"launchpad.net/snappy/helpers"
 	"launchpad.net/snappy/progress"
+	"launchpad.net/snappy/provisioning"
 	"launchpad.net/snappy/release"
 	"launchpad.net/snappy/snappy"
 
@@ -277,6 +278,9 @@
 			return err
 		}
 
+		// avoid passing more args to setup()
+		globalArgs.Revision = image.Version
+
 		if err := coreCmd.setup(img, filePathChan, len(image.Files)); err != nil {
 			return err
 		}
@@ -392,7 +396,7 @@
 		}
 	}
 
-	return nil
+	return coreCmd.writeInstallYaml(img.Boot())
 }
 
 func (coreCmd *CoreCmd) install(systemPath string) error {
@@ -586,6 +590,58 @@
 	return oem, nil
 }
 
+// Creates a YAML file inside the image that contains metadata relating
+// to the installation.
+func (coreCmd CoreCmd) writeInstallYaml(bootMountpoint string) error {
+	selfPath, err := exec.LookPath(os.Args[0])
+	if err != nil {
+		return err
+	}
+
+	bootDir := ""
+
+	switch coreCmd.oem.OEM.Hardware.Bootloader {
+	// Running systems use a bindmount for /boot/grub, but
+	// since the system isn't booted, create the file in the
+	// real location.
+	case "grub":
+		bootDir = "/EFI/ubuntu/grub"
+	}
+
+	installYamlFilePath := filepath.Join(bootMountpoint, bootDir, provisioning.InstallYamlFile)
+
+	i := provisioning.InstallYaml{
+		InstallMeta: provisioning.InstallMeta{
+			Timestamp:         time.Now(),
+			InitialVersion:    fmt.Sprintf("%d", globalArgs.Revision),
+			SystemImageServer: globalArgs.Server,
+		},
+		InstallTool: provisioning.InstallTool{
+			Name: filepath.Base(selfPath),
+			Path: selfPath,
+			// FIXME: we don't know our own version yet :)
+			// Version: "???",
+		},
+		InstallOptions: provisioning.InstallOptions{
+			Size:          coreCmd.Size,
+			SizeUnit:      "GB",
+			Output:        coreCmd.Output,
+			Channel:       coreCmd.Channel,
+			DevicePart:    coreCmd.Development.DevicePart,
+			Oem:           coreCmd.Oem,
+			DeveloperMode: coreCmd.Development.DeveloperMode,
+		},
+	}
+
+	data, err := yaml.Marshal(&i)
+	if err != nil {
+		return err
+	}
+
+	// the file isn't supposed to be modified, hence r/o.
+	return ioutil.WriteFile(installYamlFilePath, data, 0444)
+}
+
 func extractHWDescription(path string) (hw diskimage.HardwareDescription, err error) {
 	// hack to circumvent https://code.google.com/p/go/issues/detail?id=1435
 	if syscall.Getuid() == 0 {