Merge lp:~mvo/snappy/snappy-snapfs-mount into lp:~snappy-dev/snappy/snappy-moved-to-github

[Michael Vogt]

Add support to (auto)mount the snapfs based snaps. It will unpack the meta-data to disk (to speedup snappy list etc) but keep the rest in the squashfs blob and (auto)mount at runtime.

This is a bit of a RFC branch, I'm not yet super happy about it. It will put the blob into
  /apps/$pkg/$version/blob.snap
and will mount that to:
  /apps/$pkg/$version/run

The downside of this approach is of course that the binary/service paths of the deb backend and the squashfs backend are no longer identical which makes the code that writes that a bit ugly.

I think we have three option and would love to get feedback which one is best:
 1. keep it like its done in this branch given and cleanup once the clickdeb backend is killed
 2. change deb backend to write the actual (non-meta) data to "run/" as well
 3. put the blob.snap as "/apps/$pkg/$pkg_$version.snap" and auto-mount it to /apps/$pkg/$version
    (which complicates the removal a bit because its no longer enough to just rm -rf the /apps/$pkg/$version dir)

[John Lenton (chipaca)]

4. put the .snap in /var/snappy/blobs?

[John Lenton (chipaca)]

*/var/lib/snappy/blobs i meant. But /var/lib/snappy/snaps is probably better.

A small complication of remove and purge, but other than that it'd work, right?

[Michael Vogt (mvo)]

Thanks John! So with /var/lib/snappy/snaps we would setup the automount unit to mount /apps/$name/$ver - this would mean that a "snappy list" would always automount the snaps. My current approach was to only mount them when they are used and unpack the meta/* only. But maybe thats premature optimization(?).

[John Lenton (chipaca)]

I think it might be. Let me tinker with list a bit tomorrow -- i think it can be made to work without loading package.yaml's.

[John Lenton (chipaca)]

Tinkered now instead.

To make list not load package.yaml's, you'd need to

* move it to husks, or to the rest api. Latter needs doing anyway; former would be short-term quick if needed.

* add a concreter that knew about squashfs:
 - `IsInstalled` checks for presence of blob instead of package.yaml
 - `Load` loaded a RemoteSnapPart from the locally-stored remote manifest. Would be missing InstalledSize, which list doesn't output. Would probably mean moving RemoteSnapPart to its own package (\o/)

If doing it via the REST API, /1.0/packages would need to recover the ?sources=local option, and an additional flag to tell it to do things this way. Piece of cake.

So, I'd suggest do it the clean way, mounting to /$type/$name/$ver (i think we agree it's the clean way?). Then we benchmark and decide whether to do the above now, or later.

HTH,

[lp:~mvo/snappy/snappy-snapfs-mount]

754. By Michael Vogt on 2015-10-16

merged lp:snappy

755. By Michael Vogt on 2015-10-16

refactor and get rid of runPrefix

[Michael Vogt (mvo)]

Thanks, I pondered over this too and I like the approach of /var/lib/snappy/snaps and that we can use husks^Wleightweight to introspect without the full mount dance. I will update the MP accordingly.

[lp:~mvo/snappy/snappy-snapfs-mount]

756. By Michael Vogt on 2015-10-16

ensure removal removes the snapfs and refactor

757. By Michael Vogt on 2015-10-16

update comments

758. By Michael Vogt on 2015-10-16

unmount after deactivate

[Michael Vogt (mvo)]

A new look would be welcome :)

[lp:~mvo/snappy/snappy-snapfs-mount]

759. By Michael Vogt on 2015-10-16

improve comments

[John Lenton (chipaca)]

Some comments.

Need to iterate the review a little.

[Sergio Schvezov (sergiusens)]

I can't beat Chipaca on reviews but I did manage to add a bite of a comment that is not a problem today but will avoid headaches later. It is an 'if you want to do this' type of comment.

Good work

[lp:~mvo/snappy/snappy-snapfs-mount]

760. By Michael Vogt on 2015-10-16

address review comments

761. By Michael Vogt on 2015-10-16

refactor and ove MountUnit handling to systemd package

762. By Michael Vogt on 2015-10-16

add more tests

763. By Michael Vogt on 2015-10-16

add comment about little endian squashfs

[Michael Vogt (mvo)]

Thanks for the review! I addressed the comemnts, please let me know if its sufficient.

[lp:~mvo/snappy/snappy-snapfs-mount]

764. By Michael Vogt on 2015-10-16

pkg/snapfs/snapfs.go: use filepath.Clean() in BlobPath

[Michael Vogt (mvo)]

Thanks! Adressing more stuff.

[John Lenton (chipaca)]

This looks good enough, now :)

review: Approve

[Gustavo Niemeyer (niemeyer)]

I'm adding a few comments below. Please feel free to merge this once you're personally happy about it.

review: Approve

[lp:~mvo/snappy/snappy-snapfs-mount]

765. By Michael Vogt on 2015-10-20

merged lp:snappy

766. By Michael Vogt on 2015-10-20

use properly terminated sentences (thanks gustavo)

767. By Michael Vogt on 2015-10-20

more documenation and improve error reporting (thanks Gustavo)

[Michael Vogt (mvo)]

Thanks Gustavo for this excellent review. I started fixing some of the outlined issues and it looks like I'm running out of time. I will continue later but be assured that I will address every point in the review :)

Unmerged revisions

767. By Michael Vogt on 2015-10-20

more documenation and improve error reporting (thanks Gustavo)

766. By Michael Vogt on 2015-10-20

use properly terminated sentences (thanks gustavo)

765. By Michael Vogt on 2015-10-20

merged lp:snappy

764. By Michael Vogt on 2015-10-16

pkg/snapfs/snapfs.go: use filepath.Clean() in BlobPath

763. By Michael Vogt on 2015-10-16

add comment about little endian squashfs

762. By Michael Vogt on 2015-10-16

add more tests

761. By Michael Vogt on 2015-10-16

refactor and ove MountUnit handling to systemd package

760. By Michael Vogt on 2015-10-16

address review comments

759. By Michael Vogt on 2015-10-16

improve comments

758. By Michael Vogt on 2015-10-16

unmount after deactivate

=== added file 'debian/ubuntu-snappy-cli.dirs'
--- debian/ubuntu-snappy-cli.dirs	1970-01-01 00:00:00 +0000
+++ debian/ubuntu-snappy-cli.dirs	2015-10-20 15:38:57 +0000
@@ -0,0 +1,1 @@
+/var/lib/snappy/snaps

=== modified file 'dirs/dirs.go'
--- dirs/dirs.go	2015-09-25 15:27:11 +0000
+++ dirs/dirs.go	2015-10-20 15:38:57 +0000
@@ -35,6 +35,7 @@
 	LocaleDir        string
 	SnapIconsDir     string
 	SnapMetaDir      string
+	SnapBlobDir      string
 
 	SnapBinariesDir  string
 	SnapServicesDir  string
@@ -59,6 +60,7 @@
 	SnapSeccompDir = filepath.Join(rootdir, SnappyDir, "seccomp", "profiles")
 	SnapIconsDir = filepath.Join(rootdir, SnappyDir, "icons")
 	SnapMetaDir = filepath.Join(rootdir, SnappyDir, "meta")
+	SnapBlobDir = filepath.Join(rootdir, SnappyDir, "snaps")
 
 	SnapBinariesDir = filepath.Join(SnapAppsDir, "bin")
 	SnapServicesDir = filepath.Join(rootdir, "/etc/systemd/system")

=== modified file 'pkg/clickdeb/deb.go'
--- pkg/clickdeb/deb.go	2015-10-08 08:29:16 +0000
+++ pkg/clickdeb/deb.go	2015-10-20 15:38:57 +0000
@@ -40,33 +40,34 @@
 
 var (
 	// ErrSnapInvalidContent is returned if a snap package contains
-	// invalid content
+	// invalid content.
 	ErrSnapInvalidContent = errors.New("snap contains invalid content")
 
-	// ErrMemberNotFound is returned when a tar member is not found in the archive
+	// ErrMemberNotFound is returned when a tar member is not found in
+	// the archive.
 	ErrMemberNotFound = errors.New("member not found")
 )
 
-// ErrUnpackFailed is the error type for a snap unpack problem
+// ErrUnpackFailed is the error type for a snap unpack problem.
 type ErrUnpackFailed struct {
 	snapFile string
 	instDir  string
 	origErr  error
 }
 
-// ErrUnpackFailed is returned if unpacking a snap fails
+// ErrUnpackFailed is returned if unpacking a snap fails.
 func (e *ErrUnpackFailed) Error() string {
 	return fmt.Sprintf("unpack %s to %s failed with %s", e.snapFile, e.instDir, e.origErr)
 }
 
-// simple pipe based xz reader
+// Simple pipe based xz reader.
 func xzPipeReader(r io.Reader) io.Reader {
 	pr, pw := io.Pipe()
 	cmd := exec.Command("xz", "--decompress", "--stdout")
 	cmd.Stdin = r
 	cmd.Stdout = pw
 
-	// run xz in its own go-routine
+	// run xz in its own go-routine.
 	go func() {
 		pw.CloseWithError(cmd.Run())
 	}()
@@ -74,7 +75,7 @@
 	return pr
 }
 
-// simple pipe based xz writer
+// Simple pipe based xz writer.
 type xzPipeWriter struct {
 	cmd *exec.Cmd
 	w   io.Writer
@@ -93,7 +94,7 @@
 	x.cmd.Stdout = x.w
 	x.cmd.Stderr = os.Stderr
 
-	// Start is async
+	// Start is async.
 	x.cmd.Start()
 
 	return x
@@ -108,7 +109,7 @@
 	return x.cmd.Wait()
 }
 
-// ensure that the content of our data is valid:
+// Ensure that the content of our data is valid:
 // - no relative path allowed to prevent writing outside of the parent dir
 func clickVerifyContentFn(path string) (string, error) {
 	path = filepath.Clean(path)
@@ -122,7 +123,7 @@
 }
 
 // ClickDeb provides support for the "click" containers (a special kind of
-// deb package)
+// deb package).
 type ClickDeb struct {
 	file *os.File
 }
@@ -145,29 +146,34 @@
 	return &ClickDeb{f}, nil
 }
 
-// Name returns the Name of the backing file
+// Name returns the Name of the backing file.
 func (d *ClickDeb) Name() string {
 	return d.file.Name()
 }
 
-// Close closes the backing file
+// NeedsAutoMountUnit returns false.
+func (d *ClickDeb) NeedsAutoMountUnit() bool {
+	return false
+}
+
+// Close closes the backing file.
 func (d *ClickDeb) Close() error {
 	return d.file.Close()
 }
 
-// Verify checks that the clickdeb is signed
+// Verify checks that the clickdeb is signed.
 func (d *ClickDeb) Verify(allowUnauthenticated bool) error {
 	return Verify(d.Name(), allowUnauthenticated)
 }
 
 // ControlMember returns the content of the given control member file
-// (e.g. the content of the "manifest" file in the control.tar.gz ar member)
+// (e.g. the content of the "manifest" file in the control.tar.gz ar member).
 func (d *ClickDeb) ControlMember(controlMember string) (content []byte, err error) {
 	return d.member("control.tar", controlMember)
 }
 
 // MetaMember returns the content of the given meta file (e.g. the content of
-// the "package.yaml" file) from the data.tar.gz ar member's meta/ directory
+// the "package.yaml" file) from the data.tar.gz ar member's meta/ directory.
 func (d *ClickDeb) MetaMember(metaMember string) (content []byte, err error) {
 	return d.member("data.tar", filepath.Join("meta", metaMember))
 }
@@ -213,7 +219,7 @@
 }
 
 // ExtractHashes reads "hashes.yaml" from the clickdeb and writes it to
-// the given directory
+// the given directory.
 func (d *ClickDeb) ExtractHashes(dir string) error {
 	hashesFile := filepath.Join(dir, "hashes.yaml")
 	hashesData, err := d.ControlMember("hashes.yaml")
@@ -226,7 +232,7 @@
 
 // Unpack unpacks the data.tar.{gz,bz2,xz} into the given target directory
 // with click specific verification, i.e. no files will be extracted outside
-// of the targetdir (no ".." inside the data.tar is allowed)
+// of the targetdir (no ".." inside the data.tar is allowed).
 func (d *ClickDeb) Unpack(targetDir string) error {
 	var err error
 
@@ -244,7 +250,7 @@
 	return helpers.UnpackTar(dataReader, targetDir, clickVerifyContentFn)
 }
 
-// FIXME: this should move into the "ar" library itself
+// FIXME: this should move into the "ar" library itself.
 func addFileToAr(arWriter *ar.Writer, filename string) error {
 	dataF, err := os.Open(filename)
 	if err != nil {
@@ -276,7 +282,7 @@
 	return nil
 }
 
-// FIXME: this should move into the "ar" library itself
+// FIXME: this should move into the "ar" library itself.
 func addDataToAr(arWriter *ar.Writer, filename string, data []byte) error {
 	size := int64(len(data))
 	hdr := &ar.Header{
@@ -295,11 +301,11 @@
 }
 
 // tarExcludeFunc is a helper for tarCreate that is called for each file
-// that is about to be added. If it returns "false" the file is skipped
+// that is about to be added. If it returns "false" the file is skipped.
 type tarExcludeFunc func(path string) bool
 
 // tarCreate creates a tarfile for a clickdeb, all files in the archive
-// belong to root (same as dpkg-deb)
+// belong to root (same as dpkg-deb).
 func tarCreate(tarname string, sourceDir string, fn tarExcludeFunc) error {
 	w, err := os.Create(tarname)
 	if err != nil {
@@ -397,7 +403,7 @@
 }
 
 // Build takes a build debian directory with DEBIAN/ dir and creates a
-// clickdeb from it
+// clickdeb from it.
 func (d *ClickDeb) Build(sourceDir string, dataTarFinishedCallback func(dataName string) error) error {
 	var err error
 
@@ -496,7 +502,7 @@
 // target dir and drop privs when doing this.
 //
 // To do this reliably in go we need to exec a helper as we can not
-// just fork() and drop privs in the child (no support for stock fork in go)
+// just fork() and drop privs in the child (no support for stock fork in go).
 func (d *ClickDeb) UnpackWithDropPrivs(instDir, rootdir string) error {
 	// no need to drop privs, we are not root
 	if !helpers.ShouldDropPrivs() {

=== added file 'pkg/file.go'
--- pkg/file.go	1970-01-01 00:00:00 +0000
+++ pkg/file.go	2015-10-20 15:38:57 +0000
@@ -0,0 +1,81 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+
+/*
+ * Copyright (C) 2015 Canonical Ltd
+ *
+ * This program is free software: you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 3 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ */
+
+package pkg
+
+import (
+	"bytes"
+	"fmt"
+	"os"
+	"strings"
+
+	"launchpad.net/snappy/pkg/clickdeb"
+	"launchpad.net/snappy/pkg/snapfs"
+)
+
+// File is the interface to interact with the low-level snap files.
+type File interface {
+	// Verify verfies the integrity of the file.
+	// FIXME: use flags here instead of a boolean
+	Verify(allowUnauthenticated bool) error
+	// Close closes the snapfile.
+	// FIXME: this can go away once we no longer support clickdebs.
+	Close() error
+	// UnpackWithDropPrivs unpacks the given the snap to the given
+	// targetdir relative to the given rootDir.
+	// FIXME: name leaks implementation details, should be Unpack()
+	UnpackWithDropPrivs(targetDir, rootDir string) error
+	// ControlMember returns the content of snap meta data files.
+	ControlMember(name string) ([]byte, error)
+	// MetaMember returns the content of snap meta data files.
+	// FIXME: redundant
+	MetaMember(name string) ([]byte, error)
+	// ExtractHashes extracs the hashes from the snap and puts
+	// them into the filesystem for verification.
+	ExtractHashes(targetDir string) error
+
+	// NeedsAutoMountUnit determines if it's required to setup
+	// an automount unit for the snap when the snap is activated
+	NeedsAutoMountUnit() bool
+}
+
+// Open opens a given snap file with the right backend.
+func Open(path string) (File, error) {
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, fmt.Errorf("cannot open snap: %v", err)
+	}
+	defer f.Close()
+
+	// look, libmagic!
+	header := make([]byte, 20)
+	if _, err := f.ReadAt(header, 0); err != nil {
+		return nil, fmt.Errorf("cannot read snap: %v", err)
+	}
+	// Note that we only support little endian squashfs. There
+	// is nothing else with squashfs 4.0.
+	if bytes.HasPrefix(header, []byte{'h', 's', 'q', 's'}) {
+		return snapfs.New(path), nil
+	}
+	if strings.HasPrefix(string(header), "!<arch>\ndebian") {
+		return clickdeb.Open(path)
+	}
+
+	return nil, fmt.Errorf("cannot open snap: unknown header: %q", header)
+}

=== renamed file 'pkg/snapfs/pkg.go' => 'pkg/snapfs/snapfs.go'
--- pkg/snapfs/pkg.go	2015-10-08 17:38:48 +0000
+++ pkg/snapfs/snapfs.go	2015-10-20 15:38:57 +0000
@@ -27,51 +27,82 @@
 	"path/filepath"
 	"strings"
 
+	"launchpad.net/snappy/dirs"
 	"launchpad.net/snappy/helpers"
 )
 
-// Snap is the squashfs based snap
+// BlobPath is a helper that calculates the blob path from the baseDir.
+// FIXME: feels wrong (both location and approach). need something better.
+func BlobPath(instDir string) string {
+	l := strings.Split(filepath.Clean(instDir), string(filepath.Separator))
+	if len(l) < 2 {
+		panic(fmt.Sprintf("invalid path for BlobPath: %q", instDir))
+	}
+
+	return filepath.Join(dirs.SnapBlobDir, fmt.Sprintf("%s_%s.snap", l[len(l)-2], l[len(l)-1]))
+}
+
+// Snap is the squashfs based snap.
 type Snap struct {
 	path string
 }
 
-// Name returns the Name of the backing file
+// Name returns the Name of the backing file.
 func (s *Snap) Name() string {
 	return filepath.Base(s.path)
 }
 
-// New returns a new Snapfs snap
+// NeedsAutoMountUnit returns true.
+func (s *Snap) NeedsAutoMountUnit() bool {
+	return true
+}
+
+// New returns a new Snapfs snap.
 func New(path string) *Snap {
 	return &Snap{path: path}
 }
 
-// Close is not doing anything for snapfs - COMPAT
+// Close is not doing anything for snapfs. - COMPAT
 func (s *Snap) Close() error {
 	return nil
 }
 
-// ControlMember extracts from meta/ - COMPAT
+// ControlMember extracts from meta/. - COMPAT
 func (s *Snap) ControlMember(controlMember string) ([]byte, error) {
 	return s.ReadFile(filepath.Join("DEBIAN", controlMember))
 }
 
-// MetaMember extracts from meta/ - COMPAT
+// MetaMember extracts from meta/. - COMPAT
 func (s *Snap) MetaMember(metaMember string) ([]byte, error) {
 	return s.ReadFile(filepath.Join("meta", metaMember))
 }
 
-// ExtractHashes does notthing for snapfs snaps - COMAPT
+// ExtractHashes does notthing for snapfs snaps. - COMAPT
 func (s *Snap) ExtractHashes(dir string) error {
 	return nil
 }
 
-// UnpackWithDropPrivs unpacks the meta and puts stuff in place - COMAPT
+// UnpackWithDropPrivs just copies the blob into place. - COMPAT
 func (s *Snap) UnpackWithDropPrivs(instDir, rootdir string) error {
-	// FIXME: actually drop privs
-	return s.Unpack("*", instDir)
+	// FIXME: we need to unpack "meta/*" here because otherwise there
+	//        is no meta/package.yaml for "snappy list -v" for
+	//        inactive versions.
+	if err := s.UnpackMeta(instDir); err != nil {
+		return err
+	}
+
+	// ensure mount-point and blob dir.
+	for _, dir := range []string{instDir, dirs.SnapBlobDir} {
+		if err := os.MkdirAll(dir, 0755); err != nil {
+			return err
+		}
+	}
+
+	// FIXME: helpers.CopyFile() has no preserve attribute flag yet
+	return runCommand("cp", "-a", s.path, BlobPath(instDir))
 }
 
-// UnpackMeta unpacks just the meta/* directory of the given snap
+// UnpackMeta unpacks just the meta/* directory of the given snap.
 func (s *Snap) UnpackMeta(dst string) error {
 	if err := s.Unpack("meta/*", dst); err != nil {
 		return err
@@ -89,12 +120,12 @@
 	return nil
 }
 
-// Unpack unpacks the src (which may be a glob into the given target dir
+// Unpack unpacks the src (which may be a glob into the given target dir.
 func (s *Snap) Unpack(src, dstDir string) error {
 	return runCommand("unsquashfs", "-f", "-i", "-d", dstDir, s.path, src)
 }
 
-// ReadFile returns the content of a single file inside a snapfs snap
+// ReadFile returns the content of a single file inside a snapfs snap.
 func (s *Snap) ReadFile(path string) (content []byte, err error) {
 	tmpdir, err := ioutil.TempDir("", "read-file")
 	if err != nil {
@@ -110,21 +141,15 @@
 	return ioutil.ReadFile(filepath.Join(unpackDir, path))
 }
 
-// CopyBlob copies the snap to a new place
-func (s *Snap) CopyBlob(targetFile string) error {
-	// FIXME: helpers.CopyFile() has no preserve attribute flag yet
-	return runCommand("cp", "-a", s.path, targetFile)
-}
-
-// Verify verifies the snap
+// Verify verifies the snap.
 func (s *Snap) Verify(unauthOk bool) error {
 	// FIXME: there is no verification yet for snapfs packages, this
 	//        will be done via assertions later for now we rely on
-	//        the https security
+	//        the https security.
 	return nil
 }
 
-// Build builds the snap
+// Build builds the snap.
 func (s *Snap) Build(buildDir string) error {
 	fullSnapPath, err := filepath.Abs(s.path)
 	if err != nil {

=== renamed file 'pkg/snapfs/pkg_test.go' => 'pkg/snapfs/snapfs_test.go'
--- pkg/snapfs/pkg_test.go	2015-10-08 14:12:43 +0000
+++ pkg/snapfs/snapfs_test.go	2015-10-20 15:38:57 +0000
@@ -93,15 +93,6 @@
 	c.Assert(string(content), Equals, "name: foo")
 }
 
-func (s *SquashfsTestSuite) TestCopyBlob(c *C) {
-	snap := makeSnap(c, "name: foo", "")
-	dst := filepath.Join(c.MkDir(), "blob.snap")
-
-	err := snap.CopyBlob(dst)
-	c.Assert(err, IsNil)
-	c.Assert(helpers.FilesAreEqual(snap.Name(), dst), Equals, true)
-}
-
 func (s *SquashfsTestSuite) TestUnpackGlob(c *C) {
 	data := "some random data"
 	snap := makeSnap(c, "", data)

=== removed file 'snappy/pkgformat.go'
--- snappy/pkgformat.go	2015-10-08 17:42:22 +0000
+++ snappy/pkgformat.go	1970-01-01 00:00:00 +0000
@@ -1,64 +0,0 @@
-// -*- Mode: Go; indent-tabs-mode: t -*-
-
-/*
- * Copyright (C) 2014-2015 Canonical Ltd
- *
- * This program is free software: you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 3 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program.  If not, see <http://www.gnu.org/licenses/>.
- *
- */
-
-package snappy
-
-import (
-	"bytes"
-	"fmt"
-	"os"
-	"strings"
-
-	"launchpad.net/snappy/pkg/clickdeb"
-	"launchpad.net/snappy/pkg/snapfs"
-)
-
-// PackageFile is the interface to interact with the low-level snap files
-type PackageFile interface {
-	Verify(allowUnauthenticated bool) error
-	Close() error
-	UnpackWithDropPrivs(targetDir, rootDir string) error
-	ControlMember(name string) ([]byte, error)
-	MetaMember(name string) ([]byte, error)
-	ExtractHashes(targetDir string) error
-}
-
-// OpenPackageFile opens a given snap file with the right backend
-func OpenPackageFile(path string) (PackageFile, error) {
-	f, err := os.Open(path)
-	if err != nil {
-		return nil, err
-	}
-	defer f.Close()
-
-	// look, libmagic!
-	header := make([]byte, 20)
-	if _, err := f.Read(header); err != nil {
-		return nil, err
-	}
-	// note that we only support little endian squashfs for now
-	if bytes.HasPrefix(header, []byte{'h', 's', 'q', 's'}) {
-		return snapfs.New(path), nil
-	}
-	if strings.HasPrefix(string(header), "!<arch>\ndebian") {
-		return clickdeb.Open(path)
-	}
-
-	return nil, fmt.Errorf("unknown header %v", header)
-}

=== modified file 'snappy/snapp.go'
--- snappy/snapp.go	2015-10-15 08:26:11 +0000
+++ snappy/snapp.go	2015-10-20 15:38:57 +0000
@@ -44,6 +44,7 @@
 	"launchpad.net/snappy/oauth"
 	"launchpad.net/snappy/pkg"
 	"launchpad.net/snappy/pkg/remote"
+	"launchpad.net/snappy/pkg/snapfs"
 	"launchpad.net/snappy/policy"
 	"launchpad.net/snappy/progress"
 	"launchpad.net/snappy/release"
@@ -181,7 +182,7 @@
 	isActive    bool
 	isInstalled bool
 	description string
-	deb         PackageFile
+	deb         pkg.File
 	basedir     string
 }
 
@@ -430,7 +431,7 @@
 // package, as deduced from the license agreement (which might involve asking
 // the user), or an error that explains the reason why installation should not
 // proceed.
-func (m *packageYaml) checkLicenseAgreement(ag agreer, d PackageFile, currentActiveDir string) error {
+func (m *packageYaml) checkLicenseAgreement(ag agreer, d pkg.File, currentActiveDir string) error {
 	if !m.ExplicitLicenseAgreement {
 		return nil
 	}
@@ -541,7 +542,7 @@
 // Caller should call Close on the pkg.
 // TODO: expose that Close.
 func NewSnapPartFromSnapFile(snapFile string, origin string, unauthOk bool) (*SnapPart, error) {
-	d, err := OpenPackageFile(snapFile)
+	d, err := pkg.Open(snapFile)
 	if err != nil {
 		return nil, err
 	}
@@ -620,7 +621,7 @@
 	// read hash, its ok if its not there, some older versions of
 	// snappy did not write this file
 	hashesData, err := ioutil.ReadFile(filepath.Join(part.basedir, "meta", "hashes.yaml"))
-	if err != nil {
+	if err != nil && !os.IsNotExist(err) {
 		return nil, err
 	}
 
@@ -999,12 +1000,26 @@
 		}
 	}
 
+	// the hooks must run before the snap is (auto)mounted
+	// the reason is that the click hooks use:
+	//   ".click/$name.$origin.manifest"
+	// but because the origin is not known at build time it needs
+	// to be generated at runtime but will be mounted over once
+	// the snap is mounted
 	if err := installClickHooks(s.basedir, s.m, s.origin, inhibitHooks); err != nil {
 		// cleanup the failed hooks
 		removeClickHooks(s.m, s.origin, inhibitHooks)
 		return err
 	}
 
+	// generate the automount unit for the squashfs
+	// FIXME: this is ugly
+	if s.deb != nil && s.deb.NeedsAutoMountUnit() {
+		if err := s.m.addSnapfsAutomount(s.basedir, inhibitHooks, inter); err != nil {
+			return err
+		}
+	}
+
 	// generate the security policy from the package.yaml
 	if err := s.m.addSecurityPolicy(s.basedir); err != nil {
 		return err
@@ -1018,7 +1033,6 @@
 	if err := s.m.addPackageServices(s.basedir, inhibitHooks, inter); err != nil {
 		return err
 	}
-
 	if err := os.Remove(currentActiveSymlink); err != nil && !os.IsNotExist(err) {
 		logger.Noticef("Failed to remove %q: %v", currentActiveSymlink, err)
 	}
@@ -1068,6 +1082,9 @@
 	if err := s.m.removeSecurityPolicy(s.basedir); err != nil {
 		return err
 	}
+	if err := s.m.removeSnapfsAutomount(s.basedir, inter); err != nil {
+		return err
+	}
 
 	if s.Type() == pkg.TypeFramework {
 		if err := policy.Remove(s.Name(), s.basedir, dirs.GlobalRootDir); err != nil {
@@ -1132,11 +1149,19 @@
 		return err
 	}
 
+	// unmount squashfs but ignore errors as its ok if the fs is not mounted
+	exec.Command("unmount", "--lazy", filepath.Join(s.basedir)).CombinedOutput()
+
 	err = os.RemoveAll(s.basedir)
 	if err != nil {
 		return err
 	}
 
+	err = os.RemoveAll(snapfs.BlobPath(s.basedir))
+	if err != nil {
+		return err
+	}
+
 	// best effort(?)
 	os.Remove(filepath.Dir(s.basedir))
 
@@ -1996,3 +2021,46 @@
 
 	return env
 }
+
+func (m *packageYaml) addSnapfsAutomount(baseDir string, inhibitHooks bool, inter interacter) error {
+	squashfsPath := stripGlobalRootDir(snapfs.BlobPath(baseDir))
+	whereDir := stripGlobalRootDir(baseDir)
+
+	sysd := systemd.New(dirs.GlobalRootDir, inter)
+	_, err := sysd.WriteMountUnitFile(m.Name, squashfsPath, whereDir)
+	if err != nil {
+		return err
+	}
+	autoMountUnitName, err := sysd.WriteAutoMountUnitFile(m.Name, whereDir)
+	if err != nil {
+		return err
+	}
+
+	// we always enable the mount unit even in inhibit hooks
+	if err := sysd.Enable(autoMountUnitName); err != nil {
+		return err
+	}
+
+	if !inhibitHooks {
+		return sysd.Start(autoMountUnitName)
+	}
+
+	return nil
+}
+
+func (m *packageYaml) removeSnapfsAutomount(baseDir string, inter interacter) error {
+	sysd := systemd.New(dirs.GlobalRootDir, inter)
+	for _, s := range []string{"mount", "automount"} {
+		unit := systemd.MountUnitPath(stripGlobalRootDir(baseDir), s)
+		if helpers.FileExists(unit) {
+			// we ignore errors, nothing should stop removals
+			_ = sysd.Disable(filepath.Base(unit))
+			_ = sysd.Stop(filepath.Base(unit), time.Duration(1*time.Second))
+			if err := os.Remove(unit); err != nil {
+				return err
+			}
+		}
+	}
+
+	return nil
+}

=== modified file 'snappy/snapp_snapfs_test.go'
--- snappy/snapp_snapfs_test.go	2015-10-08 10:33:37 +0000
+++ snappy/snapp_snapfs_test.go	2015-10-20 15:38:57 +0000
@@ -20,11 +20,14 @@
 package snappy
 
 import (
+	"io/ioutil"
+	"os"
 	"path/filepath"
 
 	"launchpad.net/snappy/dirs"
 	"launchpad.net/snappy/helpers"
 	"launchpad.net/snappy/pkg/snapfs"
+	"launchpad.net/snappy/systemd"
 
 	. "gopkg.in/check.v1"
 )
@@ -36,6 +39,12 @@
 	// mocks
 	aaClickHookCmd = "/bin/true"
 	dirs.SetRootDir(c.MkDir())
+	os.MkdirAll(filepath.Join(dirs.SnapServicesDir, "multi-user.target.wants"), 0755)
+
+	// ensure we do not run a real systemd (slows down tests)
+	systemd.SystemctlCmd = func(cmd ...string) ([]byte, error) {
+		return []byte("ActiveState=inactive\n"), nil
+	}
 
 	// ensure we use the right builder func (snapfs)
 	snapBuilderFunc = BuildSnapfsSnap
@@ -70,15 +79,88 @@
 	_, err = part.Install(&MockProgressMeter{}, 0)
 	c.Assert(err, IsNil)
 
-	// after install its just on disk for now, note that this will
-	// change once the mounting gets added
-	base := filepath.Join(dirs.SnapAppsDir, "hello-app.origin", "1.10")
-	for _, needle := range []string{
-		"bin/foo",
-		"meta/package.yaml",
-		".click/info/hello-app.origin.manifest",
-	} {
-		println(needle)
-		c.Assert(helpers.FileExists(filepath.Join(base, needle)), Equals, true)
-	}
+	// after install the blob is in the right dir
+	c.Assert(helpers.FileExists(filepath.Join(dirs.SnapBlobDir, "hello-app.origin_1.10.snap")), Equals, true)
+
+	// ensure the right unit is created
+	mup := systemd.MountUnitPath("/apps/hello-app.origin/1.10", "mount")
+	content, err := ioutil.ReadFile(mup)
+	c.Assert(err, IsNil)
+	c.Assert(string(content), Matches, "(?ms).*^Where=/apps/hello-app.origin/1.10")
+	c.Assert(string(content), Matches, "(?ms).*^What=/var/lib/snappy/snaps/hello-app.origin_1.10.snap")
+}
+
+func (s *SnapfsTestSuite) TestAddSnapfsAutomount(c *C) {
+	m := packageYaml{
+		Name:          "foo.origin",
+		Version:       "1.0",
+		Architectures: []string{"all"},
+	}
+	inter := &MockProgressMeter{}
+	err := m.addSnapfsAutomount(filepath.Join(dirs.SnapAppsDir, "foo.origin/1.0"), true, inter)
+	c.Assert(err, IsNil)
+
+	// ensure correct mount unit
+	mount, err := ioutil.ReadFile(filepath.Join(dirs.SnapServicesDir, "apps-foo.origin-1.0.mount"))
+	c.Assert(err, IsNil)
+	c.Assert(string(mount), Equals, `[Unit]
+Description=Snapfs mount unit for foo.origin
+
+[Mount]
+What=/var/lib/snappy/snaps/foo.origin_1.0.snap
+Where=/apps/foo.origin/1.0
+`)
+
+	// and correct automount unit
+	automount, err := ioutil.ReadFile(filepath.Join(dirs.SnapServicesDir, "apps-foo.origin-1.0.automount"))
+	c.Assert(err, IsNil)
+	c.Assert(string(automount), Equals, `[Unit]
+Description=Snapfs automount unit for foo.origin
+
+[Automount]
+Where=/apps/foo.origin/1.0
+TimeoutIdleSec=30
+
+[Install]
+WantedBy=multi-user.target
+`)
+}
+
+func (s *SnapfsTestSuite) TestRemoveSnapfsAutomount(c *C) {
+	m := packageYaml{}
+	inter := &MockProgressMeter{}
+	err := m.addSnapfsAutomount(filepath.Join(dirs.SnapAppsDir, "foo.origin/1.0"), true, inter)
+	c.Assert(err, IsNil)
+
+	// ensure we have the files
+	for _, ext := range []string{"mount", "automount"} {
+		p := filepath.Join(dirs.SnapServicesDir, "apps-foo.origin-1.0.") + ext
+		c.Assert(helpers.FileExists(p), Equals, true)
+	}
+
+	// now call remove and ensure they are gone
+	err = m.removeSnapfsAutomount(filepath.Join(dirs.SnapAppsDir, "foo.origin/1.0"), inter)
+	c.Assert(err, IsNil)
+	for _, ext := range []string{"mount", "automount"} {
+		p := filepath.Join(dirs.SnapServicesDir, "apps-foo.origin-1.0.") + ext
+		c.Assert(helpers.FileExists(p), Equals, false)
+	}
+}
+
+func (s *SnapfsTestSuite) TestRemoveViaSnapfsWorks(c *C) {
+	snapPkg := makeTestSnapPackage(c, packageHello)
+	part, err := NewSnapPartFromSnapFile(snapPkg, "origin", true)
+	c.Assert(err, IsNil)
+
+	_, err = part.Install(&MockProgressMeter{}, 0)
+	c.Assert(err, IsNil)
+
+	// after install the blob is in the right dir
+	c.Assert(helpers.FileExists(filepath.Join(dirs.SnapBlobDir, "hello-app.origin_1.10.snap")), Equals, true)
+
+	// now remove and ensure its gone
+	err = part.Uninstall(&MockProgressMeter{})
+	c.Assert(err, IsNil)
+	c.Assert(helpers.FileExists(filepath.Join(dirs.SnapBlobDir, "hello-app.origin_1.10.snap")), Equals, false)
+
 }

=== modified file 'systemd/systemd.go'
--- systemd/systemd.go	2015-09-15 14:53:38 +0000
+++ systemd/systemd.go	2015-10-20 15:38:57 +0000
@@ -33,6 +33,7 @@
 	"text/template"
 	"time"
 
+	"launchpad.net/snappy/dirs"
 	"launchpad.net/snappy/helpers"
 	"launchpad.net/snappy/logger"
 )
@@ -95,6 +96,8 @@
 	Status(service string) (string, error)
 	ServiceStatus(service string) (*ServiceStatus, error)
 	Logs(services []string) ([]Log, error)
+	WriteMountUnitFile(name, what, where string) (string, error)
+	WriteAutoMountUnitFile(name, where string) (string, error)
 }
 
 // A Log is a single entry in the systemd journal
@@ -510,3 +513,46 @@
 func (l Log) String() string {
 	return fmt.Sprintf("%s %s %s", l.Timestamp(), l.SID(), l.Message())
 }
+
+// MountUnitPath returns the path of a {,auto}mount unit
+func MountUnitPath(baseDir, ext string) string {
+	// FIXME: use github.com/coreos/go-systemd/unit/escape.go
+	//        once we can update go-systemd. we can not right now
+	//        because versions >= 2 are not compatible with go1.3 from
+	//        15.04
+	p, err := exec.Command("systemd-escape", "--path", baseDir).CombinedOutput()
+	if err != nil {
+		panic(fmt.Sprintf("systemd-escape failed for %s with %q", baseDir, p))
+	}
+	escapedPath := strings.TrimSpace(string(p))
+	return filepath.Join(dirs.SnapServicesDir, fmt.Sprintf("%s.%s", escapedPath, ext))
+}
+
+func (s *systemd) WriteMountUnitFile(name, what, where string) (string, error) {
+	c := fmt.Sprintf(`[Unit]
+Description=Snapfs mount unit for %s
+
+[Mount]
+What=%s
+Where=%s
+`, name, what, where)
+
+	mu := MountUnitPath(where, "mount")
+	return filepath.Base(mu), helpers.AtomicWriteFile(mu, []byte(c), 0644)
+}
+
+func (s *systemd) WriteAutoMountUnitFile(name, where string) (string, error) {
+	c := fmt.Sprintf(`[Unit]
+Description=Snapfs automount unit for %s
+
+[Automount]
+Where=%s
+TimeoutIdleSec=30
+
+[Install]
+WantedBy=multi-user.target
+`, name, where)
+
+	mu := MountUnitPath(where, "automount")
+	return filepath.Base(mu), helpers.AtomicWriteFile(MountUnitPath(where, "automount"), []byte(c), 0644)
+}

=== modified file 'systemd/systemd_test.go'
--- systemd/systemd_test.go	2015-09-14 12:29:15 +0000
+++ systemd/systemd_test.go	2015-10-20 15:38:57 +0000
@@ -21,6 +21,7 @@
 
 import (
 	"fmt"
+	"io/ioutil"
 	"os"
 	"path/filepath"
 	"testing"
@@ -28,6 +29,7 @@
 
 	. "gopkg.in/check.v1"
 
+	"launchpad.net/snappy/dirs"
 	"launchpad.net/snappy/helpers"
 )
 
@@ -411,3 +413,40 @@
 	}.String(), Equals, "1970-01-01T00:00:00.000042Z me hi")
 
 }
+
+func (s *SystemdTestSuite) TestMountUnitPath(c *C) {
+	c.Assert(MountUnitPath("/apps/hello.origin/1.1", "mount"), Equals, filepath.Join(dirs.SnapServicesDir, "apps-hello.origin-1.1.mount"))
+}
+
+func (s *SystemdTestSuite) TestWriteMountUnit(c *C) {
+	mountUnitName, err := New("", nil).WriteMountUnitFile("foo.origin", "/var/lib/snappy/snaps/foo.origin_1.0.snap", "/apps/foo.origin/1.0")
+	c.Assert(err, IsNil)
+
+	mount, err := ioutil.ReadFile(filepath.Join(dirs.SnapServicesDir, mountUnitName))
+	c.Assert(err, IsNil)
+	c.Assert(string(mount), Equals, `[Unit]
+Description=Snapfs mount unit for foo.origin
+
+[Mount]
+What=/var/lib/snappy/snaps/foo.origin_1.0.snap
+Where=/apps/foo.origin/1.0
+`)
+}
+
+func (s *SystemdTestSuite) TestWriteAutoMountUnit(c *C) {
+	mountUnitName, err := New("", nil).WriteAutoMountUnitFile("foo.origin", "/apps/foo.origin/1.0")
+	c.Assert(err, IsNil)
+
+	automount, err := ioutil.ReadFile(filepath.Join(dirs.SnapServicesDir, mountUnitName))
+	c.Assert(err, IsNil)
+	c.Assert(string(automount), Equals, `[Unit]
+Description=Snapfs automount unit for foo.origin
+
+[Automount]
+Where=/apps/foo.origin/1.0
+TimeoutIdleSec=30
+
+[Install]
+WantedBy=multi-user.target
+`)
+}