Merge lp:~brian-murray/click/chroots into lp:click

[Brian Murray]

This branch adds chroot (using schroot) management and creation to click.

[Colin Watson (cjwatson)]

Thanks! I'm looking over this; I won't finish until tomorrow. In the meantime, do you think that instead of this pattern:

  subprocess.check_output(command).decode('utf-8')

... you could use this pattern:

  subprocess.check_output(command, universal_newlines=True)

I use this elsewhere because it tends to be a better match to other Python 2 code (click still supports Python 2.7).

[lp:~brian-murray/click/chroots]

293. By Brian Murray on 2013-10-03

call subprocess.check_output with universal_newlines=True

[Colin Watson (cjwatson)]

 review needs-fixing

This looks very nearly there; thanks a lot! There are just a few
details to clean up.

On Wed, Oct 02, 2013 at 04:13:25PM -0000, Brian Murray wrote:
> + proxy = subprocess.check_output(
> + ["apt-config", "shell", "x", "Acquire::HTTP::Proxy"]
> + ).decode('utf-8').replace('x=', '').strip()

  $ apt-config shell x Dir
  x='/'

That implies quoting may need to be involved here. Using python-apt
would be cleaner, but is an extra dependency. Hmm. How about something
like this instead:

  proxy = subprocess.check_output(
      'unset x; eval "$(apt-config shell x Acquire::HTTP::Proxy)"; echo "$x"',
      shell=True, universal_newlines=True).strip()

? I've certainly written cleaner code ... but this seems to avoid
potential quoting problems.

> + build_pkgs = [
> + "build-essential", "fakeroot",
> + "apt-utils", "pkg-create-dbgsym",
> + "pkgbinarymangler", "g++-%s" % target_tuple,
> + "pkg-config-%s" % target_tuple,
> + "dpkg-cross", "libc-dev:%s" % self.target_arch
> + ]

Let's drop pkg-create-dbgsym and pkgbinarymangler; we don't need them
for click packages.

> + shutil.copy("/etc/localtime", "%s/etc/" % mount)
> + shutil.copy("/etc/timezone", "%s/etc/" % mount)

Nit: I'd use shutil.copy2. Also, /etc/localtime may be a symlink; check
whether shutil.copy2 preserves those, and if not you'll probably need
some special-casing.

> + chroot_config = "/etc/schroot/chroot.d/%s" % self.full_name
> + with open(chroot_config, "w") as target:
> + admin_groups = "sbuild,root"

The sbuild group shouldn't be involved here.

> + print("[%s]" % self.full_name, file=target)
> + print("description=Build chroot for click packages on %s" %
> + self.target_arch, file=target)
> + for group in ["groups", "root-groups", "source-root-users",
> + "source-root-groups"]:
> + print("%s=%s" % (group, admin_groups), file=target)

I realise this is translated from mk-sbuild, but I'd prefer not to add
things described as group names to source-root-users; even if they
happen to coincide now it's a bit of a trap for the future.

For this application, I would suggest that we want the calling user to
have full access, and also the root user for good measure (it could also

> + print("type=directory", file=target)
> + print("profile=sbuild", file=target)
> + print("union-type=overlayfs", file=target)
> + print("directory=%s" % mount, file=target)

Let's drop profile=sbuild. We don't need it here, and it has unhelpful
(for this application) differences from the default profile such as not
bind-mounting the user's home directory.

> + def run(self, *args):
> + if not self.exists():
> + raise ClickChrootException(
> + "Chroot %s does not exist" % self.full_name)
> + command = ["schroot", "-c", self.full_name, "--directory=/", "--"]

Why the --directory=/ here? I'm concerned that that will make it
difficult to do simple "run make in ...

review: Needs Fixing

[lp:~brian-murray/click/chroots]

294. By Brian Murray on 2013-10-07

modifications based on reviewer feedback

295. By Brian Murray on 2013-10-07

modify how apt-config is used to check for a proxy

296. By Brian Murray on 2013-10-07

fix argparse imports

[Brian Murray (brian-murray)]

On Fri, Oct 04, 2013 at 08:45:46AM -0000, Colin Watson wrote:
> Review: Needs Fixing
>
> review needs-fixing
>
> This looks very nearly there; thanks a lot! There are just a few
> details to clean up.
>
> On Wed, Oct 02, 2013 at 04:13:25PM -0000, Brian Murray wrote:
> > + proxy = subprocess.check_output(
> > + ["apt-config", "shell", "x", "Acquire::HTTP::Proxy"]
> > + ).decode('utf-8').replace('x=', '').strip()
>
> $ apt-config shell x Dir
> x='/'
>
> That implies quoting may need to be involved here. Using python-apt
> would be cleaner, but is an extra dependency. Hmm. How about something
> like this instead:
>
> proxy = subprocess.check_output(
> 'unset x; eval "$(apt-config shell x Acquire::HTTP::Proxy)"; echo "$x"',
> shell=True, universal_newlines=True).strip()
>
> ? I've certainly written cleaner code ... but this seems to avoid
> potential quoting problems.

Changed.

>
> > + build_pkgs = [
> > + "build-essential", "fakeroot",
> > + "apt-utils", "pkg-create-dbgsym",
> > + "pkgbinarymangler", "g++-%s" % target_tuple,
> > + "pkg-config-%s" % target_tuple,
> > + "dpkg-cross", "libc-dev:%s" % self.target_arch
> > + ]
>
> Let's drop pkg-create-dbgsym and pkgbinarymangler; we don't need them
> for click packages.

Changed.

> > + shutil.copy("/etc/localtime", "%s/etc/" % mount)
> > + shutil.copy("/etc/timezone", "%s/etc/" % mount)
>
> Nit: I'd use shutil.copy2. Also, /etc/localtime may be a symlink; check
> whether shutil.copy2 preserves those, and if not you'll probably need
> some special-casing.

Done.

> > + chroot_config = "/etc/schroot/chroot.d/%s" % self.full_name
> > + with open(chroot_config, "w") as target:
> > + admin_groups = "sbuild,root"
>
> The sbuild group shouldn't be involved here.

Done.

> > + print("[%s]" % self.full_name, file=target)
> > + print("description=Build chroot for click packages on %s" %
> > + self.target_arch, file=target)
> > + for group in ["groups", "root-groups", "source-root-users",
> > + "source-root-groups"]:
> > + print("%s=%s" % (group, admin_groups), file=target)
>
> I realise this is translated from mk-sbuild, but I'd prefer not to add
> things described as group names to source-root-users; even if they
> happen to coincide now it's a bit of a trap for the future.
>
> For this application, I would suggest that we want the calling user to
> have full access, and also the root user for good measure (it could also

This seems to have been cut off.

> > + print("type=directory", file=target)
> > + print("profile=sbuild", file=target)
> > + print("union-type=overlayfs", file=target)
> > + print("directory=%s" % mount, file=target)
>
> Let's drop profile=sbuild. We don't need it here, and it has unhelpful
> (for this application) differences from the default profile such as not
> bind-mounting the user's home directory.

The sbuild profile contains a different nssdatabases profil...

[Colin Watson (cjwatson)]

On Mon, Oct 07, 2013 at 09:00:45PM -0000, Brian Murray wrote:
> On Fri, Oct 04, 2013 at 08:45:46AM -0000, Colin Watson wrote:
> > I realise this is translated from mk-sbuild, but I'd prefer not to add
> > things described as group names to source-root-users; even if they
> > happen to coincide now it's a bit of a trap for the future.
> >
> > For this application, I would suggest that we want the calling user to
> > have full access, and also the root user for good measure (it could also
>
> This seems to have been cut off.

I'm not sure what I meant to say there. I don't think it was very
important.

> > > + def run(self, *args):
> > > + if not self.exists():
> > > + raise ClickChrootException(
> > > + "Chroot %s does not exist" % self.full_name)
> > > + command = ["schroot", "-c", self.full_name, "--directory=/", "--"]
> >
> > Why the --directory=/ here? I'm concerned that that will make it
> > difficult to do simple "run make in this directory"-type things, which
> > is a primary goal of this system.
> >
> > Perhaps this was a consequence of profile=sbuild, since not having your
> > home directory mounted would be a bit of a problem. If so, try dropping
> > this argument after dropping that profile.
>
> This is still an issue with my setup because of my encrypted home
> directory. Do you have any suggestions on how we may deal with users
> that also have one?

Maybe you need to bind-mount $HOME rather than /home?

[Colin Watson (cjwatson)]

I made various further improvements and merged this. Thanks a lot!

review: Approve

=== added file 'click/chroot.py'
--- click/chroot.py	1970-01-01 00:00:00 +0000
+++ click/chroot.py	2013-10-07 20:57:16 +0000
@@ -0,0 +1,277 @@
+# Copyright (C) 2013 Canonical Ltd.
+# Authors: Colin Watson <[e-mail]>,
+#          Brian Murray <[e-mail]>
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Chroot management for building Click packages."""
+
+from __future__ import print_function
+
+__metaclass__ = type
+__all__ = [
+    "ClickChroot",
+    "ClickChrootException",
+    ]
+
+
+import os
+import shutil
+import stat
+import subprocess
+
+
+extra_packages = {
+    "ubuntu-sdk-13.10": [
+        "libqt5opengl5-dev:TARGET",
+        "libqt5webkit5-dev:TARGET",
+        "libqt5xmlpatterns5-dev:TARGET",
+        "qt5-default:TARGET",
+        "qt5-qmake:TARGET",
+        "qtbase5-dev:TARGET",
+        "qtdeclarative5-dev:TARGET",
+        "qtquick1-5-dev:TARGET",
+        "qtscript5-dev:TARGET",
+        "qtsensors5-dev:TARGET",
+        ],
+    }
+
+
+class ClickChrootException(Exception):
+    pass
+
+
+class ClickChroot:
+    def __init__(self, series, target_arch, framework, name=None):
+        if name is None:
+            name = "click"
+        self.series = series
+        self.target_arch = target_arch
+        self.framework = framework
+        self.name = name
+        self.native_arch = subprocess.check_output(
+            ["dpkg", "--print-architecture"],
+            universal_newlines=True).strip()
+        self.chroots_dir = "/var/lib/schroot/chroots"
+        # this doesn't work because we are running this under sudo
+        if 'DEBOOTSTRAP_MIRROR' in os.environ:
+            self.archive = os.environ['DEBOOTSTRAP_MIRROR']
+        else:
+            self.archive = "http://archive.ubuntu.com/ubuntu"
+
+    def _generate_sources(self, series, native_arch, target_arch, components):
+        ports_mirror = "http://ports.ubuntu.com/ubuntu-ports"
+        pockets = ['%s' % series]
+        for pocket in ['updates', 'security']:
+            pockets.append('%s-%s' % (series, pocket))
+        sources = []
+        if target_arch in ['armhf']:
+            for pocket in pockets:
+                sources.append("deb [arch=%s] %s %s %s" %
+                               (target_arch, ports_mirror, pocket, components))
+                sources.append("deb-src %s %s %s" %
+                               (ports_mirror, pocket, components))
+        if native_arch in ['i386', 'amd64']:
+            for pocket in pockets:
+                sources.append("deb [arch=%s] %s %s %s" %
+                               (native_arch, self.archive, pocket, components))
+                sources.append("deb-src %s %s %s" %
+                               (self.archive, pocket, components))
+        return sources
+
+    @property
+    def full_name(self):
+        return "%s-%s-%s-%s-%s" % (
+            self.name, self.framework, self.series, self.native_arch,
+            self.target_arch)
+
+    def exists(self):
+        command = ["schroot", "-c", self.full_name, "-i"]
+        with open("/dev/null", "w") as devnull:
+            return subprocess.call(
+                command, stdout=devnull, stderr=devnull) == 0
+
+    def create(self):
+        if self.exists():
+            raise ClickChrootException(
+                "Chroot %s already exists" % self.full_name)
+        components = ["main", "restricted", "universe", "multiverse"]
+        mount = "%s/%s" % (self.chroots_dir, self.full_name)
+        proxy = None
+        if not proxy and "http_proxy" in os.environ:
+            proxy = os.environ["http_proxy"]
+        if not proxy:
+            proxy = subprocess.check_output(
+                'unset x; eval "$(apt-config shell x Acquire::HTTP::Proxy)"; echo "$x"',
+                shell=True, universal_newlines=True).strip()
+        with open("/dev/null", "w") as devnull:
+            target_tuple = subprocess.check_output(
+                ["dpkg-architecture", "-a%s" % self.target_arch,
+                 "-qDEB_HOST_GNU_TYPE"], stderr=devnull,
+                universal_newlines=True).strip()
+        build_pkgs = [
+            "build-essential", "fakeroot",
+            "apt-utils", "g++-%s" % target_tuple,
+            "pkg-config-%s" % target_tuple,
+            "dpkg-cross", "libc-dev:%s" % self.target_arch
+            ]
+        for package in extra_packages.get(self.framework, []):
+            package = package.replace(":TARGET", ":%s" % self.target_arch)
+            build_pkgs.append(package)
+        os.makedirs(mount)
+        subprocess.check_call([
+            "debootstrap",
+            "--arch", self.native_arch,
+            "--variant=buildd",
+            "--components=%s" % ','.join(components),
+            self.series,
+            mount,
+            self.archive
+            ])
+        sources = self._generate_sources(self.series, self.native_arch,
+                                         self.target_arch,
+                                         ' '.join(components))
+        with open("%s/etc/apt/sources.list" % mount, "w") as sources_list:
+            for line in sources:
+                print(line, file=sources_list)
+        shutil.copy2("/etc/localtime", "%s/etc/" % mount)
+        shutil.copy2("/etc/timezone", "%s/etc/" % mount)
+        chroot_config = "/etc/schroot/chroot.d/%s" % self.full_name
+        with open(chroot_config, "w") as target:
+            admin_groups = "root"
+            print("[%s]" % self.full_name, file=target)
+            print("description=Build chroot for click packages on %s" %
+                  self.target_arch, file=target)
+            for group in ["groups", "root-groups", "source-root-users",
+                          "source-root-groups"]:
+                print("%s=%s" % (group, admin_groups), file=target)
+            print("type=directory", file=target)
+            print("profile=default", file=target)
+            print("# Not protocols or services see ", file=target)
+            print("# debian bug 557730", file=target)
+            print("setup.nssdatabases=sbuild/nssdatabases",
+                file=target)
+            print("union-type=overlayfs", file=target)
+            print("directory=%s" % mount, file=target)
+        daemon_policy = "%s/usr/sbin/policy-rc.d" % mount
+        with open(daemon_policy, "w") as policy:
+            print("#!/bin/sh", file=policy)
+            print("while true; do", file=policy)
+            print('    case "$1" in', file=policy)
+            print("      -*) shift ;;", file=policy)
+            print("      makedev) exit 0;;", file=policy)
+            print("      x11-common) exit 0;;", file=policy)
+            print("      *) exit 101;;", file=policy)
+            print("    esac", file=policy)
+            print("done", file=policy)
+        os.remove("%s/sbin/initctl" % mount)
+        os.symlink("%s/bin/true" % mount, "%s/sbin/initctl" % mount)
+        finish_script = "%s/finish.sh" % mount
+        with open(finish_script, 'w') as finish:
+            print("#!/bin/bash", file=finish)
+            print("set -e", file=finish)
+            if proxy:
+                print("mkdir -p /etc/apt/apt.conf.d", file=finish)
+                print("cat > /etc/apt/apt.conf.d/99-click-chroot-proxy <<EOF",
+                      file=finish)
+                print("// proxy settings copied by click-chroot", file=finish)
+                print('Acquire { HTTP { Proxy "%s"; }; };' % proxy,
+                      file=finish)
+                print("EOF", file=finish)
+            print("# Configure target arch", file=finish)
+            print("dpkg --add-architecture %s" % self.target_arch,
+                  file=finish)
+            print("# Reload package lists", file=finish)
+            print("apt-get update || true", file=finish)
+            print("# Pull down signature requirements", file=finish)
+            print("apt-get -y --force-yes install \
+gnupg ubuntu-keyring", file=finish)
+            print("# Reload package lists", file=finish)
+            print("apt-get update || true", file=finish)
+            print("# Disable debconf questions so that automated \
+builds won't prompt", file=finish)
+            print("echo set debconf/frontend Noninteractive | \
+debconf-communicate", file=finish)
+            print("echo set debconf/priority critical | \
+debconf-communicate", file=finish)
+            print("# Install basic build tool set to match buildd",
+                  file=finish)
+            print("apt-get -y --force-yes install %s"
+                  % ' '.join(build_pkgs), file=finish)
+            print("# Set up expected /dev entries", file=finish)
+            print("if [ ! -r /dev/stdin ];  \
+then ln -s /proc/self/fd/0 /dev/stdin;  fi", file=finish)
+            print("if [ ! -r /dev/stdout ]; \
+then ln -s /proc/self/fd/1 /dev/stdout; fi", file=finish)
+            print("if [ ! -r /dev/stderr ]; \
+then ln -s /proc/self/fd/2 /dev/stderr; fi", file=finish)
+            print("# Clean up", file=finish)
+            print("rm /finish.sh", file=finish)
+            print("apt-get clean", file=finish)
+        os.chmod(finish_script, stat.S_IEXEC)
+        command = ["/finish.sh"]
+        self.maint(*command)
+
+    def run(self, *args):
+        if not self.exists():
+            raise ClickChrootException(
+                "Chroot %s does not exist" % self.full_name)
+        command = ["schroot", "-c", self.full_name, "--directory=/", "--"]
+        command.extend(args)
+        subprocess.check_call(command)
+
+    def maint(self, *args):
+        command = [
+            # directory is a workaround for the click or user's home directory
+            # not existing
+            "schroot", "-c", "source:%s" % self.full_name,
+            "-u", "root", "--directory=/", "--",
+            ]
+        command.extend(args)
+        subprocess.check_call(command)
+
+    def install(self, *pkgs):
+        if not self.exists():
+            raise ClickChrootException(
+                "Chroot %s does not exist" % self.full_name)
+        self.update()
+        command = ["apt-get", "install", "--yes"]
+        command.extend(pkgs)
+        self.maint(*command)
+        self.clean()
+
+    def clean(self):
+        command = ["apt-get", "clean"]
+        self.maint(*command)
+
+    def update(self):
+        command = ["apt-get", "update", "--yes"]
+        self.maint(*command)
+
+    def upgrade(self):
+        if not self.exists():
+            raise ClickChrootException(
+                "Chroot %s does not exist" % self.full_name)
+        self.update()
+        command = ["apt-get", "dist-upgrade", "--yes"]
+        self.maint(*command)
+        self.clean()
+
+    def destroy(self):
+        if not self.exists():
+            raise ClickChrootException(
+                "Chroot %s does not exist" % self.full_name)
+        chroot_config = "/etc/schroot/chroot.d/%s" % self.full_name
+        os.remove(chroot_config)
+        mount = "%s/%s" % (self.chroots_dir, self.full_name)
+        shutil.rmtree(mount)

=== added file 'click/commands/chroot.py'
--- click/commands/chroot.py	1970-01-01 00:00:00 +0000
+++ click/commands/chroot.py	2013-10-07 20:57:16 +0000
@@ -0,0 +1,91 @@
+#! /usr/bin/python3
+
+# Copyright (C) 2013 Canonical Ltd.
+# Author: Brian Murray <[e-mail]>
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 3 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+"""Use and manage a Click chroot."""
+
+from __future__ import print_function
+
+from argparse import ArgumentParser, REMAINDER
+
+from click.chroot import ClickChroot
+
+
+def create(args):
+    ClickChroot("saucy", args.architecture,
+                "ubuntu-sdk-13.10").create()
+
+
+def install(args):
+    packages = args.packages
+    ClickChroot("saucy", args.architecture,
+                "ubuntu-sdk-13.10").install(*packages)
+
+
+def destroy(args):
+    # ask for confirmation?
+    ClickChroot("saucy", args.architecture,
+                "ubuntu-sdk-13.10").destroy()
+
+
+def execute(args):
+    # not sure what to do wrt setting up the env
+    program = args.program
+    ClickChroot("saucy", args.architecture,
+                "ubuntu-sdk-13.10").run(*program)
+
+
+def upgrade(args):
+    ClickChroot("saucy", args.architecture,
+                "ubuntu-sdk-13.10").upgrade()
+
+
+def run(argv):
+    parser = ArgumentParser("click chroot")
+    subparsers = parser.add_subparsers(
+        description="management subcommands",
+        help="valid commands")
+    parser.add_argument(
+        "-a", "--architecture", required=True,
+        help="architecture for the chroot")
+    create_parser = subparsers.add_parser(
+        "create",
+        help="create a chroot of the provided architecture")
+    create_parser.set_defaults(func=create)
+    destroy_parser = subparsers.add_parser(
+        "destroy",
+        help="destroy the chroot")
+    destroy_parser.set_defaults(func=destroy)
+    upgrade_parser = subparsers.add_parser(
+        "upgrade",
+        help="upgrade the chroot")
+    upgrade_parser.set_defaults(func=upgrade)
+    install_parser = subparsers.add_parser(
+        "install",
+        help="install packages in the chroot")
+    install_parser.add_argument(
+        "packages", nargs="+",
+        help="packages to install")
+    install_parser.set_defaults(func=install)
+    execute_parser = subparsers.add_parser(
+        "run",
+        help="run a program in the chroot")
+    execute_parser.add_argument(
+        "program", nargs=REMAINDER,
+        help="program to run with arguments")
+    execute_parser.set_defaults(func=execute)
+    args = parser.parse_args(argv)
+    args.func(args)