Merge lp:~steve-mcintyre/linaro-chromiumos/python-rewrite into lp:linaro-chromiumos

[Steve McIntyre]

Rewrite in python.
Adding more features for better control of what we're building and where.

[lp:~steve-mcintyre/linaro-chromiumos/python-rewrite]

14. By Steve McIntyre on 2011-03-23

Add creation of the top-level work area, using sudo to make sure it succeeds.

[Loïc Minier (lool)]

I submitted a branch based on yours for you to merge, as I felt it was simpler to show the changes in commits rather than explain them, then ask you to do them, then review them etc. These were all very simple changes, and overall your code was really good, thanks! See https://code.launchpad.net/~lool/linaro-chromiumos/python-rewrite/+merge/54551 for the mp.

There is one major issue with the new Python code: usage of os.system() is unchecked, which means we don't fail the run when any command fails. This really must be fixed; with the new run() wrapper I introduced, it should be simpler to add this in a single place.

Other things I had in mind:
- we should have a place to share code between the scripts
- we should move them out of remote and kill my linaro-chromiumos-build frontend
- (low priority) getopt doesn't integrate too nicely with Python in my experience, there are nicer ways to handle flags, but it works with getopt so we can keep it for now
- create_logdir() feels a bit specific; I think we should have an ensure_dir() thin wrapper around os.isdir() + os.makedirs(), and we should setup logdir early; perhaps it's easier to just expose it as an option; this would also allow merging logged_run() with run()
- I think we should use stderr consistently for our own output, but I'm open to discuss this
- the run() interface should take an array of commands rather than a command string; this is nicer to deal with when preparing commands before running them

=== removed file 'remote/run_build'
--- remote/run_build	2011-03-09 17:35:56 +0000
+++ remote/run_build	1970-01-01 00:00:00 +0000
@@ -1,14 +0,0 @@
-#!/bin/sh
-
-set -e
-
-./setup_board --board x86-generic
-./build_packages --board x86-generic
-
-# <wait a long time>
-
-./build_image --board x86-generic --noenable_rootfs_verification 
-
-# <wait a long time>
-# Success?
-

=== added file 'remote/run_build.py'
--- remote/run_build.py	1970-01-01 00:00:00 +0000
+++ remote/run_build.py	2011-03-23 14:06:22 +0000
@@ -0,0 +1,59 @@
+#!/usr/bin/python
+# 
+# Simple wrapper build script for Chromium OS builds
+#
+# Copyright (C) 2011 Linaro
+#
+# GPL v2+
+
+import os
+import getopt
+import sys
+import warnings
+import errno
+import shutil
+
+options = {}
+options['logdir'] = ''
+options['board-type'] = 'x86-generic'
+options['nousepkg'] = False
+
+try:
+    opts, args = getopt.getopt(sys.argv[1:], "",
+                               ["logdir=",
+                                "board-type=",
+                                "nousepkg="])
+except getopt.GetoptError, err:
+    # print help information and exit:
+    print str(err) # will print something like "option -a not recognized"
+    usage()
+    sys.exit(2)
+
+for o, a in opts:
+    options[o[2:]] = a
+
+
+if not options['logdir'] == '':
+    os.environ['PORT_LOGDIR'] = options['logdir']
+    try:
+        os.environ['FEATURES'] = os.environ['FEATURES'] + ' binpkg-logs'
+    except:
+        os.environ['FEATURES'] = 'binpkg-logs'
+    os.system('sudo rm -rf %s'% options['logdir'])
+    os.system('sudo mkdir %s' % options['logdir'])
+
+build_options = ''
+build_options += ' --board=%s' % options['board-type']
+if options['nousepkg']:
+    build_options += ' --nousepkg'
+
+os.system('./setup_board --force %s' % build_options)
+os.system('./build_packages %s' % build_options)
+
+# <wait a long time>
+image_options = ' --noenable_rootfs_verification '
+image_options += ' --board=%s' % options['board-type']
+os.system('./build_image %s' % image_options)
+
+# <wait a long time>
+# Success?

=== removed file 'remote/setup_build'
--- remote/setup_build	2011-03-09 17:35:56 +0000
+++ remote/setup_build	1970-01-01 00:00:00 +0000
@@ -1,39 +0,0 @@
-#!/bin/sh
-
-set -e
-
-MANIFEST=$1
-if [ "$MANIFEST"x = "" ] ; then
-   MANIFEST=http://git.chromium.org/git/manifest
-fi
-
-BASEDIR=`dirname $0`
-WDIR=/mnt/chromeos
-
-# Basic setup
-export LC_ALL=C
-sudo apt-get update
-sudo apt-get install git subversion build-essential
-
-# Make working directory
-if [ ! -d $WDIR ] ; then
-   sudo mkdir $WDIR
-   sudo chown ubuntu: $WDIR
-fi
-
-# Grab basic stuff
-cd $WDIR
-svn co http://src.chromium.org/svn/trunk/tools/depot_tools
-export PATH=$PATH:$PWD/depot_tools
-
-# Start grabbing source
-repo init -u $MANIFEST
-repo sync
-
-# Set up the working chroot
-/mnt/chromeos/src/scripts/make_chroot --chroot=chroot-x86
-cp $BASEDIR/run_build chroot-x86/src/scripts
-
-# Now chroot into it and do stuff
-script -c "/mnt/chromeos/src/scripts/enter_chroot.sh --chroot=chroot-x86 -- ./run_build" cros-build.log
-

=== added file 'remote/setup_build.py'
--- remote/setup_build.py	1970-01-01 00:00:00 +0000
+++ remote/setup_build.py	2011-03-23 14:06:22 +0000
@@ -0,0 +1,219 @@
+#!/usr/bin/python
+# 
+# Simple setup script for Chromium OS builds
+#
+# Copyright (C) 2011 Linaro
+#
+# GPL v2+
+
+import os
+import getopt
+import sys
+import warnings
+import errno
+import shutil
+
+# Set defaults for options
+options = {}
+options['build-name'] = 'build'
+options['board-type'] = 'x86-generic'
+options['use-network'] = True
+options['working-dir'] = ''
+#options['toolchain'] = 'default'
+options['manifest'] = 'http://git.chromium.org/git/manifest'
+options['nousepkg'] = False
+options['build-script'] = 'run_build.py'
+
+commands = 'create_tree get_source create_chroot build'.split()
+
+def usage():
+    print "%s\n" % sys.argv[0]
+    print "usage:\n"
+    print "  setup_build.py [options] <command>\n"
+    print "  options include:\n"
+    print "  --working-dir=<name>"
+    print "       specify a base location for a working directory"
+    print "       (default is \"%s\")" % options['working-dir']
+    print "  --build-name=<name>"
+    print "       specify a name to use for this build"
+    print "       (default is \"%s\")" % options['build-name']
+    print "  --board-type=<board>"
+    print "       specify the target board/platform"
+    print "       (default is \"%s\")" % options['board-type']
+    print "  --use-network=<true|false>"
+    print "       should network access be allowed during the build?"
+    print "       (default is %s)" % options['use-network']
+    print "  --toolchain=<toolchain>"
+    print "       choice of toolchain. Not yet implemented."
+    print "  --manifest=<manifest URL>"
+    print "       specify the manifest file to use"
+    print "       (default is \"%s\")" % options['manifest']
+    print "  --nousepkg=<true|false>"
+    print "       force building of everything from source"
+    print "       needed if --use-network==true or if using a new toolchain"
+    print "       (default is \"%s\")" % options['nousepkg']
+    print
+    print "commands:\n"
+    print "  create_tree"
+    print "       create a working tree; necessary for other commands to run"
+    print "  get_source"
+    print "       download / update source code"
+    print "  create_chroot"
+    print "       create a chroot; necessary for later commands to run"
+    print "  build"
+    print "       build Chromium OS"
+    print
+
+def do_install_depot_tools(options, tree_dir):
+    os.chdir(tree_dir)
+    os.system('svn co http://src.chromium.org/svn/trunk/tools/depot_tools')
+
+def do_create_logdir(options, tree_dir):
+    logdir = os.path.join(tree_dir, 'logs')
+    if not os.path.exists(logdir):
+        os.mkdir(logdir)
+
+def do_create_tree(options, tree_dir):
+    print 'do_create_tree'
+
+    if os.path.exists(tree_dir):
+        print 'do_create_tree: tree %s already exists. Abort.' % tree_dir
+        return errno.EEXIST
+
+    os.makedirs(tree_dir, 0755)
+    os.system('sudo apt-get update')
+    os.system('sudo apt-get install -y git subversion build-essential')
+    do_install_depot_tools(options, tree_dir)
+    do_create_logdir(options, tree_dir)
+
+def do_get_source(options, tree_dir):
+    print 'do_get_source'
+
+    if not os.path.exists(tree_dir):
+        print 'do_get_source: tree %s does not exist. Abort.' % tree_dir
+        return errno.ENOENT
+
+    basedir = os.path.dirname(os.path.realpath(sys.argv[0]))
+
+    do_create_logdir(options, tree_dir)
+    os.chdir(tree_dir)
+    # Needs the "echo" piped in here to stop "repo init" asking
+    # awkward interactive questions that you can't pass on the command
+    # line.
+    cmdline = 'script -c "'
+    cmdline += 'echo | repo init -u %s' % options['manifest']
+    cmdline += '" logs/repo-init.log'
+    os.system(cmdline)
+
+    cmdline = 'script -c "'
+    cmdline += 'repo sync'
+    cmdline += '" logs/repo-sync.log'
+    os.system(cmdline)
+
+    # Copy in our script to do the actual build work inside the chroot
+    build_script = os.path.join(basedir, options['build-script'])
+    os.chmod(build_script, 0755)
+    shutil.copy(build_script, './src/scripts')
+
+def do_create_chroot(options, tree_dir):
+    print 'do_create_chroot'
+
+    if not os.path.exists(tree_dir):
+        print 'do_create_chroot: tree %s does not exist. Abort.' % tree_dir
+        return errno.ENOENT
+
+    do_create_logdir(options, tree_dir)
+    os.chdir(tree_dir)
+    cmdline = 'script -c "'
+    cmdline += './src/scripts/make_chroot'
+    if options['nousepkg']:
+        cmdline += ' --nousepkg'
+    cmdline += '" logs/create-chroot.log'
+    os.system(cmdline)
+
+def do_build(options, tree_dir):
+    print 'do_build'
+
+    if not os.path.exists(tree_dir):
+        print 'do_build: tree %s does not exist. Abort.' % tree_dir
+        return errno.ENOENT
+
+    chroot_path = os.path.join(tree_dir, 'chroot')
+    if not os.path.exists(chroot_path):
+        print 'do_build: chroot %s does not exist. Abort.' % chroot_path
+        return errno.ENOENT
+
+    do_create_logdir(options, tree_dir)
+    os.chdir(tree_dir)
+
+    cmdline = 'script -c "'
+    cmdline += './src/scripts/enter_chroot.sh -- ./%s' % options['build-script']
+    if options['nousepkg']:
+        cmdline += ' --nousepkg=True'
+    cmdline += ' --board-type=%s' % options['board-type']
+    cmdline += ' --logdir=/var/log/saved-logs'
+    cmdline += '" logs/cros-build.log'
+    os.system(cmdline)
+
+    cmdline = 'tar c -C chroot/var/log --lzma -f logs/build-logs.tar.lzma saved-logs'
+    os.system(cmdline)
+
+try:
+    opts, args = getopt.getopt(sys.argv[1:], "",
+                               ["help",
+                                "working-dir=",
+                                "build-name=",
+                                "board-type=",
+                                "use-network=",
+                                "toolchain=",
+                                "nousepkg=",
+                                "manifest="])
+except getopt.GetoptError, err:
+    # print help information and exit:
+    print str(err) # will print something like "option -a not recognized"
+    usage()
+    sys.exit(2)
+
+for o, a in opts:
+    if o in ("--help"):
+        usage()
+        sys.exit(0)
+    else:
+        options[o[2:]] = a
+
+if options['working-dir'] == '':
+    print 'Need to specify a working dir. Abort.'
+    usage()
+    sys.exit(2)
+
+if args == []:
+    print "Not asked to do anything"
+    usage()
+    sys.exit(0)
+
+for command in args:
+    if not command in (commands):
+        print "Unknown command %s" % command
+        usage()
+        sys.exit(2)
+
+if not os.path.exists(options['working-dir']):
+    # Create working dir area; likely to be on a separate filesystem
+    # somewhere, so let's make this as simple as possible using sudo.
+    os.system('sudo mkdir -p -m 0755 %s' % options['working-dir'])
+    os.system('sudo chown %s: %s' % (os.environ['LOGNAME'], options['working-dir']))
+
+tree_dir = os.path.join(options['working-dir'], options['build-name'])
+depot_tools_dir = os.path.join(tree_dir, 'depot_tools')
+os.environ['LC_ALL'] = 'C'
+os.environ['PATH'] = os.environ['PATH'] + os.pathsep + depot_tools_dir
+
+if 'create_tree' in args:
+    err = do_create_tree(options, tree_dir)
+if 'get_source' in args:
+    err = do_get_source(options, tree_dir)
+if 'create_chroot' in args:
+    err = do_create_chroot(options, tree_dir)
+if 'build' in args:
+    err = do_build(options, tree_dir)
+