Merge lp:~jtv/launchpad/bug-532354 into lp:launchpad

[Jeroen T. Vermeulen]

Register ssh key for ppa-user; enable translation templates generation on dev systems.

= Bug 532354 =

To automate the setup for Soyuz and the build farm on local development systems, I wrote a script that creates a user, optionally uploads your real GPG key(s) for it, signs the Ubuntu code of conduct, etc.

One thing this script failed to do, however, was upload your ssh key(s). But utilities/make-lp-user does do that. So instead of re-inventing the wheel I'm now making the setup script call make-lp-user. And since it obviously belongs there next to the ssh key upload, I extended make-lp-user to take an optional email address argument. If you pass this option, the user that is created will have the email address you specify—as well as any GPG keys you have for that address. This may come in handy generally when you need to sign things like package uploads.

I also made some small improvements in options handling:
 * Don't read all the zcml unless and until you've parsed your options and know it's not just a "--help" invocation!
 * The dry-run option is gone. Can't really work now that one script invokes another.

The GPG-handling code is essentially unchanged, just moved. One small difference is that run_native_gpg figures out for itself that the command you want to run is gpg; you only pass the arguments to gpg.

Another seemingly unrelated change, but also needed for convenient buildfarm testing as far as my team is concerned, is that generation of translation templates from branches is enabled on development systems. (It's already enabled in test runs, but still disabled by default until we get the whole feature working).

Once this is all done, I'll be updating the UsingSoyuzLocally page on the dev wiki, removing tons of stuff that no longer needs to be done manually. I'll have to go over the new TryOutBuildSlave page as well.

No lint. To test (insofar as the utility scripts are ever tested),
{{{
./bin/test -vv -t sampledata-setup.txt
}}}

To Q/A, follow the updated instructions on UsingSoyuzLocally.

Jeroen

[Abel Deuring (adeuring)]

(15:28:59) adeuring: jtv: parse_fingerprints() expects untranslated gpg output: the string "Key fingerprint". Is gpg invoked without the user's language settings? (I get "Schl.-Fingerabdruck" instead when I invoke gpg --fingerprint from a shell)
(15:29:56) jtv: adeuring: gah, good point. I of all people should know better. I'll set LC_ALL to C, since I'm manipulating the environment anyway.
(15:30:12) adeuring: jtv: thanks!
(15:30:21) jtv: Thanks for pointing that out.
(15:31:49) jtv: Fixed.

review: Approve (code)

=== modified file 'configs/development/launchpad-lazr.conf'
--- configs/development/launchpad-lazr.conf	2010-02-12 19:26:23 +0000
+++ configs/development/launchpad-lazr.conf	2010-03-05 14:34:40 +0000
@@ -214,6 +214,7 @@
 
 [rosetta]
 global_suggestions_enabled: True
+generate_templates: True
 
 [rosettabranches]
 error_dir: /var/tmp/rosettabranches.test

=== modified file 'lib/canonical/testing/layers.py'
--- lib/canonical/testing/layers.py	2010-02-12 19:34:42 +0000
+++ lib/canonical/testing/layers.py	2010-03-05 14:34:40 +0000
@@ -97,7 +97,7 @@
     confirm_dbrevision, confirm_dbrevision_on_startup)
 from canonical.database.sqlbase import cursor, ZopelessTransactionManager
 from canonical.launchpad.interfaces import IMailBox, IOpenLaunchBag
-from canonical.launchpad.ftests import ANONYMOUS, login, logout, is_logged_in
+from lp.testing import ANONYMOUS, login, logout, is_logged_in
 import lp.services.mail.stub
 from lp.services.mail.mailbox import TestMailBox
 from canonical.launchpad.scripts import execute_zcml_for_scripts

=== renamed file 'lib/lp/soyuz/doc/sampledata-cleanup.txt' => 'lib/lp/soyuz/doc/sampledata-setup.txt'
=== modified file 'utilities/make-lp-user'
--- utilities/make-lp-user	2009-10-17 14:06:03 +0000
+++ utilities/make-lp-user	2010-03-05 14:34:40 +0000
@@ -1,11 +1,11 @@
 #!/usr/bin/python2.5
 #
-# Copyright 2009 Canonical Ltd.  This software is licensed under the
+# Copyright 2009-2010 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """Create a user for testing the local Launchpad.
 
-Usage: make-lp-user <username> [<team1> <team2> ...]
+Usage: make-lp-user <username> [<team1> <team2> ...] [-e email]
 
 This script will create a usable Launchpad user in the development database to
 help you test a locally running copy of Launchpad.
@@ -21,6 +21,9 @@
 In addition, this script will look in your ~/.ssh directory for public keys
 and register them for the created user.
 
+If you pass an email address, the new user will have this email address
+as well as any GPG keys you have for it.
+
 The login details will be printed to stdout.
 
 Please note that this script is for testing purposes only. Do NOT use it in
@@ -30,10 +33,14 @@
 import _pythonpath
 
 import os
+from optparse import OptionParser
+import re
+import subprocess
 import sys
-
 import transaction
 
+from storm.store import Store
+
 from zope.component import getUtility
 
 from canonical.launchpad.interfaces import (
@@ -42,7 +49,9 @@
     SSHKeyType,
     TeamMembershipStatus,
     )
+from canonical.launchpad.interfaces.gpghandler import IGPGHandler
 from canonical.launchpad.scripts import execute_zcml_for_scripts
+from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet
 from lp.testing.factory import LaunchpadObjectFactory
 
 # Shut up, pyflakes.
@@ -53,7 +62,7 @@
 factory = LaunchpadObjectFactory()
 
 
-def make_person(username):
+def make_person(username, email):
     """Create and return a person with the given username.
 
     The email address for the user will be <username>@example.com. The
@@ -61,7 +70,6 @@
 
     These details will be printed to stdout.
     """
-    email = '[e-mail]' % username
     person = factory.makePerson(
         name=username, password=DEFAULT_PASSWORD, email=email)
     print "username: %s" % (username,)
@@ -121,18 +129,115 @@
         print 'Registered SSH key: %s' % (guessed_filename,)
 
 
+def parse_fingerprints(gpg_output):
+    """Find key fingerprints in "gpg --fingerprint <email>" output."""
+    line_prefix = re.compile('\s*Key fingerprint\s*=\s*')
+    return [
+        ''.join(re.sub(line_prefix, '', line).split())
+        for line in gpg_output.splitlines()
+        if line_prefix.match(line)
+        ]
+
+
+def run_native_gpg(arguments):
+    """Run GPG using the user's real keyring."""
+    # Need to override GNUPGHOME or we'll get a dummy GPG in a temp
+    # directory, which won't find any keys.
+    env = os.environ.copy()
+    if 'GNUPGHOME' in env:
+        del env['GNUPGHOME']
+
+    # Prevent translated gpg output from messing up our parsing.
+    env['LC_ALL'] = 'C'
+
+    command_line = ['gpg'] + arguments
+    pipe = subprocess.Popen(
+        command_line, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = pipe.communicate()
+    if stderr != '':
+        print stderr
+    if pipe.returncode != 0:
+        raise Exception('GPG error during "%s"' % ' '.join(command_line))
+
+    return stdout
+
+
+def add_gpg_key(person, fingerprint):
+    """Add the GPG key with the given fingerprint to `person`."""
+    run_native_gpg([
+        '--keyserver', 'keyserver.launchpad.dev',
+        '--send-key', fingerprint
+        ])
+
+    gpghandler = getUtility(IGPGHandler)
+    key = gpghandler.retrieveKey(fingerprint)
+
+    gpgkeyset = getUtility(IGPGKeySet)
+    if gpgkeyset.getByFingerprint(fingerprint) is not None:
+        # We already have this key.
+        return
+
+    algorithm = GPGKeyAlgorithm.items[key.algorithm]
+    can_encrypt = True
+    lpkey = gpgkeyset.new(
+        person.id, key.keyid, fingerprint, key.keysize, algorithm,
+        active=True, can_encrypt=can_encrypt)
+    Store.of(person).add(lpkey)
+
+
+def attach_gpg_keys(email, person):
+    """Attach the GPG key(s) for `email` to `person`."""
+    output = run_native_gpg(['--fingerprint', email])
+
+    fingerprints = parse_fingerprints(output)
+    if len(fingerprints) == 0:
+        print "No GPG key fingerprints found!"
+    for fingerprint in fingerprints:
+        add_gpg_key(person, fingerprint)
+
+
+def parse_args(arguments):
+    """Parse command-line arguments.
+
+    :return: options object.  Among the options are username (a
+        string) and optionally teams (a list).
+    """
+    parser = OptionParser(description="Create a local Launchpad user.")
+    parser.add_option(
+        '-e', '--email', action='store', dest='email', default=None,
+        help="Email address; set to use real GPG key for this address.")
+
+    options, args = parser.parse_args(arguments)
+    if len(args) == 0:
+        print __doc__
+        sys.exit(2)
+
+    options.username = args[0]
+    options.teams = args[1:]
+
+    return options
+
+
 def main(arguments):
     """Run the script."""
-    if len(arguments) == 0:
-        print __doc__
-        return 2
+    options = parse_args(arguments)
+    if options.email is None:
+    	email = '[e-mail]' % options.username
+    else:
+        email = options.email
+
     execute_zcml_for_scripts()
-    username, teams = arguments[0], arguments[1:]
     transaction.begin()
-    person = make_person(username)
-    add_person_to_teams(person, teams)
+
+    person = make_person(options.username, email)
+    add_person_to_teams(person, options.teams)
     add_ssh_public_keys(person)
+
+    if options.email is not None:
+        attach_gpg_keys(options.email, person)
+
     transaction.commit()
+
     return 0
 
 

=== renamed file 'utilities/soyuz-sampledata-cleanup.py' => 'utilities/soyuz-sampledata-setup.py'
--- utilities/soyuz-sampledata-cleanup.py	2010-02-24 12:13:29 +0000
+++ utilities/soyuz-sampledata-setup.py	2010-03-05 14:34:40 +0000
@@ -13,6 +13,9 @@
 
 DO NOT RUN ON PRODUCTION SYSTEMS.  This script deletes lots of
 Ubuntu-related data.
+
+This script creates a user "ppa-user" (email [e-mail],
+password test) who is able to create PPAs.
 """
 
 __metaclass__ = type
@@ -20,9 +23,12 @@
 import _pythonpath
 
 from optparse import OptionParser
-from os import getenv
 import re
+import os
+import subprocess
 import sys
+from textwrap import dedent
+import transaction
 
 from zope.component import getUtility
 from zope.event import notify
@@ -40,15 +46,24 @@
 from canonical.launchpad.scripts import execute_zcml_for_scripts
 from canonical.launchpad.scripts.logger import logger, logger_options
 from canonical.launchpad.webapp.interfaces import (
-    IStoreSelector, MAIN_STORE, SLAVE_FLAVOR)
+    IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
 
+from lp.registry.interfaces.codeofconduct import ISignedCodeOfConductSet
+from lp.registry.interfaces.person import IPersonSet
 from lp.registry.interfaces.series import SeriesStatus
+from lp.registry.model.codeofconduct import SignedCodeOfConduct
 from lp.soyuz.interfaces.component import IComponentSet
+from lp.soyuz.interfaces.processor import IProcessorFamilySet
 from lp.soyuz.interfaces.section import ISectionSet
 from lp.soyuz.interfaces.sourcepackageformat import (
     ISourcePackageFormatSelectionSet, SourcePackageFormat)
 from lp.soyuz.model.section import SectionSelection
 from lp.soyuz.model.component import ComponentSelection
+from lp.testing.factory import LaunchpadObjectFactory
+
+
+user_name = 'ppa-user'
+default_email = '[e-mail]' % user_name
 
 
 class DoNotRunOnProduction(Exception):
@@ -64,13 +79,18 @@
         return max_id[0]
 
 
+def get_store(flavor=MASTER_FLAVOR):
+    """Obtain an ORM store."""
+    return getUtility(IStoreSelector).get(MAIN_STORE, flavor)
+
+
 def check_preconditions(options):
     """Try to ensure that it's safe to run.
 
     This script must not run on a production server, or anything
     remotely like it.
     """
-    store = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
+    store = get_store(SLAVE_FLAVOR)
 
     # Just a guess, but dev systems aren't likely to have ids this high
     # in this table.  Production data does.
@@ -82,7 +102,7 @@
     # For some configs it's just absolutely clear this script shouldn't
     # run.  Don't even accept --force there.
     forbidden_configs = re.compile('(edge|lpnet|production)')
-    current_config = getenv('LPCONFIG', 'an unknown config')
+    current_config = os.getenv('LPCONFIG', 'an unknown config')
     if forbidden_configs.match(current_config):
         raise DoNotRunOnProduction(
             "I won't delete Ubuntu data on %s and you can't --force me."
@@ -95,22 +115,25 @@
     :return: (options, args, logger)
     """
     parser = OptionParser(
-        description="Delete existing Ubuntu releases and set up new ones.")
+        description="Set up fresh Ubuntu series and %s identity." % user_name)
     parser.add_option('-f', '--force', action='store_true', dest='force',
         help="DANGEROUS: run even if the database looks production-like.")
-    parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
-        help="Do not commit changes.")
+    parser.add_option('-e', '--email', action='store', dest='email',
+        default=default_email,
+        help=(
+            "Email address to use for %s.  Should match your GPG key."
+            % user_name))
+
     logger_options(parser)
 
     options, args = parser.parse_args(arguments)
+
     return options, args, logger(options)
 
 
-def get_person(name):
+def get_person_set():
     """Return `IPersonSet` utility."""
-    # Avoid circular import.
-    from lp.registry.interfaces.person import IPersonSet
-    return getUtility(IPersonSet).getByName(name)
+    return getUtility(IPersonSet)
 
 
 def retire_series(distribution):
@@ -150,6 +173,20 @@
 components = main restricted universe multiverse'''
 
 
+def add_architecture(distroseries, architecture_name):
+    """Add a DistroArchSeries for the given architecture to `distroseries`."""
+    # Avoid circular import.
+    from lp.soyuz.model.distroarchseries import DistroArchSeries
+
+    store = get_store(MASTER_FLAVOR)
+    family = getUtility(IProcessorFamilySet).getByName(architecture_name)
+    archseries = DistroArchSeries(
+        distroseries=distroseries, processorfamily=family,
+        owner=distroseries.owner, official=True,
+        architecturetag=architecture_name)
+    store.add(archseries)
+
+
 def create_sections(distroseries):
     """Set up some sections for `distroseries`."""
     section_names = (
@@ -253,7 +290,7 @@
     # published binaries without corresponding sources.
 
     log.info("Deleting all items in official archives...")
-    retire_distro_archives(distribution, get_person('name16'))
+    retire_distro_archives(distribution, get_person_set().getByName('name16'))
 
     # Disable publishing of all PPAs, as they probably have broken
     # publishings too.
@@ -271,7 +308,7 @@
         utility.add(distroseries, format)
 
 
-def populate(distribution, parent_series_name, uploader_name, log):
+def populate(distribution, parent_series_name, uploader_name, options, log):
     """Set up sample data on `distribution`."""
     parent_series = distribution.getSeries(parent_series_name)
 
@@ -281,15 +318,73 @@
 
     log.info("Configuring sections...")
     create_sections(parent_series)
+    add_architecture(parent_series, 'amd64')
 
     log.info("Configuring components and permissions...")
-    create_components(parent_series, get_person(uploader_name))
+    uploader = get_person_set().getByName(uploader_name)
+    create_components(parent_series, uploader)
 
     set_source_package_format(parent_series)
 
     create_sample_series(parent_series, log)
 
 
+def sign_code_of_conduct(person, log):
+    """Sign Ubuntu Code of Conduct for `person`, if necessary."""
+    if person.is_ubuntu_coc_signer:
+        # Already signed.
+        return
+
+    log.info("Signing Ubuntu code of conduct.")
+    signedcocset = getUtility(ISignedCodeOfConductSet)
+    person_id = person.id
+    if signedcocset.searchByUser(person_id).count() == 0:
+        fake_gpg_key = LaunchpadObjectFactory().makeGPGKey(person)
+        Store.of(person).add(SignedCodeOfConduct(
+            owner=person, signingkey=fake_gpg_key,
+            signedcode="Normally a signed CoC would go here.", active=True))
+
+
+def create_ppa_user(username, options, approver, log):
+    """Create new user, with password "test," and sign code of conduct."""
+    person = get_person_set().getByName(username)
+    if person is None:
+        have_email = (options.email != default_email)
+        command_line = [
+            'utilities/make-lp-user',
+            username,
+            'ubuntu-team'
+            ]
+        if have_email:
+            command_line += ['--email', options.email]
+
+        pipe = subprocess.Popen(command_line, stderr=subprocess.PIPE)
+        stdout, stderr = pipe.communicate()
+        if stderr != '':
+            print stderr
+        if pipe.returncode != 0:
+            sys.exit(2)
+
+    transaction.commit()
+
+    person = getUtility(IPersonSet).getByName(username)
+    sign_code_of_conduct(person, log)
+
+    return person
+
+
+def create_ppa(distribution, person, name):
+    """Create a PPA for `person`."""
+    ppa = LaunchpadObjectFactory().makeArchive(
+        distribution=distribution, owner=person, name=name, virtualized=False,
+        description="Automatically created test PPA.")
+
+    series_name = distribution.currentseries.name
+    ppa.external_dependencies = (
+        "deb http://archive.ubuntu.com/ubuntu %s "
+        "main restricted universe multiverse\n") % series_name
+
+
 def main(argv):
     options, args, log = parse_args(argv[1:])
 
@@ -302,15 +397,26 @@
     clean_up(ubuntu, log)
 
     # Use Hoary as the root, as Breezy and Grumpy are broken.
-    populate(ubuntu, 'hoary', 'ubuntu-team', log)
-
-    if options.dry_run:
-        txn.abort()
-    else:
-        txn.commit()
-
+    populate(ubuntu, 'hoary', 'ubuntu-team', options, log)
+
+    admin = get_person_set().getByName('name16')
+    person = create_ppa_user(user_name, options, admin, log)
+
+    create_ppa(ubuntu, person, 'test-ppa')
+       
+    txn.commit()
     log.info("Done.")
 
+    print dedent("""
+        Now start your local Launchpad with "make run_codehosting" and log
+        into https://launchpad.dev/ as "%(email)s" with "test" as the
+        password.
+        Your user name will be %(user_name)s."""
+        % {
+            'email': options.email,
+            'user_name': user_name,
+            })
+
 
 if __name__ == "__main__":
     main(sys.argv)

=== added file 'utilities/start-dev-soyuz.sh'
--- utilities/start-dev-soyuz.sh	1970-01-01 00:00:00 +0000
+++ utilities/start-dev-soyuz.sh	2010-03-05 14:34:40 +0000
@@ -0,0 +1,21 @@
+#!/bin/sh -e
+# Start up Soyuz for local testing on a dev machine.
+
+start_twistd() {
+    # Start twistd for service $1.
+    mkdir -p "/var/tmp/$1"
+    echo "Starting $1."
+    bin/twistd \
+        --logfile "/var/tmp/development-$1.log" \
+        --pidfile "/var/tmp/development-$1.pid" \
+        -y "daemons/$1.tac"
+}
+
+start_twistd zeca
+start_twistd buildd-manager
+
+echo "Starting poppy."
+mkdir -p /var/tmp/poppy
+bin/py daemons/poppy-upload.py /var/tmp/poppy/incoming 2121 &
+
+echo "Done."