Merge lp:~jtv/launchpad/bug-527170 into lp:launchpad

[Jeroen T. Vermeulen]

Automate more manual Soyuz dev startup.

[Jeroen T. Vermeulen (jtv)]

= Bug 527170 =

This branch automates more of the manual work needed to get a working
local Soyuz setup. For the current process, see:

    https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally

I already implemented a script that does part of the work needed on the
dev sample data, based on wgrant's original, but some features are
added:

 * --amd64 option automates setup of amd64 support.

 * Creates a user ppa-user, with a fixed name.

 * Signs the Ubuntu Code of Conduct for ppa-user (with a fake key).

 * Gives you a fixed URL for adding your own GPG key.

 * --email option lets you specify your email address of choice.

 * Adds ppa-user to ubuntu-team.

The dev server won't actually be able to send email to your chosen email
address (although it will try, see the wiki page) but the choice makes
it easier for you to attach your GPG key. This is needed when you sign
uploads.

I tried automating the registration of a GPG key, or use of a dedicated
one, but no solution was really satisfactory. Well, working with gpgme
to get the real key based on your email address would have been, but it
seems that our scripting environment forces gpg to live in a temporary
directory rather than in ~/.gnupg.

This is a utilities script acting on the dev playground database, so
there's no real testing. But at least the script is exercised by the
test suite so we can detect obvious breakage:
{{{
./bin/test -vv -t sampledata-cleanup
}}}

No lint.

There's also a new script utilities/start-dev-soyuz.sh, also based on
wgrant's original. I regularlized it a bit and made sure the necessary
directories in /var/tmp are created. I have no idea how to test this
second script without (1) upsetting the test environment, or (2) porting
it to python and adding a lot of weight. For this script, used in
manual testing only and easy enough to edit, I don't think that's worth
the trouble.

Jeroen

[Jeroen T. Vermeulen (jtv)]

There's still a problem with this that escaped my notice because zeca was retaining my GPG key from previous tests.

review: Needs Fixing (code)

=== modified file 'utilities/soyuz-sampledata-cleanup.py'
--- utilities/soyuz-sampledata-cleanup.py	2010-02-24 12:13:29 +0000
+++ utilities/soyuz-sampledata-cleanup.py	2010-03-02 08:52:26 +0000
@@ -13,6 +13,9 @@
 
 DO NOT RUN ON PRODUCTION SYSTEMS.  This script deletes lots of
 Ubuntu-related data.
+
+This script creates a user "ppa-user" (email [e-mail],
+password test) who is able to create PPAs.
 """
 
 __metaclass__ = type
@@ -20,9 +23,12 @@
 import _pythonpath
 
 from optparse import OptionParser
-from os import getenv
 import re
+import os
+import subprocess
 import sys
+from textwrap import dedent
+import transaction
 
 from zope.component import getUtility
 from zope.event import notify
@@ -35,20 +41,33 @@
 
 from canonical.lp import initZopeless
 
+from canonical.launchpad.ftests.keys_for_tests import gpgkeysdir
+from canonical.launchpad.interfaces.account import AccountStatus
+from canonical.launchpad.interfaces.emailaddress import EmailAddressStatus
+from canonical.launchpad.interfaces.gpghandler import IGPGHandler
 from canonical.launchpad.interfaces.launchpad import (
     ILaunchpadCelebrities)
 from canonical.launchpad.scripts import execute_zcml_for_scripts
 from canonical.launchpad.scripts.logger import logger, logger_options
 from canonical.launchpad.webapp.interfaces import (
-    IStoreSelector, MAIN_STORE, SLAVE_FLAVOR)
+    IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
 
+from lp.registry.interfaces.codeofconduct import ISignedCodeOfConductSet
+from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet
 from lp.registry.interfaces.series import SeriesStatus
+from lp.registry.model.codeofconduct import SignedCodeOfConduct
 from lp.soyuz.interfaces.component import IComponentSet
+from lp.soyuz.interfaces.processor import IProcessorFamilySet
 from lp.soyuz.interfaces.section import ISectionSet
 from lp.soyuz.interfaces.sourcepackageformat import (
     ISourcePackageFormatSelectionSet, SourcePackageFormat)
 from lp.soyuz.model.section import SectionSelection
 from lp.soyuz.model.component import ComponentSelection
+from lp.testing.factory import LaunchpadObjectFactory
+
+
+user_name = 'ppa-user'
+default_email = '[e-mail]' % user_name
 
 
 class DoNotRunOnProduction(Exception):
@@ -82,7 +101,7 @@
     # For some configs it's just absolutely clear this script shouldn't
     # run.  Don't even accept --force there.
     forbidden_configs = re.compile('(edge|lpnet|production)')
-    current_config = getenv('LPCONFIG', 'an unknown config')
+    current_config = os.getenv('LPCONFIG', 'an unknown config')
     if forbidden_configs.match(current_config):
         raise DoNotRunOnProduction(
             "I won't delete Ubuntu data on %s and you can't --force me."
@@ -95,22 +114,31 @@
     :return: (options, args, logger)
     """
     parser = OptionParser(
-        description="Delete existing Ubuntu releases and set up new ones.")
+        description="Set up fresh Ubuntu series and %s identity." % user_name)
     parser.add_option('-f', '--force', action='store_true', dest='force',
         help="DANGEROUS: run even if the database looks production-like.")
     parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
         help="Do not commit changes.")
+    parser.add_option('-A', '--amd64', action='store_true', dest='amd64',
+        help="Support amd64 architecture.")
+    parser.add_option('-e', '--email', action='store', dest='email',
+        default=default_email,
+        help=(
+            "Email address to use for %s.  Should match your GPG key."
+            % user_name))
+
     logger_options(parser)
 
     options, args = parser.parse_args(arguments)
+
     return options, args, logger(options)
 
 
-def get_person(name):
+def get_person_set():
     """Return `IPersonSet` utility."""
     # Avoid circular import.
     from lp.registry.interfaces.person import IPersonSet
-    return getUtility(IPersonSet).getByName(name)
+    return getUtility(IPersonSet)
 
 
 def retire_series(distribution):
@@ -150,6 +178,26 @@
 components = main restricted universe multiverse'''
 
 
+def add_architecture(distroseries, architecture_name):
+    """Add a DistroArchSeries for the given architecture to `distroseries`."""
+    # Avoid circular import.
+    from lp.soyuz.model.distroarchseries import DistroArchSeries
+
+    store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
+    family = getUtility(IProcessorFamilySet).getByName(architecture_name)
+    archseries = DistroArchSeries(
+        distroseries=distroseries, processorfamily=family,
+        owner=distroseries.owner, official=True,
+        architecturetag=architecture_name)
+    store.add(archseries)
+
+
+def add_architectures(distroseries, options):
+    """Add support for additional architectures as specified by options."""
+    if options.amd64:
+        add_architecture(distroseries, 'amd64')
+    
+
 def create_sections(distroseries):
     """Set up some sections for `distroseries`."""
     section_names = (
@@ -253,7 +301,7 @@
     # published binaries without corresponding sources.
 
     log.info("Deleting all items in official archives...")
-    retire_distro_archives(distribution, get_person('name16'))
+    retire_distro_archives(distribution, get_person_set().getByName('name16'))
 
     # Disable publishing of all PPAs, as they probably have broken
     # publishings too.
@@ -271,7 +319,7 @@
         utility.add(distroseries, format)
 
 
-def populate(distribution, parent_series_name, uploader_name, log):
+def populate(distribution, parent_series_name, uploader_name, options, log):
     """Set up sample data on `distribution`."""
     parent_series = distribution.getSeries(parent_series_name)
 
@@ -281,15 +329,115 @@
 
     log.info("Configuring sections...")
     create_sections(parent_series)
+    add_architectures(parent_series, options)
 
     log.info("Configuring components and permissions...")
-    create_components(parent_series, get_person(uploader_name))
+    uploader = get_person_set().getByName(uploader_name)
+    create_components(parent_series, uploader)
 
     set_source_package_format(parent_series)
 
     create_sample_series(parent_series, log)
 
 
+def sign_code_of_conduct(person, log):
+    """Sign Ubuntu Code of Conduct for `person`, if necessary."""
+    if person.is_ubuntu_coc_signer:
+        # Already signed.
+        return
+
+    log.info("Signing Ubuntu code of conduct.")
+    signedcocset = getUtility(ISignedCodeOfConductSet)
+    person_id = person.id
+    if signedcocset.searchByUser(person_id).count() == 0:
+        fake_gpg_key = LaunchpadObjectFactory().makeGPGKey(person)
+        Store.of(person).add(SignedCodeOfConduct(
+            owner=person, signingkey=fake_gpg_key,
+            signedcode="Normally a signed CoC would go here.", active=True))
+
+
+def create_ppa_user(username, options, approver, log):
+    """Create new user, with password "test," and sign code of conduct."""
+    # Avoid circular import.
+    from lp.registry.interfaces.person import PersonCreationRationale
+
+    store = getUtility(IStoreSelector).get(MAIN_STORE, MASTER_FLAVOR)
+
+    log.info("Creating %s with email address '%s'." % (
+        user_name, options.email))
+
+    person = get_person_set().getByName(username)
+    if person is None:
+        person, email = get_person_set().createPersonAndEmail(
+            options.email, PersonCreationRationale.OWNER_CREATED_LAUNCHPAD,
+            name=username, displayname=username, password='test')
+        email.status = EmailAddressStatus.PREFERRED
+        email.account.status = AccountStatus.ACTIVE
+
+    if not options.dry_run:
+        transaction.commit()
+
+    sign_code_of_conduct(person, log)
+    get_person_set().getByName('ubuntu-team').addMember(person, approver)
+
+    return person
+
+
+def parse_fingerprints(gpg_output):
+    """Find key fingerprints in "gpg --fingerprint <email>" output."""
+    line_prefix = re.compile('\s*Key fingerprint\s*=\s*')
+    return [
+        ''.join(re.sub(line_prefix, '', line).split())
+        for line in gpg_output.splitlines()
+        if line_prefix.match(line)
+        ]
+
+
+def add_gpg_key(person, fingerprint, log):
+    """Add the GPG key with the given fingerprint to `person`."""
+    log.info("Adding GPG key %s" % fingerprint)
+    gpghandler = getUtility(IGPGHandler)
+    key = gpghandler.retrieveKey(fingerprint)
+
+    gpgkeyset = getUtility(IGPGKeySet)
+    if gpgkeyset.getByFingerprint(fingerprint) is not None:
+        # We already have this key.
+        return
+
+    algorithm = GPGKeyAlgorithm.items[key.algorithm]
+    can_encrypt = True
+    lpkey = gpgkeyset.new(
+        person.id, key.keyid, fingerprint, key.keysize, algorithm,
+        active=True, can_encrypt=can_encrypt)
+    Store.of(person).add(lpkey)
+    log.info("Created Launchpad key %s" % lpkey.displayname)
+
+
+def attach_gpg_keys(options, person, log):
+    """Attach the selected GPG key to `person`."""
+    log.info("Looking for GPG keys.")
+
+    # Need to override GNUPGHOME or we'll get a dummy GPG in a temp
+    # directory, which won't find any keys.
+    env = os.environ.copy()
+    if 'GNUPGHOME' in env:
+        del env['GNUPGHOME']
+    pipe = subprocess.Popen(
+        ['gpg', '--fingerprint', options.email], env=env,
+        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = pipe.communicate()
+    if stderr != '':
+        log.error(stderr)
+    if pipe.returncode != 0:
+        raise Exception("GPG error.")
+
+    fingerprints = parse_fingerprints(stdout)
+    if len(fingerprints) == 0:
+        log.warn("No GPG key fingerprints found!")
+    for fingerprint in fingerprints:
+        add_gpg_key(person, fingerprint, log)
+
+
 def main(argv):
     options, args, log = parse_args(argv[1:])
 
@@ -302,8 +450,24 @@
     clean_up(ubuntu, log)
 
     # Use Hoary as the root, as Breezy and Grumpy are broken.
-    populate(ubuntu, 'hoary', 'ubuntu-team', log)
-
+    populate(ubuntu, 'hoary', 'ubuntu-team', options, log)
+
+    admin = get_person_set().getByName('name16')
+    person = create_ppa_user(user_name, options, admin, log)
+
+    gave_email = (options.email != default_email)
+    print dedent("""
+        Now start your local Launchpad with "make run" and log into
+        https://launchpad.dev/ as "%(email)s" with "test" as the password.
+        Your user name will be %(user_name)s."""
+        % {
+            'email': options.email,
+            'user_name': user_name,
+            })
+
+    if gave_email:
+        attach_gpg_keys(options, person, log)
+       
     if options.dry_run:
         txn.abort()
     else:
@@ -311,6 +475,5 @@
 
     log.info("Done.")
 
-
 if __name__ == "__main__":
     main(sys.argv)

=== added file 'utilities/start-dev-soyuz.sh'
--- utilities/start-dev-soyuz.sh	1970-01-01 00:00:00 +0000
+++ utilities/start-dev-soyuz.sh	2010-03-02 08:52:26 +0000
@@ -0,0 +1,21 @@
+#!/bin/sh -e
+# Start up Soyuz for local testing on a dev machine.
+
+start_twistd() {
+    # Start twistd for service $1.
+    mkdir -p "/var/tmp/$1"
+    echo "Starting $1."
+    bin/twistd \
+        --logfile "/var/tmp/development-$1.log" \
+        --pidfile "/var/tmp/development-$1.pid" \
+        -y "daemons/$1.tac"
+}
+
+start_twistd zeca
+start_twistd buildd-manager
+
+echo "Starting poppy."
+mkdir -p /var/tmp/poppy
+bin/py daemons/poppy-upload.py /var/tmp/poppy/incoming 2121 &
+
+echo "Done."