Merge lp:~jtv/launchpad/bug-527170 into lp:launchpad

[Jeroen T. Vermeulen]

Automate much of manual Soyuz setup on dev systems.

[Jeroen T. Vermeulen (jtv)]

= Bug 527170 =

This branch automates more of the manual work needed to get a working
local Soyuz setup. For the current process, see:

    https://dev.launchpad.net/Soyuz/HowToUseSoyuzLocally

I already implemented a script that does part of the work needed on the
dev sample data, based on wgrant's original, but some features are
added:

 * --amd64 option automates setup of amd64 support.

 * Creates a user ppa-user, with a fixed name.

 * Signs the Ubuntu Code of Conduct for ppa-user (with a fake key).

 * Gives you a fixed URL for adding your own GPG key.

 * --email option lets you specify your email address of choice.

 * Adds ppa-user to ubuntu-team.

The dev server won't actually be able to send email to your chosen email
address (although it will try, see the wiki page) but the choice makes
it easier for you to attach your GPG key. This is needed when you sign
uploads.

I tried automating the registration of a GPG key, or use of a dedicated
one, but no solution was really satisfactory. Well, working with gpgme
to get the real key based on your email address would have been, but it
seems that our scripting environment forces gpg to live in a temporary
directory rather than in ~/.gnupg.

This is a utilities script acting on the dev playground database, so
there's no real testing. But at least the script is exercised by the
test suite so we can detect obvious breakage:
{{{
./bin/test -vv -t sampledata-cleanup
}}}

No lint.

There's also a new script utilities/start-dev-soyuz.sh, also based on
wgrant's original. I regularlized it a bit and made sure the necessary
directories in /var/tmp are created. I have no idea how to test this
second script without (1) upsetting the test environment, or (2) porting
it to python and adding a lot of weight. For this script, used in
manual testing only and easy enough to edit, I don't think that's worth
the trouble.

Jeroen

[Jeroen T. Vermeulen (jtv)]

There's still a problem with this that escaped my notice because zeca was retaining my GPG key from previous tests.

review: Needs Fixing (code)

[Jeroen T. Vermeulen (jtv)]

Done. A feature I had previously neglected to document: the script now goes through the steps of adding and validating your real GPG key to ppa-user for you. Saves more hassle.

[Michael Nelson (michael.nelson)]

Thanks Jeroen!

So there were a few suggestions, (1) adding the amd arch by default, (2) creating the PPA for the person and (3) adding the external dependencies. You've done (1) and were going to see how much effort is involved for (2) and (3).

12:35 < noodles775> jtv: what would be the problem with just adding amd64 as an arch always? the user still needs to target a specific series?
12:36 < jtv> noodles775: dunno frankly; I figured the safe thing to do was to stay faithful to what we already had.
12:36 < noodles775> jtv: also, do you think it should be possible to run this twice? (ie. the first time I didn't set my email, so I need to make schema before re-running right?)
12:36 < jtv> noodles775: right now you'll have to "make schema" between runs.
12:36 < jtv> wgrant: is there any conceivable reason not to add amd64 support when setting up the sampledata for soyuz?
12:37 < wgrant> jtv: No.
12:37 < jtv> noodles775: there's your answer. No reason. Fixing.
12:37 < noodles775> jtv: great.
12:39 < jtv> noodles775: testing the fix.
12:39 < noodles775> jtv: I wonder if in the future we should separate out utilities/soyuz-add-ppa-user.py?
12:40 < noodles775> (ie. it could be run multiple times).
12:41 < jtv> noodles775: we have that option. But I think we'll get a lot of documentation benefit out of sticking with ppa-user for now.
12:41 < noodles775> Right.
12:42 < noodles775> Wow... automatic addition of my GPG key :)
12:42 < jtv> yup
12:44 < jtv> noodles775: It was hard, but I hope worth it. I just pushed the version that always includes amd64 support.
12:46 < noodles775> jtv: it seems to have added a third key, in addition to the two identified by my email?
12:46 < jtv> noodles775: the bogus one is for signing the Code of Conduct. Please don't make me rework that to use the proper key (and prompt you for a passphrase)
12:46 < noodles775> jtv: ah, no worries :)
12:47 < jtv> noodles775: but glad to hear that it picked up both keys for someone who had two. :-)
12:47 < noodles775> jtv: and none of them are found in zeca when clicking on the key id?
12:48 < noodles775> jtv: actually, if I modify the final url param to op=get it finds it.
12:48 < jtv> noodles775: yup
12:48 < jtv> that would've happened with a manual upload as well, app'ly
12:48 < noodles775> but not op=index.... ok, but it's there :)
12:51 < noodles775> jtv: Another thought, it'd only be an extra 1 line to create a default PPA for the user in create_ppa_user? (you could just use factory.makeArchive(owner=person))
12:51 < jtv> noodles775: I thought that'd be a nice next feature, but I think you just saved me the research.
12:51 < noodles775> jtv: great!
12:52 -!- salgado [~salgado@canonical/launchpad/salgado] has joined #launchpad-reviews
12:53 -!- gmb changed the topic of #launchpad-reviews to: on call: gmb || reviewing: adiroiban || queue [adeuring] || This channel is logged: http://irclogs.ubuntu.com/ || https://code.edge.launchpad.net/launchpad/+activereviews
12:53 < jtv> noodles775: should I add virtualized=False?
12:54 < noodles775> jtv: yes, definitely.
12:54 < jtv> noodles775: running tests...
12:55 * noodles775 tries uploading a package with the new setup.
12:56 -!-...

review: Approve (code)

[Jeroen T. Vermeulen (jtv)]

Okay, I've added creation of the PPA ("test-ppa") and addition of an external dependency on Lucid.

[Jeroen T. Vermeulen (jtv)]

Not sure how this got turned into a review request for myself, but I approve.

review: Approve (code)

=== modified file 'lib/canonical/testing/layers.py'
--- lib/canonical/testing/layers.py	2010-02-12 19:34:42 +0000
+++ lib/canonical/testing/layers.py	2010-03-06 05:57:42 +0000
@@ -97,7 +97,7 @@
     confirm_dbrevision, confirm_dbrevision_on_startup)
 from canonical.database.sqlbase import cursor, ZopelessTransactionManager
 from canonical.launchpad.interfaces import IMailBox, IOpenLaunchBag
-from canonical.launchpad.ftests import ANONYMOUS, login, logout, is_logged_in
+from lp.testing import ANONYMOUS, login, logout, is_logged_in
 import lp.services.mail.stub
 from lp.services.mail.mailbox import TestMailBox
 from canonical.launchpad.scripts import execute_zcml_for_scripts

=== renamed file 'lib/lp/soyuz/doc/sampledata-cleanup.txt' => 'lib/lp/soyuz/doc/sampledata-setup.txt'
--- lib/lp/soyuz/doc/sampledata-cleanup.txt	2010-02-24 12:13:29 +0000
+++ lib/lp/soyuz/doc/sampledata-setup.txt	2010-03-06 05:57:42 +0000
@@ -1,18 +1,16 @@
-= Sample data cleanup =
+= Sample data setup =
 
 In order to run Soyuz locally on a development system, the sample data
 must be cleaned up and customized a bit.  This is done by a the script
-utilities/soyuz-sampledata-cleanup.py.
+utilities/soyuz-sampledata-setup.py.
 
 We only need this script for the playground sample data, so there's
 little point in inspecting what it does to the test database in detail.
 
     >>> from canonical.launchpad.ftests.script import run_script
 
-The --dry-run option makes the script roll back its changes.
-
     >>> return_code, output, error = run_script(
-    ...     'utilities/soyuz-sampledata-cleanup.py', args=['--dry-run'])
+    ...     'utilities/soyuz-sampledata-setup.py')
 
     >>> print return_code
     0
@@ -20,3 +18,6 @@
     >>> print error
     INFO ...
     INFO Done.
+
+    >>> from canonical.launchpad.ftests.harness import LaunchpadTestSetup
+    >>> LaunchpadTestSetup().force_dirty_database()

=== modified file 'utilities/soyuz-sampledata-cleanup.py'
--- utilities/soyuz-sampledata-cleanup.py	2010-02-24 12:13:29 +0000
+++ utilities/soyuz-sampledata-cleanup.py	2010-03-06 05:57:42 +0000
@@ -13,6 +13,9 @@
 
 DO NOT RUN ON PRODUCTION SYSTEMS.  This script deletes lots of
 Ubuntu-related data.
+
+This script creates a user "ppa-user" (email [e-mail],
+password test) who is able to create PPAs.
 """
 
 __metaclass__ = type
@@ -20,9 +23,12 @@
 import _pythonpath
 
 from optparse import OptionParser
-from os import getenv
 import re
+import os
+import subprocess
 import sys
+from textwrap import dedent
+import transaction
 
 from zope.component import getUtility
 from zope.event import notify
@@ -35,20 +41,32 @@
 
 from canonical.lp import initZopeless
 
+from canonical.launchpad.interfaces.account import AccountStatus
+from canonical.launchpad.interfaces.emailaddress import EmailAddressStatus
+from canonical.launchpad.interfaces.gpghandler import IGPGHandler
 from canonical.launchpad.interfaces.launchpad import (
     ILaunchpadCelebrities)
 from canonical.launchpad.scripts import execute_zcml_for_scripts
 from canonical.launchpad.scripts.logger import logger, logger_options
 from canonical.launchpad.webapp.interfaces import (
-    IStoreSelector, MAIN_STORE, SLAVE_FLAVOR)
+    IStoreSelector, MAIN_STORE, MASTER_FLAVOR, SLAVE_FLAVOR)
 
+from lp.registry.interfaces.codeofconduct import ISignedCodeOfConductSet
+from lp.registry.interfaces.gpg import GPGKeyAlgorithm, IGPGKeySet
 from lp.registry.interfaces.series import SeriesStatus
+from lp.registry.model.codeofconduct import SignedCodeOfConduct
 from lp.soyuz.interfaces.component import IComponentSet
+from lp.soyuz.interfaces.processor import IProcessorFamilySet
 from lp.soyuz.interfaces.section import ISectionSet
 from lp.soyuz.interfaces.sourcepackageformat import (
     ISourcePackageFormatSelectionSet, SourcePackageFormat)
 from lp.soyuz.model.section import SectionSelection
 from lp.soyuz.model.component import ComponentSelection
+from lp.testing.factory import LaunchpadObjectFactory
+
+
+user_name = 'ppa-user'
+default_email = '[e-mail]' % user_name
 
 
 class DoNotRunOnProduction(Exception):
@@ -64,13 +82,18 @@
         return max_id[0]
 
 
+def get_store(flavor=MASTER_FLAVOR):
+    """Obtain an ORM store."""
+    return getUtility(IStoreSelector).get(MAIN_STORE, flavor)
+
+
 def check_preconditions(options):
     """Try to ensure that it's safe to run.
 
     This script must not run on a production server, or anything
     remotely like it.
     """
-    store = getUtility(IStoreSelector).get(MAIN_STORE, SLAVE_FLAVOR)
+    store = get_store(SLAVE_FLAVOR)
 
     # Just a guess, but dev systems aren't likely to have ids this high
     # in this table.  Production data does.
@@ -82,7 +105,7 @@
     # For some configs it's just absolutely clear this script shouldn't
     # run.  Don't even accept --force there.
     forbidden_configs = re.compile('(edge|lpnet|production)')
-    current_config = getenv('LPCONFIG', 'an unknown config')
+    current_config = os.getenv('LPCONFIG', 'an unknown config')
     if forbidden_configs.match(current_config):
         raise DoNotRunOnProduction(
             "I won't delete Ubuntu data on %s and you can't --force me."
@@ -95,22 +118,29 @@
     :return: (options, args, logger)
     """
     parser = OptionParser(
-        description="Delete existing Ubuntu releases and set up new ones.")
+        description="Set up fresh Ubuntu series and %s identity." % user_name)
     parser.add_option('-f', '--force', action='store_true', dest='force',
         help="DANGEROUS: run even if the database looks production-like.")
     parser.add_option('-n', '--dry-run', action='store_true', dest='dry_run',
         help="Do not commit changes.")
+    parser.add_option('-e', '--email', action='store', dest='email',
+        default=default_email,
+        help=(
+            "Email address to use for %s.  Should match your GPG key."
+            % user_name))
+
     logger_options(parser)
 
     options, args = parser.parse_args(arguments)
+
     return options, args, logger(options)
 
 
-def get_person(name):
+def get_person_set():
     """Return `IPersonSet` utility."""
     # Avoid circular import.
     from lp.registry.interfaces.person import IPersonSet
-    return getUtility(IPersonSet).getByName(name)
+    return getUtility(IPersonSet)
 
 
 def retire_series(distribution):
@@ -150,6 +180,20 @@
 components = main restricted universe multiverse'''
 
 
+def add_architecture(distroseries, architecture_name):
+    """Add a DistroArchSeries for the given architecture to `distroseries`."""
+    # Avoid circular import.
+    from lp.soyuz.model.distroarchseries import DistroArchSeries
+
+    store = get_store(MASTER_FLAVOR)
+    family = getUtility(IProcessorFamilySet).getByName(architecture_name)
+    archseries = DistroArchSeries(
+        distroseries=distroseries, processorfamily=family,
+        owner=distroseries.owner, official=True,
+        architecturetag=architecture_name)
+    store.add(archseries)
+
+
 def create_sections(distroseries):
     """Set up some sections for `distroseries`."""
     section_names = (
@@ -253,7 +297,7 @@
     # published binaries without corresponding sources.
 
     log.info("Deleting all items in official archives...")
-    retire_distro_archives(distribution, get_person('name16'))
+    retire_distro_archives(distribution, get_person_set().getByName('name16'))
 
     # Disable publishing of all PPAs, as they probably have broken
     # publishings too.
@@ -271,7 +315,7 @@
         utility.add(distroseries, format)
 
 
-def populate(distribution, parent_series_name, uploader_name, log):
+def populate(distribution, parent_series_name, uploader_name, options, log):
     """Set up sample data on `distribution`."""
     parent_series = distribution.getSeries(parent_series_name)
 
@@ -281,15 +325,141 @@
 
     log.info("Configuring sections...")
     create_sections(parent_series)
+    add_architecture(parent_series, 'amd64')
 
     log.info("Configuring components and permissions...")
-    create_components(parent_series, get_person(uploader_name))
+    uploader = get_person_set().getByName(uploader_name)
+    create_components(parent_series, uploader)
 
     set_source_package_format(parent_series)
 
     create_sample_series(parent_series, log)
 
 
+def sign_code_of_conduct(person, log):
+    """Sign Ubuntu Code of Conduct for `person`, if necessary."""
+    if person.is_ubuntu_coc_signer:
+        # Already signed.
+        return
+
+    log.info("Signing Ubuntu code of conduct.")
+    signedcocset = getUtility(ISignedCodeOfConductSet)
+    person_id = person.id
+    if signedcocset.searchByUser(person_id).count() == 0:
+        fake_gpg_key = LaunchpadObjectFactory().makeGPGKey(person)
+        Store.of(person).add(SignedCodeOfConduct(
+            owner=person, signingkey=fake_gpg_key,
+            signedcode="Normally a signed CoC would go here.", active=True))
+
+
+def create_ppa_user(username, options, approver, log):
+    """Create new user, with password "test," and sign code of conduct."""
+    # Avoid circular import.
+    from lp.registry.interfaces.person import PersonCreationRationale
+
+    store = get_store(MASTER_FLAVOR)
+
+    log.info("Creating %s with email address '%s'." % (
+        user_name, options.email))
+
+    person = get_person_set().getByName(username)
+    if person is None:
+        person, email = get_person_set().createPersonAndEmail(
+            options.email, PersonCreationRationale.OWNER_CREATED_LAUNCHPAD,
+            name=username, displayname=username, password='test')
+        email.status = EmailAddressStatus.PREFERRED
+        email.account.status = AccountStatus.ACTIVE
+
+    if not options.dry_run:
+        transaction.commit()
+
+    sign_code_of_conduct(person, log)
+    get_person_set().getByName('ubuntu-team').addMember(person, approver)
+
+    return person
+
+
+def parse_fingerprints(gpg_output):
+    """Find key fingerprints in "gpg --fingerprint <email>" output."""
+    line_prefix = re.compile('\s*Key fingerprint\s*=\s*')
+    return [
+        ''.join(re.sub(line_prefix, '', line).split())
+        for line in gpg_output.splitlines()
+        if line_prefix.match(line)
+        ]
+
+
+def run_native_gpg(command_line, log):
+    """Run GPG using the user's real keyring."""
+    # Need to override GNUPGHOME or we'll get a dummy GPG in a temp
+    # directory, which won't find any keys.
+    env = os.environ.copy()
+    if 'GNUPGHOME' in env:
+        del env['GNUPGHOME']
+    pipe = subprocess.Popen(
+        command_line, env=env, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = pipe.communicate()
+    if stderr != '':
+        log.error(stderr)
+    if pipe.returncode != 0:
+        raise Exception('GPG error during "%s"' % ' '.join(command_line))
+
+    return stdout
+
+
+def add_gpg_key(person, fingerprint, log):
+    """Add the GPG key with the given fingerprint to `person`."""
+    log.info("Adding GPG key %s" % fingerprint)
+
+    command_line = [
+        'gpg',
+        '--keyserver', 'keyserver.launchpad.dev',
+        '--send-key', fingerprint
+        ]
+    run_native_gpg(command_line, log)
+
+    gpghandler = getUtility(IGPGHandler)
+    key = gpghandler.retrieveKey(fingerprint)
+
+    gpgkeyset = getUtility(IGPGKeySet)
+    if gpgkeyset.getByFingerprint(fingerprint) is not None:
+        # We already have this key.
+        return
+
+    algorithm = GPGKeyAlgorithm.items[key.algorithm]
+    can_encrypt = True
+    lpkey = gpgkeyset.new(
+        person.id, key.keyid, fingerprint, key.keysize, algorithm,
+        active=True, can_encrypt=can_encrypt)
+    Store.of(person).add(lpkey)
+    log.info("Created Launchpad key %s" % lpkey.displayname)
+
+
+def attach_gpg_keys(options, person, log):
+    """Attach the selected GPG key to `person`."""
+    log.info("Looking for GPG keys.")
+
+    output = run_native_gpg(['gpg', '--fingerprint', options.email], log)
+
+    fingerprints = parse_fingerprints(output)
+    if len(fingerprints) == 0:
+        log.warn("No GPG key fingerprints found!")
+    for fingerprint in fingerprints:
+        add_gpg_key(person, fingerprint, log)
+
+
+def create_ppa(distribution, person, name):
+    """Create a PPA for `person`."""
+    ppa = LaunchpadObjectFactory().makeArchive(
+        distribution=distribution, owner=person, name=name, virtualized=False,
+        description="Automatically created test PPA.")
+
+    series_name = distribution.currentseries.name
+    ppa.external_dependencies = (
+        "deb http://archive.ubuntu.com/ubuntu %s "
+        "main restricted universe multiverse\n") % series_name
+
+
 def main(argv):
     options, args, log = parse_args(argv[1:])
 
@@ -302,8 +472,17 @@
     clean_up(ubuntu, log)
 
     # Use Hoary as the root, as Breezy and Grumpy are broken.
-    populate(ubuntu, 'hoary', 'ubuntu-team', log)
-
+    populate(ubuntu, 'hoary', 'ubuntu-team', options, log)
+
+    admin = get_person_set().getByName('name16')
+    person = create_ppa_user(user_name, options, admin, log)
+
+    gave_email = (options.email != default_email)
+    if gave_email:
+        attach_gpg_keys(options, person, log)
+
+    create_ppa(ubuntu, person, 'test-ppa')
+       
     if options.dry_run:
         txn.abort()
     else:
@@ -311,6 +490,15 @@
 
     log.info("Done.")
 
+    print dedent("""
+        Now start your local Launchpad with "make run" and log into
+        https://launchpad.dev/ as "%(email)s" with "test" as the password.
+        Your user name will be %(user_name)s."""
+        % {
+            'email': options.email,
+            'user_name': user_name,
+            })
+
 
 if __name__ == "__main__":
     main(sys.argv)

=== added file 'utilities/start-dev-soyuz.sh'
--- utilities/start-dev-soyuz.sh	1970-01-01 00:00:00 +0000
+++ utilities/start-dev-soyuz.sh	2010-03-06 05:57:42 +0000
@@ -0,0 +1,21 @@
+#!/bin/sh -e
+# Start up Soyuz for local testing on a dev machine.
+
+start_twistd() {
+    # Start twistd for service $1.
+    mkdir -p "/var/tmp/$1"
+    echo "Starting $1."
+    bin/twistd \
+        --logfile "/var/tmp/development-$1.log" \
+        --pidfile "/var/tmp/development-$1.pid" \
+        -y "daemons/$1.tac"
+}
+
+start_twistd zeca
+start_twistd buildd-manager
+
+echo "Starting poppy."
+mkdir -p /var/tmp/poppy
+bin/py daemons/poppy-upload.py /var/tmp/poppy/incoming 2121 &
+
+echo "Done."